From 4a87de44af08ecbe67e2014aac04ffdb9b50de5f Mon Sep 17 00:00:00 2001 From: Kevin Howe Date: Mon, 28 Sep 2026 14:40:52 +0100 Subject: [PATCH 01/45] Define specification - requirements for a future production release --- SPECIFICATION.md | 926 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 926 insertions(+) create mode 100644 SPECIFICATION.md diff --git a/SPECIFICATION.md b/SPECIFICATION.md new file mode 100644 index 0000000..72b3e53 --- /dev/null +++ b/SPECIFICATION.md @@ -0,0 +1,926 @@ +# AdvISO Validation Toolkit: `compare` Behaviour Specification + +**Document status:** Draft for review +**Specification version:** 0.1.0-draft +**Applies to:** The `compare` command and its supporting comparison engine +**Normative language:** The terms **MUST**, **MUST NOT**, **SHOULD**, **SHOULD NOT**, and **MAY** indicate requirement strength. + +## 1. Purpose + +This document defines a specification for the AdvISO Validation Toolkit: a command-line namespace for cohesive capabilities that support bioinformatics validation and verification activities associated with ISO 15189 accreditation. + +The initial sole user-facing subcommand within the toolkit is `compare`. This subcommand compares a reference dataset with a query dataset according to an explicit comparison specification and produces deterministic, machine-readable findings and a human-readable summary suitable for review and retention as one component of a bioinformatics pipeline validation or verification record. It is intended to answer this question: + +> Given these identified inputs and these declared comparison rules, which expected and observed records and values are equivalent, different, missing, extra, or not assessable? + +Given this narrow and precisely defined scope, the toolkit cannot be used _alone_ to assert that: + +- a pipeline is clinically or analytically valid +- a pipeline conforms with ISO 15189 or any other standard +- a pipeline is fit for an unspecified intended use +- a reference or query dataset is intrinsically correct + +Responsibility for the provenance and suitability of reference material, the comparison rules, acceptance criteria, review, approval, and resulting validation decision remains with the laboratory. + +## 2. Product structure and scope + +### 2.1 Product structure + +The organisation of the toolkit within the AdvISO GitHub organisation is: + +```text +AdvISO +└── validation_toolkit + ├── compare subcommand + └── internal comparison, specification, provenance, findings, and reporting components +``` + +The **AdvISO Validation Toolkit** is the user-facing product and a minimal command namespace. **`compare`** is currently the sole specified subcommand. The other components, including the **comparison engine**, are internal reusable software components and are not separate products. + +The initial command-line form SHOULD be: + +```text +adviso-validation compare ... +``` + +The distribution, import-package, executable, and repository names MAY differ where required by their respective naming conventions, but each MUST identify its relationship to AdvISO clearly. Final distribution and executable names MUST be settled before the first stable release and thereafter follow a documented compatibility policy. + +Additional commands are not a development objective for the first release and MUST NOT be added speculatively. A future command MAY be introduced only for a demonstrated, materially distinct operation in the same validation-evidence workflow. The top-level dispatcher MUST remain thin and MUST NOT create speculative shared abstractions. The initial implementation SHOULD use ordinary modules and a small comparator registry, not a general plugin framework. Capabilities with materially different purposes, dependencies, governance, or release lifecycles SHOULD be separate AdvISO packages. + +### 2.2 Scope of the `compare` subcommand + +The first-release scope of `compare` is: + +- comparison of structured records represented in CSV, TSV, JSON, or YAML; +- explicit single-field or composite record keys; +- detection of matching, differing, missing, and extra records and fields; +- field-specific comparison rules; +- controlled column-name and value mappings; +- explicit treatment of types, nulls, and missing values; +- complete discrepancy collection rather than stopping at the first difference; +- structured JSON results, human-readable reports, and automation-friendly exit codes; +- provenance sufficient to reproduce and review a comparison; +- safe handling of data that may contain sensitive clinical information. + +### 2.3 Initially out of scope + +Unless introduced by a separately specified extension, the `compare` command does not: + +- establish analytical sensitivity, analytical specificity, precision, limit of detection, reportable range, clinical validity, or clinical utility; +- generate or adjudicate truth data; +- determine whether a reference dataset is representative or fit for purpose; +- replace laboratory review or approval; +- compare unstructured prose, images, binary files, or specialist bioinformatics formats (e.g. SAM, VCF, FASTQ); +- infer biological equivalence of variants, transcripts, genomic intervals, taxonomic calls, or other domain objects; +- provide an electronic-signature or laboratory document-management system. + +No bioinformatics-specific or domain-aware comparator is required for the first release. Such comparators MAY be added later under explicit, independently versioned specifications. + +## 3. Design principles + +The following principles govern all requirements: + +1. **Explicit over inferred:** Clinically material comparison behaviour MUST be declared, not guessed from observed data. +2. **No silent data loss:** Input records, duplicate keys, unmapped values, and unsupported structures MUST NOT be silently discarded or overwritten. +3. **Symmetry:** Missing and extra content MUST be assessed in both directions. +4. **Separation of concerns:** Execution success, comparison outcome, and laboratory acceptance decision are distinct concepts. +5. **Complete evidence:** A normal comparison failure MUST still produce a complete report. +6. **Determinism:** Identical inputs, configuration, and tool version MUST produce semantically identical results. +7. **Traceability:** Findings MUST identify the rule, record, and field that produced them. +8. **Privacy by design:** Reports MUST include only the sensitive data necessary for their intended purpose. +9. **Fail clearly:** Invalid or ambiguous input MUST produce an explicit error rather than an assumed interpretation. +10. **Testability:** Each normative requirement SHOULD be traceable to one or more automated tests. + +## 4. Terminology + +- **Reference dataset:** The dataset containing the approved expected or comparator records. “Reference” does not imply intrinsic truth. +- **Query dataset:** The dataset being evaluated against the reference. +- **Comparison specification:** The versioned configuration declaring the data contract, normalisation, comparison, and acceptance rules. +- **Comparison scope:** The final cumulative comparison phase requested: `contract`, `records`, or `full`. +- **Not assessed:** Deliberately outside the selected scope or excluded by an explicit field policy. +- **Not assessable:** Inside the selected scope, but unable to be compared safely because a prerequisite was not satisfied. +- **Contract comparison:** Assessment of whether inputs satisfy the declared data contract, including fields, logical types, key definitions, nullability, mappings, and configured constraints. +- **Records comparison:** Alignment and assessment of valid record identities, including matching, missing, and extra records. Invalid, missing, null, or duplicate record keys are execution errors rather than record-comparison findings. +- **Value comparison:** Application of declared comparators to values in records that can be aligned meaningfully. +- **Record:** One logical unit of comparison. +- **Record key:** One field or an ordered combination of fields that uniquely identifies a record within a dataset. +- **Field:** A named value within a record. +- **Finding:** A structured observation generated by applying one comparison rule to a record, field, dataset, or run. +- **Execution status:** Whether the requested toolkit command completed correctly. +- **Comparison outcome:** The result of applying configured acceptance criteria: `pass`, `fail`, or `inconclusive`. +- **Laboratory decision:** The externally reviewed conclusion about validation or verification. The toolkit does not make this decision. +- **Missing:** Expected in the reference but absent from the query. +- **Extra:** Present in the query but absent from the reference. +- **Null:** A present field with an explicitly empty/no-value representation defined by the input format or specification. +- **Absent:** A field or record that is not present at all. +- **Normalisation:** A declared, non-lossy or intentionally controlled transformation performed before comparison. +- **Mapping:** A declared translation of field names or values from one representation to another. + +## 5. Inputs + +### 5.1 Required inputs + +A comparison run MUST receive: + +- one reference dataset; +- one query dataset; +- an output destination. + +A custom comparison specification MAY be supplied. If it is omitted, the bundled strict-exact specification defined in Section 6.8 MUST be used. + +### 5.2 Supported formats + +The initial implementation MUST support: + +- CSV conforming to the configured dialect; +- TSV conforming to the configured dialect; +- JSON using a documented record-oriented structure; +- YAML using the same logical structure as JSON. + +The reference and query formats MAY be selected independently through explicit command-line options. If a format is not supplied, it MAY be detected only from a recognised, unambiguous extension: `.csv`, `.tsv`, `.json`, `.yaml`, or `.yml`. In either case: + +- detected format MUST be recorded; +- ambiguous or unsupported content MUST fail validation; +- an unknown or ambiguous extension, including `.txt`, MUST require an explicit format; +- content MUST be validated against the selected format. + +### 5.3 Character encoding and text handling + +- UTF-8 MUST be the default and documented encoding. +- Alternative encodings MAY be configured explicitly. +- Invalid byte sequences MUST cause an input error unless an explicit replacement policy is configured and recorded. +- Field names and string values MUST be preserved exactly before declared normalisation. +- Line endings MUST NOT affect comparison results. + +### 5.4 Tabular parsing + +For CSV and TSV: + +- a header row MUST be required; +- duplicate or empty column names MUST be rejected; +- delimiters, quoting, escaping, and embedded newlines MUST be handled by a standards-compliant parser; +- rows with too many or too few columns MUST be input errors; +- all physical input rows MUST be accounted for; +- source row numbers SHOULD be retained for traceability; +- input values MUST initially remain strings unless a custom specification declares a null representation or type conversion; in the bundled strict-exact specification, an empty cell is an empty string. + +### 5.5 JSON and YAML parsing + +- JSON and YAML inputs MUST use a schema-validated list of record objects: + +```json +[ + {"sample_id": "S1", "gene": "BRAF", "result": "detected"}, + {"sample_id": "S2", "gene": "KRAS", "result": "not_detected"} +] +``` + +- YAML scalar resolution MUST use a documented restricted YAML 1.2-compatible schema in which only lowercase `true`, `false`, and `null` receive implicit non-numeric scalar types. +- Values such as `yes`, `no`, `on`, `off`, uppercase boolean-like values, and timestamp-like strings MUST remain strings unless a custom specification explicitly maps or parses them. +- JSON and YAML numerical tokens that may be used as decimals MUST be loaded without an intermediate binary floating-point conversion, and their source lexical representations MUST remain available. +- Root scalars, heterogeneous root structures, and records that are not objects MUST be rejected with actionable errors. + +### 5.6 Preserved parser-native values + +For `preserved_exact` comparison, parsers MUST expose a common, deterministic set of parser-native value categories: + +- `string`, `integer`, `decimal`, `boolean`, `null`, `array`, and `object`; +- CSV and TSV cells MUST be strings, including empty strings, unless a custom specification declares further processing; +- JSON numbers without a fractional or exponent component MUST be integers, while other finite JSON numbers MUST be exact decimals; +- YAML scalars MUST resolve under the restricted schema in Section 5.5 to strings, arbitrary-precision integers, exact finite decimals, booleans, or nulls; timestamps MUST NOT be resolved implicitly; +- JSON arrays and YAML sequences MUST be arrays, and JSON or YAML mappings used as field values MUST be objects; and +- equality MUST require the same parser-native category and recursively equal content. Array order MUST be significant and object member order MUST not be significant. + +The source lexical representation of numerical values MUST remain available as evidence even though numerically equivalent representations within the same parser-native numerical category compare equally. Non-finite numerical values MUST be rejected. + +### 5.7 Empty inputs + +- Empty files and datasets with zero records MUST be recognised explicitly. +- A custom comparison specification MUST declare whether two empty datasets satisfy its records-phase acceptance criteria. +- A non-empty reference and empty query MUST report every expected record as missing. +- An empty reference and non-empty query MUST report every observed record as extra. + +## 6. Comparison specifications + +### 6.1 General requirements + +Every comparison uses an effective comparison specification. A custom specification MUST: + +- use a published, independently versioned specification schema supported by the installed toolkit release; +- declare its schema version and a stable specification identifier and revision; +- be validated fully before either dataset is compared; +- reject unknown configuration keys by default; +- provide actionable validation errors with configuration locations; +- be identified in provenance by a cryptographic digest; +- be retained or embedded in the evidence package, subject to policy. + +A valid specification MUST declare at least: + +- record key fields; +- field definitions or an explicit policy for undeclared fields; +- comparison rules; +- missing and extra record policy; +- missing and extra field policy; +- duplicate-key policy; +- phase-specific acceptance criteria. + +### 6.2 Record keys + +- A record key MUST contain one or more named fields. +- Composite-key field order MUST be stable and recorded. +- Key fields MUST be present and non-null. +- Key values MUST NOT undergo implicit numeric or boolean conversion. +- Normalisation of key values MUST be explicitly configured. +- Alternative key names and composite keys MUST be declared in a custom specification. +- Key names MUST NOT be inferred, and command-line key overrides MUST NOT be supported in the first release. +- The normalised key MUST uniquely identify one record. +- Duplicate keys MUST be input errors in every profile and scope. +- Duplicate records MUST never overwrite one another silently. +- Findings and reports MUST retain enough information to distinguish every source record. + +### 6.3 Field declarations and preserved values + +The comparison engine MUST support both preserved, untyped values and explicitly declared logical types. Custom specifications MAY declare logical field types. A type declaration is required for a field that uses typed parsing, normalisation, mapping, or a type-specific comparator. Every custom specification MUST explicitly declare how undeclared fields are treated: `compare`, `ignore`, `warn`, or `fail`. There MUST be no implicit policy. An `ignore` policy MUST still be represented as an assessment exclusion in accordance with Section 7.5. + +A field definition MAY declare: + +- canonical name; +- reference and query source names; +- required or optional status on each side; +- logical type; +- null policy; +- source-text and typed-value normalisation steps; +- comparator and comparator parameters; +- controlled vocabulary or value mapping; +- sensitivity or redaction classification; +- severity or acceptance contribution. + +### 6.4 Logical types and typed parsing + +The comparison engine MUST support custom specifications declaring scalar logical types `string`, `integer`, `decimal`, `boolean`, `date`, and `datetime`, and typed `list`, `set`, and `multiset` collections. Collection fields MUST declare their element type. Null is a distinct value state governed by field policy, not a logical type. + +Type parsing MUST be field-specific and declared. The tool MUST NOT infer a logical type from observed text. Values of different logical types MUST NOT be implicitly equal. Intended equivalence between differing source representations requires both sides to be parsed as the same declared destination logical type, with an explicit value mapping where the source representation cannot be parsed directly. The original source value or lexical representation MUST remain available in evidence, subject to redaction policy. + +An invalid typed value in a non-key field that remains attributable to an unambiguous record MUST be a contract finding and MUST make that value `not_assessable`; it MUST NOT be an execution error. Defects that prevent construction of a complete and unambiguous record model are classified under Section 8.3. + +#### 6.4.1 String + +- A string MUST preserve its Unicode code points exactly by default. +- `typed_exact` equality MUST be code-point-for-code-point equality. +- Empty strings, whitespace-only strings, nulls, and absent fields MUST remain distinct. + +#### 6.4.2 Integer + +- Textual integers MUST use signed base-10 notation consisting of an optional `+` or `-` followed by one or more ASCII digits. +- Decimal points, exponent notation, booleans, and other characters MUST be invalid after source-text normalisation and mapping. +- Leading zeros MUST be accepted and have no numerical significance. Identifiers for which leading zeros are significant MUST be declared as strings. +- Integer representation MUST support arbitrary precision within documented resource limits. + +#### 6.4.3 Decimal + +- Decimals MUST be parsed and represented using decimal arithmetic without an intermediate binary floating-point conversion. +- Plain and scientific decimal notation MUST be accepted. +- Numerically equivalent finite representations such as `1.0`, `1.00`, and `1e0` MUST have the same canonical value. +- Parsing MUST preserve all supplied numerical precision. +- `NaN`, positive infinity, and negative infinity MUST be invalid in the first release. + +#### 6.4.4 Boolean + +- Native JSON and YAML booleans MUST be accepted. +- Textual boolean parsing MUST accept only lowercase `true` and `false`. +- Integers, uppercase boolean-like values, `yes`, `no`, `on`, `off`, and other representations MUST require an explicit value mapping. +- Booleans MUST NOT be treated as integers. + +#### 6.4.5 Date + +- Dates MUST use the ISO 8601 full-date form `YYYY-MM-DD`. +- Values MUST represent valid Gregorian calendar dates. +- Partial dates and locale-dependent forms MUST be invalid. +- The canonical value MUST be the represented calendar date. + +#### 6.4.6 Datetime + +- Datetimes MUST use an RFC 3339-compatible representation containing a full date, time, and `Z` or numeric UTC offset. +- Local datetimes without an offset MUST be invalid. +- Equivalent instants expressed with different offsets MUST have the same canonical value. +- Fractional seconds MUST be preserved unless explicit precision normalisation is configured. +- Leap-second representations MUST be invalid in the first release. + +#### 6.4.7 Collections + +- Every collection MUST declare its element logical type and element comparator. +- A `list` MUST preserve order and duplicates. +- A `set` MUST be unordered and MUST reject duplicate source elements rather than silently collapse them. +- A `multiset` MUST be unordered and MUST preserve element occurrence counts. +- JSON and YAML collections MUST use array or sequence values. +- A CSV or TSV collection cell MUST use valid JSON-array syntax; ordinary delimited text MUST NOT be inferred to be a collection. +- Nested collections are outside the first-release scope. + +### 6.5 Null, blank, and absent values + +The specification MUST distinguish absent fields, explicit nulls, empty strings, whitespace-only strings, and configured missing tokens such as `NA`, `.`, or `null`. No two states are equivalent unless a field-specific policy declares them equivalent. Applied equivalence rules MUST be recorded. + +- Two explicit nulls MUST match only when the field permits null and its policy declares them equivalent. +- Null MUST NOT implicitly equal empty string, absent field, zero, boolean false, or a missing token. +- A missing token MUST become null only through an explicit field policy. +- Two absent required fields MUST produce contract findings and MUST NOT constitute a value match. +- A null and non-null pair MUST be a value mismatch when both states are valid and comparable. +- A prohibited null MUST be a contract finding and MUST make the affected value `not_assessable` unless another treatment is explicitly configured. + +The bundled strict-exact specification permits parser-native nulls. Under that specification, null matches only null and remains distinct from an absent field, an empty string, and every non-null value. + +### 6.6 Mappings + +#### 6.6.1 Column mappings + +- Reference and query fields MAY map to a canonical field name. +- Mappings MUST be one-to-one unless an explicitly defined transformation supports another cardinality. +- Conflicting target mappings MUST be rejected. +- Unknown source fields MUST follow the undeclared-field policy. +- Required mapped fields that are absent MUST be reported. +- All applied mappings MUST be identifiable in the evidence. + +#### 6.6.2 Value mappings + +- A value mapping MUST operate on a preserved source value before typed parsing. +- The mapping output MUST be parsed as the declared destination logical type. +- A map MUST declare its source representation and destination logical type. +- Unknown source values MUST default to a contract finding, not pass through silently. +- Many-to-one mappings MUST be allowed only when intentionally declared. +- Mapping to null MUST be distinguished from a missing map entry. +- Mapping keys MUST be parsed without format-dependent scalar ambiguity. +- Applied mappings MUST be identifiable in the evidence. + +### 6.7 Normalisation and processing order + +Normalisation MUST be opt-in, field-specific, ordered deterministically, and recorded. Source-text normalisation is distinct from typed-value normalisation. + +Source-text normalisation MAY include Unicode normalisation, leading or trailing whitespace trimming, and case folding. Typed-value normalisation MAY include datetime offset normalisation, declared precision reduction, and stable ordering of collections declared to be unordered. Destructive or clinically material transformations SHOULD require an independently specified domain comparator. + +Typed processing MUST occur in this order: + +1. preserve the source value and representation; +2. apply declared source-text normalisation; +3. recognise configured null or missing tokens; +4. apply a declared source-value mapping; +5. parse the declared logical type; +6. apply declared typed-value normalisation; +7. compare canonical values. + +Column-name mapping MUST occur before field-value processing. A failure at any stage MUST identify the stage and attributable source value, subject to redaction policy. + +### 6.8 Bundled strict-exact specification + +If `--specification` is omitted, the tool MUST select the bundled strict-exact specification. It MUST be distributed as a documented template from which laboratories can develop controlled custom specifications, and the selected effective specification MUST be recorded in provenance. Supplying `--specification` MUST select the named custom specification; sentinel path values such as `DEFAULT` MUST NOT have special meaning. + +The bundled default specification MUST: + +- declare one key whose canonical, reference, and query field names are all `record_id`; +- include every shared non-key field without requiring field-specific declarations; +- preserve parser-native types and use `preserved_exact` comparison; +- perform no type inference, implicit coercion, mapping, normalisation, or tolerance; +- permit parser-native null values and compare them according to Section 6.5; +- treat two syntactically valid zero-record datasets as satisfying all bundled acceptance criteria; +- treat reference-only and query-only fields as contract findings; +- require zero contract findings, zero missing or extra records, and zero value mismatches. + +Because parser-native types differ between formats, strict-exact comparison across different input formats MAY report type mismatches for lexically similar values. A custom typed specification is required when cross-format logical equivalence is intended. + +## 7. Comparison behaviour + +### 7.1 Cumulative phases and scopes + +Comparison phases MUST be cumulative and ordered: + +1. **Contract:** assess fields, logical types, key definitions, nullability, mappings, and declared data-contract constraints. +2. **Records:** align valid record identities and assess matching, missing, and extra records. +3. **Values:** compare corresponding values using declared comparators. + +Public scope names identify the final requested phase: + +| Scope | Contract | Records | Values | +|---|---|---|---| +| `contract` | Full | Not assessed | Not assessed | +| `records` | Full | Full | Not assessed | +| `full` | Full | Full | Full | + +`full` MUST be the default. Failure of an earlier phase MUST NOT automatically prevent later phases. Later comparison MUST continue wherever its prerequisites remain satisfied. Content inside the selected scope that cannot be compared safely MUST be `not_assessable`; content deliberately outside the scope or excluded by policy MUST be `not_assessed`. + +A run MUST conceptually: + +1. validate the effective specification; +2. identify and hash inputs; +3. parse inputs without silent loss; +4. perform the contract phase; +5. align and compare records when the scope is `records` or `full`; +6. compare values for safely aligned records when the scope is `full`; +7. evaluate applicable phase acceptance criteria; +8. write results and provenance atomically. + +### 7.2 Contract comparison + +The contract phase MUST assess declared and observed fields from both directions for every record. It MUST distinguish: + +- field present and valid; +- required field missing from the reference or query; +- unexpected reference or query field; +- invalid logical type, null state, mapping, or controlled value; +- field deliberately `not_assessed`; +- field `not_assessable` because a prerequisite failed. + +One record’s observed field set MUST NOT be assumed to represent every other record. Key presence, validity, and uniqueness MUST be assessed in every scope because later interpretation depends on an unambiguous record model. + +### 7.3 Records comparison + +For the union of normalised valid record keys, the records phase MUST classify each key as matching, missing from the query, or extra in the query. Duplicate or otherwise invalid keys are execution errors under Section 8.3, not ordinary record findings. + +Missing and extra records MUST be reported individually unless the selected privacy mode permits only aggregate reporting. Query-only records MUST NOT be ignored. Missing or extra records MUST NOT prevent value comparison of the safely aligned intersection when the scope is `full`. + +### 7.4 Values comparison and comparators + +The values phase MUST compare every in-scope, valid field in safely aligned records. A value whose contract or alignment prerequisite failed MUST be `not_assessable`; other values MUST continue to be compared. + +The first release MUST support the following deterministic comparators with stable identifiers. + +#### 7.4.1 `preserved_exact` + +Used by the bundled strict-exact specification. Values match only when their parser-native types and values are identical. No inference, coercion, mapping, normalisation, or tolerance is applied. Textual `"1"`, integer `1`, decimal `1.0`, and boolean `true` are distinct. + +#### 7.4.2 `typed_exact` + +Values match only when both are present, valid under their field policies, have the same declared logical type, and have equal canonical values. Null handling follows Section 6.5. Cross-type equality is not implicit. + +#### 7.4.3 `normalised_string` + +Only configured string normalisations are applied in their declared order. Equality is exact code-point equality after transformation. Applied transformations MUST be identifiable in the evidence. + +#### 7.4.4 `numeric_exact` + +Integer or decimal comparison MUST use canonical arbitrary-precision values. Integer and decimal remain different logical types. Differing source representations are comparable only when both are explicitly parsed as the same declared logical type, using a mapping where required. + +#### 7.4.5 `numeric_tolerance` + +```text +difference = abs(reference - query) +allowed_difference = max(absolute_tolerance, relative_tolerance * abs(reference)) +match if difference <= allowed_difference +``` + +- At least one tolerance MUST be supplied; an omitted tolerance contributes zero. +- Tolerances MUST be non-negative finite decimals. +- The boundary MUST be inclusive. +- Relative tolerance MUST be directional and calculated against the absolute reference value. +- Absolute tolerance MUST govern behaviour around a zero reference value. +- A numeric field MAY declare a unit as metadata. If unit semantics affect equivalence or acceptance, the unit MUST be declared explicitly and MUST be compatible on both sides. Automatic unit conversion is outside the first-release scope. +- Rounding or significant-figure normalisation MUST occur explicitly before comparison. + +#### 7.4.6 `date_exact` and `datetime_exact` + +Dates match when their canonical calendar dates are equal. Datetimes match when they represent the same instant after UTC-offset normalisation. Any precision reduction MUST be explicitly configured before comparison. + +#### 7.4.7 `list_exact`, `set_exact`, and `multiset_exact` + +- Lists use ordered element-by-element comparison and their lengths MUST match. +- Sets use order-independent comparison of unique elements; duplicate source elements are invalid under Section 6.4.7. +- Multisets use order-independent comparison that preserves occurrence counts. +- Elements use the declared element comparator. + +### 7.5 Completeness, subsets, and determinism + +- Comparison findings MUST NOT terminate a normal run. +- All assessable differences MUST be collected. +- A configurable finding limit MAY protect resources and reaching it MUST be reported prominently as truncation. If a conclusive criterion failure has already been established, the outcome MUST remain `fail`; otherwise truncation MUST make the outcome `inconclusive`. A truncated run MUST NOT make claims about unexamined content. +- A malformed record that prevents a complete and unambiguous record model MUST be an input error and MUST NOT disappear silently. +- A field subset MUST be declared in the specification or invocation parameters retained in provenance. +- Key fields MUST always be processed. +- Individual findings for fields outside a configured subset MAY be omitted only under a documented report policy. The canonical report MUST still record a stable exclusion code, the exclusion rule, the affected field names or configured subset, and the number of excluded records or values where calculable. +- The report MUST identify that only a subset was assessed. +- An unknown requested field MUST be an error. +- Record order MUST NOT affect results unless explicitly declared significant. +- Output ordering MUST be deterministic and independent of hash-map ordering and locale. + +## 8. Findings and outcomes + +### 8.1 Finding structure + +Every finding MUST be structured and SHOULD contain: + +- stable finding code; +- category; +- severity; +- record key, or privacy-preserving surrogate; +- reference and query source locations where available; +- canonical field name; +- expected and observed presence state; +- expected and observed logical type; +- expected and observed value, subject to redaction policy; +- comparator and parameters; +- mappings/normalisations applied; +- concise human-readable explanation; +- related specification requirement or rule identifier. + +Human-readable messages MUST NOT be the sole machine-readable representation of a finding. Successful field and record matches MUST be represented in summary counts and MUST NOT produce individual findings. + +### 8.2 Minimum finding categories + +Stable result codes MUST exist for at least: + +- missing record; +- extra record; +- invalid record key; +- missing field; +- extra field; +- type mismatch; +- value mismatch; +- mapping failure; +- parse or data-contract error; +- invalid specification; +- not assessed; +- not assessable; +- comparison truncated; +- internal execution error. + +### 8.3 Execution status + +Execution status MUST be separate from comparison outcome and MUST include at least: + +- `completed`: comparison ran as specified and the result package was written; +- `error`: the tool could not establish a complete and unambiguous record model or could not complete the required output operation. + +An execution error includes an invalid invocation or specification, malformed or unreadable input syntax or structure, missing, null, invalid, or duplicate record keys, input/output failure, or an unexpected internal failure. No comparison outcome MUST be produced for an execution error. An error report SHOULD be written where possible and MUST be identified as such. + +Content that is parseable and attributable to an unambiguous record but violates the declared data contract MUST be recorded as a comparison finding rather than an execution error. Comparisons that remain assessable MUST continue. A completed comparison MAY have a failing or inconclusive comparison outcome. + +### 8.4 Comparison outcome + +Comparison outcome MUST be one of: + +- `pass`: all configured acceptance criteria were met; +- `fail`: one or more configured acceptance criteria were not met; +- `inconclusive`: criteria could not be fully evaluated, including truncation or configured recoverable data errors. + +A completed run is `inconclusive` only when the inputs remain sufficiently trustworthy for comparison, one or more requested assessments cannot be completed, and the available evidence cannot establish a defensible pass or fail. It MUST identify all unassessed content and MUST never produce a pass. + +Each executed phase MUST have its own outcome. A conclusive failure of any applicable criterion MUST produce overall `fail`. If no criterion conclusively fails but one or more applicable criteria cannot be evaluated, the overall outcome MUST be `inconclusive`. Overall `pass` requires every applicable criterion to be evaluated and satisfied. The outcome MUST be calculated only from explicit acceptance criteria and MUST NOT be inferred merely from whether warnings occurred. + +### 8.5 Acceptance criteria + +The specification MUST associate acceptance criteria with the individual `contract`, `records`, and `values` phases. It MUST NOT define a separate `full` acceptance section. The `contract` scope evaluates contract criteria; `records` evaluates contract and records criteria; and `full` evaluates all three phase sections. + +The specification MUST define how findings contribute to outcome. Criteria MAY include: + +- zero missing or extra records; +- zero mismatches in critical fields; +- a maximum number or proportion of specified differences; +- metric-specific thresholds; +- permitted known differences identified by stable rule and rationale. + +Criteria based on counts or proportions MUST define their denominator and the treatment of deliberately `not_assessed` content. `not_assessable` content MUST NOT be counted as a match or silently removed from a denominator. A criterion that depends on `not_assessable` content MUST be unevaluable unless its failure can be established conclusively from the assessable evidence. + +Exceptions and permitted differences MUST be explicit, versioned, reviewable, and reported. A custom specification missing acceptance criteria for any phase applicable to the selected scope MUST be invalid. + +### 8.6 Laboratory decision + +As established in Section 1, the calculated comparison outcome is not a laboratory validation or approval decision. Reports MUST label these concepts separately and MUST NOT describe the output as “ISO 15189 compliant”, “clinically validated”, or equivalent. + +## 9. Output and evidence + +A completed comparison MUST create an evidence-package directory at the destination selected by `--output`. The package MUST contain, at minimum, the canonical JSON result and the HTML report. A CSV findings export MAY also be included. Package member names and layout MUST be stable and documented. + +### 9.1 Machine-readable output + +JSON MUST be the canonical output format. It MUST have a published, versioned schema containing: + +- report schema version; +- execution status and comparison outcome; +- run identifier; +- timestamps; +- tool identity and version; +- specification identity, version, and digest; +- input identities and digests; +- effective comparison settings; +- summary counts; +- structured findings; +- errors and warnings; +- truncation status; +- privacy/redaction mode. + +Consumers MUST be able to distinguish absent, null, and redacted values. + +### 9.2 Human-readable output + +The first release MUST produce an HTML human-readable report. The report SHOULD include: + +- purpose and scope of the run; +- prominent outcome and execution status; +- identities of reference, query, and specification; +- summary counts; +- acceptance criteria and evaluation; +- grouped discrepancy details; +- limitations, exclusions, and unassessed content; +- provenance and reproduction information; +- space or linkage for review and approval outside the tool. + +CSV MAY be provided as an export of findings, but MUST: + +- be produced using a conforming CSV writer; +- preserve record identifiers or approved surrogates; +- quote delimiters, quotes, and newlines correctly; +- use a stable documented column schema; +- mitigate spreadsheet formula injection when intended for spreadsheet use; +- never be the only canonical evidence format. + +### 9.3 Summary counts + +Summary counts MUST reconcile with emitted findings and SHOULD include: + +- input record counts; +- valid unique key counts; +- matched, missing, and extra record counts; +- fields assessed and not assessed; +- findings by code and severity; +- pass/fail/inconclusive criteria counts. + +### 9.4 Provenance + +Provenance MUST include: + +- SHA-256 or stronger digest of each input and the specification; +- tool version and build/release identity; +- report schema version; +- invocation parameters after redaction; +- start and end timestamps with time-zone offset; +- runtime environment sufficient for reproduction; +- selected input formats and encodings; +- all non-default policies in effect. + +Absolute paths and the raw command line MUST NOT be included by default because they may contain sensitive information or credentials. Safe display names MAY be supplied separately. + +### 9.5 Atomicity and overwrite protection + +- Output files MUST be written atomically where supported. +- The complete evidence-package directory MUST be published atomically where the platform supports it. +- An existing destination MUST NOT be overwritten unless the user supplies `--overwrite`. +- The actual selected output directory MUST be reported clearly. +- File suffixes MUST be handled structurally, not with character stripping. +- Partial evidence packages MUST be identifiable as incomplete. + +## 10. Command-line interface + +### 10.1 General behaviour + +- A subcommand MUST be required. +- Invalid argument combinations MUST be rejected before input processing. +- Modes and formats MUST use enumerated choices. +- `--help` and `--version` MUST work without other arguments. +- Errors MUST be concise on standard error and actionable. +- Normal summaries SHOULD be written to standard output. +- Comparison progress SHOULD be optional and MUST NOT contaminate machine-readable output. + +### 10.2 Proposed comparison interface + +The conceptual interface is: + +```text +adviso-validation compare \ + --reference REFERENCE \ + --query QUERY \ + --output OUTPUT \ + [--scope {contract,records,full}] \ + [--specification SPECIFICATION] \ + [--reference-format {csv,tsv,json,yaml}] \ + [--query-format {csv,tsv,json,yaml}] \ + [--overwrite] +``` + +`--scope` MUST accept `contract`, `records`, or `full`, and MUST default to `full`. Scope selection and any content made `not_assessable` by contract incompatibility MUST be prominent in the report. + +`--specification` MUST be optional. Its omission MUST select the bundled strict-exact specification; when present, its value MUST be treated as the path to a custom specification. `--reference-format` and `--query-format` MUST follow Section 5.2. `--output` MUST name the evidence-package directory described in Section 9, and `--overwrite` MUST provide the explicit overwrite authorisation required by Section 9.5. + +Convenience flags MAY override non-critical settings only when the effective configuration is captured in provenance. Clinically material overrides SHOULD require an explicit opt-in and be prominent in the report. + +### 10.3 Exit codes + +Exit codes MUST distinguish at least: + +| Code | Meaning | +|---:|---| +| 0 | Execution completed and comparison outcome is `pass` | +| 1 | Execution completed and comparison outcome is `fail` | +| 2 | Execution completed but comparison outcome is `inconclusive` | +| 3 | Invocation, specification, input, or I/O error | +| 4 | Unexpected internal error | + +These exit-code meanings MUST be stable and documented. A comparison failure MUST still write a report when possible. + +### 10.4 Additional commands + +No additional command is specified for the initial release. In particular, the prototype’s format-conversion operation is excluded because it does not presently justify a place in the validation-evidence workflow. Any future command requires its own approved behaviour specification and MUST satisfy the product-boundary criteria in Section 2.1. + +## 11. Privacy, security, and data governance + +- The tool MUST assume that datasets may contain confidential patient or laboratory information. +- It MUST operate locally by default and MUST NOT transmit input or result data over a network without explicit functionality and authorisation. +- Reports MUST support configurable redaction or hashing of record identifiers and sensitive values. +- Hashing used as pseudonymisation MUST support a laboratory-controlled keyed method where resistance to dictionary attacks is required. +- Secrets MUST be redacted from recorded invocation details. +- Temporary files MUST be created securely and removed according to documented policy. +- Logs MUST NOT contain more sensitive information than the selected report policy permits. +- YAML MUST be parsed using a safe loader. +- Spreadsheet-oriented exports MUST mitigate formula injection. +- Resource limits SHOULD protect against unexpectedly large or malicious inputs without silently truncating results. +- The tool SHOULD document filesystem permission expectations and leave access control to an appropriate deployment wrapper where necessary. + +## 12. Deployment, accessibility, performance, and reliability + +The toolkit is intended to be usable in low-resource and LMIC settings. Therefore: + +- all core comparison functionality MUST operate fully offline; +- the toolkit MUST NOT require a hosted service, account, licence server, or telemetry connection; +- the base installation SHOULD minimise runtime dependency number and size; +- specialised dependencies SHOULD be optional and MUST NOT be required for strict structured-data comparison; +- an offline-installable artefact with integrity information SHOULD be provided; +- a container image MAY be supplied but MUST NOT be the only deployment method; +- reports MUST use open formats and MUST NOT require proprietary software; +- errors and documentation SHOULD be usable without specialist Python knowledge; +- documentation and examples SHOULD be available offline; +- localisation SHOULD remain possible without changing result semantics. + +Behaviour MUST be correct before optimisation. Releases SHOULD state: + +- the modest hardware profiles on which they were tested; +- representative dataset sizes; +- expected time and memory use; +- enforced resource or input-size limits. + +Large inputs MUST either be processed within documented limits or rejected without producing a false pass. Resource exhaustion MUST produce an execution error or `inconclusive` outcome according to whether a trustworthy partial record model remains. Interrupted runs MUST NOT leave apparently complete reports. Results MUST remain deterministic across supported Python versions and operating systems. Internal exceptions MUST be captured in an error report where safe, with concise diagnostics presented to the user. + +## 13. Software quality and release requirements + +### 13.1 Requirements traceability + +- Normative requirements intended for implementation MUST receive stable identifiers before the specification is approved. +- Tests MUST reference relevant requirement identifiers. +- A traceability report SHOULD show requirement-to-test and test-to-requirement coverage. + +### 13.2 Testing + +The project MUST include: + +- unit tests for parsing, typing, mapping, normalisation, and comparators; +- behavioural contract tests for result semantics; +- end-to-end CLI tests including exit codes; +- schema validation tests for specifications and reports; +- installation tests from wheel and source distribution; +- regression tests for every corrected material defect; +- tests across all supported Python versions; +- deterministic-output tests; +- representative malformed and adversarial inputs; +- tests for duplicate/composite keys and missing/extra content in both directions; +- boundary tests for tolerances and acceptance criteria. + +Property-based testing SHOULD be used for parsing and comparison invariants. Mutation testing SHOULD be used to assess whether the suite detects deliberately introduced comparison defects. + +### 13.3 Test data + +Controlled test cases SHOULD record: + +- stable scenario identifier; +- requirement(s) tested; +- purpose and rationale; +- input data and comparison specification; +- expected findings, counts, status, outcome, and exit code; +- fixture provenance or generation method; +- fixture version and digest; +- independent review status where used as formal validation evidence. + +Generated fixtures MUST be reproducible. Clinically derived fixtures MUST be appropriately de-identified and governed. Controlled test fixtures, including test comparison specifications, MUST be maintained by the development team under version control and its documented review process. In operational use, each laboratory is responsible for creating, approving, maintaining, and retaining its own comparison specifications under its local quality-management arrangements; the AdvISO project does not centrally approve or manage laboratory specifications. + +### 13.4 Continuous integration + +CI MUST: + +- run on proposed changes and protected branches; +- install the built package, not rely solely on the source checkout; +- test the console entry point; +- test the supported Python matrix; +- fail on enforced lint, type, test, and build errors; +- validate that generated schemas and examples are current; +- avoid publishing releases when required checks fail. + +### 13.5 Releases and change control + +Each release SHOULD provide: + +- an immutable semantic version; +- release notes and known limitations; +- compatibility statements for specification and report schemas; +- documented change-impact assessment; +- signed or otherwise attestable source and build artefacts where practical; +- a reproducible lockfile or equivalent dependency record; +- evidence that required verification tests passed. + +Specification-schema versions and toolkit versions MUST be independent. Each toolkit release MUST document the specification-schema and report-schema versions it supports. An unsupported schema version MUST be rejected clearly rather than interpreted approximately. The project MUST publish migration and compatibility rules before changing either schema and MUST provide a migration path or documented transition period for any incompatible change. + +## 14. Acceptance-test catalogue + +This catalogue summarises minimum executable scenarios for the first release. It verifies, but does not independently define, the normative behaviour above. When normative requirement identifiers are assigned, every scenario MUST reference the requirements it verifies. + +### 14.1 Input and contract scenarios + +- **AT-CON-001:** Identical typed datasets satisfy their contract. +- **AT-CON-002:** Required missing and unexpected extra fields are reported in both directions and across heterogeneous records. +- **AT-CON-003:** Leading-zero identifiers remain strings unless explicitly transformed. +- **AT-CON-004:** Null, empty string, whitespace-only string, absent field, and configured missing tokens remain distinct according to policy. +- **AT-CON-005:** Unknown, conflicting, and incomplete mappings fail clearly; mapping to null differs from a missing map entry. +- **AT-CON-006:** Malformed syntax or structure, empty or duplicate column names, and irregular tabular rows receive their specified error classification without silent data loss. +- **AT-CON-007:** Two syntactically valid empty datasets pass under the bundled strict-exact specification; one-sided empty datasets produce all applicable missing or extra record findings. +- **AT-CON-008:** YAML scalar resolution and cross-format typed parsing are deterministic across supported environments. +- **AT-CON-009:** The bundled strict-exact specification treats parser-native null as equal only to null and distinguishes null, absent fields, and empty strings. +- **AT-CON-010:** Every custom specification explicitly declares an undeclared-field policy; omission is rejected. + +### 14.2 Record scenarios + +- **AT-REC-001:** Every reference-only record is reported as missing and every query-only record as extra, including when both occur together. +- **AT-REC-002:** Duplicate single and composite keys are rejected without overwriting data. +- **AT-REC-003:** Composite keys preserve multiple legitimate records belonging to one sample. +- **AT-REC-004:** Record order does not alter findings or outcomes unless explicitly declared significant. + +### 14.3 Typed value and comparator scenarios + +- **AT-VAL-001:** Boolean `true`, integer `1`, decimal `1.0`, and textual `"1"` remain distinct under the applicable exact comparator. +- **AT-VAL-002:** Typed strings remain code-point-exact unless configured normalisations are applied in declared order. +- **AT-VAL-003:** Integer parsing accepts signed ASCII decimal integers and leading zeros but rejects decimal points, exponents, booleans, and invalid text. +- **AT-VAL-004:** Decimal parsing is exact, accepts scientific notation, and rejects `NaN` and infinities. +- **AT-VAL-005:** Boolean parsing accepts native booleans and lowercase textual `true` or `false`; other representations require a mapping. +- **AT-VAL-006:** Dates reject invalid or non-ISO full dates; datetimes require offsets and compare as instants. +- **AT-VAL-007:** Lists preserve order and duplicates, sets reject duplicates and ignore order, and multisets ignore order while preserving occurrence counts. +- **AT-VAL-008:** Typed transformations follow Section 6.7 and retain evidence of the attributable source value. +- **AT-VAL-009:** Numeric tolerance uses exact decimal arithmetic, a reference-relative component, and an inclusive boundary, including below, at, and above boundary cases. +- **AT-VAL-010:** All assessable differences are collected in one run. +- **AT-VAL-011:** Subset comparison identifies its limited coverage, records the required exclusion summary when individual findings are omitted, and rejects unknown requested fields. +- **AT-VAL-012:** `preserved_exact` follows the parser-native category table, including deterministic same-format and cross-format comparisons. + +### 14.4 Scope and outcome scenarios + +- **AT-OUT-001:** `contract` assesses only the contract phase. +- **AT-OUT-002:** `records` assesses contract and records without comparing values. +- **AT-OUT-003:** `full` is the default and assesses contract, records, and values. +- **AT-OUT-004:** Failure of an earlier phase does not prevent later assessment where prerequisites remain satisfied. +- **AT-OUT-005:** Content excluded deliberately is `not_assessed`; in-scope content blocked by a failed prerequisite is `not_assessable`. +- **AT-OUT-006:** A conclusive criterion failure produces `fail`; otherwise an unevaluable applicable criterion produces `inconclusive`; every applicable criterion must pass for overall `pass`. +- **AT-OUT-007:** Pass, fail, inconclusive, invocation or input error, and internal error produce their documented exit behaviour. +- **AT-OUT-008:** Truncation preserves an already established conclusive failure; otherwise it produces `inconclusive` and identifies unexamined content. +- **AT-OUT-009:** Counts and proportions use declared denominators; `not_assessable` content is never treated as a match or silently excluded. + +### 14.5 Evidence, CLI, and packaging scenarios + +- **AT-EVD-001:** Identical runs produce semantically identical canonical results except for explicitly variable run metadata. +- **AT-EVD-002:** Reports contain input and specification digests without exposing absolute paths or raw secrets by default. +- **AT-EVD-003:** CSV findings containing commas, quotes, newlines, or formula prefixes are exported safely. +- **AT-EVD-004:** An output failure or interrupted write cannot leave an evidence package that appears complete. +- **AT-EVD-005:** A completed passing comparison exits with code 0; comparison failures still produce reports. +- **AT-EVD-006:** Omitting `--specification` selects and records the bundled strict-exact specification; a supplied value is treated only as a custom-specification path. +- **AT-EVD-007:** Explicit reference and query formats override extension detection; recognised extensions are detected and recorded; unknown or ambiguous extensions require explicit formats. +- **AT-EVD-008:** A successful run creates the documented JSON and HTML evidence package, refuses an existing destination by default, and replaces it only with `--overwrite`. +- **AT-EVD-009:** Invocation, specification, input, and output errors write a clearly identified error report when an output location can be established. +- **AT-PKG-001:** A built wheel contains every required subpackage, and its installed CLI completes an end-to-end comparison. + +## 15. Compatibility with the current prototype + +The current `File_IO`, `FileCompare`, and `TableValidator` module names and APIs MUST NOT constrain the new core data model or public terminology. In particular, the new model MUST NOT use a single dictionary keyed only by sample ID as its sole record representation. + +Compatibility MAY be provided temporarily through an adapter, provided that: + +- legacy semantics are labelled clearly; +- unsafe behaviour such as silent duplicate overwrite is not preserved; +- the adapter has a documented removal policy; +- new evidence reports identify use of legacy compatibility mode. + +The current free-form result strings MUST NOT become the canonical result API. + +## 16. Decisions required before approval + +The following non-kernel decisions remain open. They do not prevent implementation of the comparison kernel, but MUST be resolved at the appropriate point before the specification and first release are approved. + +1. What default privacy and redaction mode is appropriate for the intended deployment environment? +2. Which provenance fields are mandatory in local interactive use, CI, and formal retained evidence? +3. What maximum input sizes and performance targets are operationally relevant? +4. What compatibility period, if any, is required for the prototype CLI and outputs? +5. What objective hardware profile should define “modest” deployment and performance testing? + +## 17. Suggested approval criteria for this specification + +Before implementation begins, reviewers SHOULD confirm that: + +- the intended use and non-claims are accurate; +- terminology matches laboratory practice; +- the record model covers representative pipeline outputs; +- missing, extra, duplicate, null, and malformed-data policies are unambiguous; +- comparison and acceptance are clearly separated; +- privacy and provenance requirements fit local governance; +- every first-milestone scenario has an agreed expected result; +- unresolved decisions have named owners and target dates; +- approved requirements have stable identifiers for test traceability. + + From cc8ed5a67c58b2c43f9cb42c159f7ab27758932f Mon Sep 17 00:00:00 2001 From: Kevin Howe Date: Thu, 15 Oct 2026 17:40:11 +0100 Subject: [PATCH 02/45] More detail in spec --- SPECIFICATION.md | 414 +++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 347 insertions(+), 67 deletions(-) diff --git a/SPECIFICATION.md b/SPECIFICATION.md index 72b3e53..957f865 100644 --- a/SPECIFICATION.md +++ b/SPECIFICATION.md @@ -29,7 +29,7 @@ Responsibility for the provenance and suitability of reference material, the com The organisation of the toolkit within the AdvISO GitHub organisation is: ```text -AdvISO +adviso-project └── validation_toolkit ├── compare subcommand └── internal comparison, specification, provenance, findings, and reporting components @@ -51,7 +51,7 @@ Additional commands are not a development objective for the first release and MU The first-release scope of `compare` is: -- comparison of structured records represented in CSV, TSV, JSON, or YAML; +- comparison of structured records represented in CSV, TSV or JSON; - explicit single-field or composite record keys; - detection of matching, differing, missing, and extra records and fields; - field-specific comparison rules; @@ -80,11 +80,11 @@ No bioinformatics-specific or domain-aware comparator is required for the first The following principles govern all requirements: -1. **Explicit over inferred:** Clinically material comparison behaviour MUST be declared, not guessed from observed data. +1. **Explicit over inferred:** Comparison behaviour MUST be declared, not guessed from observed data. 2. **No silent data loss:** Input records, duplicate keys, unmapped values, and unsupported structures MUST NOT be silently discarded or overwritten. 3. **Symmetry:** Missing and extra content MUST be assessed in both directions. -4. **Separation of concerns:** Execution success, comparison outcome, and laboratory acceptance decision are distinct concepts. -5. **Complete evidence:** A normal comparison failure MUST still produce a complete report. +4. **Separation of concerns:** Execution success and comparison outcome are distinct concepts. +5. **Complete evidence:** A normal comparison mismatch MUST still produce a complete report. 6. **Determinism:** Identical inputs, configuration, and tool version MUST produce semantically identical results. 7. **Traceability:** Findings MUST identify the rule, record, and field that produced them. 8. **Privacy by design:** Reports MUST include only the sensitive data necessary for their intended purpose. @@ -126,7 +126,7 @@ A comparison run MUST receive: - one query dataset; - an output destination. -A custom comparison specification MAY be supplied. If it is omitted, the bundled strict-exact specification defined in Section 6.8 MUST be used. +A custom comparison specification MAY be supplied. If it is omitted, the bundled strict-exact specification defined in Section 6.9 MUST be used. ### 5.2 Supported formats @@ -134,10 +134,9 @@ The initial implementation MUST support: - CSV conforming to the configured dialect; - TSV conforming to the configured dialect; -- JSON using a documented record-oriented structure; -- YAML using the same logical structure as JSON. +- JSON using a documented record-oriented structure. -The reference and query formats MAY be selected independently through explicit command-line options. If a format is not supplied, it MAY be detected only from a recognised, unambiguous extension: `.csv`, `.tsv`, `.json`, `.yaml`, or `.yml`. In either case: +The reference and query formats MAY be selected independently through explicit command-line options. If a format is not supplied, it MAY be detected only from a recognised, unambiguous extension: `.csv`, `.tsv` or `.json`. In either case: - detected format MUST be recorded; - ambiguous or unsupported content MUST fail validation; @@ -164,9 +163,9 @@ For CSV and TSV: - source row numbers SHOULD be retained for traceability; - input values MUST initially remain strings unless a custom specification declares a null representation or type conversion; in the bundled strict-exact specification, an empty cell is an empty string. -### 5.5 JSON and YAML parsing +### 5.5 JSON parsing -- JSON and YAML inputs MUST use a schema-validated list of record objects: +- JSON inputs MUST use a schema-validated list of record objects: ```json [ @@ -175,28 +174,24 @@ For CSV and TSV: ] ``` -- YAML scalar resolution MUST use a documented restricted YAML 1.2-compatible schema in which only lowercase `true`, `false`, and `null` receive implicit non-numeric scalar types. - Values such as `yes`, `no`, `on`, `off`, uppercase boolean-like values, and timestamp-like strings MUST remain strings unless a custom specification explicitly maps or parses them. -- JSON and YAML numerical tokens that may be used as decimals MUST be loaded without an intermediate binary floating-point conversion, and their source lexical representations MUST remain available. +- JSON numerical tokens that may be used as decimals MUST be loaded without an intermediate binary floating-point conversion, and their source lexical representations MUST remain available. - Root scalars, heterogeneous root structures, and records that are not objects MUST be rejected with actionable errors. ### 5.6 Preserved parser-native values -For `preserved_exact` comparison, parsers MUST expose a common, deterministic set of parser-native value categories: - -- `string`, `integer`, `decimal`, `boolean`, `null`, `array`, and `object`; +- For `preserved_exact` comparison, parsers MUST expose a common, deterministic set of parser-native value categories: `string`, `integer`, `decimal`, `boolean`, `null`, `array`, and `object`; - CSV and TSV cells MUST be strings, including empty strings, unless a custom specification declares further processing; - JSON numbers without a fractional or exponent component MUST be integers, while other finite JSON numbers MUST be exact decimals; -- YAML scalars MUST resolve under the restricted schema in Section 5.5 to strings, arbitrary-precision integers, exact finite decimals, booleans, or nulls; timestamps MUST NOT be resolved implicitly; -- JSON arrays and YAML sequences MUST be arrays, and JSON or YAML mappings used as field values MUST be objects; and -- equality MUST require the same parser-native category and recursively equal content. Array order MUST be significant and object member order MUST not be significant. - -The source lexical representation of numerical values MUST remain available as evidence even though numerically equivalent representations within the same parser-native numerical category compare equally. Non-finite numerical values MUST be rejected. +- JSON arrays MUST be arrays, and JSON mappings used as field values MUST be objects; and +- Values are equal only when they have the same parser-native category and recursively equal content. For array-valued fields, element order is significant under `preserved_exact`; for object-valued fields, member names and values must match but member order is not significant. Top-level records are aligned by their declared record keys, and their input order does not affect comparison. +- Numerical values MUST be retained exactly as they appeared in the input for reporting and traceability. However, two values in the same numerical category compare as equal when they represent the same number; for example, decimal `1.0` and decimal `1.00` are equal. +- `NaN`, positive infinity, and negative infinity are not valid numerical values. In an attributable non-key field, they MUST produce a contract finding and make that value `not_assessable`; other assessable content MUST still be compared. If such a value occurs in a record key or prevents construction of an unambiguous record model, it MUST produce an execution error. ### 5.7 Empty inputs - Empty files and datasets with zero records MUST be recognised explicitly. -- A custom comparison specification MUST declare whether two empty datasets satisfy its records-phase acceptance criteria. +- A custom comparison specification MUST state whether a comparison in which both datasets contain no records is acceptable. It MAY accept two empty datasets as having no missing or extra records, or require a minimum number of records. - A non-empty reference and empty query MUST report every expected record as missing. - An empty reference and non-empty query MUST report every observed record as extra. @@ -224,7 +219,301 @@ A valid specification MUST declare at least: - duplicate-key policy; - phase-specific acceptance criteria. -### 6.2 Record keys +### 6.2 Custom comparison-specification YAML format + +Custom comparison specifications MUST be YAML documents conforming to the format defined in this section. YAML is used only for the comparison specification; it is not a supported dataset format. A specification file SHOULD use the `.yaml` extension; `.yml` MAY also be accepted. + +#### 6.2.1 YAML parsing and validation + +- The document MUST be UTF-8 encoded and contain exactly one YAML mapping at its root. +- YAML 1.2 scalar resolution MUST be used with a safe loader. +- Duplicate mapping keys, merge keys, aliases, anchors, explicit tags, and executable or implementation-specific object constructors MUST be rejected. +- Unknown keys MUST be rejected unless a future schema version explicitly defines an extension mechanism. +- Required keys MUST NOT be supplied as YAML null values. +- `schema_version`, `id`, `revision`, criterion identifiers, exception identifiers, field names, operation names, comparator names, enumerated values, decimal lexical values, dates, and datetimes SHOULD be quoted to prevent unintended YAML scalar conversion. +- The complete document MUST be validated against the published schema identified by `schema_version` before either dataset is read. +- Validation errors MUST identify the YAML path and the violated requirement. + +#### 6.2.2 Complete structural template + +The following template defines the complete first-release structure. Keys marked “optional” in comments MAY be omitted; other displayed keys are required when their parent construct is used, except where a conditional omission is explicitly permitted below. Empty lists are valid only where stated below. + +```yaml +schema_version: "1.0" + +specification: + id: "org.example.pipeline-output-comparison" + revision: "1.0.0" + title: "Example pipeline-output comparison" # optional + description: "Approved comparison rules" # optional + +records: + keys: + - "record_id" + policies: + duplicate_keys: "error" + missing_records: "finding" + extra_records: "finding" + empty_datasets: "accept" + +fields: + include: "all" + undeclared: "fail" + policies: + missing_fields: "finding" + extra_fields: "finding" + definitions: + record_id: + source: + reference: "sample_id" + query: "specimen_id" + required: + reference: true + query: true + logical_type: "string" + null: + allowed: false + nulls_equal: false + missing_tokens: + reference: [] + query: [] + source_normalisation: [] + value_mapping: + reference: [] + query: [] + typed_normalisation: [] + comparator: + name: "typed_exact" + parameters: {} + unit: "laboratory-defined-unit" # optional + sensitivity: "identifier" # optional label + finding_severity: "error" # optional + +acceptance: + contract: + - id: "no-contract-findings" + metric: "contract_finding_count" + operator: "equal" + threshold: 0 + records: + - id: "no-missing-records" + metric: "missing_record_count" + operator: "equal" + threshold: 0 + - id: "no-extra-records" + metric: "extra_record_count" + operator: "equal" + threshold: 0 + values: + - id: "no-value-mismatches" + metric: "value_mismatch_count" + operator: "equal" + threshold: 0 + +exceptions: [] + +reporting: + finding_limit: 10000 # optional + excluded_field_detail: "summary" +``` + +The template is illustrative rather than an instruction to use these particular rules. In particular, laboratories MUST choose identifiers, field definitions, policies, criteria, thresholds, exceptions, and reporting limits appropriate to the intended comparison. The published machine-readable schema MUST encode the same structure and constraints as this section; where the schema and this document disagree, the normative behaviour specification MUST be corrected before release rather than silently choosing one interpretation. + +#### 6.2.3 Top-level keys and identity + +The only permitted top-level keys are `schema_version`, `specification`, `records`, `fields`, `acceptance`, `exceptions`, and `reporting`. + +- `schema_version` MUST be a quoted string identifying a supported comparison-specification schema version. +- `specification.id` MUST be a stable, non-empty identifier. It SHOULD use a collision-resistant organisational namespace. +- `specification.revision` MUST be a non-empty revision string controlled by the specification owner. Any behaviourally material change MUST produce a new revision. +- `specification.title` and `specification.description` are optional human-readable strings and MUST NOT alter comparison behaviour. + +#### 6.2.4 Record configuration + +- `records.keys` MUST be a non-empty ordered list of canonical field names present in `fields.definitions`. List order defines composite-key order. +- Every key field MUST be required on both sides, non-nullable, and declared with a scalar logical type. A key field MAY use explicitly declared source normalisation or value mapping, but MUST use `typed_exact` after transformation. +- `records.policies.duplicate_keys` MUST be `error`; `missing_records` and `extra_records` MUST be `finding`. These values are explicit even though no alternative is supported in the first release. +- `records.policies.empty_datasets` MUST be `accept` or `reject`. `accept` means that two syntactically valid zero-record datasets satisfy the records-phase empty-dataset rule; it does not waive independently assessable contract requirements. `reject` creates a records-phase finding and is itself a conclusive records-phase acceptance failure when both datasets contain zero records. This generated criterion and its result MUST be included in the report. + +#### 6.2.5 Field selection, declaration, and source names + +- `fields.include` MUST be either the string `all` or a non-empty list of canonical field names. Key fields are always included whether or not they appear in the list. +- `fields.undeclared` MUST be one of `compare`, `ignore`, `warn`, or `fail`. + - `compare` compares shared undeclared fields with `preserved_exact`; an undeclared field present on only one side follows the missing/extra-field policy. + - `ignore` excludes undeclared fields and records the exclusion as `not_assessed`. + - `warn` behaves as `ignore` and additionally emits a structured warning. + - `fail` creates a contract finding for every undeclared field. +- `fields.policies.missing_fields` and `extra_fields` MUST be `finding`. Required missing fields and unexpected extra fields are therefore reported; acceptance criteria determine their effect on the outcome. +- `fields.definitions` MUST be a mapping keyed by unique canonical field name and MUST contain a definition for every field referenced by `records.keys`. Definitions for non-key fields MAY be omitted only to the extent permitted by `fields.undeclared`. +- `source.reference` and `source.query` MUST each contain the source field name used on that side. Omitting either source name is permitted only for a field that is not required on that side. +- `required.reference` and `required.query` MUST be booleans. Absence of a required field creates a contract finding; absence of an optional field does not. +- `sensitivity`, when present, is a stable non-empty classification label interpreted by the selected reporting/privacy policy. It does not itself define redaction behaviour. +- `finding_severity`, when present, MUST be `info`, `warning`, or `error`. Severity is report metadata and MUST NOT determine the comparison outcome unless referenced by an explicit acceptance criterion. +- `unit`, when present, MUST be a non-empty string. It records the declared unit but does not request automatic unit conversion. + +#### 6.2.6 Logical types and null policy + +For a scalar field, `logical_type` MUST be one of `string`, `integer`, `decimal`, `boolean`, `date`, or `datetime`. It MAY be omitted only when the comparator is `preserved_exact` and no typed parsing, mapping, or typed normalisation is configured. + +For a collection field, `logical_type` MUST use this form: + +```yaml +logical_type: + name: "list" # list, set, or multiset + element_type: "string" # scalar logical type + element_comparator: + name: "typed_exact" + parameters: {} +``` + +Nested collection types MUST be rejected. The comparator named for the field itself MUST correspond to the collection kind: `list_exact`, `set_exact`, or `multiset_exact`. + +Every declared field MUST contain a `null` mapping: + +- `allowed` and `nulls_equal` MUST be booleans. +- `nulls_equal` MUST be `false` when `allowed` is `false`. +- `missing_tokens.reference` and `missing_tokens.query` MUST each be a list of source-value descriptors. A matched token is converted to null at processing step 3 in Section 6.8. +- An empty list declares that there are no textual missing tokens on that side. Parser-native JSON null remains null independently of this list. + +A source-value descriptor has one of these forms: + +```yaml +category: "string" +value: "NA" +``` + +```yaml +category: "decimal" +value: "1.00" +``` + +```yaml +category: "null" +``` + +`category` MUST be `string`, `integer`, `decimal`, `boolean`, or `null`. `value` MUST be present for every category except `null`; it MUST be a quoted lexical string for `string`, `integer`, and `decimal`, and `true` or `false` for `boolean`. The lexical value MUST be valid for its stated category. Descriptors are matched using parser-native category and value equality under Section 5.6; the retained lexical form remains evidence but does not create a distinct numerical value. This explicit representation prevents YAML scalar resolution from changing mapping keys or missing tokens. + +#### 6.2.7 Normalisation and value mapping + +`source_normalisation` and `typed_normalisation` MUST be ordered lists. Each entry MUST have this form: + +```yaml +- operation: "trim" + parameters: + side: "both" +``` + +`parameters` MAY be omitted only when the registered operation has no parameters. `operation` MUST name an operation defined by the applicable schema version, and its parameters MUST validate against that operation’s published parameter schema. An unknown operation or parameter MUST invalidate the specification. List order is execution order. The permitted processing stages and their overall order are fixed by Section 6.8. + +The first-release source-text operation identifiers are: + +- `unicode_normalise`, requiring `form` with one of `NFC`, `NFD`, `NFKC`, or `NFKD`; +- `trim`, requiring `side` with one of `leading`, `trailing`, or `both`; and +- `case_fold`, which takes no parameters. + +The first-release typed-value operation identifiers are: + +- `decimal_quantise`, requiring a non-negative integer `places` and `rounding` with one of `half_even`, `half_up`, or `down`; and +- `datetime_truncate`, requiring `precision` with one of `second`, `millisecond`, or `microsecond`. + +A source-text operation MUST appear only in `source_normalisation`; a typed-value operation MUST appear only in `typed_normalisation`. Operation semantics, including the Unicode-data version used by Unicode operations, MUST be published with the schema and recorded sufficiently for reproducibility. + +`value_mapping.reference` and `value_mapping.query` MUST be ordered lists of entries with this form: + +```yaml +- from: + category: "string" + value: "resistant" + to: + category: "string" + value: "R" +``` + +- `from` and `to` MUST be source-value descriptors as defined in Section 6.2.6. +- A `to` descriptor with `category: "null"` explicitly maps to null and is distinct from absence of a mapping entry. +- Each `from` value MUST occur at most once within one side’s mapping. Conflicting or unreachable entries MUST invalidate the specification. +- A non-empty value mapping requires a scalar `logical_type`; value mapping of collection fields is not supported in the first release. Its `to` values are parsed as the destination type after mapping. +- When a side’s value-mapping list is non-empty, a source value with no mapping entry creates a contract finding. Pass-through of unmapped values is not supported in the first release. +- Empty lists declare that no value mapping is applied on that side. + +#### 6.2.8 Comparator declarations + +Every field definition MUST contain `comparator.name` and `comparator.parameters`. `name` MUST be one of the comparator identifiers in Section 7.4, and `parameters` MUST be a mapping containing only parameters valid for that comparator. + +- `preserved_exact`, `typed_exact`, `normalised_string`, `numeric_exact`, `date_exact`, `datetime_exact`, `list_exact`, `set_exact`, and `multiset_exact` take no field-level parameters in the first release; their `parameters` mapping MUST be empty. +- `numeric_tolerance` permits only `absolute_tolerance` and `relative_tolerance`. At least one MUST be present. Each MUST be a quoted, non-negative finite decimal string; omission of one means it is treated as zero as specified in Section 7.4.5. +- The comparator MUST be compatible with `logical_type`: string comparators with `string`, numeric comparators with `integer` or `decimal`, date comparators with their corresponding temporal type, and collection comparators with their corresponding collection type. +- `normalised_string` requires at least one `source_normalisation` or `typed_normalisation` operation applicable to strings. + +#### 6.2.9 Acceptance criteria + +`acceptance` MUST contain `contract`, `records`, and `values`, each an ordered list of criterion mappings. A list MAY be empty only when that phase cannot be reached under the approved intended use; selecting a scope that includes a phase with no criterion MUST invalidate the run as required by Section 8.5. + +Every criterion MUST contain: + +- a stable `id`, unique within the specification; +- `metric`, naming a metric defined by the applicable schema version; +- `operator`, one of `equal`, `not_equal`, `less_than`, `less_than_or_equal`, `greater_than`, or `greater_than_or_equal`; and +- `threshold`, expressed as a non-negative integer for count metrics or a quoted finite decimal string between `0` and `1`, inclusive, for proportion metrics. + +The first-release core metrics are `contract_finding_count`, `missing_record_count`, `extra_record_count`, `value_mismatch_count`, `not_assessable_count`, `reference_record_count`, and `query_record_count`. It also supports the corresponding proportion metrics `contract_finding_proportion` and `value_mismatch_proportion`. + +A criterion MAY additionally contain: + +```yaml +fields: ["result", "confidence"] +finding_codes: ["VALUE_MISMATCH"] +parameters: + side: "reference" +denominator: + population: "in_scope_values" + not_assessed: "exclude" +exceptions: ["known-difference-001"] +``` + +- `fields` restricts the metric to the listed canonical fields; omission means all fields applicable to the metric. +- `finding_codes` restricts a finding-based metric to the listed stable codes; omission means all codes applicable to that metric. +- `parameters` MUST contain only parameters published for the chosen metric. `side` is required for a metric whose population is not otherwise unambiguous. +- A filter or parameter that is not meaningful for the chosen metric MUST invalidate the specification rather than be ignored. +- `denominator` is required for proportion metrics and MUST be absent for count metrics. `population` MUST name a population supported by the metric. `not_assessed` MUST be `include` or `exclude`, making its treatment explicit. `not_assessable` content is governed by Section 8.5 and cannot be configured as a match or silently excluded. +- `exceptions` lists exception identifiers applied before the criterion is evaluated; omission is equivalent to an empty list. +- A referenced exception excludes a matching finding from that criterion’s failing metric but MUST NOT remove the finding from the report, summary counts, or provenance. +- Criteria are evaluated in list order for reporting, but order MUST NOT alter any phase or overall outcome. + +#### 6.2.10 Permitted exceptions + +`exceptions` MUST be a list and MAY be empty. Each exception MUST contain a unique stable `id`, a non-empty `rationale`, and an `applies_to` mapping containing at least one of `finding_codes`, `fields`, or `record_keys`. It MAY contain `revision`, `approved_by`, and `review_date` as provenance strings. These metadata fields do not replace the laboratory’s external approval controls. + +```yaml +exceptions: + - id: "known-difference-001" + revision: "1" + rationale: "Approved legacy representation" + applies_to: + finding_codes: ["VALUE_MISMATCH"] + fields: ["result"] + record_keys: + - record_id: + category: "string" + value: "S001" + effect: "permitted_difference" + approved_by: "approval-record-reference" + review_date: "2027-01-31" +``` + +- `finding_codes`, `fields`, and `record_keys` are cumulative filters: a finding must satisfy every filter that is present. +- `record_keys` MUST be a list of mappings containing exactly the canonical key fields declared by `records.keys`. Each field value MUST be expressed as a value descriptor in the form defined by Section 6.2.6 and MUST represent the canonical post-transformation key value. +- `effect` MUST be `permitted_difference`; no exception may suppress parsing, contract assessment, reporting, or provenance. +- Every applied exception MUST remain visible in findings and criterion evaluation. +- An exception has no effect unless its `id` is referenced by an acceptance criterion. + +#### 6.2.11 Reporting controls + +- `reporting.finding_limit`, when present, MUST be a positive integer. Omission means that no comparison-finding limit is configured, although documented resource limits still apply. Reaching a configured limit has the outcome consequences specified in Sections 7.5 and 8.4. +- `reporting.excluded_field_detail` MUST be `summary` or `individual`. `summary` permits omission of individual `not_assessed` findings only when the exclusion summary required by Section 7.5 is emitted; `individual` requires a finding for each excluded field occurrence. +- Privacy and redaction settings are not defined by this schema version. They MUST be supplied by a separately approved reporting policy and recorded in provenance. + +### 6.3 Record keys - A record key MUST contain one or more named fields. - Composite-key field order MUST be stable and recorded. @@ -238,24 +527,13 @@ A valid specification MUST declare at least: - Duplicate records MUST never overwrite one another silently. - Findings and reports MUST retain enough information to distinguish every source record. -### 6.3 Field declarations and preserved values +### 6.4 Field declarations and preserved values -The comparison engine MUST support both preserved, untyped values and explicitly declared logical types. Custom specifications MAY declare logical field types. A type declaration is required for a field that uses typed parsing, normalisation, mapping, or a type-specific comparator. Every custom specification MUST explicitly declare how undeclared fields are treated: `compare`, `ignore`, `warn`, or `fail`. There MUST be no implicit policy. An `ignore` policy MUST still be represented as an assessment exclusion in accordance with Section 7.5. +The comparison engine MUST support both preserved, untyped values and explicitly declared logical types. A type declaration is required for a field that uses typed parsing, normalisation, mapping, or a type-specific comparator. Every custom specification MUST explicitly declare how undeclared fields are treated: `compare`, `ignore`, `warn`, or `fail`. There MUST be no implicit policy. An `ignore` policy MUST still be represented as an assessment exclusion in accordance with Section 7.5. -A field definition MAY declare: +Every declared field MUST define its canonical name, reference and query source names where applicable, required status on each side, null policy, ordered normalisation and mapping lists, and comparator. It MUST declare a logical type except where Section 6.2.6 permits omission for `preserved_exact`. Unit, sensitivity classification, and finding severity are optional. The authoritative YAML representation and validation rules are defined in Section 6.2. -- canonical name; -- reference and query source names; -- required or optional status on each side; -- logical type; -- null policy; -- source-text and typed-value normalisation steps; -- comparator and comparator parameters; -- controlled vocabulary or value mapping; -- sensitivity or redaction classification; -- severity or acceptance contribution. - -### 6.4 Logical types and typed parsing +### 6.5 Logical types and typed parsing The comparison engine MUST support custom specifications declaring scalar logical types `string`, `integer`, `decimal`, `boolean`, `date`, and `datetime`, and typed `list`, `set`, and `multiset` collections. Collection fields MUST declare their element type. Null is a distinct value state governed by field policy, not a logical type. @@ -263,42 +541,42 @@ Type parsing MUST be field-specific and declared. The tool MUST NOT infer a logi An invalid typed value in a non-key field that remains attributable to an unambiguous record MUST be a contract finding and MUST make that value `not_assessable`; it MUST NOT be an execution error. Defects that prevent construction of a complete and unambiguous record model are classified under Section 8.3. -#### 6.4.1 String +#### 6.5.1 String - A string MUST preserve its Unicode code points exactly by default. - `typed_exact` equality MUST be code-point-for-code-point equality. - Empty strings, whitespace-only strings, nulls, and absent fields MUST remain distinct. -#### 6.4.2 Integer +#### 6.5.2 Integer - Textual integers MUST use signed base-10 notation consisting of an optional `+` or `-` followed by one or more ASCII digits. - Decimal points, exponent notation, booleans, and other characters MUST be invalid after source-text normalisation and mapping. - Leading zeros MUST be accepted and have no numerical significance. Identifiers for which leading zeros are significant MUST be declared as strings. - Integer representation MUST support arbitrary precision within documented resource limits. -#### 6.4.3 Decimal +#### 6.5.3 Decimal - Decimals MUST be parsed and represented using decimal arithmetic without an intermediate binary floating-point conversion. - Plain and scientific decimal notation MUST be accepted. - Numerically equivalent finite representations such as `1.0`, `1.00`, and `1e0` MUST have the same canonical value. - Parsing MUST preserve all supplied numerical precision. -- `NaN`, positive infinity, and negative infinity MUST be invalid in the first release. +- `NaN`, positive infinity, and negative infinity MUST be invalid. -#### 6.4.4 Boolean +#### 6.5.4 Boolean -- Native JSON and YAML booleans MUST be accepted. +- Native JSON booleans MUST be accepted. - Textual boolean parsing MUST accept only lowercase `true` and `false`. - Integers, uppercase boolean-like values, `yes`, `no`, `on`, `off`, and other representations MUST require an explicit value mapping. - Booleans MUST NOT be treated as integers. -#### 6.4.5 Date +#### 6.5.5 Date - Dates MUST use the ISO 8601 full-date form `YYYY-MM-DD`. - Values MUST represent valid Gregorian calendar dates. - Partial dates and locale-dependent forms MUST be invalid. - The canonical value MUST be the represented calendar date. -#### 6.4.6 Datetime +#### 6.5.6 Datetime - Datetimes MUST use an RFC 3339-compatible representation containing a full date, time, and `Z` or numeric UTC offset. - Local datetimes without an offset MUST be invalid. @@ -306,17 +584,17 @@ An invalid typed value in a non-key field that remains attributable to an unambi - Fractional seconds MUST be preserved unless explicit precision normalisation is configured. - Leap-second representations MUST be invalid in the first release. -#### 6.4.7 Collections +#### 6.5.7 Collections - Every collection MUST declare its element logical type and element comparator. - A `list` MUST preserve order and duplicates. - A `set` MUST be unordered and MUST reject duplicate source elements rather than silently collapse them. - A `multiset` MUST be unordered and MUST preserve element occurrence counts. -- JSON and YAML collections MUST use array or sequence values. +- JSON collections MUST use arrays. - A CSV or TSV collection cell MUST use valid JSON-array syntax; ordinary delimited text MUST NOT be inferred to be a collection. - Nested collections are outside the first-release scope. -### 6.5 Null, blank, and absent values +### 6.6 Null, blank, and absent values The specification MUST distinguish absent fields, explicit nulls, empty strings, whitespace-only strings, and configured missing tokens such as `NA`, `.`, or `null`. No two states are equivalent unless a field-specific policy declares them equivalent. Applied equivalence rules MUST be recorded. @@ -329,9 +607,9 @@ The specification MUST distinguish absent fields, explicit nulls, empty strings, The bundled strict-exact specification permits parser-native nulls. Under that specification, null matches only null and remains distinct from an absent field, an empty string, and every non-null value. -### 6.6 Mappings +### 6.7 Mappings -#### 6.6.1 Column mappings +#### 6.7.1 Column mappings - Reference and query fields MAY map to a canonical field name. - Mappings MUST be one-to-one unless an explicitly defined transformation supports another cardinality. @@ -340,7 +618,7 @@ The bundled strict-exact specification permits parser-native nulls. Under that s - Required mapped fields that are absent MUST be reported. - All applied mappings MUST be identifiable in the evidence. -#### 6.6.2 Value mappings +#### 6.7.2 Value mappings - A value mapping MUST operate on a preserved source value before typed parsing. - The mapping output MUST be parsed as the declared destination logical type. @@ -351,7 +629,7 @@ The bundled strict-exact specification permits parser-native nulls. Under that s - Mapping keys MUST be parsed without format-dependent scalar ambiguity. - Applied mappings MUST be identifiable in the evidence. -### 6.7 Normalisation and processing order +### 6.8 Normalisation and processing order Normalisation MUST be opt-in, field-specific, ordered deterministically, and recorded. Source-text normalisation is distinct from typed-value normalisation. @@ -369,7 +647,7 @@ Typed processing MUST occur in this order: Column-name mapping MUST occur before field-value processing. A failure at any stage MUST identify the stage and attributable source value, subject to redaction policy. -### 6.8 Bundled strict-exact specification +### 6.9 Bundled strict-exact specification If `--specification` is omitted, the tool MUST select the bundled strict-exact specification. It MUST be distributed as a documented template from which laboratories can develop controlled custom specifications, and the selected effective specification MUST be recorded in provenance. Supplying `--specification` MUST select the named custom specification; sentinel path values such as `DEFAULT` MUST NOT have special meaning. @@ -379,7 +657,7 @@ The bundled default specification MUST: - include every shared non-key field without requiring field-specific declarations; - preserve parser-native types and use `preserved_exact` comparison; - perform no type inference, implicit coercion, mapping, normalisation, or tolerance; -- permit parser-native null values and compare them according to Section 6.5; +- permit parser-native null values and compare them according to Section 6.6; - treat two syntactically valid zero-record datasets as satisfying all bundled acceptance criteria; - treat reference-only and query-only fields as contract findings; - require zero contract findings, zero missing or extra records, and zero value mismatches. @@ -399,7 +677,7 @@ Comparison phases MUST be cumulative and ordered: Public scope names identify the final requested phase: | Scope | Contract | Records | Values | -|---|---|---|---| +| --- | --- | --- | --- | | `contract` | Full | Not assessed | Not assessed | | `records` | Full | Full | Not assessed | | `full` | Full | Full | Full | @@ -448,7 +726,7 @@ Used by the bundled strict-exact specification. Values match only when their par #### 7.4.2 `typed_exact` -Values match only when both are present, valid under their field policies, have the same declared logical type, and have equal canonical values. Null handling follows Section 6.5. Cross-type equality is not implicit. +Values match only when both are present, valid under their field policies, have the same declared logical type, and have equal canonical values. Null handling follows Section 6.6. Cross-type equality is not implicit. #### 7.4.3 `normalised_string` @@ -466,7 +744,7 @@ allowed_difference = max(absolute_tolerance, relative_tolerance * abs(reference) match if difference <= allowed_difference ``` -- At least one tolerance MUST be supplied; an omitted tolerance contributes zero. +- At least one of `absolute_tolerance` or `relative_tolerance` MUST be specified. If only one is specified, the other MUST be treated as zero when calculating the allowed difference. - Tolerances MUST be non-negative finite decimals. - The boundary MUST be inclusive. - Relative tolerance MUST be directional and calculated against the absolute reference value. @@ -481,7 +759,7 @@ Dates match when their canonical calendar dates are equal. Datetimes match when #### 7.4.7 `list_exact`, `set_exact`, and `multiset_exact` - Lists use ordered element-by-element comparison and their lengths MUST match. -- Sets use order-independent comparison of unique elements; duplicate source elements are invalid under Section 6.4.7. +- Sets use order-independent comparison of unique elements; duplicate source elements are invalid under Section 6.5.7. - Multisets use order-independent comparison that preserves occurrence counts. - Elements use the declared element comparator. @@ -688,8 +966,8 @@ adviso-validation compare \ --output OUTPUT \ [--scope {contract,records,full}] \ [--specification SPECIFICATION] \ - [--reference-format {csv,tsv,json,yaml}] \ - [--query-format {csv,tsv,json,yaml}] \ + [--reference-format {csv,tsv,json}] \ + [--query-format {csv,tsv,json}] \ [--overwrite] ``` @@ -704,7 +982,7 @@ Convenience flags MAY override non-critical settings only when the effective con Exit codes MUST distinguish at least: | Code | Meaning | -|---:|---| +| ---: | --- | | 0 | Execution completed and comparison outcome is `pass` | | 1 | Execution completed and comparison outcome is `fail` | | 2 | Execution completed but comparison outcome is `inconclusive` | @@ -726,7 +1004,7 @@ No additional command is specified for the initial release. In particular, the p - Secrets MUST be redacted from recorded invocation details. - Temporary files MUST be created securely and removed according to documented policy. - Logs MUST NOT contain more sensitive information than the selected report policy permits. -- YAML MUST be parsed using a safe loader. +- Comparison-specification YAML MUST be parsed using the safe-loader rules in Section 6.2.1. - Spreadsheet-oriented exports MUST mitigate formula injection. - Resource limits SHOULD protect against unexpectedly large or malicious inputs without silently truncating results. - The tool SHOULD document filesystem permission expectations and leave access control to an appropriate deployment wrapper where necessary. @@ -835,9 +1113,13 @@ This catalogue summarises minimum executable scenarios for the first release. It - **AT-CON-005:** Unknown, conflicting, and incomplete mappings fail clearly; mapping to null differs from a missing map entry. - **AT-CON-006:** Malformed syntax or structure, empty or duplicate column names, and irregular tabular rows receive their specified error classification without silent data loss. - **AT-CON-007:** Two syntactically valid empty datasets pass under the bundled strict-exact specification; one-sided empty datasets produce all applicable missing or extra record findings. -- **AT-CON-008:** YAML scalar resolution and cross-format typed parsing are deterministic across supported environments. +- **AT-CON-008:** Comparison-specification YAML scalar resolution and cross-format typed dataset parsing are deterministic across supported environments. - **AT-CON-009:** The bundled strict-exact specification treats parser-native null as equal only to null and distinguishes null, absent fields, and empty strings. - **AT-CON-010:** Every custom specification explicitly declares an undeclared-field policy; omission is rejected. +- **AT-CON-011:** A complete custom YAML specification conforming to Section 6.2 validates and produces the declared effective configuration. +- **AT-CON-012:** Duplicate or unknown YAML keys, unsafe YAML features, unsupported schema versions, invalid enumerations, and incompatible type/comparator combinations are rejected with precise configuration paths. +- **AT-CON-013:** Source-value descriptors distinguish strings, numbers, booleans, and null; mapping to null remains distinct from an absent mapping entry. +- **AT-CON-014:** Each selected comparison phase requires applicable acceptance criteria, and count and proportion criteria validate their operators, thresholds, filters, and denominators. ### 14.2 Record scenarios @@ -855,7 +1137,7 @@ This catalogue summarises minimum executable scenarios for the first release. It - **AT-VAL-005:** Boolean parsing accepts native booleans and lowercase textual `true` or `false`; other representations require a mapping. - **AT-VAL-006:** Dates reject invalid or non-ISO full dates; datetimes require offsets and compare as instants. - **AT-VAL-007:** Lists preserve order and duplicates, sets reject duplicates and ignore order, and multisets ignore order while preserving occurrence counts. -- **AT-VAL-008:** Typed transformations follow Section 6.7 and retain evidence of the attributable source value. +- **AT-VAL-008:** Typed transformations follow Section 6.8 and retain evidence of the attributable source value. - **AT-VAL-009:** Numeric tolerance uses exact decimal arithmetic, a reference-relative component, and an inclusive boundary, including below, at, and above boundary cases. - **AT-VAL-010:** All assessable differences are collected in one run. - **AT-VAL-011:** Subset comparison identifies its limited coverage, records the required exclusion summary when individual findings are omitted, and rejects unknown requested fields. @@ -922,5 +1204,3 @@ Before implementation begins, reviewers SHOULD confirm that: - every first-milestone scenario has an agreed expected result; - unresolved decisions have named owners and target dates; - approved requirements have stable identifiers for test traceability. - - From cd4fe102f0140ce314c7aaea1fca15dc6175d1d5 Mon Sep 17 00:00:00 2001 From: Kevin Howe Date: Tue, 20 Oct 2026 16:32:38 +0100 Subject: [PATCH 03/45] More refinement of Spec, and addition of JSON schema for configuration file --- SPECIFICATION.md | 704 ++++++++---------- src/adviso_validation/__init__.py | 1 + src/adviso_validation/schemas/__init__.py | 1 + .../comparison-configuration-v1.0.schema.json | 638 ++++++++++++++++ 4 files changed, 960 insertions(+), 384 deletions(-) create mode 100644 src/adviso_validation/__init__.py create mode 100644 src/adviso_validation/schemas/__init__.py create mode 100644 src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json diff --git a/SPECIFICATION.md b/SPECIFICATION.md index 957f865..d8e8cd5 100644 --- a/SPECIFICATION.md +++ b/SPECIFICATION.md @@ -7,9 +7,9 @@ ## 1. Purpose -This document defines a specification for the AdvISO Validation Toolkit: a command-line namespace for cohesive capabilities that support bioinformatics validation and verification activities associated with ISO 15189 accreditation. +This document defines a specification for version 1.0 of the AdvISO Validation Toolkit: a command-line namespace for cohesive capabilities that support bioinformatics validation and verification activities associated with ISO 15189 accreditation. -The initial sole user-facing subcommand within the toolkit is `compare`. This subcommand compares a reference dataset with a query dataset according to an explicit comparison specification and produces deterministic, machine-readable findings and a human-readable summary suitable for review and retention as one component of a bioinformatics pipeline validation or verification record. It is intended to answer this question: +The initial sole user-facing subcommand within the toolkit is `compare`. This subcommand compares a reference dataset with a query dataset according to an explicit comparison configuration and produces deterministic, machine-readable findings and a human-readable summary suitable for review and retention as one component of a bioinformatics pipeline validation or verification record. It is intended to answer this question: > Given these identified inputs and these declared comparison rules, which expected and observed records and values are equivalent, different, missing, extra, or not assessable? @@ -30,20 +30,22 @@ The organisation of the toolkit within the AdvISO GitHub organisation is: ```text adviso-project -└── validation_toolkit - ├── compare subcommand - └── internal comparison, specification, provenance, findings, and reporting components +└── validation_toolkit repository + └── adviso-validation distribution and executable + └── adviso_validation Python import package + ├── internal comparison, configuration, provenance, findings, and reporting components + └── compare subcommand ``` -The **AdvISO Validation Toolkit** is the user-facing product and a minimal command namespace. **`compare`** is currently the sole specified subcommand. The other components, including the **comparison engine**, are internal reusable software components and are not separate products. +The **AdvISO Validation Toolkit** is the user-facing product and a minimal command namespace. Its installable distribution package and command-line executable MUST both be named **`adviso-validation`**. **`compare`** is currently the sole specified subcommand. The other components, including the **comparison engine**, are internal reusable software components and are not separate products. -The initial command-line form SHOULD be: +The command-line form MUST be: ```text adviso-validation compare ... ``` -The distribution, import-package, executable, and repository names MAY differ where required by their respective naming conventions, but each MUST identify its relationship to AdvISO clearly. Final distribution and executable names MUST be settled before the first stable release and thereafter follow a documented compatibility policy. +The Python import package MUST be named `adviso_validation`. The repository name MAY differ from the distribution and import-package names, but it MUST be documented clearly. Once released, changes to the distribution, executable, or import-package name MUST follow the documented compatibility policy. Additional commands are not a development objective for the first release and MUST NOT be added speculatively. A future command MAY be introduced only for a demonstrated, materially distinct operation in the same validation-evidence workflow. The top-level dispatcher MUST remain thin and MUST NOT create speculative shared abstractions. The initial implementation SHOULD use ordinary modules and a small comparator registry, not a general plugin framework. Capabilities with materially different purposes, dependencies, governance, or release lifecycles SHOULD be separate AdvISO packages. @@ -62,9 +64,9 @@ The first-release scope of `compare` is: - provenance sufficient to reproduce and review a comparison; - safe handling of data that may contain sensitive clinical information. -### 2.3 Initially out of scope +### 2.3 Out of scope -Unless introduced by a separately specified extension, the `compare` command does not: +The `compare` command does not: - establish analytical sensitivity, analytical specificity, precision, limit of detection, reportable range, clinical validity, or clinical utility; - generate or adjudicate truth data; @@ -72,10 +74,9 @@ Unless introduced by a separately specified extension, the `compare` command doe - replace laboratory review or approval; - compare unstructured prose, images, binary files, or specialist bioinformatics formats (e.g. SAM, VCF, FASTQ); - infer biological equivalence of variants, transcripts, genomic intervals, taxonomic calls, or other domain objects; +- provide automated redaction, identifier pseudonymisation, keyed hashing, or field-level sensitivity policies; - provide an electronic-signature or laboratory document-management system. -No bioinformatics-specific or domain-aware comparator is required for the first release. Such comparators MAY be added later under explicit, independently versioned specifications. - ## 3. Design principles The following principles govern all requirements: @@ -83,11 +84,11 @@ The following principles govern all requirements: 1. **Explicit over inferred:** Comparison behaviour MUST be declared, not guessed from observed data. 2. **No silent data loss:** Input records, duplicate keys, unmapped values, and unsupported structures MUST NOT be silently discarded or overwritten. 3. **Symmetry:** Missing and extra content MUST be assessed in both directions. -4. **Separation of concerns:** Execution success and comparison outcome are distinct concepts. +4. **Separation of concerns:** Run status, comparison evidence, and the laboratory's acceptance decision are distinct concepts. 5. **Complete evidence:** A normal comparison mismatch MUST still produce a complete report. 6. **Determinism:** Identical inputs, configuration, and tool version MUST produce semantically identical results. 7. **Traceability:** Findings MUST identify the rule, record, and field that produced them. -8. **Privacy by design:** Reports MUST include only the sensitive data necessary for their intended purpose. +8. **Data minimisation:** Reports MUST include only the source identifiers and values necessary to explain and review the comparison; routine logs and diagnostics MUST not reproduce them by default. 9. **Fail clearly:** Invalid or ambiguous input MUST produce an explicit error rather than an assumed interpretation. 10. **Testability:** Each normative requirement SHOULD be traceable to one or more automated tests. @@ -95,10 +96,9 @@ The following principles govern all requirements: - **Reference dataset:** The dataset containing the approved expected or comparator records. “Reference” does not imply intrinsic truth. - **Query dataset:** The dataset being evaluated against the reference. -- **Comparison specification:** The versioned configuration declaring the data contract, normalisation, comparison, and acceptance rules. -- **Comparison scope:** The final cumulative comparison phase requested: `contract`, `records`, or `full`. -- **Not assessed:** Deliberately outside the selected scope or excluded by an explicit field policy. -- **Not assessable:** Inside the selected scope, but unable to be compared safely because a prerequisite was not satisfied. +- **Comparison configuration:** The versioned YAML configuration declaring the data contract, normalisation, and comparison behaviour. Its permitted structure and semantics are defined by the comparison-configuration schema and this behaviour specification. +- **Not assessed:** Deliberately excluded by an explicit field policy. +- **Not assessable:** Included in the configured comparison, but unable to be compared safely because a prerequisite was not satisfied. - **Contract comparison:** Assessment of whether inputs satisfy the declared data contract, including fields, logical types, key definitions, nullability, mappings, and configured constraints. - **Records comparison:** Alignment and assessment of valid record identities, including matching, missing, and extra records. Invalid, missing, null, or duplicate record keys are execution errors rather than record-comparison findings. - **Value comparison:** Application of declared comparators to values in records that can be aligned meaningfully. @@ -106,12 +106,11 @@ The following principles govern all requirements: - **Record key:** One field or an ordered combination of fields that uniquely identifies a record within a dataset. - **Field:** A named value within a record. - **Finding:** A structured observation generated by applying one comparison rule to a record, field, dataset, or run. -- **Execution status:** Whether the requested toolkit command completed correctly. -- **Comparison outcome:** The result of applying configured acceptance criteria: `pass`, `fail`, or `inconclusive`. +- **Run status:** Whether the requested comparison is `complete`, `incomplete`, or ended in `error`. - **Laboratory decision:** The externally reviewed conclusion about validation or verification. The toolkit does not make this decision. - **Missing:** Expected in the reference but absent from the query. - **Extra:** Present in the query but absent from the reference. -- **Null:** A present field with an explicitly empty/no-value representation defined by the input format or specification. +- **Null:** A present field with an explicitly empty/no-value representation defined by the input format or comparison configuration. - **Absent:** A field or record that is not present at all. - **Normalisation:** A declared, non-lossy or intentionally controlled transformation performed before comparison. - **Mapping:** A declared translation of field names or values from one representation to another. @@ -126,7 +125,7 @@ A comparison run MUST receive: - one query dataset; - an output destination. -A custom comparison specification MAY be supplied. If it is omitted, the bundled strict-exact specification defined in Section 6.9 MUST be used. +A custom comparison configuration MAY be supplied. If it is omitted, the bundled strict-exact configuration defined in Section 6.9 MUST be used. ### 5.2 Supported formats @@ -139,7 +138,7 @@ The initial implementation MUST support: The reference and query formats MAY be selected independently through explicit command-line options. If a format is not supplied, it MAY be detected only from a recognised, unambiguous extension: `.csv`, `.tsv` or `.json`. In either case: - detected format MUST be recorded; -- ambiguous or unsupported content MUST fail validation; +- ambiguous or unsupported content MUST fail execution; - an unknown or ambiguous extension, including `.txt`, MUST require an explicit format; - content MUST be validated against the selected format. @@ -156,12 +155,12 @@ The reference and query formats MAY be selected independently through explicit c For CSV and TSV: - a header row MUST be required; -- duplicate or empty column names MUST be rejected; +- duplicate or empty column names MUST cause an input error; - delimiters, quoting, escaping, and embedded newlines MUST be handled by a standards-compliant parser; - rows with too many or too few columns MUST be input errors; - all physical input rows MUST be accounted for; - source row numbers SHOULD be retained for traceability; -- input values MUST initially remain strings unless a custom specification declares a null representation or type conversion; in the bundled strict-exact specification, an empty cell is an empty string. +- input values MUST initially remain strings unless a custom configuration declares a null representation or type conversion; in the bundled strict-exact configuration, an empty cell is an empty string. ### 5.5 JSON parsing @@ -174,94 +173,80 @@ For CSV and TSV: ] ``` -- Values such as `yes`, `no`, `on`, `off`, uppercase boolean-like values, and timestamp-like strings MUST remain strings unless a custom specification explicitly maps or parses them. -- JSON numerical tokens that may be used as decimals MUST be loaded without an intermediate binary floating-point conversion, and their source lexical representations MUST remain available. -- Root scalars, heterogeneous root structures, and records that are not objects MUST be rejected with actionable errors. +- Values such as `yes`, `no`, `on`, `off`, uppercase boolean-like values, and timestamp-like strings MUST remain strings unless a custom configuration explicitly maps or parses them. +- JSON decimal numbers - i.e. numbers containing a fractional or exponent component - MUST be parsed using exact decimal arithmetic, without first converting them to binary floating-point values. The tool MUST also retain the original text of each number — for example, distinguishing the representations `0.100`, `0.1`, and `1e-1` — for inclusion in evidence and diagnostic output when required. +- A JSON input MUST contain either one object representing a single record or an array of objects representing multiple records. A root-level scalar, or a root-level array containing any item that is not an object, MUST be an input error. The error MUST identify the invalid root value or array item and explain the required structure. -### 5.6 Preserved parser-native values +### 5.6 Parser-native value model -- For `preserved_exact` comparison, parsers MUST expose a common, deterministic set of parser-native value categories: `string`, `integer`, `decimal`, `boolean`, `null`, `array`, and `object`; -- CSV and TSV cells MUST be strings, including empty strings, unless a custom specification declares further processing; -- JSON numbers without a fractional or exponent component MUST be integers, while other finite JSON numbers MUST be exact decimals; -- JSON arrays MUST be arrays, and JSON mappings used as field values MUST be objects; and -- Values are equal only when they have the same parser-native category and recursively equal content. For array-valued fields, element order is significant under `preserved_exact`; for object-valued fields, member names and values must match but member order is not significant. Top-level records are aligned by their declared record keys, and their input order does not affect comparison. +- When an input file is read, and before any configured type conversion, mapping, or normalisation is applied, each input value MUST be represented internally using one of the following parser-native value categories: `string`, `integer`, `decimal`, `boolean`, `null`, `array`, and `object`. +- CSV and TSV cells MUST be strings, including empty strings, unless a custom configuration declares further processing. +- JSON numbers without a fractional or exponent component MUST be integers, while other finite JSON numbers MUST be exact decimals. +- JSON arrays used as field values MUST be represented internally as `array` values, and JSON objects used as field values MUST be represented internally as `object` values. +- Parser-native values are equal only when they have the same category and recursively equal content. For array-valued fields, element order is significant; for object-valued fields, member names and values must match but member order is not significant. Top-level records are aligned by their declared record keys, and their input order does not affect comparison. - Numerical values MUST be retained exactly as they appeared in the input for reporting and traceability. However, two values in the same numerical category compare as equal when they represent the same number; for example, decimal `1.0` and decimal `1.00` are equal. - `NaN`, positive infinity, and negative infinity are not valid numerical values. In an attributable non-key field, they MUST produce a contract finding and make that value `not_assessable`; other assessable content MUST still be compared. If such a value occurs in a record key or prevents construction of an unambiguous record model, it MUST produce an execution error. ### 5.7 Empty inputs - Empty files and datasets with zero records MUST be recognised explicitly. -- A custom comparison specification MUST state whether a comparison in which both datasets contain no records is acceptable. It MAY accept two empty datasets as having no missing or extra records, or require a minimum number of records. +- If both datasets are syntactically valid and contain no records, the records phase MUST report no missing or extra records, and the values phase MUST perform no comparisons. Contract requirements that can be evaluated without records MUST still be checked. The tool MUST NOT infer that an empty dataset is unsuitable merely because it is empty; any minimum-size acceptance criterion belongs to the external laboratory decision process. - A non-empty reference and empty query MUST report every expected record as missing. - An empty reference and non-empty query MUST report every observed record as extra. -## 6. Comparison specifications +## 6. Comparison configurations ### 6.1 General requirements -Every comparison uses an effective comparison specification. A custom specification MUST: +Every comparison uses an effective comparison configuration. A custom configuration MUST: -- use a published, independently versioned specification schema supported by the installed toolkit release; -- declare its schema version and a stable specification identifier and revision; +- use a published, independently versioned comparison-configuration schema supported by the installed toolkit release; +- declare its schema version and a stable configuration identifier and revision; - be validated fully before either dataset is compared; - reject unknown configuration keys by default; - provide actionable validation errors with configuration locations; - be identified in provenance by a cryptographic digest; - be retained or embedded in the evidence package, subject to policy. -A valid specification MUST declare at least: +A valid comparison configuration MUST declare at least: - record key fields; -- field definitions or an explicit policy for undeclared fields; -- comparison rules; -- missing and extra record policy; -- missing and extra field policy; -- duplicate-key policy; -- phase-specific acceptance criteria. +- field definitions and an explicit policy for undeclared fields; +- comparison rules. -### 6.2 Custom comparison-specification YAML format +### 6.2 Custom comparison-configuration YAML format -Custom comparison specifications MUST be YAML documents conforming to the format defined in this section. YAML is used only for the comparison specification; it is not a supported dataset format. A specification file SHOULD use the `.yaml` extension; `.yml` MAY also be accepted. +Custom comparison configurations MUST be YAML documents conforming to the format defined in this section. YAML is used only for the comparison configuration; it is not a supported dataset format. A configuration file SHOULD use the `.yaml` or `.yml` extension. #### 6.2.1 YAML parsing and validation - The document MUST be UTF-8 encoded and contain exactly one YAML mapping at its root. -- YAML 1.2 scalar resolution MUST be used with a safe loader. -- Duplicate mapping keys, merge keys, aliases, anchors, explicit tags, and executable or implementation-specific object constructors MUST be rejected. -- Unknown keys MUST be rejected unless a future schema version explicitly defines an extension mechanism. +- The configuration MUST be parsed according to the YAML 1.2 rules for interpreting unquoted scalar values. +- The parser MUST use a safe loading mode that cannot construct arbitrary application objects or execute code. +- Duplicate mapping keys, merge keys, aliases, anchors, explicit tags, and executable or implementation-specific object constructors MUST be rejected as configuration errors. +- Unknown keys MUST be rejected as configuration errors. - Required keys MUST NOT be supplied as YAML null values. -- `schema_version`, `id`, `revision`, criterion identifiers, exception identifiers, field names, operation names, comparator names, enumerated values, decimal lexical values, dates, and datetimes SHOULD be quoted to prevent unintended YAML scalar conversion. +- `schema_version`, `id`, `revision`, field names, operation names, comparator names, enumerated values, decimal lexical values, dates, and datetimes SHOULD be quoted to prevent unintended YAML scalar conversion. - The complete document MUST be validated against the published schema identified by `schema_version` before either dataset is read. - Validation errors MUST identify the YAML path and the violated requirement. #### 6.2.2 Complete structural template -The following template defines the complete first-release structure. Keys marked “optional” in comments MAY be omitted; other displayed keys are required when their parent construct is used, except where a conditional omission is explicitly permitted below. Empty lists are valid only where stated below. +The following template defines the complete first-release structure. Keys marked “optional” in comments MAY be omitted; other displayed keys are required when their parent construct is used, except where a conditional omission is explicitly permitted below. Optional processing properties are omitted rather than represented by empty lists. ```yaml schema_version: "1.0" -specification: +metadata: id: "org.example.pipeline-output-comparison" revision: "1.0.0" title: "Example pipeline-output comparison" # optional description: "Approved comparison rules" # optional -records: - keys: - - "record_id" - policies: - duplicate_keys: "error" - missing_records: "finding" - extra_records: "finding" - empty_datasets: "accept" - fields: - include: "all" + record_key: + - "record_id" undeclared: "fail" - policies: - missing_fields: "finding" - extra_fields: "finding" definitions: record_id: source: @@ -271,85 +256,129 @@ fields: reference: true query: true logical_type: "string" - null: + nulls: allowed: false nulls_equal: false + comparator: + name: "typed_exact" + field_example_1: + source: + reference: "expected_result" + query: "reported_result" + required: + reference: true + query: true + logical_type: "decimal" + nulls: + allowed: false + nulls_equal: false + typed_normalisation: + - operation: "decimal_quantise" + parameters: + places: 3 + rounding: "half_even" + comparator: + name: "numeric_tolerance" + parameters: + absolute_tolerance: "0.01" + relative_tolerance: "0.05" + review_priority: "high" # optional; default: normal + field_example_2: + source: + reference: "expected_interpretation" + query: "reported_interpretation" + required: + reference: true + query: true + logical_type: "string" + nulls: + allowed: true + nulls_equal: true missing_tokens: - reference: [] - query: [] - source_normalisation: [] + reference: + - category: "string" + value: "na" + query: + - category: "string" + value: "." + source_normalisation: + reference: + - operation: "trim" + parameters: + side: "both" + - operation: "case_fold" + query: + - operation: "trim" + parameters: + side: "both" value_mapping: - reference: [] - query: [] - typed_normalisation: [] + reference: + - from: + category: "string" + value: "detected" + to: + category: "string" + value: "detected" + - from: + category: "string" + value: "not detected" + to: + category: "string" + value: "not_detected" + query: + - from: + category: "string" + value: "pos" + to: + category: "string" + value: "detected" + - from: + category: "string" + value: "neg" + to: + category: "string" + value: "not_detected" comparator: name: "typed_exact" - parameters: {} - unit: "laboratory-defined-unit" # optional - sensitivity: "identifier" # optional label - finding_severity: "error" # optional - -acceptance: - contract: - - id: "no-contract-findings" - metric: "contract_finding_count" - operator: "equal" - threshold: 0 - records: - - id: "no-missing-records" - metric: "missing_record_count" - operator: "equal" - threshold: 0 - - id: "no-extra-records" - metric: "extra_record_count" - operator: "equal" - threshold: 0 - values: - - id: "no-value-mismatches" - metric: "value_mismatch_count" - operator: "equal" - threshold: 0 - -exceptions: [] - -reporting: - finding_limit: 10000 # optional - excluded_field_detail: "summary" + review_priority: "low" # optional; default: normal + ``` -The template is illustrative rather than an instruction to use these particular rules. In particular, laboratories MUST choose identifiers, field definitions, policies, criteria, thresholds, exceptions, and reporting limits appropriate to the intended comparison. The published machine-readable schema MUST encode the same structure and constraints as this section; where the schema and this document disagree, the normative behaviour specification MUST be corrected before release rather than silently choosing one interpretation. +The template is illustrative rather than an instruction to use these particular rules. In particular, laboratories MUST choose identifiers, field definitions, and policies appropriate to the intended comparison. The published machine-readable schema MUST encode the same structure and constraints as this section; where the schema and this document disagree, the normative behaviour specification MUST be corrected before release rather than silently choosing one interpretation. + +The three field definitions illustrate different configuration roles: `record_id` is a renamed exact key; `field_example_1` is parsed as an exact decimal and quantised before tolerance comparison; and `field_example_2` demonstrates source-text normalisation, side-specific missing tokens, and mapping different source vocabularies to common canonical string values. The deliberately generic names make clear that these are user-defined canonical field names rather than reserved configuration keywords. These example rules are not domain recommendations. #### 6.2.3 Top-level keys and identity -The only permitted top-level keys are `schema_version`, `specification`, `records`, `fields`, `acceptance`, `exceptions`, and `reporting`. +The only permitted top-level keys are `schema_version`, `metadata`, and `fields`. -- `schema_version` MUST be a quoted string identifying a supported comparison-specification schema version. -- `specification.id` MUST be a stable, non-empty identifier. It SHOULD use a collision-resistant organisational namespace. -- `specification.revision` MUST be a non-empty revision string controlled by the specification owner. Any behaviourally material change MUST produce a new revision. -- `specification.title` and `specification.description` are optional human-readable strings and MUST NOT alter comparison behaviour. +- `schema_version` MUST be a quoted string identifying a supported comparison-configuration schema version. +- `metadata.id` MUST be a stable, non-empty identifier for the comparison configuration. It SHOULD use a collision-resistant organisational namespace. +- `metadata.revision` MUST be a non-empty revision string controlled by the configuration owner. Any behaviourally material change elsewhere in the configuration MUST produce a new revision. +- `metadata.title` and `metadata.description` are optional human-readable strings. +- The `metadata` properties MUST NOT directly alter comparison behaviour. Any change to them nevertheless changes the configuration document and therefore its provenance digest. -#### 6.2.4 Record configuration +#### 6.2.4 Record-key selection -- `records.keys` MUST be a non-empty ordered list of canonical field names present in `fields.definitions`. List order defines composite-key order. +- `fields.record_key` MUST be a non-empty ordered list of canonical field names present in `fields.definitions`. The list defines one logical record key, and list order defines the component order of a composite key. - Every key field MUST be required on both sides, non-nullable, and declared with a scalar logical type. A key field MAY use explicitly declared source normalisation or value mapping, but MUST use `typed_exact` after transformation. -- `records.policies.duplicate_keys` MUST be `error`; `missing_records` and `extra_records` MUST be `finding`. These values are explicit even though no alternative is supported in the first release. -- `records.policies.empty_datasets` MUST be `accept` or `reject`. `accept` means that two syntactically valid zero-record datasets satisfy the records-phase empty-dataset rule; it does not waive independently assessable contract requirements. `reject` creates a records-phase finding and is itself a conclusive records-phase acceptance failure when both datasets contain zero records. This generated criterion and its result MUST be included in the report. #### 6.2.5 Field selection, declaration, and source names -- `fields.include` MUST be either the string `all` or a non-empty list of canonical field names. Key fields are always included whether or not they appear in the list. -- `fields.undeclared` MUST be one of `compare`, `ignore`, `warn`, or `fail`. - - `compare` compares shared undeclared fields with `preserved_exact`; an undeclared field present on only one side follows the missing/extra-field policy. - - `ignore` excludes undeclared fields and records the exclusion as `not_assessed`. - - `warn` behaves as `ignore` and additionally emits a structured warning. +- Every field in `fields.definitions` is in scope. Field definitions are the sole mechanism for selecting explicitly configured fields in version 1. +- `fields.undeclared` is required and MUST be one of `compare`, `exclude`, or `fail`. + - `compare` compares shared undeclared fields with `preserved_exact`; an undeclared field present on only one side creates a contract finding because no corresponding value can be compared. + - `exclude` does not compare undeclared fields and records them in the exclusion summary as `not_assessed`. - `fail` creates a contract finding for every undeclared field. -- `fields.policies.missing_fields` and `extra_fields` MUST be `finding`. Required missing fields and unexpected extra fields are therefore reported; acceptance criteria determine their effect on the outcome. -- `fields.definitions` MUST be a mapping keyed by unique canonical field name and MUST contain a definition for every field referenced by `records.keys`. Definitions for non-key fields MAY be omitted only to the extent permitted by `fields.undeclared`. -- `source.reference` and `source.query` MUST each contain the source field name used on that side. Omitting either source name is permitted only for a field that is not required on that side. -- `required.reference` and `required.query` MUST be booleans. Absence of a required field creates a contract finding; absence of an optional field does not. -- `sensitivity`, when present, is a stable non-empty classification label interpreted by the selected reporting/privacy policy. It does not itself define redaction behaviour. -- `finding_severity`, when present, MUST be `info`, `warning`, or `error`. Severity is report metadata and MUST NOT determine the comparison outcome unless referenced by an explicit acceptance criterion. -- `unit`, when present, MUST be a non-empty string. It records the declared unit but does not request automatic unit conversion. +- Undeclared reference and query fields MUST be aligned only when their input field names are identical. No name inference or source-name mapping is applied to undeclared fields. Fields with different names MUST be declared explicitly under a common canonical field definition if they are intended to correspond. +- `fields.definitions` MUST be a mapping keyed by unique canonical field name and MUST contain a definition for every field referenced by `fields.record_key`. +- `source.reference` and `source.query` MUST each contain a non-empty source field name. A declared field always defines correspondence between one reference field and one query field, regardless of whether either field is required to be present in every record. +- `required.reference` and `required.query` MUST be booleans. Absence of a field required on a particular side MUST create a contract finding. Absence of an optional field MUST NOT create a missing-field finding. +- A field observed in an input but absent from `fields.definitions` is an undeclared field and MUST be handled only according to `fields.undeclared`. +- `review_priority`, when present, MUST be `low`, `normal`, or `high`. When omitted, it defaults to `normal` in the effective configuration. +- A declared field's resolved `review_priority` MUST be attached to every finding attributable to that canonical field, including missing-field, null-policy, type, parsing, mapping, and value-mismatch findings. It MUST NOT be inherited by findings that cannot be attributed to that field, such as missing or extra records, truncation, invalid-configuration findings, or internal failures. +- Review priority is presentation and review-workflow metadata only. It MUST NOT change comparison behaviour, determine whether a finding is emitted, alter counts, affect run or phase status, affect exit codes, or constitute a laboratory acceptance decision. +- Human-readable reports MAY use review priority for grouping, ordering, filtering, icons, or colour coding. Colour MUST NOT be the only means of communicating priority. Reports and documentation MUST NOT describe review priority as clinical significance or as an execution-error classification. #### 6.2.6 Logical types and null policy @@ -363,17 +392,16 @@ logical_type: element_type: "string" # scalar logical type element_comparator: name: "typed_exact" - parameters: {} ``` Nested collection types MUST be rejected. The comparator named for the field itself MUST correspond to the collection kind: `list_exact`, `set_exact`, or `multiset_exact`. -Every declared field MUST contain a `null` mapping: +Every declared field MUST contain a `nulls` mapping: - `allowed` and `nulls_equal` MUST be booleans. - `nulls_equal` MUST be `false` when `allowed` is `false`. -- `missing_tokens.reference` and `missing_tokens.query` MUST each be a list of source-value descriptors. A matched token is converted to null at processing step 3 in Section 6.8. -- An empty list declares that there are no textual missing tokens on that side. Parser-native JSON null remains null independently of this list. +- `missing_tokens` MAY be omitted; omission means that no additional source values are interpreted as null. Parser-native JSON null remains null independently of this property. +- When `missing_tokens` is present, it MUST contain `reference`, `query`, or both. Each present side MUST be a non-empty list of source-value descriptors. An omitted side has no configured textual missing tokens. A matched token is converted to null at processing step 3 in Section 6.8. A source-value descriptor has one of these forms: @@ -395,7 +423,9 @@ category: "null" #### 6.2.7 Normalisation and value mapping -`source_normalisation` and `typed_normalisation` MUST be ordered lists. Each entry MUST have this form: +`source_normalisation` MAY be omitted; omission means that no source-text normalisation is applied. When present, it MUST contain `reference`, `query`, or both. Each present side MUST be a non-empty ordered list of operations applied only to values from that dataset; an omitted side has no source-text operations. + +`typed_normalisation` MAY be omitted; omission means that no typed-value normalisation is applied. When present, it MUST be a non-empty ordered list applied to the canonical typed values on both sides. Each operation entry has this form: ```yaml - operation: "trim" @@ -403,12 +433,12 @@ category: "null" side: "both" ``` -`parameters` MAY be omitted only when the registered operation has no parameters. `operation` MUST name an operation defined by the applicable schema version, and its parameters MUST validate against that operation’s published parameter schema. An unknown operation or parameter MUST invalidate the specification. List order is execution order. The permitted processing stages and their overall order are fixed by Section 6.8. +`parameters` MAY be omitted only when the registered operation has no parameters. `operation` MUST name an operation defined by the applicable schema version, and its parameters MUST validate against that operation’s published parameter schema. An unknown operation or parameter MUST invalidate the comparison configuration. List order is execution order. The permitted processing stages and their overall order are fixed by Section 6.8. The first-release source-text operation identifiers are: - `unicode_normalise`, requiring `form` with one of `NFC`, `NFD`, `NFKC`, or `NFKD`; -- `trim`, requiring `side` with one of `leading`, `trailing`, or `both`; and +- `trim`, requiring `side` with one of `leading`, `trailing`, or `both`; here `side` identifies which end or ends of the text are trimmed, not the reference or query dataset; and - `case_fold`, which takes no parameters. The first-release typed-value operation identifiers are: @@ -416,9 +446,9 @@ The first-release typed-value operation identifiers are: - `decimal_quantise`, requiring a non-negative integer `places` and `rounding` with one of `half_even`, `half_up`, or `down`; and - `datetime_truncate`, requiring `precision` with one of `second`, `millisecond`, or `microsecond`. -A source-text operation MUST appear only in `source_normalisation`; a typed-value operation MUST appear only in `typed_normalisation`. Operation semantics, including the Unicode-data version used by Unicode operations, MUST be published with the schema and recorded sufficiently for reproducibility. +A source-text operation MUST appear only in `source_normalisation.reference` or `source_normalisation.query` and applies only to that dataset. A typed-value operation MUST appear only in `typed_normalisation` and applies identically to both datasets. Operation semantics, including the Unicode-data version used by Unicode operations, MUST be published with the schema and recorded sufficiently for reproducibility. -`value_mapping.reference` and `value_mapping.query` MUST be ordered lists of entries with this form: +`value_mapping` MAY be omitted; omission means that no value mapping is applied. When present, it MUST contain `reference`, `query`, or both. Each present side MUST be a non-empty ordered list of entries with this form; an omitted side has no value mapping: ```yaml - from: @@ -430,114 +460,64 @@ A source-text operation MUST appear only in `source_normalisation`; a typed-valu ``` - `from` and `to` MUST be source-value descriptors as defined in Section 6.2.6. +- A `from` descriptor is matched against the source value after the applicable `source_normalisation` operations and missing-token recognition have been performed, in accordance with the processing order in Section 6.8. - A `to` descriptor with `category: "null"` explicitly maps to null and is distinct from absence of a mapping entry. -- Each `from` value MUST occur at most once within one side’s mapping. Conflicting or unreachable entries MUST invalidate the specification. -- A non-empty value mapping requires a scalar `logical_type`; value mapping of collection fields is not supported in the first release. Its `to` values are parsed as the destination type after mapping. -- When a side’s value-mapping list is non-empty, a source value with no mapping entry creates a contract finding. Pass-through of unmapped values is not supported in the first release. -- Empty lists declare that no value mapping is applied on that side. +- Each `from` value MUST occur at most once within one side’s mapping. Conflicting or unreachable entries MUST invalidate the comparison configuration. +- A value mapping requires a scalar `logical_type`; value mapping of collection fields is not supported in the first release. Its `to` values are parsed as the destination type after mapping. +- When a side has a value mapping, a source value with no mapping entry creates a contract finding. Pass-through of unmapped values is not supported in the first release. + +Before comparison, the tool MUST expand omitted optional processing properties and omitted dataset sides into explicit empty internal structures in the effective configuration. The user-authored configuration and the expanded effective configuration MUST remain distinguishable in provenance. Omission changes no behaviour beyond declaring that the corresponding processing step is absent. #### 6.2.8 Comparator declarations -Every field definition MUST contain `comparator.name` and `comparator.parameters`. `name` MUST be one of the comparator identifiers in Section 7.4, and `parameters` MUST be a mapping containing only parameters valid for that comparator. +Every field definition MUST contain `comparator.name`. `name` MUST be one of the comparator identifiers in Section 7.4. A `parameters` mapping MUST be present only when the selected comparator requires parameters and MUST contain only parameters valid for that comparator. -- `preserved_exact`, `typed_exact`, `normalised_string`, `numeric_exact`, `date_exact`, `datetime_exact`, `list_exact`, `set_exact`, and `multiset_exact` take no field-level parameters in the first release; their `parameters` mapping MUST be empty. -- `numeric_tolerance` permits only `absolute_tolerance` and `relative_tolerance`. At least one MUST be present. Each MUST be a quoted, non-negative finite decimal string; omission of one means it is treated as zero as specified in Section 7.4.5. +- `preserved_exact`, `typed_exact`, `normalised_string`, `numeric_exact`, `date_exact`, `datetime_exact`, `list_exact`, `set_exact`, and `multiset_exact` take no field-level parameters in the first release; `parameters` MUST be omitted for these comparators. +- `numeric_tolerance` MUST contain a `parameters` mapping that permits only `absolute_tolerance` and `relative_tolerance`. At least one tolerance MUST be present. Each MUST be a quoted, non-negative finite decimal string; omission of one means it is treated as zero as specified in Section 7.4.5. - The comparator MUST be compatible with `logical_type`: string comparators with `string`, numeric comparators with `integer` or `decimal`, date comparators with their corresponding temporal type, and collection comparators with their corresponding collection type. -- `normalised_string` requires at least one `source_normalisation` or `typed_normalisation` operation applicable to strings. +- `normalised_string` requires at least one operation in `source_normalisation.reference`, `source_normalisation.query`, or `typed_normalisation` that is applicable to strings. -#### 6.2.9 Acceptance criteria +#### 6.2.9 Normative machine-readable schema and semantic validation -`acceptance` MUST contain `contract`, `records`, and `values`, each an ordered list of criterion mappings. A list MAY be empty only when that phase cannot be reached under the approved intended use; selecting a scope that includes a phase with no criterion MUST invalidate the run as required by Section 8.5. +The normative JSON Schema Draft 2020-12 definition of comparison-configuration schema version `1.0` is [`src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json`](src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json). A YAML configuration MUST first be loaded using Section 6.2.1. The resulting JSON-compatible data structure MUST then validate against that schema before either input dataset is read. -Every criterion MUST contain: +The standalone schema file is the authoritative machine-readable definition. It MUST be version controlled in the source repository, included as package data in the installed `adviso_validation.schemas` package, and loaded at runtime through the package-resource API rather than through a path relative to the working directory. CI MUST verify its presence in built distributions and validate every documented example and controlled configuration fixture against it. Any inconsistency between the human-readable requirements in this document and the machine-readable schema is a specification defect that MUST be resolved before release. -- a stable `id`, unique within the specification; -- `metric`, naming a metric defined by the applicable schema version; -- `operator`, one of `equal`, `not_equal`, `less_than`, `less_than_or_equal`, `greater_than`, or `greater_than_or_equal`; and -- `threshold`, expressed as a non-negative integer for count metrics or a quoted finite decimal string between `0` and `1`, inclusive, for proportion metrics. +JSON Schema cannot reliably express every relationship involving user-defined mapping keys, uniqueness across separate arrays, or references between distant parts of a document. After schema validation, the implementation MUST therefore apply the following semantic checks before reading either dataset: -The first-release core metrics are `contract_finding_count`, `missing_record_count`, `extra_record_count`, `value_mismatch_count`, `not_assessable_count`, `reference_record_count`, and `query_record_count`. It also supports the corresponding proportion metrics `contract_finding_proportion` and `value_mismatch_proportion`. +- every name in `fields.record_key` exists in `fields.definitions`; +- every key field is required on both sides, non-nullable, scalar, and uses `typed_exact`; +- a field without `logical_type` uses `preserved_exact` and has no typed normalisation or value mapping; +- each comparator and normalisation operation is compatible with the field’s logical type and processing stage; +- source-value descriptors contain lexically valid values for their declared categories, and each side’s value mapping has unique `from` values; -A criterion MAY additionally contain: - -```yaml -fields: ["result", "confidence"] -finding_codes: ["VALUE_MISMATCH"] -parameters: - side: "reference" -denominator: - population: "in_scope_values" - not_assessed: "exclude" -exceptions: ["known-difference-001"] -``` - -- `fields` restricts the metric to the listed canonical fields; omission means all fields applicable to the metric. -- `finding_codes` restricts a finding-based metric to the listed stable codes; omission means all codes applicable to that metric. -- `parameters` MUST contain only parameters published for the chosen metric. `side` is required for a metric whose population is not otherwise unambiguous. -- A filter or parameter that is not meaningful for the chosen metric MUST invalidate the specification rather than be ignored. -- `denominator` is required for proportion metrics and MUST be absent for count metrics. `population` MUST name a population supported by the metric. `not_assessed` MUST be `include` or `exclude`, making its treatment explicit. `not_assessable` content is governed by Section 8.5 and cannot be configured as a match or silently excluded. -- `exceptions` lists exception identifiers applied before the criterion is evaluated; omission is equivalent to an empty list. -- A referenced exception excludes a matching finding from that criterion’s failing metric but MUST NOT remove the finding from the report, summary counts, or provenance. -- Criteria are evaluated in list order for reporting, but order MUST NOT alter any phase or overall outcome. - -#### 6.2.10 Permitted exceptions - -`exceptions` MUST be a list and MAY be empty. Each exception MUST contain a unique stable `id`, a non-empty `rationale`, and an `applies_to` mapping containing at least one of `finding_codes`, `fields`, or `record_keys`. It MAY contain `revision`, `approved_by`, and `review_date` as provenance strings. These metadata fields do not replace the laboratory’s external approval controls. - -```yaml -exceptions: - - id: "known-difference-001" - revision: "1" - rationale: "Approved legacy representation" - applies_to: - finding_codes: ["VALUE_MISMATCH"] - fields: ["result"] - record_keys: - - record_id: - category: "string" - value: "S001" - effect: "permitted_difference" - approved_by: "approval-record-reference" - review_date: "2027-01-31" -``` - -- `finding_codes`, `fields`, and `record_keys` are cumulative filters: a finding must satisfy every filter that is present. -- `record_keys` MUST be a list of mappings containing exactly the canonical key fields declared by `records.keys`. Each field value MUST be expressed as a value descriptor in the form defined by Section 6.2.6 and MUST represent the canonical post-transformation key value. -- `effect` MUST be `permitted_difference`; no exception may suppress parsing, contract assessment, reporting, or provenance. -- Every applied exception MUST remain visible in findings and criterion evaluation. -- An exception has no effect unless its `id` is referenced by an acceptance criterion. - -#### 6.2.11 Reporting controls - -- `reporting.finding_limit`, when present, MUST be a positive integer. Omission means that no comparison-finding limit is configured, although documented resource limits still apply. Reaching a configured limit has the outcome consequences specified in Sections 7.5 and 8.4. -- `reporting.excluded_field_detail` MUST be `summary` or `individual`. `summary` permits omission of individual `not_assessed` findings only when the exclusion summary required by Section 7.5 is emitted; `individual` requires a finding for each excluded field occurrence. -- Privacy and redaction settings are not defined by this schema version. They MUST be supplied by a separately approved reporting policy and recorded in provenance. +A failure of any semantic check MUST be reported as an invalid comparison configuration with the relevant configuration path and violated rule. ### 6.3 Record keys - A record key MUST contain one or more named fields. - Composite-key field order MUST be stable and recorded. -- Key fields MUST be present and non-null. +- Key fields MUST be present, valid under their declared field definitions, and non-null. - Key values MUST NOT undergo implicit numeric or boolean conversion. - Normalisation of key values MUST be explicitly configured. -- Alternative key names and composite keys MUST be declared in a custom specification. +- Alternative key names and composite keys MUST be declared in a custom configuration. - Key names MUST NOT be inferred, and command-line key overrides MUST NOT be supported in the first release. -- The normalised key MUST uniquely identify one record. -- Duplicate keys MUST be input errors in every profile and scope. +- The normalised key MUST uniquely identify one record within its dataset. +- A missing, null, invalid, or duplicate record key prevents construction of an unambiguous record model and MUST be an input error in every comparison. The resulting execution error MUST identify the affected dataset and the key or key field involved and, where available, the relevant source-record locations. - Duplicate records MUST never overwrite one another silently. - Findings and reports MUST retain enough information to distinguish every source record. ### 6.4 Field declarations and preserved values -The comparison engine MUST support both preserved, untyped values and explicitly declared logical types. A type declaration is required for a field that uses typed parsing, normalisation, mapping, or a type-specific comparator. Every custom specification MUST explicitly declare how undeclared fields are treated: `compare`, `ignore`, `warn`, or `fail`. There MUST be no implicit policy. An `ignore` policy MUST still be represented as an assessment exclusion in accordance with Section 7.5. +The comparison engine MUST support both preserved, untyped values and explicitly declared logical types. A type declaration is required for a field that uses typed parsing, normalisation, mapping, or a type-specific comparator. Every custom configuration MUST explicitly declare how undeclared fields are treated: `compare`, `exclude`, or `fail`. There MUST be no implicit policy. An `exclude` policy MUST be represented as an assessment exclusion in accordance with Section 7.5. -Every declared field MUST define its canonical name, reference and query source names where applicable, required status on each side, null policy, ordered normalisation and mapping lists, and comparator. It MUST declare a logical type except where Section 6.2.6 permits omission for `preserved_exact`. Unit, sensitivity classification, and finding severity are optional. The authoritative YAML representation and validation rules are defined in Section 6.2. +Every declared field MUST define its canonical name, reference and query source names, required status on each side, null policy, and comparator. It MUST declare a logical type except where Section 6.2.6 permits omission for `preserved_exact`. Normalisation, value mapping, and review priority are optional. An omitted review priority resolves to `normal` in the effective configuration. The authoritative YAML representation and validation rules are defined in Section 6.2. ### 6.5 Logical types and typed parsing -The comparison engine MUST support custom specifications declaring scalar logical types `string`, `integer`, `decimal`, `boolean`, `date`, and `datetime`, and typed `list`, `set`, and `multiset` collections. Collection fields MUST declare their element type. Null is a distinct value state governed by field policy, not a logical type. +The comparison engine MUST support custom configurations declaring scalar logical types `string`, `integer`, `decimal`, `boolean`, `date`, and `datetime`, and typed `list`, `set`, and `multiset` collections. Collection fields MUST declare their element type. Null is a distinct value state governed by field policy, not a logical type. -Type parsing MUST be field-specific and declared. The tool MUST NOT infer a logical type from observed text. Values of different logical types MUST NOT be implicitly equal. Intended equivalence between differing source representations requires both sides to be parsed as the same declared destination logical type, with an explicit value mapping where the source representation cannot be parsed directly. The original source value or lexical representation MUST remain available in evidence, subject to redaction policy. +Type parsing MUST be field-specific and declared. The tool MUST NOT infer a logical type from observed text. Values of different logical types MUST NOT be implicitly equal. Intended equivalence between differing source representations requires both sides to be parsed as the same declared destination logical type, with an explicit value mapping where the source representation cannot be parsed directly. The original source value or lexical representation MUST remain available to the comparison engine and MAY be included in evidence when required to explain a finding. An invalid typed value in a non-key field that remains attributable to an unambiguous record MUST be a contract finding and MUST make that value `not_assessable`; it MUST NOT be an execution error. Defects that prevent construction of a complete and unambiguous record model are classified under Section 8.3. @@ -596,7 +576,7 @@ An invalid typed value in a non-key field that remains attributable to an unambi ### 6.6 Null, blank, and absent values -The specification MUST distinguish absent fields, explicit nulls, empty strings, whitespace-only strings, and configured missing tokens such as `NA`, `.`, or `null`. No two states are equivalent unless a field-specific policy declares them equivalent. Applied equivalence rules MUST be recorded. +The comparison configuration MUST distinguish absent fields, explicit nulls, empty strings, whitespace-only strings, and configured missing tokens such as `NA`, `.`, or `null`. No two states are equivalent unless a field-specific policy declares them equivalent. Applied equivalence rules MUST be recorded. - Two explicit nulls MUST match only when the field permits null and its policy declares them equivalent. - Null MUST NOT implicitly equal empty string, absent field, zero, boolean false, or a missing token. @@ -605,7 +585,7 @@ The specification MUST distinguish absent fields, explicit nulls, empty strings, - A null and non-null pair MUST be a value mismatch when both states are valid and comparable. - A prohibited null MUST be a contract finding and MUST make the affected value `not_assessable` unless another treatment is explicitly configured. -The bundled strict-exact specification permits parser-native nulls. Under that specification, null matches only null and remains distinct from an absent field, an empty string, and every non-null value. +The bundled strict-exact configuration permits parser-native nulls. Under that configuration, null matches only null and remains distinct from an absent field, an empty string, and every non-null value. ### 6.7 Mappings @@ -626,73 +606,64 @@ The bundled strict-exact specification permits parser-native nulls. Under that s - Unknown source values MUST default to a contract finding, not pass through silently. - Many-to-one mappings MUST be allowed only when intentionally declared. - Mapping to null MUST be distinguished from a missing map entry. -- Mapping keys MUST be parsed without format-dependent scalar ambiguity. +- Values used in mappings MUST be represented using the typed source-value descriptor. The parser MUST NOT infer their intended category from YAML scalar syntax. - Applied mappings MUST be identifiable in the evidence. ### 6.8 Normalisation and processing order Normalisation MUST be opt-in, field-specific, ordered deterministically, and recorded. Source-text normalisation is distinct from typed-value normalisation. -Source-text normalisation MAY include Unicode normalisation, leading or trailing whitespace trimming, and case folding. Typed-value normalisation MAY include datetime offset normalisation, declared precision reduction, and stable ordering of collections declared to be unordered. Destructive or clinically material transformations SHOULD require an independently specified domain comparator. +Source-text normalisation MAY include Unicode normalisation, leading or trailing whitespace trimming, and case folding. Typed-value normalisation MAY include datetime offset normalisation, declared precision reduction, and stable ordering of collections declared to be unordered. Typed processing MUST occur in this order: 1. preserve the source value and representation; -2. apply declared source-text normalisation; +2. apply the source-text normalisation declared for the applicable dataset side; 3. recognise configured null or missing tokens; 4. apply a declared source-value mapping; 5. parse the declared logical type; 6. apply declared typed-value normalisation; 7. compare canonical values. -Column-name mapping MUST occur before field-value processing. A failure at any stage MUST identify the stage and attributable source value, subject to redaction policy. +Column-name mapping MUST occur before field-value processing. A failure at any stage MUST identify the stage and attributable record and field. The source value MUST be included in the evidence report only when required to explain the failure and MUST NOT be reproduced in routine logs or terminal diagnostics by default. -### 6.9 Bundled strict-exact specification +### 6.9 Bundled strict-exact configuration -If `--specification` is omitted, the tool MUST select the bundled strict-exact specification. It MUST be distributed as a documented template from which laboratories can develop controlled custom specifications, and the selected effective specification MUST be recorded in provenance. Supplying `--specification` MUST select the named custom specification; sentinel path values such as `DEFAULT` MUST NOT have special meaning. +If `--configuration` is omitted, the tool MUST select the bundled strict-exact configuration. It MUST be distributed as a documented template from which laboratories can develop controlled custom configurations, and the selected effective configuration MUST be recorded in provenance. Supplying `--configuration` MUST select the named custom configuration. -The bundled default specification MUST: +The bundled default configuration MUST: - declare one key whose canonical, reference, and query field names are all `record_id`; - include every shared non-key field without requiring field-specific declarations; - preserve parser-native types and use `preserved_exact` comparison; - perform no type inference, implicit coercion, mapping, normalisation, or tolerance; - permit parser-native null values and compare them according to Section 6.6; -- treat two syntactically valid zero-record datasets as satisfying all bundled acceptance criteria; -- treat reference-only and query-only fields as contract findings; -- require zero contract findings, zero missing or extra records, and zero value mismatches. +- treat two syntactically valid zero-record datasets as having zero missing records, zero extra records, and zero value mismatches, while evaluating contract findings as required by Sections 5.7 and 7.2; +- treat reference-only and query-only fields as contract findings. -Because parser-native types differ between formats, strict-exact comparison across different input formats MAY report type mismatches for lexically similar values. A custom typed specification is required when cross-format logical equivalence is intended. +Because parser-native types differ between formats, strict-exact comparison across different input formats MAY report type mismatches for lexically similar values. A custom typed configuration is required when cross-format logical equivalence is intended. ## 7. Comparison behaviour -### 7.1 Cumulative phases and scopes +### 7.1 Comparison phases -Comparison phases MUST be cumulative and ordered: +Every comparison MUST perform all three phases in this order: 1. **Contract:** assess fields, logical types, key definitions, nullability, mappings, and declared data-contract constraints. 2. **Records:** align valid record identities and assess matching, missing, and extra records. 3. **Values:** compare corresponding values using declared comparators. -Public scope names identify the final requested phase: - -| Scope | Contract | Records | Values | -| --- | --- | --- | --- | -| `contract` | Full | Not assessed | Not assessed | -| `records` | Full | Full | Not assessed | -| `full` | Full | Full | Full | - -`full` MUST be the default. Failure of an earlier phase MUST NOT automatically prevent later phases. Later comparison MUST continue wherever its prerequisites remain satisfied. Content inside the selected scope that cannot be compared safely MUST be `not_assessable`; content deliberately outside the scope or excluded by policy MUST be `not_assessed`. +A finding or `not_assessable` item in an earlier phase MUST NOT automatically prevent later assessment. Later comparison MUST continue wherever its prerequisites remain satisfied. Configured content that cannot be compared safely MUST be `not_assessable`; content deliberately excluded by field policy MUST be `not_assessed`. A run MUST conceptually: -1. validate the effective specification; +1. validate the effective comparison configuration; 2. identify and hash inputs; 3. parse inputs without silent loss; 4. perform the contract phase; -5. align and compare records when the scope is `records` or `full`; -6. compare values for safely aligned records when the scope is `full`; -7. evaluate applicable phase acceptance criteria; +5. align and compare records; +6. compare values for safely aligned records; +7. determine whether every configured assessment was completed; 8. write results and provenance atomically. ### 7.2 Contract comparison @@ -701,28 +672,30 @@ The contract phase MUST assess declared and observed fields from both directions - field present and valid; - required field missing from the reference or query; -- unexpected reference or query field; +- observed undeclared field and the treatment required by `fields.undeclared`; - invalid logical type, null state, mapping, or controlled value; - field deliberately `not_assessed`; - field `not_assessable` because a prerequisite failed. -One record’s observed field set MUST NOT be assumed to represent every other record. Key presence, validity, and uniqueness MUST be assessed in every scope because later interpretation depends on an unambiguous record model. +One record’s observed field set MUST NOT be assumed to represent every other record. Key presence, validity, and uniqueness MUST always be assessed because later interpretation depends on an unambiguous record model. + +For a zero-record dataset, contract requirements that can be evaluated from available structural information, such as CSV or TSV headers, MUST still be assessed. A requirement that depends on examining a record and cannot be evaluated because no records exist MUST be reported as `not_assessable`; it MUST NOT be treated as satisfied merely because no contradictory record was observed. ### 7.3 Records comparison For the union of normalised valid record keys, the records phase MUST classify each key as matching, missing from the query, or extra in the query. Duplicate or otherwise invalid keys are execution errors under Section 8.3, not ordinary record findings. -Missing and extra records MUST be reported individually unless the selected privacy mode permits only aggregate reporting. Query-only records MUST NOT be ignored. Missing or extra records MUST NOT prevent value comparison of the safely aligned intersection when the scope is `full`. +Missing and extra records MUST be reported individually. Query-only records MUST NOT be ignored. Missing or extra records MUST NOT prevent value comparison of the safely aligned intersection. ### 7.4 Values comparison and comparators -The values phase MUST compare every in-scope, valid field in safely aligned records. A value whose contract or alignment prerequisite failed MUST be `not_assessable`; other values MUST continue to be compared. +The values phase MUST compare every configured, valid field in safely aligned records. A value whose contract or alignment prerequisite failed MUST be `not_assessable`; other values MUST continue to be compared. The first release MUST support the following deterministic comparators with stable identifiers. #### 7.4.1 `preserved_exact` -Used by the bundled strict-exact specification. Values match only when their parser-native types and values are identical. No inference, coercion, mapping, normalisation, or tolerance is applied. Textual `"1"`, integer `1`, decimal `1.0`, and boolean `true` are distinct. +Used by the bundled strict-exact configuration. Values match according to the parser-native value model and equality rules in Section 5.6. No inference, coercion, mapping, normalisation, or tolerance is applied. Textual `"1"`, integer `1`, decimal `1.0`, and boolean `true` are distinct. #### 7.4.2 `typed_exact` @@ -749,7 +722,6 @@ match if difference <= allowed_difference - The boundary MUST be inclusive. - Relative tolerance MUST be directional and calculated against the absolute reference value. - Absolute tolerance MUST govern behaviour around a zero reference value. -- A numeric field MAY declare a unit as metadata. If unit semantics affect equivalence or acceptance, the unit MUST be declared explicitly and MUST be compatible on both sides. Automatic unit conversion is outside the first-release scope. - Rounding or significant-figure normalisation MUST occur explicitly before comparison. #### 7.4.6 `date_exact` and `datetime_exact` @@ -767,17 +739,14 @@ Dates match when their canonical calendar dates are equal. Datetimes match when - Comparison findings MUST NOT terminate a normal run. - All assessable differences MUST be collected. -- A configurable finding limit MAY protect resources and reaching it MUST be reported prominently as truncation. If a conclusive criterion failure has already been established, the outcome MUST remain `fail`; otherwise truncation MUST make the outcome `inconclusive`. A truncated run MUST NOT make claims about unexamined content. +- A documented implementation safeguard MAY stop assessment to protect resources, but it MUST NOT be configurable through the comparison configuration. Reaching such a safeguard MUST be reported prominently as truncation and MUST make the run `incomplete`. A truncated run MUST NOT make claims about unexamined content. - A malformed record that prevents a complete and unambiguous record model MUST be an input error and MUST NOT disappear silently. -- A field subset MUST be declared in the specification or invocation parameters retained in provenance. - Key fields MUST always be processed. -- Individual findings for fields outside a configured subset MAY be omitted only under a documented report policy. The canonical report MUST still record a stable exclusion code, the exclusion rule, the affected field names or configured subset, and the number of excluded records or values where calculable. -- The report MUST identify that only a subset was assessed. -- An unknown requested field MUST be an error. +- Under `fields.undeclared: "exclude"`, undeclared fields MUST be represented by a summary rather than by an individual finding for every excluded occurrence. The canonical report MUST record a stable exclusion code, the policy, the affected input field names, and the number of excluded records or values where calculable. - Record order MUST NOT affect results unless explicitly declared significant. - Output ordering MUST be deterministic and independent of hash-map ordering and locale. -## 8. Findings and outcomes +## 8. Findings and run status ### 8.1 Finding structure @@ -785,13 +754,13 @@ Every finding MUST be structured and SHOULD contain: - stable finding code; - category; -- severity; -- record key, or privacy-preserving surrogate; +- review priority, when the finding is attributable to a declared field; +- record key; - reference and query source locations where available; - canonical field name; - expected and observed presence state; - expected and observed logical type; -- expected and observed value, subject to redaction policy; +- expected and observed value when required to explain the finding; - comparator and parameters; - mappings/normalisations applied; - concise human-readable explanation; @@ -807,93 +776,69 @@ Stable result codes MUST exist for at least: - extra record; - invalid record key; - missing field; -- extra field; +- undeclared field; - type mismatch; - value mismatch; - mapping failure; - parse or data-contract error; -- invalid specification; +- invalid comparison configuration; - not assessed; - not assessable; - comparison truncated; - internal execution error. -### 8.3 Execution status +### 8.3 Run status -Execution status MUST be separate from comparison outcome and MUST include at least: +Run status MUST be one of: -- `completed`: comparison ran as specified and the result package was written; -- `error`: the tool could not establish a complete and unambiguous record model or could not complete the required output operation. +- `complete`: every configured assessment was performed wherever the configuration and inputs made it applicable; +- `incomplete`: the inputs and record model remain sufficiently trustworthy for a useful comparison, but one or more configured assessments could not be completed safely or assessment was truncated; +- `error`: no trustworthy comparison result could be produced because of an invalid invocation or comparison configuration, malformed or unreadable input syntax or structure, a missing, null, invalid, or duplicate record key, an incomplete or ambiguous record model, an input/output failure, or an unexpected internal failure. -An execution error includes an invalid invocation or specification, malformed or unreadable input syntax or structure, missing, null, invalid, or duplicate record keys, input/output failure, or an unexpected internal failure. No comparison outcome MUST be produced for an execution error. An error report SHOULD be written where possible and MUST be identified as such. +Differences between valid, assessable reference and query content—including contract findings, missing or extra records, and value mismatches—MUST NOT by themselves change a run from `complete` to `incomplete` or `error`. The run status describes completeness and operational validity, not acceptability. -Content that is parseable and attributable to an unambiguous record but violates the declared data contract MUST be recorded as a comparison finding rather than an execution error. Comparisons that remain assessable MUST continue. A completed comparison MAY have a failing or inconclusive comparison outcome. +Any configured content classified as `not_assessable`, or any content left unexamined because assessment was truncated, MUST make the run `incomplete`. Deliberately `not_assessed` content MUST NOT do so. An incomplete report MUST identify the affected or unexamined content and MUST not make claims about it. -### 8.4 Comparison outcome +Content that is parseable and attributable to an unambiguous record but violates the declared data contract MUST be recorded as a comparison finding rather than an error. Comparisons that remain assessable MUST continue. An error report SHOULD be written where possible and MUST be identified as such. -Comparison outcome MUST be one of: +Each phase MUST record whether it was `complete` or `incomplete`. Overall status is `complete` only when all three phases are complete. The tool MUST report raw findings and metrics without applying acceptance thresholds or producing a pass/fail verdict. -- `pass`: all configured acceptance criteria were met; -- `fail`: one or more configured acceptance criteria were not met; -- `inconclusive`: criteria could not be fully evaluated, including truncation or configured recoverable data errors. +### 8.4 Laboratory decision -A completed run is `inconclusive` only when the inputs remain sufficiently trustworthy for comparison, one or more requested assessments cannot be completed, and the available evidence cannot establish a defensible pass or fail. It MUST identify all unassessed content and MUST never produce a pass. - -Each executed phase MUST have its own outcome. A conclusive failure of any applicable criterion MUST produce overall `fail`. If no criterion conclusively fails but one or more applicable criteria cannot be evaluated, the overall outcome MUST be `inconclusive`. Overall `pass` requires every applicable criterion to be evaluated and satisfied. The outcome MUST be calculated only from explicit acceptance criteria and MUST NOT be inferred merely from whether warnings occurred. - -### 8.5 Acceptance criteria - -The specification MUST associate acceptance criteria with the individual `contract`, `records`, and `values` phases. It MUST NOT define a separate `full` acceptance section. The `contract` scope evaluates contract criteria; `records` evaluates contract and records criteria; and `full` evaluates all three phase sections. - -The specification MUST define how findings contribute to outcome. Criteria MAY include: - -- zero missing or extra records; -- zero mismatches in critical fields; -- a maximum number or proportion of specified differences; -- metric-specific thresholds; -- permitted known differences identified by stable rule and rationale. - -Criteria based on counts or proportions MUST define their denominator and the treatment of deliberately `not_assessed` content. `not_assessable` content MUST NOT be counted as a match or silently removed from a denominator. A criterion that depends on `not_assessable` content MUST be unevaluable unless its failure can be established conclusively from the assessable evidence. - -Exceptions and permitted differences MUST be explicit, versioned, reviewable, and reported. A custom specification missing acceptance criteria for any phase applicable to the selected scope MUST be invalid. - -### 8.6 Laboratory decision - -As established in Section 1, the calculated comparison outcome is not a laboratory validation or approval decision. Reports MUST label these concepts separately and MUST NOT describe the output as “ISO 15189 compliant”, “clinically validated”, or equivalent. +The run status and reported differences do not constitute a laboratory validation, approval, or acceptance decision. Any acceptance criteria and their application belong to a separately governed process outside the tool. Reports MUST label these concepts separately and MUST NOT describe the output as “ISO 15189 compliant”, “clinically validated”, or equivalent. ## 9. Output and evidence -A completed comparison MUST create an evidence-package directory at the destination selected by `--output`. The package MUST contain, at minimum, the canonical JSON result and the HTML report. A CSV findings export MAY also be included. Package member names and layout MUST be stable and documented. +A non-error comparison, whether `complete` or `incomplete`, MUST create an evidence-package directory at the destination selected by `--output`. The package MUST contain, at minimum, the canonical JSON result and the HTML report. A CSV findings export MAY also be included. Package member names and layout MUST be stable and documented. ### 9.1 Machine-readable output JSON MUST be the canonical output format. It MUST have a published, versioned schema containing: - report schema version; -- execution status and comparison outcome; +- run status and per-phase completion status; - run identifier; - timestamps; - tool identity and version; -- specification identity, version, and digest; +- comparison-configuration identity, revision, schema version, and digest; - input identities and digests; - effective comparison settings; - summary counts; - structured findings; - errors and warnings; - truncation status; -- privacy/redaction mode. -Consumers MUST be able to distinguish absent, null, and redacted values. +Consumers MUST be able to distinguish absent and null values. ### 9.2 Human-readable output The first release MUST produce an HTML human-readable report. The report SHOULD include: -- purpose and scope of the run; -- prominent outcome and execution status; -- identities of reference, query, and specification; +- purpose of the run and the three comparison phases performed; +- prominent run status and per-phase completion status; +- identities of reference, query, and comparison configuration; - summary counts; -- acceptance criteria and evaluation; +- measured counts and findings, without an acceptance verdict; - grouped discrepancy details; - limitations, exclusions, and unassessed content; - provenance and reproduction information; @@ -902,7 +847,7 @@ The first release MUST produce an HTML human-readable report. The report SHOULD CSV MAY be provided as an export of findings, but MUST: - be produced using a conforming CSV writer; -- preserve record identifiers or approved surrogates; +- preserve record identifiers; - quote delimiters, quotes, and newlines correctly; - use a stable documented column schema; - mitigate spreadsheet formula injection when intended for spreadsheet use; @@ -910,29 +855,29 @@ CSV MAY be provided as an export of findings, but MUST: ### 9.3 Summary counts -Summary counts MUST reconcile with emitted findings and SHOULD include: +Summary counts MUST reconcile with emitted findings and MUST include at least: - input record counts; - valid unique key counts; - matched, missing, and extra record counts; - fields assessed and not assessed; -- findings by code and severity; -- pass/fail/inconclusive criteria counts. +- findings by code and, where applicable, review priority; +- complete/incomplete status for each of the three phases. ### 9.4 Provenance Provenance MUST include: -- SHA-256 or stronger digest of each input and the specification; +- SHA-256 or stronger digest of each input and the comparison configuration; - tool version and build/release identity; - report schema version; -- invocation parameters after redaction; +- invocation parameters with secrets omitted; - start and end timestamps with time-zone offset; - runtime environment sufficient for reproduction; - selected input formats and encodings; - all non-default policies in effect. -Absolute paths and the raw command line MUST NOT be included by default because they may contain sensitive information or credentials. Safe display names MAY be supplied separately. +Absolute paths and the raw command line MUST NOT be included. Safe display names MAY be supplied separately. ### 9.5 Atomicity and overwrite protection @@ -964,16 +909,13 @@ adviso-validation compare \ --reference REFERENCE \ --query QUERY \ --output OUTPUT \ - [--scope {contract,records,full}] \ - [--specification SPECIFICATION] \ + [--configuration CONFIGURATION] \ [--reference-format {csv,tsv,json}] \ [--query-format {csv,tsv,json}] \ [--overwrite] ``` -`--scope` MUST accept `contract`, `records`, or `full`, and MUST default to `full`. Scope selection and any content made `not_assessable` by contract incompatibility MUST be prominent in the report. - -`--specification` MUST be optional. Its omission MUST select the bundled strict-exact specification; when present, its value MUST be treated as the path to a custom specification. `--reference-format` and `--query-format` MUST follow Section 5.2. `--output` MUST name the evidence-package directory described in Section 9, and `--overwrite` MUST provide the explicit overwrite authorisation required by Section 9.5. +`--configuration` MUST be optional. Its omission MUST select the bundled strict-exact configuration; when present, its value MUST be treated as the path to a custom comparison-configuration file. `--reference-format` and `--query-format` MUST follow Section 5.2. `--output` MUST name the evidence-package directory described in Section 9, and `--overwrite` MUST provide the explicit overwrite authorisation required by Section 9.5. Convenience flags MAY override non-critical settings only when the effective configuration is captured in provenance. Clinically material overrides SHOULD require an explicit opt-in and be prominent in the report. @@ -983,55 +925,44 @@ Exit codes MUST distinguish at least: | Code | Meaning | | ---: | --- | -| 0 | Execution completed and comparison outcome is `pass` | -| 1 | Execution completed and comparison outcome is `fail` | -| 2 | Execution completed but comparison outcome is `inconclusive` | -| 3 | Invocation, specification, input, or I/O error | -| 4 | Unexpected internal error | +| 0 | Run status is `complete`, whether or not differences were found | +| 1 | Run status is `incomplete` | +| 2 | Invocation, comparison-configuration, input, or I/O error | +| 3 | Unexpected internal error | -These exit-code meanings MUST be stable and documented. A comparison failure MUST still write a report when possible. +These exit-code meanings MUST be stable and documented. A complete comparison that finds differences MUST still exit with code 0 and write the evidence package. Error classes represented by exit codes 2 and 3 have run status `error`; the distinct codes allow automation to distinguish expected operational errors from unexpected implementation failures. ### 10.4 Additional commands No additional command is specified for the initial release. In particular, the prototype’s format-conversion operation is excluded because it does not presently justify a place in the validation-evidence workflow. Any future command requires its own approved behaviour specification and MUST satisfy the product-boundary criteria in Section 2.1. -## 11. Privacy, security, and data governance +## 11. Security and data governance -- The tool MUST assume that datasets may contain confidential patient or laboratory information. -- It MUST operate locally by default and MUST NOT transmit input or result data over a network without explicit functionality and authorisation. -- Reports MUST support configurable redaction or hashing of record identifiers and sensitive values. -- Hashing used as pseudonymisation MUST support a laboratory-controlled keyed method where resistance to dictionary attacks is required. -- Secrets MUST be redacted from recorded invocation details. +- The tool MUST assume that datasets and evidence packages may contain confidential patient or laboratory information. +- It MUST operate locally and MUST NOT transmit input data, comparison configurations, logs, or results over a network. +- Routine logs, progress output, and terminal diagnostics MUST NOT contain record identifiers or source values by default. A user MAY explicitly request diagnostic detail, and that choice MUST be prominent. +- Evidence reports MAY contain record identifiers and source values only where required to identify or explain findings. Reports MUST state prominently that the evidence package may contain sensitive information. +- Absolute paths, raw command lines, environment variables, and secrets MUST NOT be included in reports or routine logs. Safe display names and a structured record of non-secret invocation parameters MAY be included. - Temporary files MUST be created securely and removed according to documented policy. -- Logs MUST NOT contain more sensitive information than the selected report policy permits. -- Comparison-specification YAML MUST be parsed using the safe-loader rules in Section 6.2.1. +- Comparison-configuration YAML MUST be parsed using the safe-loader rules in Section 6.2.1. - Spreadsheet-oriented exports MUST mitigate formula injection. - Resource limits SHOULD protect against unexpectedly large or malicious inputs without silently truncating results. -- The tool SHOULD document filesystem permission expectations and leave access control to an appropriate deployment wrapper where necessary. - -## 12. Deployment, accessibility, performance, and reliability +- The tool SHOULD document filesystem permission expectations. The laboratory is responsible for supplying appropriately pseudonymised input data where required and for controlling access to, retention of, and disposal of inputs and evidence packages. +- Automated redaction, identifier pseudonymisation, keyed hashing, and field-level sensitivity policies are outside the scope of version 1.0. -The toolkit is intended to be usable in low-resource and LMIC settings. Therefore: +## 12. Deployment, accessibility, and reliability - all core comparison functionality MUST operate fully offline; - the toolkit MUST NOT require a hosted service, account, licence server, or telemetry connection; -- the base installation SHOULD minimise runtime dependency number and size; -- specialised dependencies SHOULD be optional and MUST NOT be required for strict structured-data comparison; - an offline-installable artefact with integrity information SHOULD be provided; -- a container image MAY be supplied but MUST NOT be the only deployment method; - reports MUST use open formats and MUST NOT require proprietary software; - errors and documentation SHOULD be usable without specialist Python knowledge; - documentation and examples SHOULD be available offline; - localisation SHOULD remain possible without changing result semantics. -Behaviour MUST be correct before optimisation. Releases SHOULD state: - -- the modest hardware profiles on which they were tested; -- representative dataset sizes; -- expected time and memory use; -- enforced resource or input-size limits. +Releases SHOULD state any enforced resource or input-size limits. -Large inputs MUST either be processed within documented limits or rejected without producing a false pass. Resource exhaustion MUST produce an execution error or `inconclusive` outcome according to whether a trustworthy partial record model remains. Interrupted runs MUST NOT leave apparently complete reports. Results MUST remain deterministic across supported Python versions and operating systems. Internal exceptions MUST be captured in an error report where safe, with concise diagnostics presented to the user. +Large inputs MUST either be processed within documented limits or stopped without presenting an incomplete assessment as complete. Resource exhaustion before construction of a complete and unambiguous record model MUST produce an error. Resource exhaustion after that model has been constructed MAY produce an `incomplete` run only when the evidence package identifies the unexamined content and otherwise satisfies Section 8.3. Interrupted runs MUST NOT leave apparently complete reports. Results MUST remain deterministic across supported Python versions and operating systems. Internal exceptions MUST be captured in an error report where safe, with concise diagnostics presented to the user. ## 13. Software quality and release requirements @@ -1048,14 +979,14 @@ The project MUST include: - unit tests for parsing, typing, mapping, normalisation, and comparators; - behavioural contract tests for result semantics; - end-to-end CLI tests including exit codes; -- schema validation tests for specifications and reports; +- schema validation tests for comparison configurations and reports; - installation tests from wheel and source distribution; - regression tests for every corrected material defect; - tests across all supported Python versions; - deterministic-output tests; - representative malformed and adversarial inputs; - tests for duplicate/composite keys and missing/extra content in both directions; -- boundary tests for tolerances and acceptance criteria. +- boundary tests for comparator tolerances. Property-based testing SHOULD be used for parsing and comparison invariants. Mutation testing SHOULD be used to assess whether the suite detects deliberately introduced comparison defects. @@ -1066,13 +997,13 @@ Controlled test cases SHOULD record: - stable scenario identifier; - requirement(s) tested; - purpose and rationale; -- input data and comparison specification; -- expected findings, counts, status, outcome, and exit code; +- input data and comparison configuration; +- expected findings, counts, run status, per-phase statuses, and exit code; - fixture provenance or generation method; - fixture version and digest; - independent review status where used as formal validation evidence. -Generated fixtures MUST be reproducible. Clinically derived fixtures MUST be appropriately de-identified and governed. Controlled test fixtures, including test comparison specifications, MUST be maintained by the development team under version control and its documented review process. In operational use, each laboratory is responsible for creating, approving, maintaining, and retaining its own comparison specifications under its local quality-management arrangements; the AdvISO project does not centrally approve or manage laboratory specifications. +Generated fixtures MUST be reproducible. Clinically derived fixtures MUST be appropriately de-identified and governed. Controlled test fixtures, including test comparison configurations, MUST be maintained by the development team under version control and its documented review process. In operational use, each laboratory is responsible for creating, approving, maintaining, and retaining its own comparison configurations under its local quality-management arrangements; the AdvISO project does not centrally approve or manage laboratory configurations. ### 13.4 Continuous integration @@ -1092,13 +1023,13 @@ Each release SHOULD provide: - an immutable semantic version; - release notes and known limitations; -- compatibility statements for specification and report schemas; +- compatibility statements for comparison-configuration and report schemas; - documented change-impact assessment; - signed or otherwise attestable source and build artefacts where practical; - a reproducible lockfile or equivalent dependency record; - evidence that required verification tests passed. -Specification-schema versions and toolkit versions MUST be independent. Each toolkit release MUST document the specification-schema and report-schema versions it supports. An unsupported schema version MUST be rejected clearly rather than interpreted approximately. The project MUST publish migration and compatibility rules before changing either schema and MUST provide a migration path or documented transition period for any incompatible change. +Comparison-configuration schema versions and toolkit versions MUST be independent. Each toolkit release MUST document the comparison-configuration-schema and report-schema versions it supports. An unsupported schema version MUST be rejected clearly rather than interpreted approximately. The project MUST publish migration and compatibility rules before changing either schema and MUST provide a migration path or documented transition period for any incompatible change. ## 14. Acceptance-test catalogue @@ -1107,26 +1038,31 @@ This catalogue summarises minimum executable scenarios for the first release. It ### 14.1 Input and contract scenarios - **AT-CON-001:** Identical typed datasets satisfy their contract. -- **AT-CON-002:** Required missing and unexpected extra fields are reported in both directions and across heterogeneous records. +- **AT-CON-002:** Required missing fields produce contract findings in both directions and across heterogeneous records; optional missing fields do not. Observed undeclared fields are handled according to each `fields.undeclared` mode and aligned only by identical input names under `compare`. - **AT-CON-003:** Leading-zero identifiers remain strings unless explicitly transformed. - **AT-CON-004:** Null, empty string, whitespace-only string, absent field, and configured missing tokens remain distinct according to policy. - **AT-CON-005:** Unknown, conflicting, and incomplete mappings fail clearly; mapping to null differs from a missing map entry. - **AT-CON-006:** Malformed syntax or structure, empty or duplicate column names, and irregular tabular rows receive their specified error classification without silent data loss. -- **AT-CON-007:** Two syntactically valid empty datasets pass under the bundled strict-exact specification; one-sided empty datasets produce all applicable missing or extra record findings. -- **AT-CON-008:** Comparison-specification YAML scalar resolution and cross-format typed dataset parsing are deterministic across supported environments. -- **AT-CON-009:** The bundled strict-exact specification treats parser-native null as equal only to null and distinguishes null, absent fields, and empty strings. -- **AT-CON-010:** Every custom specification explicitly declares an undeclared-field policy; omission is rejected. -- **AT-CON-011:** A complete custom YAML specification conforming to Section 6.2 validates and produces the declared effective configuration. +- **AT-CON-007:** For two syntactically valid zero-record datasets, the records phase reports no missing or extra records and the values phase performs no comparisons under every configuration. Independently assessable contract requirements are still evaluated, and requirements that depend on examining a record are `not_assessable`. One-sided empty datasets produce all applicable missing or extra record findings. +- **AT-CON-008:** Comparison-configuration YAML scalar resolution and cross-format typed dataset parsing are deterministic across supported environments. +- **AT-CON-009:** The bundled strict-exact configuration treats parser-native null as equal only to null and distinguishes null, absent fields, and empty strings. +- **AT-CON-010:** `fields.undeclared` is required and accepts only `compare`, `exclude`, or `fail`; omission and unknown policies are rejected. +- **AT-CON-011:** A complete custom YAML configuration conforming to Section 6.2 validates; omitted optional processing properties and dataset sides expand to empty internal structures in the effective configuration, while explicitly empty placeholder lists or mappings are rejected. - **AT-CON-012:** Duplicate or unknown YAML keys, unsafe YAML features, unsupported schema versions, invalid enumerations, and incompatible type/comparator combinations are rejected with precise configuration paths. - **AT-CON-013:** Source-value descriptors distinguish strings, numbers, booleans, and null; mapping to null remains distinct from an absent mapping entry. -- **AT-CON-014:** Each selected comparison phase requires applicable acceptance criteria, and count and proportion criteria validate their operators, thresholds, filters, and denominators. +- **AT-CON-014:** The authoritative schema is present in the source tree and built distribution, can be loaded through the installed package-resource API, and validates every complete configuration example and controlled configuration fixture. +- **AT-CON-015:** Configurations that are structurally schema-valid but violate a semantic rule in Section 6.2.9 are rejected with the relevant configuration path and rule. +- **AT-CON-016:** `review_priority` accepts only `low`, `normal`, or `high`; omission resolves to `normal` in the effective configuration. +- **AT-CON-017:** `fields.record_key` is required, non-empty, ordered, and free of duplicate component fields; every entry names a field in `fields.definitions` that satisfies all key-field constraints. +- **AT-CON-018:** `metadata.id` and `metadata.revision` are required non-empty strings; title and description are optional, metadata does not alter comparison behaviour, and every metadata change alters the configuration digest. +- **AT-CON-019:** Every declared field supplies non-empty `source.reference` and `source.query` names, including when the field is optional on either side; the `required` flags govern record-level presence rather than whether a correspondence is declared. ### 14.2 Record scenarios - **AT-REC-001:** Every reference-only record is reported as missing and every query-only record as extra, including when both occur together. -- **AT-REC-002:** Duplicate single and composite keys are rejected without overwriting data. +- **AT-REC-002:** Duplicate single and composite keys produce execution errors without overwriting data; each error identifies the affected dataset, duplicate normalised key, and available source-record locations. - **AT-REC-003:** Composite keys preserve multiple legitimate records belonging to one sample. -- **AT-REC-004:** Record order does not alter findings or outcomes unless explicitly declared significant. +- **AT-REC-004:** Record order does not alter findings, metrics, or run and phase statuses. ### 14.3 Typed value and comparator scenarios @@ -1140,33 +1076,36 @@ This catalogue summarises minimum executable scenarios for the first release. It - **AT-VAL-008:** Typed transformations follow Section 6.8 and retain evidence of the attributable source value. - **AT-VAL-009:** Numeric tolerance uses exact decimal arithmetic, a reference-relative component, and an inclusive boundary, including below, at, and above boundary cases. - **AT-VAL-010:** All assessable differences are collected in one run. -- **AT-VAL-011:** Subset comparison identifies its limited coverage, records the required exclusion summary when individual findings are omitted, and rejects unknown requested fields. +- **AT-VAL-011:** Every defined field is included in comparison; under `fields.undeclared: "exclude"`, undeclared fields are `not_assessed` and represented by the required exclusion summary. - **AT-VAL-012:** `preserved_exact` follows the parser-native category table, including deterministic same-format and cross-format comparisons. -### 14.4 Scope and outcome scenarios +### 14.4 Phase and run-status scenarios -- **AT-OUT-001:** `contract` assesses only the contract phase. -- **AT-OUT-002:** `records` assesses contract and records without comparing values. -- **AT-OUT-003:** `full` is the default and assesses contract, records, and values. -- **AT-OUT-004:** Failure of an earlier phase does not prevent later assessment where prerequisites remain satisfied. -- **AT-OUT-005:** Content excluded deliberately is `not_assessed`; in-scope content blocked by a failed prerequisite is `not_assessable`. -- **AT-OUT-006:** A conclusive criterion failure produces `fail`; otherwise an unevaluable applicable criterion produces `inconclusive`; every applicable criterion must pass for overall `pass`. -- **AT-OUT-007:** Pass, fail, inconclusive, invocation or input error, and internal error produce their documented exit behaviour. -- **AT-OUT-008:** Truncation preserves an already established conclusive failure; otherwise it produces `inconclusive` and identifies unexamined content. -- **AT-OUT-009:** Counts and proportions use declared denominators; `not_assessable` content is never treated as a match or silently excluded. +- **AT-OUT-001:** Every comparison performs the contract, records, and values phases in order and records a completion status for each. +- **AT-OUT-002:** A finding or `not_assessable` item in an earlier phase does not prevent later assessment where prerequisites remain satisfied. +- **AT-OUT-003:** Content excluded deliberately is `not_assessed`; configured content blocked by a failed prerequisite is `not_assessable`. +- **AT-OUT-004:** A fully performed comparison is `complete` whether it finds no differences or any combination of contract findings, missing records, extra records, and value mismatches. +- **AT-OUT-005:** Complete, incomplete, invocation or input error, and unexpected internal error produce their documented exit behaviour. +- **AT-OUT-006:** Truncation produces `incomplete`, identifies unexamined content, and makes no claims about it. +- **AT-OUT-007:** Configured `not_assessable` content produces `incomplete`; deliberately `not_assessed` content does not. +- **AT-OUT-008:** Invalid configuration, malformed or unreadable input, and an incomplete or ambiguous record model produce `error`, never `incomplete`. ### 14.5 Evidence, CLI, and packaging scenarios - **AT-EVD-001:** Identical runs produce semantically identical canonical results except for explicitly variable run metadata. -- **AT-EVD-002:** Reports contain input and specification digests without exposing absolute paths or raw secrets by default. -- **AT-EVD-003:** CSV findings containing commas, quotes, newlines, or formula prefixes are exported safely. +- **AT-EVD-002:** Reports contain input and comparison-configuration digests without exposing absolute paths, raw command lines, environment variables, or secrets. +- **AT-EVD-003:** If the optional CSV findings export is provided, findings containing commas, quotes, newlines, or formula prefixes are exported safely. - **AT-EVD-004:** An output failure or interrupted write cannot leave an evidence package that appears complete. -- **AT-EVD-005:** A completed passing comparison exits with code 0; comparison failures still produce reports. -- **AT-EVD-006:** Omitting `--specification` selects and records the bundled strict-exact specification; a supplied value is treated only as a custom-specification path. +- **AT-EVD-005:** Every complete comparison exits with code 0 and produces its evidence package, including when differences are reported. +- **AT-EVD-006:** Omitting `--configuration` selects and records the bundled strict-exact configuration; a supplied value is treated only as a custom comparison-configuration path. - **AT-EVD-007:** Explicit reference and query formats override extension detection; recognised extensions are detected and recorded; unknown or ambiguous extensions require explicit formats. -- **AT-EVD-008:** A successful run creates the documented JSON and HTML evidence package, refuses an existing destination by default, and replaces it only with `--overwrite`. -- **AT-EVD-009:** Invocation, specification, input, and output errors write a clearly identified error report when an output location can be established. -- **AT-PKG-001:** A built wheel contains every required subpackage, and its installed CLI completes an end-to-end comparison. +- **AT-EVD-008:** A complete or incomplete run creates the documented JSON and HTML evidence package, refuses an existing destination by default, and replaces it only with `--overwrite`. +- **AT-EVD-009:** When an output location can be established and an error report can be written, invocation, configuration, input, and output errors produce a clearly identified error report. +- **AT-EVD-010:** Routine logs, progress output, and terminal diagnostics omit record identifiers and source values. +- **AT-EVD-011:** Evidence packages warn that they may contain sensitive information, and findings contain record identifiers or source values only where needed to identify or explain the finding. +- **AT-EVD-012:** Core comparison completes without network access and makes no attempt to transmit inputs, comparison configurations, logs, or results. +- **AT-EVD-013:** Findings attributable to declared fields inherit their resolved review priority; other findings do not. HTML presentation may distinguish priorities but does not rely on colour alone, and priority never changes findings, counts, status, or exit code. +- **AT-PKG-001:** The built distribution is named `adviso-validation`, contains an importable `adviso_validation` package and every required subpackage, installs the `adviso-validation` executable, and completes an end-to-end comparison through that executable. ## 15. Compatibility with the current prototype @@ -1185,11 +1124,8 @@ The current free-form result strings MUST NOT become the canonical result API. The following non-kernel decisions remain open. They do not prevent implementation of the comparison kernel, but MUST be resolved at the appropriate point before the specification and first release are approved. -1. What default privacy and redaction mode is appropriate for the intended deployment environment? -2. Which provenance fields are mandatory in local interactive use, CI, and formal retained evidence? -3. What maximum input sizes and performance targets are operationally relevant? -4. What compatibility period, if any, is required for the prototype CLI and outputs? -5. What objective hardware profile should define “modest” deployment and performance testing? +1. Which provenance fields are mandatory in local interactive use, CI, and formal retained evidence? +2. What compatibility period, if any, is required for the prototype CLI and outputs? ## 17. Suggested approval criteria for this specification @@ -1198,9 +1134,9 @@ Before implementation begins, reviewers SHOULD confirm that: - the intended use and non-claims are accurate; - terminology matches laboratory practice; - the record model covers representative pipeline outputs; -- missing, extra, duplicate, null, and malformed-data policies are unambiguous; -- comparison and acceptance are clearly separated; -- privacy and provenance requirements fit local governance; +- missing- and extra-content policies, and the handling of duplicate, null, and malformed data, are unambiguous; +- comparison evidence, run status, and the laboratory decision are clearly separated; +- data-handling and provenance requirements fit local governance; - every first-milestone scenario has an agreed expected result; - unresolved decisions have named owners and target dates; - approved requirements have stable identifiers for test traceability. diff --git a/src/adviso_validation/__init__.py b/src/adviso_validation/__init__.py new file mode 100644 index 0000000..76eb0da --- /dev/null +++ b/src/adviso_validation/__init__.py @@ -0,0 +1 @@ +"""AdvISO validation toolkit.""" diff --git a/src/adviso_validation/schemas/__init__.py b/src/adviso_validation/schemas/__init__.py new file mode 100644 index 0000000..3011ae2 --- /dev/null +++ b/src/adviso_validation/schemas/__init__.py @@ -0,0 +1 @@ +"""Versioned machine-readable schemas distributed with adviso-validation.""" diff --git a/src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json b/src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json new file mode 100644 index 0000000..9c59da4 --- /dev/null +++ b/src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json @@ -0,0 +1,638 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "title": "AdvISO comparison configuration", + "description": "Normative structural schema for comparison-configuration schema version 1.0.", + "type": "object", + "additionalProperties": false, + "required": [ + "schema_version", + "metadata", + "fields" + ], + "properties": { + "schema_version": { + "const": "1.0" + }, + "metadata": { + "$ref": "#/$defs/configurationMetadata" + }, + "fields": { + "$ref": "#/$defs/fields" + } + }, + "$defs": { + "nonEmptyString": { + "type": "string", + "minLength": 1 + }, + "uniqueNonEmptyStrings": { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { + "$ref": "#/$defs/nonEmptyString" + } + }, + "configurationMetadata": { + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "revision" + ], + "properties": { + "id": { + "$ref": "#/$defs/nonEmptyString" + }, + "revision": { + "$ref": "#/$defs/nonEmptyString" + }, + "title": { + "type": "string" + }, + "description": { + "type": "string" + } + } + }, + "fields": { + "type": "object", + "additionalProperties": false, + "required": [ + "record_key", + "undeclared", + "definitions" + ], + "properties": { + "record_key": { + "$ref": "#/$defs/uniqueNonEmptyStrings" + }, + "undeclared": { + "enum": [ + "compare", + "exclude", + "fail" + ] + }, + "definitions": { + "type": "object", + "minProperties": 1, + "propertyNames": { + "$ref": "#/$defs/nonEmptyString" + }, + "additionalProperties": { + "$ref": "#/$defs/fieldDefinition" + } + } + } + }, + "fieldDefinition": { + "type": "object", + "additionalProperties": false, + "required": [ + "source", + "required", + "nulls", + "comparator" + ], + "properties": { + "source": { + "type": "object", + "additionalProperties": false, + "required": [ + "reference", + "query" + ], + "properties": { + "reference": { + "$ref": "#/$defs/nonEmptyString" + }, + "query": { + "$ref": "#/$defs/nonEmptyString" + } + } + }, + "required": { + "type": "object", + "additionalProperties": false, + "required": [ + "reference", + "query" + ], + "properties": { + "reference": { + "type": "boolean" + }, + "query": { + "type": "boolean" + } + } + }, + "logical_type": { + "$ref": "#/$defs/logicalType" + }, + "nulls": { + "$ref": "#/$defs/nullPolicy" + }, + "source_normalisation": { + "type": "object", + "additionalProperties": false, + "minProperties": 1, + "properties": { + "reference": { + "type": "array", + "minItems": 1, + "items": { + "$ref": "#/$defs/sourceNormalisation" + } + }, + "query": { + "type": "array", + "minItems": 1, + "items": { + "$ref": "#/$defs/sourceNormalisation" + } + } + } + }, + "value_mapping": { + "$ref": "#/$defs/valueMapping" + }, + "typed_normalisation": { + "type": "array", + "minItems": 1, + "items": { + "$ref": "#/$defs/typedNormalisation" + } + }, + "comparator": { + "$ref": "#/$defs/comparator" + }, + "review_priority": { + "default": "normal", + "enum": [ + "low", + "normal", + "high" + ] + } + }, + "allOf": [ + { + "if": { + "not": { + "required": [ + "logical_type" + ] + } + }, + "then": { + "properties": { + "comparator": { + "properties": { + "name": { + "const": "preserved_exact" + } + } + } + }, + "not": { + "anyOf": [ + { + "required": [ + "value_mapping" + ] + }, + { + "required": [ + "typed_normalisation" + ] + } + ] + } + } + } + ] + }, + "scalarLogicalType": { + "enum": [ + "string", + "integer", + "decimal", + "boolean", + "date", + "datetime" + ] + }, + "logicalType": { + "oneOf": [ + { + "$ref": "#/$defs/scalarLogicalType" + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "name", + "element_type", + "element_comparator" + ], + "properties": { + "name": { + "enum": [ + "list", + "set", + "multiset" + ] + }, + "element_type": { + "$ref": "#/$defs/scalarLogicalType" + }, + "element_comparator": { + "$ref": "#/$defs/comparator" + } + } + } + ] + }, + "nullPolicy": { + "type": "object", + "additionalProperties": false, + "required": [ + "allowed", + "nulls_equal" + ], + "properties": { + "allowed": { + "type": "boolean" + }, + "nulls_equal": { + "type": "boolean" + }, + "missing_tokens": { + "type": "object", + "additionalProperties": false, + "minProperties": 1, + "properties": { + "reference": { + "type": "array", + "minItems": 1, + "items": { + "$ref": "#/$defs/sourceValueDescriptor" + } + }, + "query": { + "type": "array", + "minItems": 1, + "items": { + "$ref": "#/$defs/sourceValueDescriptor" + } + } + } + } + }, + "allOf": [ + { + "if": { + "properties": { + "allowed": { + "const": false + } + }, + "required": [ + "allowed" + ] + }, + "then": { + "properties": { + "nulls_equal": { + "const": false + } + } + } + } + ] + }, + "sourceValueDescriptor": { + "oneOf": [ + { + "type": "object", + "additionalProperties": false, + "required": [ + "category", + "value" + ], + "properties": { + "category": { + "const": "string" + }, + "value": { + "type": "string" + } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "category", + "value" + ], + "properties": { + "category": { + "const": "integer" + }, + "value": { + "type": "string", + "pattern": "^-?(?:0|[1-9][0-9]*)$" + } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "category", + "value" + ], + "properties": { + "category": { + "const": "decimal" + }, + "value": { + "type": "string", + "pattern": "^-?(?:(?:0|[1-9][0-9]*)\\.[0-9]+|(?:0|[1-9][0-9]*)(?:[eE][+-]?[0-9]+)|(?:0|[1-9][0-9]*)\\.[0-9]+(?:[eE][+-]?[0-9]+))$" + } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "category", + "value" + ], + "properties": { + "category": { + "const": "boolean" + }, + "value": { + "type": "boolean" + } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "category" + ], + "properties": { + "category": { + "const": "null" + } + } + } + ] + }, + "sourceNormalisation": { + "oneOf": [ + { + "type": "object", + "additionalProperties": false, + "required": [ + "operation", + "parameters" + ], + "properties": { + "operation": { + "const": "unicode_normalise" + }, + "parameters": { + "type": "object", + "additionalProperties": false, + "required": [ + "form" + ], + "properties": { + "form": { + "enum": [ + "NFC", + "NFD", + "NFKC", + "NFKD" + ] + } + } + } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "operation", + "parameters" + ], + "properties": { + "operation": { + "const": "trim" + }, + "parameters": { + "type": "object", + "additionalProperties": false, + "required": [ + "side" + ], + "properties": { + "side": { + "enum": [ + "leading", + "trailing", + "both" + ] + } + } + } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "operation" + ], + "properties": { + "operation": { + "const": "case_fold" + }, + "parameters": { + "type": "object", + "maxProperties": 0 + } + } + } + ] + }, + "typedNormalisation": { + "oneOf": [ + { + "type": "object", + "additionalProperties": false, + "required": [ + "operation", + "parameters" + ], + "properties": { + "operation": { + "const": "decimal_quantise" + }, + "parameters": { + "type": "object", + "additionalProperties": false, + "required": [ + "places", + "rounding" + ], + "properties": { + "places": { + "type": "integer", + "minimum": 0 + }, + "rounding": { + "enum": [ + "half_even", + "half_up", + "down" + ] + } + } + } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "operation", + "parameters" + ], + "properties": { + "operation": { + "const": "datetime_truncate" + }, + "parameters": { + "type": "object", + "additionalProperties": false, + "required": [ + "precision" + ], + "properties": { + "precision": { + "enum": [ + "second", + "millisecond", + "microsecond" + ] + } + } + } + } + } + ] + }, + "valueMapping": { + "type": "object", + "additionalProperties": false, + "minProperties": 1, + "properties": { + "reference": { + "type": "array", + "minItems": 1, + "items": { + "$ref": "#/$defs/valueMappingEntry" + } + }, + "query": { + "type": "array", + "minItems": 1, + "items": { + "$ref": "#/$defs/valueMappingEntry" + } + } + } + }, + "valueMappingEntry": { + "type": "object", + "additionalProperties": false, + "required": [ + "from", + "to" + ], + "properties": { + "from": { + "$ref": "#/$defs/sourceValueDescriptor" + }, + "to": { + "$ref": "#/$defs/sourceValueDescriptor" + } + } + }, + "comparator": { + "oneOf": [ + { + "type": "object", + "additionalProperties": false, + "required": [ + "name" + ], + "properties": { + "name": { + "enum": [ + "preserved_exact", + "typed_exact", + "normalised_string", + "numeric_exact", + "date_exact", + "datetime_exact", + "list_exact", + "set_exact", + "multiset_exact" + ] + } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "name", + "parameters" + ], + "properties": { + "name": { + "const": "numeric_tolerance" + }, + "parameters": { + "type": "object", + "additionalProperties": false, + "minProperties": 1, + "properties": { + "absolute_tolerance": { + "$ref": "#/$defs/nonNegativeDecimalString" + }, + "relative_tolerance": { + "$ref": "#/$defs/nonNegativeDecimalString" + } + } + } + } + } + ] + }, + "nonNegativeDecimalString": { + "type": "string", + "pattern": "^(?:0|[1-9][0-9]*)(?:\\.[0-9]+)?(?:[eE][+-]?[0-9]+)?$" + } + } +} From 52dd09a7dd594d18af3f511df289fb75d8c20376 Mon Sep 17 00:00:00 2001 From: Kevin Howe Date: Wed, 21 Oct 2026 16:56:50 +0100 Subject: [PATCH 04/45] More refinement of SPEC (including output schema) --- SPECIFICATION.md | 1333 +++++++++++------ .../comparison-configuration-v1.0.schema.json | 1 + .../comparison-results-v1.0.schema.json | 1272 ++++++++++++++++ 3 files changed, 2114 insertions(+), 492 deletions(-) create mode 100644 src/adviso_validation/schemas/comparison-results-v1.0.schema.json diff --git a/SPECIFICATION.md b/SPECIFICATION.md index d8e8cd5..e5960e5 100644 --- a/SPECIFICATION.md +++ b/SPECIFICATION.md @@ -37,17 +37,29 @@ adviso-project └── compare subcommand ``` -The **AdvISO Validation Toolkit** is the user-facing product and a minimal command namespace. Its installable distribution package and command-line executable MUST both be named **`adviso-validation`**. **`compare`** is currently the sole specified subcommand. The other components, including the **comparison engine**, are internal reusable software components and are not separate products. +**REQ-PRD-001.** The **AdvISO Validation Toolkit** is the user-facing product and a minimal command namespace. The installable distribution package MUST be named **`adviso-validation`**. The command-line executable MUST be named **`adviso-validation`**. **`compare`** is currently the sole specified subcommand. The other components, including the **comparison engine**, are internal reusable software components and are not separate products. -The command-line form MUST be: +**REQ-PRD-002.** The command-line form MUST be: ```text adviso-validation compare ... ``` -The Python import package MUST be named `adviso_validation`. The repository name MAY differ from the distribution and import-package names, but it MUST be documented clearly. Once released, changes to the distribution, executable, or import-package name MUST follow the documented compatibility policy. +**REQ-PRD-003.** The Python import package MUST be named `adviso_validation`. -Additional commands are not a development objective for the first release and MUST NOT be added speculatively. A future command MAY be introduced only for a demonstrated, materially distinct operation in the same validation-evidence workflow. The top-level dispatcher MUST remain thin and MUST NOT create speculative shared abstractions. The initial implementation SHOULD use ordinary modules and a small comparator registry, not a general plugin framework. Capabilities with materially different purposes, dependencies, governance, or release lifecycles SHOULD be separate AdvISO packages. +**REQ-PRD-004.** The repository name MAY differ from the distribution and import-package names. Any difference between these names MUST be documented clearly. + +**REQ-PRD-005.** Once released, changes to the distribution name MUST follow the documented compatibility policy. Changes to the executable name MUST follow that policy. Changes to the import-package name MUST follow that policy. + +**REQ-PRD-006.** Additional commands are not a development objective for the first release. They MUST NOT be added speculatively. + +**REQ-PRD-007.** A future command MAY be introduced only for a demonstrated, materially distinct operation in the same validation-evidence workflow. + +**REQ-PRD-008.** The top-level dispatcher MUST remain thin. It MUST NOT create speculative shared abstractions. + +**REQ-PRD-009.** The initial implementation SHOULD use ordinary modules and a small comparator registry rather than a general plugin framework. + +**REQ-PRD-010.** Capabilities with materially different purposes, dependencies, governance, or release lifecycles SHOULD be separate AdvISO packages. ### 2.2 Scope of the `compare` subcommand @@ -81,16 +93,15 @@ The `compare` command does not: The following principles govern all requirements: -1. **Explicit over inferred:** Comparison behaviour MUST be declared, not guessed from observed data. -2. **No silent data loss:** Input records, duplicate keys, unmapped values, and unsupported structures MUST NOT be silently discarded or overwritten. -3. **Symmetry:** Missing and extra content MUST be assessed in both directions. +1. **REQ-PRN-001.** **Explicit over inferred:** Comparison behaviour MUST be declared, not guessed from observed data. +2. **REQ-PRN-002.** **No silent data loss:** Input records, duplicate keys, unmapped values, and unsupported structures MUST NOT be silently discarded or overwritten. +3. **REQ-PRN-003.** **Symmetry:** Missing and extra content MUST be assessed in both directions. 4. **Separation of concerns:** Run status, comparison evidence, and the laboratory's acceptance decision are distinct concepts. -5. **Complete evidence:** A normal comparison mismatch MUST still produce a complete report. -6. **Determinism:** Identical inputs, configuration, and tool version MUST produce semantically identical results. -7. **Traceability:** Findings MUST identify the rule, record, and field that produced them. -8. **Data minimisation:** Reports MUST include only the source identifiers and values necessary to explain and review the comparison; routine logs and diagnostics MUST not reproduce them by default. -9. **Fail clearly:** Invalid or ambiguous input MUST produce an explicit error rather than an assumed interpretation. -10. **Testability:** Each normative requirement SHOULD be traceable to one or more automated tests. +5. **REQ-PRN-004.** **Complete evidence:** A normal comparison mismatch MUST still produce a complete report. +6. **REQ-PRN-005.** **Determinism:** Identical inputs, configuration, and tool version MUST produce semantically identical results. +7. **REQ-PRN-006.** **Traceability:** Findings MUST identify the rule, record, and field that produced them. +8. **REQ-PRN-007.** **Data minimisation:** Reports MUST include only the source identifiers and values necessary to explain and review the comparison. Routine logs, progress output, terminal diagnostics, and execution-error diagnostics MUST NOT reproduce them. +9. **REQ-PRN-008.** **Fail clearly:** Invalid or ambiguous input MUST produce an explicit error rather than an assumed interpretation. ## 4. Terminology @@ -119,52 +130,63 @@ The following principles govern all requirements: ### 5.1 Required inputs -A comparison run MUST receive: +**REQ-INP-001.** A comparison run MUST receive: + +- **REQ-INP-060.** one reference dataset; +- **REQ-INP-061.** one query dataset; +- **REQ-INP-062.** an output destination. -- one reference dataset; -- one query dataset; -- an output destination. +**REQ-INP-002.** A custom comparison configuration MAY be supplied. -A custom comparison configuration MAY be supplied. If it is omitted, the bundled strict-exact configuration defined in Section 6.9 MUST be used. +**REQ-INP-003.** If a custom comparison configuration is omitted, the bundled strict-exact configuration defined in Section 6.9 MUST be used. ### 5.2 Supported formats -The initial implementation MUST support: +**REQ-INP-004.** The initial implementation MUST support: -- CSV conforming to the configured dialect; -- TSV conforming to the configured dialect; -- JSON using a documented record-oriented structure. +- **REQ-INP-063.** CSV using the fixed version 1 parsing rules in Section 5.4; +- **REQ-INP-064.** TSV using the fixed version 1 parsing rules in Section 5.4; +- **REQ-INP-065.** JSON using a documented record-oriented structure. -The reference and query formats MAY be selected independently through explicit command-line options. If a format is not supplied, it MAY be detected only from a recognised, unambiguous extension: `.csv`, `.tsv` or `.json`. In either case: +For each input: -- detected format MUST be recorded; -- ambiguous or unsupported content MUST fail execution; -- an unknown or ambiguous extension, including `.txt`, MUST require an explicit format; -- content MUST be validated against the selected format. +- **REQ-INP-005.** its format MUST be determined from its case-insensitive filename extension: `.csv`, `.tsv` or `.json`; +- **REQ-INP-006.** the detected format MUST be recorded; +- **REQ-INP-007.** a missing, unknown, or ambiguous extension, including `.txt`, MUST cause an input error; +- **REQ-INP-008.** its content MUST be validated against the format indicated by the extension; +- **REQ-INP-009.** content that does not conform to the indicated format MUST cause an input error. ### 5.3 Character encoding and text handling -- UTF-8 MUST be the default and documented encoding. -- Alternative encodings MAY be configured explicitly. -- Invalid byte sequences MUST cause an input error unless an explicit replacement policy is configured and recorded. -- Field names and string values MUST be preserved exactly before declared normalisation. -- Line endings MUST NOT affect comparison results. +- **REQ-INP-010.** UTF-8 MUST be the only supported input encoding in version 1. +- **REQ-INP-011.** Invalid UTF-8 byte sequences MUST cause an input error. +- **REQ-INP-012.** Invalid UTF-8 byte sequences MUST NOT be replaced silently. +- **REQ-INP-013.** Field names and string values MUST be preserved exactly before declared normalisation. +- **REQ-INP-014.** Line endings MUST NOT affect comparison results. ### 5.4 Tabular parsing For CSV and TSV: -- a header row MUST be required; -- duplicate or empty column names MUST cause an input error; -- delimiters, quoting, escaping, and embedded newlines MUST be handled by a standards-compliant parser; -- rows with too many or too few columns MUST be input errors; -- all physical input rows MUST be accounted for; -- source row numbers SHOULD be retained for traceability; -- input values MUST initially remain strings unless a custom configuration declares a null representation or type conversion; in the bundled strict-exact configuration, an empty cell is an empty string. +- **REQ-INP-015.** CSV MUST use comma as its delimiter. +- **REQ-INP-016.** TSV MUST use tab as its delimiter. +- **REQ-INP-017.** CSV and TSV MUST use double quote as the quote character. +- **REQ-INP-018.** A double quote within a quoted field MUST be escaped by doubling it. +- **REQ-INP-019.** CRLF and LF record endings MUST be accepted. +- **REQ-INP-020.** Embedded delimiters and line breaks MUST be accepted only within quoted fields. +- No separate backslash or alternative escape mechanism is supported in version 1. +- **REQ-INP-021.** a header row MUST be required; +- **REQ-INP-022.** duplicate or empty column names MUST cause an input error; +- **REQ-INP-023.** delimiters, quoting, escaping, and embedded newlines MUST be handled by a standards-compliant parser; +- **REQ-INP-024.** rows with too many or too few columns MUST be input errors; +- **REQ-INP-025.** all physical input rows MUST be accounted for; +- **REQ-INP-026.** source row numbers SHOULD be retained for traceability; +- **REQ-INP-027.** Input values MUST initially remain strings unless a custom configuration declares a null representation or type conversion. +- **REQ-INP-028.** In the bundled strict-exact configuration, an empty cell MUST be an empty string. ### 5.5 JSON parsing -- JSON inputs MUST use a schema-validated list of record objects: +- **REQ-INP-029.** JSON inputs MUST use a schema-validated list of record objects: ```json [ @@ -173,66 +195,86 @@ For CSV and TSV: ] ``` -- Values such as `yes`, `no`, `on`, `off`, uppercase boolean-like values, and timestamp-like strings MUST remain strings unless a custom configuration explicitly maps or parses them. -- JSON decimal numbers - i.e. numbers containing a fractional or exponent component - MUST be parsed using exact decimal arithmetic, without first converting them to binary floating-point values. The tool MUST also retain the original text of each number — for example, distinguishing the representations `0.100`, `0.1`, and `1e-1` — for inclusion in evidence and diagnostic output when required. -- A JSON input MUST contain either one object representing a single record or an array of objects representing multiple records. A root-level scalar, or a root-level array containing any item that is not an object, MUST be an input error. The error MUST identify the invalid root value or array item and explain the required structure. +- **REQ-INP-030.** Values such as `yes`, `no`, `on`, `off`, uppercase boolean-like values, and timestamp-like strings MUST remain strings unless a custom configuration explicitly maps or parses them. +- **REQ-INP-031.** JSON decimal numbers - i.e. numbers containing a fractional or exponent component - MUST be parsed using exact decimal arithmetic. +- **REQ-INP-032.** JSON decimal numbers MUST NOT first be converted to binary floating-point values. +- **REQ-INP-033.** The tool MUST retain the original text of each JSON number — for example, distinguishing the representations `0.100`, `0.1`, and `1e-1` — for inclusion in evidence when required. +- **REQ-INP-034.** A JSON input MUST contain either one object representing a single record or an array of objects representing multiple records. +- **REQ-INP-035.** A root-level scalar MUST cause an input error. +- **REQ-INP-036.** A root-level array containing any item that is not an object MUST cause an input error. +- **REQ-INP-037.** An invalid-root error MUST identify the invalid root value or array item. +- **REQ-INP-038.** An invalid-root error MUST explain the required structure. ### 5.6 Parser-native value model -- When an input file is read, and before any configured type conversion, mapping, or normalisation is applied, each input value MUST be represented internally using one of the following parser-native value categories: `string`, `integer`, `decimal`, `boolean`, `null`, `array`, and `object`. -- CSV and TSV cells MUST be strings, including empty strings, unless a custom configuration declares further processing. -- JSON numbers without a fractional or exponent component MUST be integers, while other finite JSON numbers MUST be exact decimals. -- JSON arrays used as field values MUST be represented internally as `array` values, and JSON objects used as field values MUST be represented internally as `object` values. +- **REQ-INP-039.** When an input file is read, and before any configured type conversion, mapping, or normalisation is applied, each input value MUST be represented internally using one of the following parser-native value categories: `string`, `integer`, `decimal`, `boolean`, `null`, `array`, and `object`. +- **REQ-INP-040.** CSV and TSV cells MUST be strings, including empty strings, unless a custom configuration declares further processing. +- **REQ-INP-041.** JSON numbers without a fractional or exponent component MUST be integers. +- **REQ-INP-042.** Other finite JSON numbers MUST be exact decimals. +- **REQ-INP-043.** JSON arrays used as field values MUST be represented internally as `array` values. +- **REQ-INP-044.** JSON objects used as field values MUST be represented internally as `object` values. - Parser-native values are equal only when they have the same category and recursively equal content. For array-valued fields, element order is significant; for object-valued fields, member names and values must match but member order is not significant. Top-level records are aligned by their declared record keys, and their input order does not affect comparison. -- Numerical values MUST be retained exactly as they appeared in the input for reporting and traceability. However, two values in the same numerical category compare as equal when they represent the same number; for example, decimal `1.0` and decimal `1.00` are equal. -- `NaN`, positive infinity, and negative infinity are not valid numerical values. In an attributable non-key field, they MUST produce a contract finding and make that value `not_assessable`; other assessable content MUST still be compared. If such a value occurs in a record key or prevents construction of an unambiguous record model, it MUST produce an execution error. +- **REQ-INP-045.** Numerical values MUST be retained exactly as they appeared in the input for reporting and traceability. +- **REQ-INP-046.** Two values in the same numerical category MUST compare as equal when they represent the same number; for example, decimal `1.0` and decimal `1.00` are equal. +- `NaN`, positive infinity, and negative infinity are not valid numerical values. +- **REQ-INP-047.** A non-finite value in an attributable non-key field MUST produce a contract finding. +- **REQ-INP-048.** A non-finite value in an attributable non-key field MUST make that value `not_assessable`. +- **REQ-INP-049.** Other assessable content MUST still be compared when a non-finite value occurs in an attributable non-key field. +- **REQ-INP-050.** A non-finite value in a record key MUST produce an execution error. +- **REQ-INP-051.** A non-finite value that prevents construction of an unambiguous record model MUST produce an execution error. ### 5.7 Empty inputs -- Empty files and datasets with zero records MUST be recognised explicitly. -- If both datasets are syntactically valid and contain no records, the records phase MUST report no missing or extra records, and the values phase MUST perform no comparisons. Contract requirements that can be evaluated without records MUST still be checked. The tool MUST NOT infer that an empty dataset is unsuitable merely because it is empty; any minimum-size acceptance criterion belongs to the external laboratory decision process. -- A non-empty reference and empty query MUST report every expected record as missing. -- An empty reference and non-empty query MUST report every observed record as extra. +- **REQ-INP-052.** Empty files and datasets with zero records MUST be recognised explicitly. +- **REQ-INP-053.** If both datasets are syntactically valid and contain no records, the records phase MUST report no missing records. +- **REQ-INP-054.** If both datasets are syntactically valid and contain no records, the records phase MUST report no extra records. +- **REQ-INP-055.** If both datasets are syntactically valid and contain no records, the values phase MUST perform no comparisons. +- **REQ-INP-056.** Contract requirements that can be evaluated without records MUST still be checked. +- **REQ-INP-057.** The tool MUST NOT infer that an empty dataset is unsuitable merely because it is empty. +- Any minimum-size acceptance criterion belongs to the external laboratory decision process. +- **REQ-INP-058.** A non-empty reference and empty query MUST report every expected record as missing. +- **REQ-INP-059.** An empty reference and non-empty query MUST report every observed record as extra. ## 6. Comparison configurations ### 6.1 General requirements -Every comparison uses an effective comparison configuration. A custom configuration MUST: +**REQ-CFG-001.** Every comparison uses an effective comparison configuration. A custom configuration MUST: -- use a published, independently versioned comparison-configuration schema supported by the installed toolkit release; -- declare its schema version and a stable configuration identifier and revision; -- be validated fully before either dataset is compared; -- reject unknown configuration keys by default; -- provide actionable validation errors with configuration locations; -- be identified in provenance by a cryptographic digest; -- be retained or embedded in the evidence package, subject to policy. +- **REQ-CFG-095.** use a published, independently versioned comparison-configuration schema supported by the installed toolkit release; +- **REQ-CFG-096.** declare its schema version and a stable configuration identifier and revision; +- **REQ-CFG-097.** be validated fully before either dataset is compared; +- **REQ-CFG-098.** reject unknown configuration keys by default; +- **REQ-CFG-099.** provide actionable validation errors with configuration locations; +- **REQ-CFG-100.** be embedded in the canonical comparison result. -A valid comparison configuration MUST declare at least: +**REQ-CFG-002.** A valid comparison configuration MUST declare at least: -- record key fields; -- field definitions and an explicit policy for undeclared fields; -- comparison rules. +- **REQ-CFG-101.** record key fields; +- **REQ-CFG-102.** field definitions and an explicit policy for undeclared fields; +- **REQ-CFG-103.** comparison rules. ### 6.2 Custom comparison-configuration YAML format -Custom comparison configurations MUST be YAML documents conforming to the format defined in this section. YAML is used only for the comparison configuration; it is not a supported dataset format. A configuration file SHOULD use the `.yaml` or `.yml` extension. +**REQ-CFG-003.** Custom comparison configurations MUST be YAML documents conforming to the format defined in this section. YAML is used only for the comparison configuration; it is not a supported dataset format. + +**REQ-CFG-004.** A configuration file SHOULD use the `.yaml` or `.yml` extension. #### 6.2.1 YAML parsing and validation -- The document MUST be UTF-8 encoded and contain exactly one YAML mapping at its root. -- The configuration MUST be parsed according to the YAML 1.2 rules for interpreting unquoted scalar values. -- The parser MUST use a safe loading mode that cannot construct arbitrary application objects or execute code. -- Duplicate mapping keys, merge keys, aliases, anchors, explicit tags, and executable or implementation-specific object constructors MUST be rejected as configuration errors. -- Unknown keys MUST be rejected as configuration errors. -- Required keys MUST NOT be supplied as YAML null values. -- `schema_version`, `id`, `revision`, field names, operation names, comparator names, enumerated values, decimal lexical values, dates, and datetimes SHOULD be quoted to prevent unintended YAML scalar conversion. -- The complete document MUST be validated against the published schema identified by `schema_version` before either dataset is read. -- Validation errors MUST identify the YAML path and the violated requirement. +- **REQ-CFG-005.** The document MUST be UTF-8 encoded and contain exactly one YAML mapping at its root. +- **REQ-CFG-006.** The configuration MUST be parsed according to the YAML 1.2 rules for interpreting unquoted scalar values. +- **REQ-CFG-007.** The parser MUST use a safe loading mode that cannot construct arbitrary application objects or execute code. +- **REQ-CFG-008.** Duplicate mapping keys, merge keys, aliases, anchors, explicit tags, and executable or implementation-specific object constructors MUST be rejected as configuration errors. +- **REQ-CFG-009.** Unknown keys MUST be rejected as configuration errors. +- **REQ-CFG-010.** Required keys MUST NOT be supplied as YAML null values. +- **REQ-CFG-011.** `schema_version`, `id`, `revision`, field names, operation names, comparator names, enumerated values, decimal lexical values, dates, and datetimes SHOULD be quoted to prevent unintended YAML scalar conversion. +- **REQ-CFG-012.** The complete document MUST be validated against the published schema identified by `schema_version` before either dataset is read. +- **REQ-CFG-013.** Validation errors MUST identify the YAML path and the violated requirement. #### 6.2.2 Complete structural template -The following template defines the complete first-release structure. Keys marked “optional” in comments MAY be omitted; other displayed keys are required when their parent construct is used, except where a conditional omission is explicitly permitted below. Optional processing properties are omitted rather than represented by empty lists. +**REQ-CFG-014.** The following template defines the complete first-release structure. Keys marked “optional” in comments MAY be omitted; other displayed keys are required when their parent construct is used, except where a conditional omission is explicitly permitted below. Optional processing properties are omitted rather than represented by empty lists. ```yaml schema_version: "1.0" @@ -281,7 +323,6 @@ fields: name: "numeric_tolerance" parameters: absolute_tolerance: "0.01" - relative_tolerance: "0.05" review_priority: "high" # optional; default: normal field_example_2: source: @@ -344,7 +385,9 @@ fields: ``` -The template is illustrative rather than an instruction to use these particular rules. In particular, laboratories MUST choose identifiers, field definitions, and policies appropriate to the intended comparison. The published machine-readable schema MUST encode the same structure and constraints as this section; where the schema and this document disagree, the normative behaviour specification MUST be corrected before release rather than silently choosing one interpretation. +**REQ-CFG-015.** The template is illustrative rather than an instruction to use these particular rules. Laboratories MUST choose identifiers, field definitions, and policies appropriate to the intended comparison. + +**REQ-CFG-016.** The published machine-readable schema MUST encode the complete version 1 comparison-configuration structure and constraints. The three field definitions illustrate different configuration roles: `record_id` is a renamed exact key; `field_example_1` is parsed as an exact decimal and quantised before tolerance comparison; and `field_example_2` demonstrates source-text normalisation, side-specific missing tokens, and mapping different source vocabularies to common canonical string values. The deliberately generic names make clear that these are user-defined canonical field names rather than reserved configuration keywords. These example rules are not domain recommendations. @@ -352,39 +395,57 @@ The three field definitions illustrate different configuration roles: `record_id The only permitted top-level keys are `schema_version`, `metadata`, and `fields`. -- `schema_version` MUST be a quoted string identifying a supported comparison-configuration schema version. -- `metadata.id` MUST be a stable, non-empty identifier for the comparison configuration. It SHOULD use a collision-resistant organisational namespace. -- `metadata.revision` MUST be a non-empty revision string controlled by the configuration owner. Any behaviourally material change elsewhere in the configuration MUST produce a new revision. +- **REQ-CFG-017.** `schema_version` MUST be a quoted string identifying a supported comparison-configuration schema version. +- **REQ-CFG-018.** `metadata.id` MUST be a stable, non-empty identifier for the comparison configuration. +- **REQ-CFG-019.** `metadata.id` SHOULD use a collision-resistant organisational namespace. +- **REQ-CFG-020.** `metadata.revision` MUST be a non-empty revision string controlled by the configuration owner. +- **REQ-CFG-021.** Any behaviourally material change elsewhere in the configuration MUST produce a new revision. - `metadata.title` and `metadata.description` are optional human-readable strings. -- The `metadata` properties MUST NOT directly alter comparison behaviour. Any change to them nevertheless changes the configuration document and therefore its provenance digest. +- **REQ-CFG-022.** The `metadata` properties MUST NOT directly alter comparison behaviour. +- **REQ-CFG-023.** The `metadata` properties MUST be preserved in the configuration embedded in the canonical result. #### 6.2.4 Record-key selection -- `fields.record_key` MUST be a non-empty ordered list of canonical field names present in `fields.definitions`. The list defines one logical record key, and list order defines the component order of a composite key. -- Every key field MUST be required on both sides, non-nullable, and declared with a scalar logical type. A key field MAY use explicitly declared source normalisation or value mapping, but MUST use `typed_exact` after transformation. +- **REQ-CFG-024.** `fields.record_key` MUST be a non-empty ordered list of canonical field names present in `fields.definitions`. The list defines one logical record key, and list order defines the component order of a composite key. +- **REQ-CFG-025.** Every key field MUST be required on both sides. +- **REQ-CFG-026.** Every key field MUST be non-nullable. +- **REQ-CFG-027.** Every key field MUST be declared with a scalar logical type. +- **REQ-CFG-028.** A key field MAY use explicitly declared source normalisation or value mapping. +- **REQ-CFG-029.** Every key field MUST use `typed_exact` after transformation. #### 6.2.5 Field selection, declaration, and source names - Every field in `fields.definitions` is in scope. Field definitions are the sole mechanism for selecting explicitly configured fields in version 1. -- `fields.undeclared` is required and MUST be one of `compare`, `exclude`, or `fail`. +- **REQ-CFG-030.** `fields.undeclared` is required and MUST be one of `compare`, `exclude`, or `fail`. - `compare` compares shared undeclared fields with `preserved_exact`; an undeclared field present on only one side creates a contract finding because no corresponding value can be compared. - `exclude` does not compare undeclared fields and records them in the exclusion summary as `not_assessed`. - `fail` creates a contract finding for every undeclared field. -- Undeclared reference and query fields MUST be aligned only when their input field names are identical. No name inference or source-name mapping is applied to undeclared fields. Fields with different names MUST be declared explicitly under a common canonical field definition if they are intended to correspond. -- `fields.definitions` MUST be a mapping keyed by unique canonical field name and MUST contain a definition for every field referenced by `fields.record_key`. -- `source.reference` and `source.query` MUST each contain a non-empty source field name. A declared field always defines correspondence between one reference field and one query field, regardless of whether either field is required to be present in every record. -- `required.reference` and `required.query` MUST be booleans. Absence of a field required on a particular side MUST create a contract finding. Absence of an optional field MUST NOT create a missing-field finding. -- A field observed in an input but absent from `fields.definitions` is an undeclared field and MUST be handled only according to `fields.undeclared`. -- `review_priority`, when present, MUST be `low`, `normal`, or `high`. When omitted, it defaults to `normal` in the effective configuration. -- A declared field's resolved `review_priority` MUST be attached to every finding attributable to that canonical field, including missing-field, null-policy, type, parsing, mapping, and value-mismatch findings. It MUST NOT be inherited by findings that cannot be attributed to that field, such as missing or extra records, truncation, invalid-configuration findings, or internal failures. -- Review priority is presentation and review-workflow metadata only. It MUST NOT change comparison behaviour, determine whether a finding is emitted, alter counts, affect run or phase status, affect exit codes, or constitute a laboratory acceptance decision. -- Human-readable reports MAY use review priority for grouping, ordering, filtering, icons, or colour coding. Colour MUST NOT be the only means of communicating priority. Reports and documentation MUST NOT describe review priority as clinical significance or as an execution-error classification. +- **REQ-CFG-031.** Undeclared reference and query fields MUST be aligned only when their input field names are identical. +- No name inference or source-name mapping is applied to undeclared fields. +- **REQ-CFG-032.** Fields with different names MUST be declared explicitly under a common canonical field definition if they are intended to correspond. +- **REQ-CFG-033.** `fields.definitions` MUST be a mapping keyed by unique canonical field name. +- **REQ-CFG-034.** `fields.definitions` MUST contain a definition for every field referenced by `fields.record_key`. +- **REQ-CFG-035.** `source.reference` and `source.query` MUST each contain a non-empty source field name. A declared field always defines correspondence between one reference field and one query field, regardless of whether either field is required to be present in every record. +- **REQ-CFG-036.** `required.reference` and `required.query` MUST be booleans. +- **REQ-CFG-037.** Absence of a field required on a particular side MUST create a contract finding. +- **REQ-CFG-038.** Absence of an optional field MUST NOT create a missing-field finding. +- **REQ-CFG-039.** A field observed in an input but absent from `fields.definitions` is an undeclared field and MUST be handled only according to `fields.undeclared`. +- **REQ-CFG-040.** `review_priority`, when present, MUST be `low`, `normal`, or `high`. When omitted, it defaults to `normal` in the effective configuration. +- **REQ-CFG-041.** A declared field's resolved `review_priority` MUST be attached to every finding attributable to that canonical field, including missing-field, null-policy, type, parsing, mapping, and value-mismatch findings. +- **REQ-CFG-042.** Review priority MUST NOT be inherited by findings that cannot be attributed to a declared field, such as missing or extra records. +- **REQ-CFG-043.** Review priority is presentation and review-workflow metadata only. It MUST NOT change comparison behaviour, determine whether a finding is emitted, alter counts, affect run or phase status, affect exit codes, or constitute a laboratory acceptance decision. +- **REQ-CFG-044.** Human-readable reports MAY use review priority for grouping, ordering, filtering, icons, or colour coding. +- **REQ-CFG-045.** Colour MUST NOT be the only means of communicating review priority. +- **REQ-CFG-046.** Reports and documentation MUST NOT describe review priority as clinical significance. +- **REQ-CFG-047.** Reports and documentation MUST NOT describe review priority as an execution-error classification. #### 6.2.6 Logical types and null policy -For a scalar field, `logical_type` MUST be one of `string`, `integer`, `decimal`, `boolean`, `date`, or `datetime`. It MAY be omitted only when the comparator is `preserved_exact` and no typed parsing, mapping, or typed normalisation is configured. +**REQ-CFG-048.** For a scalar field, `logical_type` MUST be one of `string`, `integer`, `decimal`, `boolean`, `date`, or `datetime`. -For a collection field, `logical_type` MUST use this form: +**REQ-CFG-049.** `logical_type` MAY be omitted only when the comparator is `preserved_exact` and no typed parsing, mapping, or typed normalisation is configured. + +**REQ-CFG-050.** For a collection field, `logical_type` MUST use this form: ```yaml logical_type: @@ -394,14 +455,16 @@ logical_type: name: "typed_exact" ``` -Nested collection types MUST be rejected. The comparator named for the field itself MUST correspond to the collection kind: `list_exact`, `set_exact`, or `multiset_exact`. +**REQ-CFG-051.** Nested collection types MUST be rejected. + +**REQ-CFG-052.** The comparator named for the field itself MUST correspond to the collection kind: `list_exact`, `set_exact`, or `multiset_exact`. -Every declared field MUST contain a `nulls` mapping: +**REQ-CFG-053.** Every declared field MUST contain a `nulls` mapping: -- `allowed` and `nulls_equal` MUST be booleans. -- `nulls_equal` MUST be `false` when `allowed` is `false`. -- `missing_tokens` MAY be omitted; omission means that no additional source values are interpreted as null. Parser-native JSON null remains null independently of this property. -- When `missing_tokens` is present, it MUST contain `reference`, `query`, or both. Each present side MUST be a non-empty list of source-value descriptors. An omitted side has no configured textual missing tokens. A matched token is converted to null at processing step 3 in Section 6.8. +- **REQ-CFG-054.** `allowed` and `nulls_equal` MUST be booleans. +- **REQ-CFG-055.** `nulls_equal` MUST be `false` when `allowed` is `false`. +- **REQ-CFG-056.** `missing_tokens` MAY be omitted; omission means that no additional source values are interpreted as null. Parser-native JSON null remains null independently of this property. +- **REQ-CFG-057.** When `missing_tokens` is present, it MUST contain `reference`, `query`, or both. Each present side MUST be a non-empty list of source-value descriptors. An omitted side has no configured textual missing tokens. A matched token is converted to null at processing step 3 in Section 6.8. A source-value descriptor has one of these forms: @@ -419,13 +482,21 @@ value: "1.00" category: "null" ``` -`category` MUST be `string`, `integer`, `decimal`, `boolean`, or `null`. `value` MUST be present for every category except `null`; it MUST be a quoted lexical string for `string`, `integer`, and `decimal`, and `true` or `false` for `boolean`. The lexical value MUST be valid for its stated category. Descriptors are matched using parser-native category and value equality under Section 5.6; the retained lexical form remains evidence but does not create a distinct numerical value. This explicit representation prevents YAML scalar resolution from changing mapping keys or missing tokens. +**REQ-CFG-058.** `category` MUST be `string`, `integer`, `decimal`, `boolean`, or `null`. + +**REQ-CFG-059.** `value` MUST be present for every category except `null`. It MUST be a quoted lexical string for `string`, `integer`, and `decimal`. It MUST be `true` or `false` for `boolean`. The lexical value MUST be valid for its stated category. + +Descriptors are matched using parser-native category and value equality under Section 5.6. The retained lexical form remains evidence but does not create a distinct numerical value. This explicit representation prevents YAML scalar resolution from changing mapping keys or missing tokens. #### 6.2.7 Normalisation and value mapping -`source_normalisation` MAY be omitted; omission means that no source-text normalisation is applied. When present, it MUST contain `reference`, `query`, or both. Each present side MUST be a non-empty ordered list of operations applied only to values from that dataset; an omitted side has no source-text operations. +**REQ-CFG-060.** `source_normalisation` MAY be omitted. Omission means that no source-text normalisation is applied. + +**REQ-CFG-061.** When present, `source_normalisation` MUST contain `reference`, `query`, or both. Each present side MUST be a non-empty ordered list of operations applied only to values from that dataset. An omitted side has no source-text operations. -`typed_normalisation` MAY be omitted; omission means that no typed-value normalisation is applied. When present, it MUST be a non-empty ordered list applied to the canonical typed values on both sides. Each operation entry has this form: +**REQ-CFG-062.** `typed_normalisation` MAY be omitted. Omission means that no typed-value normalisation is applied. + +**REQ-CFG-063.** When present, `typed_normalisation` MUST be a non-empty ordered list applied to the canonical typed values on both sides. Each operation entry has this form: ```yaml - operation: "trim" @@ -433,7 +504,11 @@ category: "null" side: "both" ``` -`parameters` MAY be omitted only when the registered operation has no parameters. `operation` MUST name an operation defined by the applicable schema version, and its parameters MUST validate against that operation’s published parameter schema. An unknown operation or parameter MUST invalidate the comparison configuration. List order is execution order. The permitted processing stages and their overall order are fixed by Section 6.8. +**REQ-CFG-064.** `parameters` MAY be omitted only when the registered operation has no parameters. + +**REQ-CFG-065.** `operation` MUST name an operation defined by the applicable schema version. Its parameters MUST validate against that operation’s published parameter schema. An unknown operation MUST invalidate the comparison configuration. An unknown parameter MUST invalidate the comparison configuration. + +List order is execution order. The permitted processing stages and their overall order are fixed by Section 6.8. The first-release source-text operation identifiers are: @@ -446,9 +521,15 @@ The first-release typed-value operation identifiers are: - `decimal_quantise`, requiring a non-negative integer `places` and `rounding` with one of `half_even`, `half_up`, or `down`; and - `datetime_truncate`, requiring `precision` with one of `second`, `millisecond`, or `microsecond`. -A source-text operation MUST appear only in `source_normalisation.reference` or `source_normalisation.query` and applies only to that dataset. A typed-value operation MUST appear only in `typed_normalisation` and applies identically to both datasets. Operation semantics, including the Unicode-data version used by Unicode operations, MUST be published with the schema and recorded sufficiently for reproducibility. +**REQ-CFG-066.** A source-text operation MUST appear only in `source_normalisation.reference` or `source_normalisation.query`. It applies only to the corresponding dataset. + +**REQ-CFG-067.** A typed-value operation MUST appear only in `typed_normalisation`. It applies identically to both datasets. -`value_mapping` MAY be omitted; omission means that no value mapping is applied. When present, it MUST contain `reference`, `query`, or both. Each present side MUST be a non-empty ordered list of entries with this form; an omitted side has no value mapping: +**REQ-CFG-068.** Operation semantics MUST be published with the schema and recorded sufficiently for reproducibility. This information MUST include the Unicode-data version used by Unicode operations. + +**REQ-CFG-069.** `value_mapping` MAY be omitted. Omission means that no value mapping is applied. + +**REQ-CFG-070.** When present, `value_mapping` MUST contain `reference`, `query`, or both. Each present side MUST be a non-empty ordered list of entries with the following form. An omitted side has no value mapping: ```yaml - from: @@ -459,163 +540,210 @@ A source-text operation MUST appear only in `source_normalisation.reference` or value: "R" ``` -- `from` and `to` MUST be source-value descriptors as defined in Section 6.2.6. +- **REQ-CFG-071.** `from` and `to` MUST be source-value descriptors as defined in Section 6.2.6. - A `from` descriptor is matched against the source value after the applicable `source_normalisation` operations and missing-token recognition have been performed, in accordance with the processing order in Section 6.8. - A `to` descriptor with `category: "null"` explicitly maps to null and is distinct from absence of a mapping entry. -- Each `from` value MUST occur at most once within one side’s mapping. Conflicting or unreachable entries MUST invalidate the comparison configuration. +- **REQ-CFG-072.** Each `from` value MUST occur at most once within one side’s mapping. +- **REQ-CFG-073.** Conflicting entries MUST invalidate the comparison configuration. +- **REQ-CFG-074.** Unreachable entries MUST invalidate the comparison configuration. - A value mapping requires a scalar `logical_type`; value mapping of collection fields is not supported in the first release. Its `to` values are parsed as the destination type after mapping. - When a side has a value mapping, a source value with no mapping entry creates a contract finding. Pass-through of unmapped values is not supported in the first release. -Before comparison, the tool MUST expand omitted optional processing properties and omitted dataset sides into explicit empty internal structures in the effective configuration. The user-authored configuration and the expanded effective configuration MUST remain distinguishable in provenance. Omission changes no behaviour beyond declaring that the corresponding processing step is absent. +**REQ-CFG-075.** Before comparison, the tool MAY expand omitted optional processing properties and omitted dataset sides into explicit empty internal structures for internal use. + +**REQ-CFG-076.** Such expansion is implementation state. It MUST NOT replace the validated configuration document selected for the run and embedded in the canonical result. It MUST NOT alter that document. Omission changes no behaviour beyond declaring that the corresponding processing step is absent. #### 6.2.8 Comparator declarations -Every field definition MUST contain `comparator.name`. `name` MUST be one of the comparator identifiers in Section 7.4. A `parameters` mapping MUST be present only when the selected comparator requires parameters and MUST contain only parameters valid for that comparator. +**REQ-CFG-077.** Every field definition MUST contain `comparator.name`. + +**REQ-CFG-078.** `name` MUST be one of the comparator identifiers in Section 7.4. + +**REQ-CFG-079.** A `parameters` mapping MUST be present only when the selected comparator requires parameters. It MUST contain only parameters valid for that comparator. -- `preserved_exact`, `typed_exact`, `normalised_string`, `numeric_exact`, `date_exact`, `datetime_exact`, `list_exact`, `set_exact`, and `multiset_exact` take no field-level parameters in the first release; `parameters` MUST be omitted for these comparators. -- `numeric_tolerance` MUST contain a `parameters` mapping that permits only `absolute_tolerance` and `relative_tolerance`. At least one tolerance MUST be present. Each MUST be a quoted, non-negative finite decimal string; omission of one means it is treated as zero as specified in Section 7.4.5. -- The comparator MUST be compatible with `logical_type`: string comparators with `string`, numeric comparators with `integer` or `decimal`, date comparators with their corresponding temporal type, and collection comparators with their corresponding collection type. +- **REQ-CFG-080.** `preserved_exact`, `typed_exact`, `normalised_string`, `numeric_exact`, `date_exact`, `datetime_exact`, `list_exact`, `set_exact`, and `multiset_exact` take no field-level parameters in the first release; `parameters` MUST be omitted for these comparators. +- **REQ-CFG-081.** `numeric_tolerance` MUST contain a `parameters` mapping with exactly one of `absolute_tolerance` or `relative_tolerance`. +- **REQ-CFG-082.** Supplying neither tolerance MUST invalidate the comparison configuration. +- **REQ-CFG-083.** Supplying both tolerances MUST invalidate the comparison configuration. +- **REQ-CFG-084.** The supplied tolerance MUST be a quoted, non-negative finite decimal string. +- **REQ-CFG-085.** The comparator MUST be compatible with `logical_type`: string comparators with `string`, numeric comparators with `integer` or `decimal`, date comparators with their corresponding temporal type, and collection comparators with their corresponding collection type. - `normalised_string` requires at least one operation in `source_normalisation.reference`, `source_normalisation.query`, or `typed_normalisation` that is applicable to strings. #### 6.2.9 Normative machine-readable schema and semantic validation -The normative JSON Schema Draft 2020-12 definition of comparison-configuration schema version `1.0` is [`src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json`](src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json). A YAML configuration MUST first be loaded using Section 6.2.1. The resulting JSON-compatible data structure MUST then validate against that schema before either input dataset is read. +The normative JSON Schema Draft 2020-12 definition of comparison-configuration schema version `1.0` is [`src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json`](src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json). -The standalone schema file is the authoritative machine-readable definition. It MUST be version controlled in the source repository, included as package data in the installed `adviso_validation.schemas` package, and loaded at runtime through the package-resource API rather than through a path relative to the working directory. CI MUST verify its presence in built distributions and validate every documented example and controlled configuration fixture against it. Any inconsistency between the human-readable requirements in this document and the machine-readable schema is a specification defect that MUST be resolved before release. +**REQ-CFG-086.** A YAML configuration MUST first be loaded using Section 6.2.1. The resulting JSON-compatible data structure MUST then validate against that schema before either input dataset is read. -JSON Schema cannot reliably express every relationship involving user-defined mapping keys, uniqueness across separate arrays, or references between distant parts of a document. After schema validation, the implementation MUST therefore apply the following semantic checks before reading either dataset: +The standalone schema file is the authoritative machine-readable definition. -- every name in `fields.record_key` exists in `fields.definitions`; -- every key field is required on both sides, non-nullable, scalar, and uses `typed_exact`; -- a field without `logical_type` uses `preserved_exact` and has no typed normalisation or value mapping; -- each comparator and normalisation operation is compatible with the field’s logical type and processing stage; -- source-value descriptors contain lexically valid values for their declared categories, and each side’s value mapping has unique `from` values; +**REQ-CFG-087.** It MUST be version controlled in the source repository. It MUST be included as package data in the installed `adviso_validation.schemas` package. It MUST be loaded at runtime through the package-resource API rather than through a path relative to the working directory. -A failure of any semantic check MUST be reported as an invalid comparison configuration with the relevant configuration path and violated rule. +**REQ-CFG-088.** CI MUST verify the schema's presence in built distributions. CI MUST validate every documented example against it. CI MUST validate every controlled configuration fixture against it. + +**REQ-CFG-090.** JSON Schema cannot reliably express every relationship involving user-defined mapping keys, uniqueness across separate arrays, or references between distant parts of a document. After schema validation, the implementation MUST therefore apply the following semantic checks before reading either dataset: + +- **REQ-CFG-104.** every name in `fields.record_key` exists in `fields.definitions`; +- **REQ-CFG-105.** every key field is required on both sides, non-nullable, scalar, and uses `typed_exact`; +- **REQ-CFG-106.** a field without `logical_type` uses `preserved_exact` and has no typed normalisation or value mapping; +- **REQ-CFG-107.** each comparator and normalisation operation is compatible with the field’s logical type and processing stage; +- **REQ-CFG-108.** source-value descriptors contain lexically valid values for their declared categories, and each side’s value mapping has unique `from` values; + +**REQ-CFG-091.** A failure of any semantic check MUST be reported as an invalid comparison configuration with the relevant configuration path and violated rule. ### 6.3 Record keys -- A record key MUST contain one or more named fields. -- Composite-key field order MUST be stable and recorded. -- Key fields MUST be present, valid under their declared field definitions, and non-null. -- Key values MUST NOT undergo implicit numeric or boolean conversion. -- Normalisation of key values MUST be explicitly configured. -- Alternative key names and composite keys MUST be declared in a custom configuration. -- Key names MUST NOT be inferred, and command-line key overrides MUST NOT be supported in the first release. -- The normalised key MUST uniquely identify one record within its dataset. -- A missing, null, invalid, or duplicate record key prevents construction of an unambiguous record model and MUST be an input error in every comparison. The resulting execution error MUST identify the affected dataset and the key or key field involved and, where available, the relevant source-record locations. -- Duplicate records MUST never overwrite one another silently. -- Findings and reports MUST retain enough information to distinguish every source record. +- **REQ-REC-001.** A record key MUST contain one or more named fields. +- **REQ-REC-002.** Composite-key field order MUST be stable and recorded. +- **REQ-REC-003.** Key fields MUST be present, valid under their declared field definitions, and non-null. +- **REQ-REC-004.** Key values MUST NOT undergo implicit numeric or boolean conversion. +- **REQ-REC-005.** Normalisation of key values MUST be explicitly configured. +- **REQ-REC-006.** Alternative key names and composite keys MUST be declared in a custom configuration. +- **REQ-REC-007.** Key names MUST NOT be inferred. +- **REQ-REC-008.** Command-line key overrides MUST NOT be supported in the first release. +- **REQ-REC-009.** The normalised key MUST uniquely identify one record within its dataset. +- **REQ-REC-010.** A missing, null, invalid, or duplicate record key prevents construction of an unambiguous record model and MUST be an input error in every comparison. +- **REQ-REC-011.** The resulting execution error MUST identify the affected dataset. +- **REQ-REC-012.** The resulting execution error MUST identify the key or key field involved. +- **REQ-REC-013.** Where available, the resulting execution error MUST identify the relevant source-record locations. +- **REQ-REC-014.** Duplicate records MUST never overwrite one another silently. +- **REQ-REC-015.** Findings and reports MUST retain enough information to distinguish every source record. ### 6.4 Field declarations and preserved values -The comparison engine MUST support both preserved, untyped values and explicitly declared logical types. A type declaration is required for a field that uses typed parsing, normalisation, mapping, or a type-specific comparator. Every custom configuration MUST explicitly declare how undeclared fields are treated: `compare`, `exclude`, or `fail`. There MUST be no implicit policy. An `exclude` policy MUST be represented as an assessment exclusion in accordance with Section 7.5. +**REQ-VAL-001.** The comparison engine MUST support preserved, untyped values. It MUST also support explicitly declared logical types. + +A type declaration is required for a field that uses typed parsing, normalisation, mapping, or a type-specific comparator. -Every declared field MUST define its canonical name, reference and query source names, required status on each side, null policy, and comparator. It MUST declare a logical type except where Section 6.2.6 permits omission for `preserved_exact`. Normalisation, value mapping, and review priority are optional. An omitted review priority resolves to `normal` in the effective configuration. The authoritative YAML representation and validation rules are defined in Section 6.2. +**REQ-VAL-002.** Every custom configuration MUST explicitly declare how undeclared fields are treated: `compare`, `exclude`, or `fail`. There MUST be no implicit policy. An `exclude` policy MUST be represented as an assessment exclusion in accordance with Section 7.5. + +**REQ-VAL-003.** Every declared field MUST define its canonical name, reference source name, query source name, required status on each side, null policy, and comparator. + +**REQ-VAL-004.** Every declared field MUST declare a logical type except where Section 6.2.6 permits omission for `preserved_exact`. + +Normalisation, value mapping, and review priority are optional. An omitted review priority resolves to `normal` in the effective configuration. The authoritative YAML representation and validation rules are defined in Section 6.2. ### 6.5 Logical types and typed parsing -The comparison engine MUST support custom configurations declaring scalar logical types `string`, `integer`, `decimal`, `boolean`, `date`, and `datetime`, and typed `list`, `set`, and `multiset` collections. Collection fields MUST declare their element type. Null is a distinct value state governed by field policy, not a logical type. +**REQ-VAL-005.** The comparison engine MUST support custom configurations declaring scalar logical types `string`, `integer`, `decimal`, `boolean`, `date`, and `datetime`. + +**REQ-VAL-006.** The comparison engine MUST support typed `list`, `set`, and `multiset` collections. Collection fields MUST declare their element type. Null is a distinct value state governed by field policy, not a logical type. + +**REQ-VAL-007.** Type parsing MUST be field-specific and declared. + +**REQ-VAL-008.** The tool MUST NOT infer a logical type from observed text. + +**REQ-VAL-009.** Values of different logical types MUST NOT be implicitly equal. + +Intended equivalence between differing source representations requires both sides to be parsed as the same declared destination logical type, with an explicit value mapping where the source representation cannot be parsed directly. -Type parsing MUST be field-specific and declared. The tool MUST NOT infer a logical type from observed text. Values of different logical types MUST NOT be implicitly equal. Intended equivalence between differing source representations requires both sides to be parsed as the same declared destination logical type, with an explicit value mapping where the source representation cannot be parsed directly. The original source value or lexical representation MUST remain available to the comparison engine and MAY be included in evidence when required to explain a finding. +**REQ-VAL-010.** The original source value or lexical representation MUST remain available to the comparison engine. It MAY be included in evidence when required to explain a finding. -An invalid typed value in a non-key field that remains attributable to an unambiguous record MUST be a contract finding and MUST make that value `not_assessable`; it MUST NOT be an execution error. Defects that prevent construction of a complete and unambiguous record model are classified under Section 8.3. +**REQ-VAL-011.** An invalid typed value in a non-key field that remains attributable to an unambiguous record MUST be a contract finding. It MUST make that value `not_assessable`. It MUST NOT be an execution error. #### 6.5.1 String -- A string MUST preserve its Unicode code points exactly by default. -- `typed_exact` equality MUST be code-point-for-code-point equality. -- Empty strings, whitespace-only strings, nulls, and absent fields MUST remain distinct. +- **REQ-VAL-012.** A string MUST preserve its Unicode code points exactly by default. +- **REQ-VAL-013.** `typed_exact` equality MUST be code-point-for-code-point equality. +- **REQ-VAL-014.** Empty strings, whitespace-only strings, nulls, and absent fields MUST remain distinct. #### 6.5.2 Integer -- Textual integers MUST use signed base-10 notation consisting of an optional `+` or `-` followed by one or more ASCII digits. -- Decimal points, exponent notation, booleans, and other characters MUST be invalid after source-text normalisation and mapping. -- Leading zeros MUST be accepted and have no numerical significance. Identifiers for which leading zeros are significant MUST be declared as strings. -- Integer representation MUST support arbitrary precision within documented resource limits. +- **REQ-VAL-015.** Textual integers MUST use signed base-10 notation consisting of an optional `+` or `-` followed by one or more ASCII digits. +- **REQ-VAL-016.** Decimal points, exponent notation, booleans, and other characters MUST be invalid after source-text normalisation and mapping. +- **REQ-VAL-017.** Leading zeros MUST be accepted and have no numerical significance. Identifiers for which leading zeros are significant MUST be declared as strings. +- **REQ-VAL-018.** Integer representation MUST support arbitrary precision within documented resource limits. #### 6.5.3 Decimal -- Decimals MUST be parsed and represented using decimal arithmetic without an intermediate binary floating-point conversion. -- Plain and scientific decimal notation MUST be accepted. -- Numerically equivalent finite representations such as `1.0`, `1.00`, and `1e0` MUST have the same canonical value. -- Parsing MUST preserve all supplied numerical precision. -- `NaN`, positive infinity, and negative infinity MUST be invalid. +- **REQ-VAL-019.** Decimals MUST be parsed and represented using decimal arithmetic without an intermediate binary floating-point conversion. +- **REQ-VAL-020.** Plain and scientific decimal notation MUST be accepted. +- **REQ-VAL-021.** Numerically equivalent finite representations such as `1.0`, `1.00`, and `1e0` MUST have the same canonical value. +- **REQ-VAL-022.** Parsing MUST preserve all supplied numerical precision. +- **REQ-VAL-023.** `NaN`, positive infinity, and negative infinity MUST be invalid. #### 6.5.4 Boolean -- Native JSON booleans MUST be accepted. -- Textual boolean parsing MUST accept only lowercase `true` and `false`. -- Integers, uppercase boolean-like values, `yes`, `no`, `on`, `off`, and other representations MUST require an explicit value mapping. -- Booleans MUST NOT be treated as integers. +- **REQ-VAL-024.** Native JSON booleans MUST be accepted. +- **REQ-VAL-025.** Textual boolean parsing MUST accept only lowercase `true` and `false`. +- **REQ-VAL-026.** Integers, uppercase boolean-like values, `yes`, `no`, `on`, `off`, and other representations MUST require an explicit value mapping. +- **REQ-VAL-027.** Booleans MUST NOT be treated as integers. #### 6.5.5 Date -- Dates MUST use the ISO 8601 full-date form `YYYY-MM-DD`. -- Values MUST represent valid Gregorian calendar dates. -- Partial dates and locale-dependent forms MUST be invalid. -- The canonical value MUST be the represented calendar date. +- **REQ-VAL-028.** Dates MUST use the ISO 8601 full-date form `YYYY-MM-DD`. +- **REQ-VAL-029.** Values MUST represent valid Gregorian calendar dates. +- **REQ-VAL-030.** Partial dates and locale-dependent forms MUST be invalid. +- **REQ-VAL-031.** The canonical value MUST be the represented calendar date. #### 6.5.6 Datetime -- Datetimes MUST use an RFC 3339-compatible representation containing a full date, time, and `Z` or numeric UTC offset. -- Local datetimes without an offset MUST be invalid. -- Equivalent instants expressed with different offsets MUST have the same canonical value. -- Fractional seconds MUST be preserved unless explicit precision normalisation is configured. -- Leap-second representations MUST be invalid in the first release. +- **REQ-VAL-032.** Datetimes MUST use an RFC 3339-compatible representation containing a full date, time, and `Z` or numeric UTC offset. +- **REQ-VAL-033.** Local datetimes without an offset MUST be invalid. +- **REQ-VAL-034.** Equivalent instants expressed with different offsets MUST have the same canonical value. +- **REQ-VAL-035.** Fractional seconds MUST be preserved unless explicit precision normalisation is configured. +- **REQ-VAL-036.** Leap-second representations MUST be invalid in the first release. #### 6.5.7 Collections -- Every collection MUST declare its element logical type and element comparator. -- A `list` MUST preserve order and duplicates. -- A `set` MUST be unordered and MUST reject duplicate source elements rather than silently collapse them. -- A `multiset` MUST be unordered and MUST preserve element occurrence counts. -- JSON collections MUST use arrays. -- A CSV or TSV collection cell MUST use valid JSON-array syntax; ordinary delimited text MUST NOT be inferred to be a collection. +- **REQ-VAL-037.** Every collection MUST declare its element logical type and element comparator. +- **REQ-VAL-038.** A `list` MUST preserve order and duplicates. +- **REQ-VAL-039.** A `set` MUST be unordered. +- **REQ-VAL-040.** A `set` MUST reject duplicate source elements rather than silently collapse them. +- **REQ-VAL-041.** A `multiset` MUST be unordered. +- **REQ-VAL-042.** A `multiset` MUST preserve element occurrence counts. +- **REQ-VAL-043.** JSON collections MUST use arrays. +- **REQ-VAL-044.** A CSV or TSV collection cell MUST use valid JSON-array syntax. +- **REQ-VAL-045.** Ordinary delimited text MUST NOT be inferred to be a collection. +- **REQ-VAL-046.** Collection elements MUST be non-null in version 1. A parser-native null element MUST produce an `invalid_field_value` finding, prevent construction of a canonical collection value, and make that field comparison `not_assessable`. - Nested collections are outside the first-release scope. ### 6.6 Null, blank, and absent values -The comparison configuration MUST distinguish absent fields, explicit nulls, empty strings, whitespace-only strings, and configured missing tokens such as `NA`, `.`, or `null`. No two states are equivalent unless a field-specific policy declares them equivalent. Applied equivalence rules MUST be recorded. +**REQ-VAL-047.** The comparison configuration MUST distinguish absent fields, explicit nulls, empty strings, whitespace-only strings, and configured missing tokens such as `NA`, `.`, or `null`. No two states are equivalent unless a field-specific policy declares them equivalent. Applied equivalence rules MUST be recorded. -- Two explicit nulls MUST match only when the field permits null and its policy declares them equivalent. -- Null MUST NOT implicitly equal empty string, absent field, zero, boolean false, or a missing token. -- A missing token MUST become null only through an explicit field policy. -- Two absent required fields MUST produce contract findings and MUST NOT constitute a value match. -- A null and non-null pair MUST be a value mismatch when both states are valid and comparable. -- A prohibited null MUST be a contract finding and MUST make the affected value `not_assessable` unless another treatment is explicitly configured. +- **REQ-VAL-048.** Two explicit nulls MUST match only when the field permits null and its policy declares them equivalent. +- **REQ-VAL-049.** Null MUST NOT implicitly equal empty string, absent field, zero, boolean false, or a missing token. +- **REQ-VAL-050.** A missing token MUST become null only through an explicit field policy. +- **REQ-VAL-051.** Two absent required fields MUST produce contract findings. +- **REQ-VAL-052.** Two absent required fields MUST NOT constitute a value match. +- **REQ-VAL-053.** A null and non-null pair MUST be a value mismatch when both states are valid and comparable. +- **REQ-VAL-054.** A prohibited null MUST be a contract finding. +- **REQ-VAL-055.** A prohibited null MUST make the affected value `not_assessable` unless another treatment is explicitly configured. The bundled strict-exact configuration permits parser-native nulls. Under that configuration, null matches only null and remains distinct from an absent field, an empty string, and every non-null value. +The field-level null policy governs the collection value as a whole and does not permit null collection elements. Version 1 has no separate element-null policy. + ### 6.7 Mappings #### 6.7.1 Column mappings -- Reference and query fields MAY map to a canonical field name. -- Mappings MUST be one-to-one unless an explicitly defined transformation supports another cardinality. -- Conflicting target mappings MUST be rejected. -- Unknown source fields MUST follow the undeclared-field policy. -- Required mapped fields that are absent MUST be reported. -- All applied mappings MUST be identifiable in the evidence. +- **REQ-VAL-056.** Reference and query fields MAY map to a canonical field name. +- **REQ-VAL-057.** Mappings MUST be one-to-one unless an explicitly defined transformation supports another cardinality. +- **REQ-VAL-058.** Conflicting target mappings MUST be rejected. +- **REQ-VAL-059.** Unknown source fields MUST follow the undeclared-field policy. +- **REQ-VAL-060.** Required mapped fields that are absent MUST be reported. +- **REQ-VAL-061.** All applied mappings MUST be identifiable in the evidence. #### 6.7.2 Value mappings -- A value mapping MUST operate on a preserved source value before typed parsing. -- The mapping output MUST be parsed as the declared destination logical type. -- A map MUST declare its source representation and destination logical type. -- Unknown source values MUST default to a contract finding, not pass through silently. -- Many-to-one mappings MUST be allowed only when intentionally declared. -- Mapping to null MUST be distinguished from a missing map entry. -- Values used in mappings MUST be represented using the typed source-value descriptor. The parser MUST NOT infer their intended category from YAML scalar syntax. -- Applied mappings MUST be identifiable in the evidence. +- **REQ-VAL-062.** A value mapping MUST operate on a preserved source value before typed parsing. +- **REQ-VAL-063.** The mapping output MUST be parsed as the declared destination logical type. +- **REQ-VAL-064.** A map MUST declare its source representation and destination logical type. +- **REQ-VAL-065.** Unknown source values MUST default to a contract finding, not pass through silently. +- **REQ-VAL-066.** Many-to-one mappings MUST be allowed only when intentionally declared. +- **REQ-VAL-067.** Mapping to null MUST be distinguished from a missing map entry. +- **REQ-VAL-068.** Values used in mappings MUST be represented using the typed source-value descriptor. The parser MUST NOT infer their intended category from YAML scalar syntax. +- **REQ-VAL-069.** Applied mappings MUST be identifiable in the evidence. ### 6.8 Normalisation and processing order -Normalisation MUST be opt-in, field-specific, ordered deterministically, and recorded. Source-text normalisation is distinct from typed-value normalisation. +**REQ-VAL-070.** Normalisation MUST be opt-in, field-specific, ordered deterministically, and recorded. Source-text normalisation is distinct from typed-value normalisation. -Source-text normalisation MAY include Unicode normalisation, leading or trailing whitespace trimming, and case folding. Typed-value normalisation MAY include datetime offset normalisation, declared precision reduction, and stable ordering of collections declared to be unordered. +**REQ-VAL-071.** Source-text normalisation MAY include Unicode normalisation, leading or trailing whitespace trimming, and case folding. Typed-value normalisation MAY include datetime offset normalisation, declared precision reduction, and stable ordering of collections declared to be unordered. -Typed processing MUST occur in this order: +**REQ-VAL-072.** Typed processing MUST occur in this order: 1. preserve the source value and representation; 2. apply the source-text normalisation declared for the applicable dataset side; @@ -625,37 +753,45 @@ Typed processing MUST occur in this order: 6. apply declared typed-value normalisation; 7. compare canonical values. -Column-name mapping MUST occur before field-value processing. A failure at any stage MUST identify the stage and attributable record and field. The source value MUST be included in the evidence report only when required to explain the failure and MUST NOT be reproduced in routine logs or terminal diagnostics by default. +**REQ-VAL-073.** Column-name mapping MUST occur before field-value processing. + +**REQ-VAL-074.** A failure at any stage MUST identify the stage and attributable record and field. + +**REQ-VAL-075.** The source value MUST be included in the evidence package only when required to explain the failure. It MUST NOT be reproduced in routine logs, progress output, terminal diagnostics, or execution-error diagnostics. ### 6.9 Bundled strict-exact configuration -If `--configuration` is omitted, the tool MUST select the bundled strict-exact configuration. It MUST be distributed as a documented template from which laboratories can develop controlled custom configurations, and the selected effective configuration MUST be recorded in provenance. Supplying `--configuration` MUST select the named custom configuration. +**REQ-CFG-092.** If `--configuration` is omitted, the tool MUST select the bundled strict-exact configuration. It MUST be distributed as a documented template from which laboratories can develop controlled custom configurations, and the selected effective configuration MUST be recorded in provenance. Supplying `--configuration` MUST select the named custom configuration. -The bundled default configuration MUST: +**REQ-CFG-093.** The bundled default configuration MUST: -- declare one key whose canonical, reference, and query field names are all `record_id`; -- include every shared non-key field without requiring field-specific declarations; -- preserve parser-native types and use `preserved_exact` comparison; -- perform no type inference, implicit coercion, mapping, normalisation, or tolerance; -- permit parser-native null values and compare them according to Section 6.6; -- treat two syntactically valid zero-record datasets as having zero missing records, zero extra records, and zero value mismatches, while evaluating contract findings as required by Sections 5.7 and 7.2; -- treat reference-only and query-only fields as contract findings. +- **REQ-CFG-109.** declare one key whose canonical, reference, and query field names are all `record_id`; +- **REQ-CFG-110.** include every shared non-key field without requiring field-specific declarations; +- **REQ-CFG-111.** preserve parser-native types and use `preserved_exact` comparison; +- **REQ-CFG-112.** perform no type inference, implicit coercion, mapping, normalisation, or tolerance; +- **REQ-CFG-113.** permit parser-native null values and compare them according to Section 6.6; +- **REQ-CFG-114.** treat two syntactically valid zero-record datasets as having zero missing records, zero extra records, and zero value mismatches, while evaluating contract findings as required by Sections 5.7 and 7.2; +- **REQ-CFG-115.** treat reference-only and query-only fields as contract findings. -Because parser-native types differ between formats, strict-exact comparison across different input formats MAY report type mismatches for lexically similar values. A custom typed configuration is required when cross-format logical equivalence is intended. +**REQ-CFG-094.** Because parser-native types differ between formats, strict-exact comparison across different input formats MAY report type mismatches for lexically similar values. A custom typed configuration is required when cross-format logical equivalence is intended. ## 7. Comparison behaviour ### 7.1 Comparison phases -Every comparison MUST perform all three phases in this order: +**REQ-CMP-001.** Every comparison MUST perform all three phases in this order: 1. **Contract:** assess fields, logical types, key definitions, nullability, mappings, and declared data-contract constraints. 2. **Records:** align valid record identities and assess matching, missing, and extra records. 3. **Values:** compare corresponding values using declared comparators. -A finding or `not_assessable` item in an earlier phase MUST NOT automatically prevent later assessment. Later comparison MUST continue wherever its prerequisites remain satisfied. Configured content that cannot be compared safely MUST be `not_assessable`; content deliberately excluded by field policy MUST be `not_assessed`. +**REQ-CMP-002.** A finding or `not_assessable` item in an earlier phase MUST NOT automatically prevent later assessment. + +**REQ-CMP-003.** Later comparison MUST continue wherever its prerequisites remain satisfied. + +**REQ-CMP-004.** Configured content that cannot be compared safely MUST be `not_assessable`. Content deliberately excluded by field policy MUST be `not_assessed`. -A run MUST conceptually: +**REQ-CMP-005.** A run MUST conceptually: 1. validate the effective comparison configuration; 2. identify and hash inputs; @@ -668,30 +804,36 @@ A run MUST conceptually: ### 7.2 Contract comparison -The contract phase MUST assess declared and observed fields from both directions for every record. It MUST distinguish: +**REQ-CMP-006.** The contract phase MUST assess declared and observed fields from both directions for every record. It MUST distinguish: -- field present and valid; -- required field missing from the reference or query; -- observed undeclared field and the treatment required by `fields.undeclared`; -- invalid logical type, null state, mapping, or controlled value; -- field deliberately `not_assessed`; -- field `not_assessable` because a prerequisite failed. +- **REQ-CMP-034.** field present and valid; +- **REQ-CMP-035.** required field missing from the reference or query; +- **REQ-CMP-036.** observed undeclared field and the treatment required by `fields.undeclared`; +- **REQ-CMP-037.** invalid logical type, null state, mapping, or controlled value; +- **REQ-CMP-038.** field deliberately `not_assessed`; +- **REQ-CMP-039.** field `not_assessable` because a prerequisite failed. -One record’s observed field set MUST NOT be assumed to represent every other record. Key presence, validity, and uniqueness MUST always be assessed because later interpretation depends on an unambiguous record model. +**REQ-CMP-007.** One record’s observed field set MUST NOT be assumed to represent every other record. -For a zero-record dataset, contract requirements that can be evaluated from available structural information, such as CSV or TSV headers, MUST still be assessed. A requirement that depends on examining a record and cannot be evaluated because no records exist MUST be reported as `not_assessable`; it MUST NOT be treated as satisfied merely because no contradictory record was observed. +**REQ-CMP-008.** Key presence, validity, and uniqueness MUST always be assessed because later interpretation depends on an unambiguous record model. + +**REQ-CMP-009.** For a zero-record dataset, contract requirements that can be evaluated from available structural information, such as CSV or TSV headers, MUST still be assessed. + +**REQ-CMP-010.** A requirement that depends on examining a record and cannot be evaluated because no records exist MUST be reported as `not_assessable`. It MUST NOT be treated as satisfied merely because no contradictory record was observed. ### 7.3 Records comparison -For the union of normalised valid record keys, the records phase MUST classify each key as matching, missing from the query, or extra in the query. Duplicate or otherwise invalid keys are execution errors under Section 8.3, not ordinary record findings. +**REQ-CMP-011.** For the union of normalised valid record keys, the records phase MUST classify each key as matching, missing from the query, or extra in the query. Duplicate or otherwise invalid keys are execution errors under Section 8.3, not ordinary record findings. -Missing and extra records MUST be reported individually. Query-only records MUST NOT be ignored. Missing or extra records MUST NOT prevent value comparison of the safely aligned intersection. +**REQ-CMP-012.** Missing records MUST be reported individually. Extra records MUST be reported individually. Query-only records MUST NOT be ignored. Missing or extra records MUST NOT prevent value comparison of the safely aligned intersection. ### 7.4 Values comparison and comparators -The values phase MUST compare every configured, valid field in safely aligned records. A value whose contract or alignment prerequisite failed MUST be `not_assessable`; other values MUST continue to be compared. +**REQ-CMP-013.** The values phase MUST compare every configured, valid field in safely aligned records. -The first release MUST support the following deterministic comparators with stable identifiers. +**REQ-CMP-014.** A value whose contract or alignment prerequisite failed MUST be `not_assessable`. Other values MUST continue to be compared. + +**REQ-CMP-015.** The first release MUST support the following deterministic comparators with stable identifiers. #### 7.4.1 `preserved_exact` @@ -703,202 +845,317 @@ Values match only when both are present, valid under their field policies, have #### 7.4.3 `normalised_string` -Only configured string normalisations are applied in their declared order. Equality is exact code-point equality after transformation. Applied transformations MUST be identifiable in the evidence. +**REQ-CMP-016.** Only configured string normalisations are applied in their declared order. Equality is exact code-point equality after transformation. Applied transformations MUST be identifiable in the evidence. #### 7.4.4 `numeric_exact` -Integer or decimal comparison MUST use canonical arbitrary-precision values. Integer and decimal remain different logical types. Differing source representations are comparable only when both are explicitly parsed as the same declared logical type, using a mapping where required. +**REQ-CMP-017.** Integer or decimal comparison MUST use canonical arbitrary-precision values. Integer and decimal remain different logical types. Differing source representations are comparable only when both are explicitly parsed as the same declared logical type, using a mapping where required. #### 7.4.5 `numeric_tolerance` ```text difference = abs(reference - query) -allowed_difference = max(absolute_tolerance, relative_tolerance * abs(reference)) -match if difference <= allowed_difference ``` -- At least one of `absolute_tolerance` or `relative_tolerance` MUST be specified. If only one is specified, the other MUST be treated as zero when calculating the allowed difference. -- Tolerances MUST be non-negative finite decimals. -- The boundary MUST be inclusive. -- Relative tolerance MUST be directional and calculated against the absolute reference value. -- Absolute tolerance MUST govern behaviour around a zero reference value. -- Rounding or significant-figure normalisation MUST occur explicitly before comparison. +- **REQ-CMP-018.** Exactly one of `absolute_tolerance` or `relative_tolerance` MUST be specified. +- With `absolute_tolerance`, values match when `difference <= absolute_tolerance`. +- With `relative_tolerance`, values match when `difference <= relative_tolerance * abs(reference)`. A relative tolerance is expressed as a fraction; for example, `"0.05"` means five percent. +- **REQ-CMP-019.** Tolerances MUST be non-negative finite decimals. +- **REQ-CMP-020.** The boundary MUST be inclusive. +- **REQ-CMP-021.** Relative tolerance MUST be calculated using the magnitude of the reference value: `relative_tolerance × abs(reference)`. The reference and query therefore have distinct roles in this calculation. The same tolerance applies to query values above and below the reference. +- **REQ-CMP-022.** When the reference value is zero, a relative tolerance permits only an exactly equal query value. An absolute tolerance MUST be used when non-zero differences around a zero reference are intended to match. +- **REQ-CMP-023.** Rounding or significant-figure normalisation MUST occur explicitly before comparison. #### 7.4.6 `date_exact` and `datetime_exact` -Dates match when their canonical calendar dates are equal. Datetimes match when they represent the same instant after UTC-offset normalisation. Any precision reduction MUST be explicitly configured before comparison. +**REQ-CMP-024.** Dates match when their canonical calendar dates are equal. Datetimes match when they represent the same instant after UTC-offset normalisation. Any precision reduction MUST be explicitly configured before comparison. #### 7.4.7 `list_exact`, `set_exact`, and `multiset_exact` -- Lists use ordered element-by-element comparison and their lengths MUST match. +- **REQ-CMP-025.** Lists use ordered element-by-element comparison and their lengths MUST match. - Sets use order-independent comparison of unique elements; duplicate source elements are invalid under Section 6.5.7. - Multisets use order-independent comparison that preserves occurrence counts. - Elements use the declared element comparator. ### 7.5 Completeness, subsets, and determinism -- Comparison findings MUST NOT terminate a normal run. -- All assessable differences MUST be collected. -- A documented implementation safeguard MAY stop assessment to protect resources, but it MUST NOT be configurable through the comparison configuration. Reaching such a safeguard MUST be reported prominently as truncation and MUST make the run `incomplete`. A truncated run MUST NOT make claims about unexamined content. -- A malformed record that prevents a complete and unambiguous record model MUST be an input error and MUST NOT disappear silently. -- Key fields MUST always be processed. -- Under `fields.undeclared: "exclude"`, undeclared fields MUST be represented by a summary rather than by an individual finding for every excluded occurrence. The canonical report MUST record a stable exclusion code, the policy, the affected input field names, and the number of excluded records or values where calculable. -- Record order MUST NOT affect results unless explicitly declared significant. -- Output ordering MUST be deterministic and independent of hash-map ordering and locale. +- **REQ-CMP-026.** Comparison findings MUST NOT terminate a normal run. +- **REQ-CMP-027.** All assessable differences MUST be collected. +- **REQ-CMP-028.** A malformed record that prevents a complete and unambiguous record model MUST be an input error. +- **REQ-CMP-029.** A malformed record MUST NOT disappear silently. +- **REQ-CMP-030.** Key fields MUST always be processed. +- **REQ-CMP-031.** Under `fields.undeclared: "exclude"`, undeclared fields MUST be represented by a summary rather than by an individual finding for every excluded occurrence. The canonical report MUST record a stable exclusion code, the policy, the affected input field names, and the number of excluded records or values where calculable. +- **REQ-CMP-032.** Record order MUST NOT affect results unless explicitly declared significant. +- **REQ-CMP-033.** Output ordering MUST be deterministic and independent of hash-map ordering and locale. ## 8. Findings and run status ### 8.1 Finding structure -Every finding MUST be structured and SHOULD contain: +**REQ-FND-001.** Every finding MUST follow the structured representation in Section 9.1.2 and the normative results schema. + +**REQ-FND-002.** Every finding MUST contain a stable finding code, deterministic report-local ID, phase, affected side, and human-readable message. -- stable finding code; -- category; -- review priority, when the finding is attributable to a declared field; -- record key; -- reference and query source locations where available; -- canonical field name; -- expected and observed presence state; -- expected and observed logical type; -- expected and observed value when required to explain the finding; -- comparator and parameters; -- mappings/normalisations applied; -- concise human-readable explanation; -- related specification requirement or rule identifier. +**REQ-FND-003.** Record identity, source locations, canonical or source field name, review priority, and value evidence MUST be present exactly where required by the finding code and data-minimisation rules. -Human-readable messages MUST NOT be the sole machine-readable representation of a finding. Successful field and record matches MUST be represented in summary counts and MUST NOT produce individual findings. +**REQ-FND-004.** Human-readable messages MUST NOT be the sole machine-readable representation of a finding. + +**REQ-FND-005.** Successful field matches MUST be represented in summary counts. Successful record matches MUST be represented in summary counts. Successful matches MUST NOT produce individual findings. ### 8.2 Minimum finding categories -Stable result codes MUST exist for at least: - -- missing record; -- extra record; -- invalid record key; -- missing field; -- undeclared field; -- type mismatch; -- value mismatch; -- mapping failure; -- parse or data-contract error; -- invalid comparison configuration; -- not assessed; -- not assessable; -- comparison truncated; -- internal execution error. +**REQ-FND-006.** Stable finding codes MUST exist for at least: + +- **REQ-FND-008.** missing record; +- **REQ-FND-009.** extra record; +- **REQ-FND-010.** missing required field; +- **REQ-FND-011.** undeclared field; +- **REQ-FND-012.** type mismatch; +- **REQ-FND-013.** invalid field value, including an attributable field-level parse or data-contract failure; +- **REQ-FND-014.** null-policy violation; +- **REQ-FND-015.** mapping failure; +- **REQ-FND-016.** value mismatch; +- **REQ-FND-017.** not assessable; + +**REQ-FND-007.** Execution errors MUST NOT be represented as findings or assigned finding codes. In particular, invalid invocation or configuration, malformed input structure, invalid record keys, resource failure, I/O failure, and internal failure are execution errors governed by Section 8.3. ### 8.3 Run status -Run status MUST be one of: +**REQ-RUN-001.** Run status MUST be one of: + +- **REQ-RUN-015.** `complete`: every configured assessment was performed wherever the configuration and inputs made it applicable; +- **REQ-RUN-016.** `incomplete`: the inputs and record model remain sufficiently trustworthy for a useful comparison, but one or more configured assessments could not be completed safely because a comparison prerequisite was not satisfied; +- **REQ-RUN-017.** `error`: no trustworthy comparison result could be produced because of an invalid invocation or comparison configuration, malformed or unreadable input syntax or structure, a missing, null, invalid, or duplicate record key, an incomplete or ambiguous record model, an enforced resource limit, resource exhaustion, an input/output failure, or an unexpected internal failure. + +**REQ-RUN-002.** Differences between valid, assessable reference and query content — including contract findings, missing or extra records, and value mismatches — MUST NOT by themselves change a run from `complete` to `incomplete` or `error`. The run status describes completeness and operational validity, not acceptability. + +**REQ-RUN-003.** Any configured content classified as `not_assessable` MUST make the run `incomplete`. + +**REQ-RUN-004.** Deliberately `not_assessed` content MUST NOT make the run `incomplete`. + +**REQ-RUN-005.** An incomplete report MUST identify the affected content. It MUST identify why the comparison prerequisite was not satisfied. + +**REQ-RUN-006.** Content that is parseable and attributable to an unambiguous record but violates the declared data contract MUST be recorded as a comparison finding rather than an error. -- `complete`: every configured assessment was performed wherever the configuration and inputs made it applicable; -- `incomplete`: the inputs and record model remain sufficiently trustworthy for a useful comparison, but one or more configured assessments could not be completed safely or assessment was truncated; -- `error`: no trustworthy comparison result could be produced because of an invalid invocation or comparison configuration, malformed or unreadable input syntax or structure, a missing, null, invalid, or duplicate record key, an incomplete or ambiguous record model, an input/output failure, or an unexpected internal failure. +**REQ-RUN-007.** Comparisons that remain assessable MUST continue. -Differences between valid, assessable reference and query content—including contract findings, missing or extra records, and value mismatches—MUST NOT by themselves change a run from `complete` to `incomplete` or `error`. The run status describes completeness and operational validity, not acceptability. +**REQ-RUN-008.** An error report SHOULD be written where possible. When written, it MUST be identified as an error report. -Any configured content classified as `not_assessable`, or any content left unexamined because assessment was truncated, MUST make the run `incomplete`. Deliberately `not_assessed` content MUST NOT do so. An incomplete report MUST identify the affected or unexamined content and MUST not make claims about it. +An error report is a diagnostic artefact, not a comparison result or evidence package. -Content that is parseable and attributable to an unambiguous record but violates the declared data contract MUST be recorded as a comparison finding rather than an error. Comparisons that remain assessable MUST continue. An error report SHOULD be written where possible and MUST be identified as such. +**REQ-RUN-009.** When an error report can be written safely, it MUST contain `run_status: error`, a stable execution error code and category, a concise actionable message, and sufficient non-sensitive context to locate the cause. -Each phase MUST record whether it was `complete` or `incomplete`. Overall status is `complete` only when all three phases are complete. The tool MUST report raw findings and metrics without applying acceptance thresholds or producing a pass/fail verdict. +**REQ-RUN-010.** An error report MUST NOT contain findings, finding codes, comparison summary metrics, or phase statuses. + +**REQ-RUN-011.** Findings accumulated internally before the error occurred MUST NOT be published as a partial comparison result. + +**REQ-RUN-012.** Each phase MUST record whether it was `complete` or `incomplete`. + +Overall status is `complete` only when all three phases are complete. + +**REQ-RUN-013.** The tool MUST report raw findings and metrics without applying acceptance thresholds. It MUST NOT produce a pass/fail verdict. ### 8.4 Laboratory decision -The run status and reported differences do not constitute a laboratory validation, approval, or acceptance decision. Any acceptance criteria and their application belong to a separately governed process outside the tool. Reports MUST label these concepts separately and MUST NOT describe the output as “ISO 15189 compliant”, “clinically validated”, or equivalent. +The run status and reported differences do not constitute a laboratory validation, approval, or acceptance decision. Any acceptance criteria and their application belong to a separately governed process outside the tool. + +**REQ-RUN-014.** Reports MUST label these concepts separately. Reports MUST NOT describe the output as “ISO 15189 compliant”, “clinically validated”, or equivalent. ## 9. Output and evidence -A non-error comparison, whether `complete` or `incomplete`, MUST create an evidence-package directory at the destination selected by `--output`. The package MUST contain, at minimum, the canonical JSON result and the HTML report. A CSV findings export MAY also be included. Package member names and layout MUST be stable and documented. +**REQ-OUT-001.** A non-error comparison, whether `complete` or `incomplete`, MUST create an evidence-package directory at the destination selected by `--output`. + +**REQ-OUT-002.** The evidence package MUST contain the canonical JSON result. It MUST contain the HTML report. A CSV findings export MAY also be included. + +**REQ-OUT-003.** Package member names MUST be stable and documented. Package layout MUST be stable and documented. ### 9.1 Machine-readable output -JSON MUST be the canonical output format. It MUST have a published, versioned schema containing: +**REQ-OUT-004.** JSON MUST be the canonical output format. Its normative JSON Schema Draft 2020-12 definition is [`src/adviso_validation/schemas/comparison-results-v1.0.schema.json`](src/adviso_validation/schemas/comparison-results-v1.0.schema.json). -- report schema version; -- run status and per-phase completion status; -- run identifier; -- timestamps; -- tool identity and version; -- comparison-configuration identity, revision, schema version, and digest; -- input identities and digests; -- effective comparison settings; -- summary counts; -- structured findings; -- errors and warnings; -- truncation status; +**REQ-OUT-005.** The standalone results-schema file MUST be version controlled. It MUST be included as package data in `adviso_validation.schemas`. It MUST be resolvable together with the comparison-configuration schema that it references. -Consumers MUST be able to distinguish absent and null values. +The result-schema version and comparison-configuration-schema version are independent. -### 9.2 Human-readable output +**REQ-OUT-006.** The canonical result MUST contain exactly these top-level properties: + +- **REQ-OUT-050.** `results_schema_version`; +- **REQ-OUT-051.** `run`; +- **REQ-OUT-052.** `phases`; +- **REQ-OUT-053.** `software`; +- **REQ-OUT-054.** `runtime`; +- **REQ-OUT-055.** `inputs`; +- **REQ-OUT-056.** `record_key_fields`; +- **REQ-OUT-057.** `comparison_configuration`; +- **REQ-OUT-058.** `summary`; +- **REQ-OUT-059.** `exclusions`; +- **REQ-OUT-060.** `findings`. + +Every property is required, including empty `exclusions` and `findings` arrays. + +**REQ-OUT-007.** Unknown properties MUST be rejected at every defined object level. + +**REQ-OUT-008.** The schema accepts only non-error results with run status `complete` or `incomplete`. An execution-error diagnostic under Section 8.3 is not a canonical comparison result and MUST NOT validate against this schema. + +**REQ-OUT-009.** The `run` object MUST contain a randomly generated UUID `id`, `status`, and `completed_at`. + +**REQ-OUT-010.** `completed_at` MUST be an RFC 3339 UTC timestamp using `Z`. + +**REQ-OUT-011.** Start time MUST NOT be included. Elapsed duration MUST NOT be included. + +**REQ-OUT-012.** Run ID and completion time are deliberately variable metadata. They MAY be excluded from semantic-determinism comparisons. + +**REQ-OUT-013.** The `phases` object MUST contain `contract`, `records`, and `values`, each with status `complete` or `incomplete`. -The first release MUST produce an HTML human-readable report. The report SHOULD include: +**REQ-OUT-014.** Overall run status MUST be `complete` if and only if all three phases are complete. -- purpose of the run and the three comparison phases performed; -- prominent run status and per-phase completion status; -- identities of reference, query, and comparison configuration; -- summary counts; -- measured counts and findings, without an acceptance verdict; -- grouped discrepancy details; -- limitations, exclusions, and unassessed content; -- provenance and reproduction information; -- space or linkage for review and approval outside the tool. +**REQ-OUT-015.** The `software` object MUST identify `adviso-validation` and its version. A build identifier MAY be included when available. -CSV MAY be provided as an export of findings, but MUST: +**REQ-OUT-016.** The `runtime` object MUST record Python version, a normalized non-identifying platform identifier, and the Unicode-data version. -- be produced using a conforming CSV writer; -- preserve record identifiers; -- quote delimiters, quotes, and newlines correctly; -- use a stable documented column schema; -- mitigate spreadsheet formula injection when intended for spreadsheet use; -- never be the only canonical evidence format. +**REQ-OUT-017.** Hostname, username, environment variables, working directory, absolute paths, raw command line, and a full dependency inventory MUST NOT be included. -### 9.3 Summary counts +**REQ-OUT-018.** Each input object MUST contain its automatically captured basename, selected format, UTF-8 encoding, byte size, and SHA-256 digest of the exact input bytes. -Summary counts MUST reconcile with emitted findings and MUST include at least: +**REQ-OUT-019.** Absolute paths MUST NOT be included. -- input record counts; -- valid unique key counts; -- matched, missing, and extra record counts; -- fields assessed and not assessed; -- findings by code and, where applicable, review priority; -- complete/incomplete status for each of the three phases. +Basenames form part of the potentially sensitive evidence covered by Section 11. -### 9.4 Provenance +**REQ-OUT-020.** V1 MUST use one fixed documented interpretation for CSV. It MUST use one fixed documented interpretation for TSV. Because format uniquely determines that interpretation, no dialect identifier appears in the result. -Provenance MUST include: +**REQ-OUT-021.** `record_key_fields` MUST declare the ordered source key-field names for the reference and query and the ordered canonical key-field names. Findings identify records through corresponding ordered arrays of source lexical values and canonical textual values. A single-field key is represented by a one-element array. Each identity array MUST have the same length as its corresponding field-name array. The reference or query identity is JSON null when that record is absent on that side; the canonical identity is always present. -- SHA-256 or stronger digest of each input and the comparison configuration; -- tool version and build/release identity; -- report schema version; -- invocation parameters with secrets omitted; -- start and end timestamps with time-zone offset; -- runtime environment sufficient for reproduction; -- selected input formats and encodings; -- all non-default policies in effect. +**REQ-OUT-022.** `comparison_configuration.document` MUST contain the complete validated JSON-compatible representation of the comparison configuration, including the bundled configuration when no custom file was supplied. -Absolute paths and the raw command line MUST NOT be included. Safe display names MAY be supplied separately. +**REQ-OUT-023.** The embedded document MUST validate against the referenced comparison-configuration schema. -### 9.5 Atomicity and overwrite protection +No comparison-configuration digest or command-invocation object is included in version 1. -- Output files MUST be written atomically where supported. -- The complete evidence-package directory MUST be published atomically where the platform supports it. -- An existing destination MUST NOT be overwritten unless the user supplies `--overwrite`. -- The actual selected output directory MUST be reported clearly. -- File suffixes MUST be handled structurally, not with character stripping. -- Partial evidence packages MUST be identifiable as incomplete. +**REQ-OUT-024.** Consumers MUST validate `results_schema_version` before interpreting a result. + +**REQ-OUT-025.** Consumers MUST use structured properties and finding codes for program logic. They MUST NOT infer semantics by parsing human-readable finding messages. + +#### 9.1.1 Value evidence + +**REQ-OUT-026.** When a finding includes field evidence, `values.reference` and `values.query` MUST each state whether the field is `present` or `absent`. An absent side has no source or canonical value. An explicit null is present and is represented explicitly rather than as absence. + +**REQ-OUT-027.** Parser-native source values MUST use typed objects. Strings retain their text; integer and decimal values retain their source lexical text; booleans use JSON booleans; null has no `value`; arrays contain recursively typed source values; and objects map member names to recursively typed source values. + +**REQ-OUT-028.** Object member order has no semantic significance. Object members MUST be serialized in Unicode code-point order. + +**REQ-OUT-029.** Canonical scalar values MUST contain their logical type and value. Strings use JSON strings, booleans use JSON booleans, and dates and datetimes use canonical ISO text. Integers and decimals use normalized exact decimal text: no leading plus sign or unnecessary leading or trailing zeros, zero represented as `"0"`, and normalized lowercase scientific notation permitted where needed. Canonical collections contain the collection logical type, scalar element type, and an array of compact canonical element values. Lists preserve order; sets and multisets use deterministic canonical order; multisets retain duplicates. Null uses the declared logical type with a JSON null value. A `preserved_exact` field has no typed canonical value. + +**REQ-OUT-030.** Values MUST be included only as follows: + +- **REQ-OUT-061.** missing and extra record findings contain no field values; +- **REQ-OUT-062.** missing-required-field findings contain presence states but do not reproduce the present side's value; +- **REQ-OUT-063.** undeclared-field findings contain no field value; +- **REQ-OUT-064.** type-mismatch findings contain source values for affected sides and canonical values only where parsing succeeded; +- **REQ-OUT-065.** invalid-field-value and mapping-failure findings contain the affected source value and no failed canonical value; +- **REQ-OUT-066.** null-policy-violation findings contain the affected source and canonical null representation; +- **REQ-OUT-067.** value-mismatch findings contain source and canonical values for both sides; +- **REQ-OUT-068.** not-assessable findings contain only values necessary to explain the failed prerequisite. + +**REQ-OUT-031.** Implementations MUST NOT add additional source values merely for convenience. + +#### 9.1.2 Findings + +**REQ-OUT-032.** Every finding MUST contain a deterministic report-local ID, finding code, phase, human-readable English message, and affected side. + +**REQ-OUT-033.** A message is presentation text derived deterministically from structured properties. It MUST NOT add facts absent from those properties. Consumers MUST NOT parse it for program logic. + +Specification-requirement identifiers and a separate finding category are not included in version 1. + +The closed version 1 finding-code vocabulary is: + +- `missing_record`; +- `extra_record`; +- `missing_required_field`; +- `undeclared_field`; +- `type_mismatch`; +- `invalid_field_value`; +- `null_policy_violation`; +- `mapping_failure`; +- `value_mismatch`; +- `not_assessable`. + +**REQ-OUT-034.** `affected_side` MUST be `reference`, `query`, or `both`. A finding attributable to a declared canonical field MUST contain that field name and its resolved review priority. An undeclared-field finding instead contains the source field name. Record-specific findings MUST contain record identity and source locations; dataset-level findings omit record identity. + +For CSV and TSV, a source location records the one-based data-record number where applicable, the one-based starting physical line number, and the source field name where applicable. For JSON, it records an RFC 6901 JSON Pointer. The reference and query locations are represented independently; a location is null when the corresponding record or field does not exist. + +**REQ-OUT-035.** Findings MUST be sorted deterministically by phase, canonical record identity, canonical field name, finding code, affected side, and source location, using locale-independent Unicode code-point ordering. IDs `F000001`, `F000002`, and so on MUST be assigned after sorting. IDs are stable within one result but are not persistent identifiers across materially different comparisons. + +**REQ-OUT-036.** Comparator parameters, mappings, and normalisation declarations MUST NOT be repeated in findings. Consumers and the HTML renderer obtain them from the embedded configuration using the canonical field name. Source and canonical values show their effect for the affected record. + +#### 9.1.3 Summary and exclusions + +**REQ-OUT-037.** The summary MUST contain complete fixed-shape counts, including zero-valued entries. Record counts comprise reference records, query records, matched records, missing records, and extra records. Value counts comprise matched, mismatched, and not-assessable in-scope non-key value opportunities. Finding counts comprise total findings, every version 1 finding code, and review priorities `low`, `normal`, `high`, and `unassigned`. + +**REQ-OUT-038.** Record counts MUST reconcile as follows: + +```text +reference = matched + missing +query = matched + extra +``` + +**REQ-OUT-039.** Every in-scope non-key value opportunity MUST be classified exactly once as matched, mismatched, or not assessable. Record-key fields are not compared or counted again in the values phase. For declared non-key fields in aligned records: + +- **REQ-OUT-069.** both valid present values are compared normally; +- **REQ-OUT-070.** two absent optional fields match; +- **REQ-OUT-071.** one absent optional field and one present field mismatch; +- **REQ-OUT-072.** absence on a required side is not assessable; +- **REQ-OUT-073.** an invalid or prohibited value that prevents comparison is not assessable. + +Under `fields.undeclared: compare`, undeclared non-key fields in aligned records are value opportunities: fields present on both sides use `preserved_exact`; one-sided fields create an undeclared-field finding and are not assessable. Under `exclude`, they are not value opportunities. Under `fail`, they create undeclared-field findings, are not compared, and are counted as not assessable. + +**REQ-OUT-040.** `summary.findings.total` MUST equal the number of emitted findings. Counts by code and counts by review priority MUST each sum to the total. Findings not attributable to a declared field, including record and undeclared-field findings, count as `unassigned` review priority. + +**REQ-OUT-041.** `exclusions` MUST always be an array. Under `fields.undeclared: exclude`, it contains one entry per source field and dataset side, recording code `undeclared_field_excluded`, side, source field name, and the number of records containing the field. Entries MUST be ordered by side and source field name. Exclusions are not findings, do not have review priorities, and do not contribute to value counts. A column known from a zero-record CSV or TSV header has record count zero. + +### 9.2 Human-readable output + +**REQ-OUT-042.** The first release MUST produce an HTML human-readable report. The report SHOULD include: + +- **REQ-OUT-074.** purpose of the run and the three comparison phases performed; +- **REQ-OUT-075.** prominent run status and per-phase completion status; +- **REQ-OUT-076.** identities of reference, query, and comparison configuration; +- **REQ-OUT-077.** summary counts; +- **REQ-OUT-078.** measured counts and findings, without an acceptance verdict; +- **REQ-OUT-079.** grouped discrepancy details; +- **REQ-OUT-080.** limitations, exclusions, and unassessed content; +- **REQ-OUT-081.** provenance and reproduction information; +- **REQ-OUT-082.** space or linkage for review and approval outside the tool. + +**REQ-OUT-043.** CSV MAY be provided as an export of findings, but MUST: + +- **REQ-OUT-083.** be produced using a conforming CSV writer; +- **REQ-OUT-084.** preserve record identifiers; +- **REQ-OUT-085.** quote delimiters, quotes, and newlines correctly; +- **REQ-OUT-086.** use a stable documented column schema; +- **REQ-OUT-087.** mitigate spreadsheet formula injection when intended for spreadsheet use; +- **REQ-OUT-088.** never be the only canonical evidence format. + +### 9.3 Atomicity and overwrite protection + +- **REQ-OUT-044.** Output files MUST be written atomically where supported. +- **REQ-OUT-045.** The complete evidence-package directory MUST be published atomically where the platform supports it. +- **REQ-OUT-046.** An existing destination MUST NOT be overwritten unless the user supplies `--overwrite`. +- **REQ-OUT-047.** The actual selected output directory MUST be reported clearly. +- **REQ-OUT-048.** File suffixes MUST be handled structurally, not with character stripping. +- **REQ-OUT-049.** A failed or interrupted write MUST NOT publish a partial directory at the selected destination as if it were a complete evidence package. Temporary output MUST be identifiable as temporary and cleaned up where possible. ## 10. Command-line interface ### 10.1 General behaviour -- A subcommand MUST be required. -- Invalid argument combinations MUST be rejected before input processing. -- Modes and formats MUST use enumerated choices. -- `--help` and `--version` MUST work without other arguments. -- Errors MUST be concise on standard error and actionable. -- Normal summaries SHOULD be written to standard output. -- Comparison progress SHOULD be optional and MUST NOT contaminate machine-readable output. +- **REQ-CLI-001.** A subcommand MUST be required. +- **REQ-CLI-002.** Invalid argument combinations MUST be rejected before input processing. +- **REQ-CLI-003.** Modes and formats MUST use enumerated choices. +- **REQ-CLI-004.** `--help` and `--version` MUST work without other arguments. +- **REQ-CLI-005.** Errors MUST be concise on standard error and actionable. +- **REQ-CLI-006.** Normal summaries SHOULD be written to standard output. +- **REQ-CLI-007.** Comparison progress SHOULD be optional. +- **REQ-CLI-008.** Comparison progress MUST NOT contaminate machine-readable output. ### 10.2 Proposed comparison interface @@ -910,233 +1167,325 @@ adviso-validation compare \ --query QUERY \ --output OUTPUT \ [--configuration CONFIGURATION] \ - [--reference-format {csv,tsv,json}] \ - [--query-format {csv,tsv,json}] \ [--overwrite] ``` -`--configuration` MUST be optional. Its omission MUST select the bundled strict-exact configuration; when present, its value MUST be treated as the path to a custom comparison-configuration file. `--reference-format` and `--query-format` MUST follow Section 5.2. `--output` MUST name the evidence-package directory described in Section 9, and `--overwrite` MUST provide the explicit overwrite authorisation required by Section 9.5. +**REQ-CLI-009.** `--configuration` MUST be optional. + +**REQ-CLI-010.** Omission of `--configuration` MUST select the bundled strict-exact configuration. When `--configuration` is present, its value MUST be treated as the path to a custom comparison-configuration file. + +**REQ-CLI-011.** `--output` MUST name the evidence-package directory described in Section 9. -Convenience flags MAY override non-critical settings only when the effective configuration is captured in provenance. Clinically material overrides SHOULD require an explicit opt-in and be prominent in the report. +**REQ-CLI-012.** `--overwrite` MUST provide the explicit overwrite authorisation required by Section 9.3. + +**REQ-CLI-013.** Input formats MUST be determined as specified in Section 5.2. Version 1 provides no command-line format overrides. ### 10.3 Exit codes -Exit codes MUST distinguish at least: +**REQ-CLI-014.** Exit codes MUST distinguish at least: | Code | Meaning | | ---: | --- | | 0 | Run status is `complete`, whether or not differences were found | | 1 | Run status is `incomplete` | -| 2 | Invocation, comparison-configuration, input, or I/O error | +| 2 | Invocation, comparison-configuration, input, resource-limit, resource-exhaustion, or I/O error | | 3 | Unexpected internal error | -These exit-code meanings MUST be stable and documented. A complete comparison that finds differences MUST still exit with code 0 and write the evidence package. Error classes represented by exit codes 2 and 3 have run status `error`; the distinct codes allow automation to distinguish expected operational errors from unexpected implementation failures. +**REQ-CLI-015.** These exit-code meanings MUST be stable. They MUST be documented. + +**REQ-CLI-016.** A complete comparison that finds differences MUST still exit with code 0. It MUST write the evidence package. + +Error classes represented by exit codes 2 and 3 have run status `error`. The distinct codes allow automation to distinguish expected operational errors from unexpected implementation failures. ### 10.4 Additional commands -No additional command is specified for the initial release. In particular, the prototype’s format-conversion operation is excluded because it does not presently justify a place in the validation-evidence workflow. Any future command requires its own approved behaviour specification and MUST satisfy the product-boundary criteria in Section 2.1. +**REQ-CLI-017.** The version 1 executable MUST expose no user-facing subcommand other than `compare`. ## 11. Security and data governance -- The tool MUST assume that datasets and evidence packages may contain confidential patient or laboratory information. -- It MUST operate locally and MUST NOT transmit input data, comparison configurations, logs, or results over a network. -- Routine logs, progress output, and terminal diagnostics MUST NOT contain record identifiers or source values by default. A user MAY explicitly request diagnostic detail, and that choice MUST be prominent. -- Evidence reports MAY contain record identifiers and source values only where required to identify or explain findings. Reports MUST state prominently that the evidence package may contain sensitive information. -- Absolute paths, raw command lines, environment variables, and secrets MUST NOT be included in reports or routine logs. Safe display names and a structured record of non-secret invocation parameters MAY be included. -- Temporary files MUST be created securely and removed according to documented policy. -- Comparison-configuration YAML MUST be parsed using the safe-loader rules in Section 6.2.1. -- Spreadsheet-oriented exports MUST mitigate formula injection. -- Resource limits SHOULD protect against unexpectedly large or malicious inputs without silently truncating results. -- The tool SHOULD document filesystem permission expectations. The laboratory is responsible for supplying appropriately pseudonymised input data where required and for controlling access to, retention of, and disposal of inputs and evidence packages. +- **REQ-SEC-001.** Input datasets and evidence packages MAY contain sensitive patient or laboratory information. Version 1 provides only the data-minimisation and non-disclosure safeguards specified below; it does not de-identify data or replace the laboratory's access-control and information-governance arrangements. +- **REQ-SEC-002.** The tool MUST operate locally. +- **REQ-SEC-003.** The tool MUST NOT transmit input data, comparison configurations, logs, or results over a network. +- **REQ-SEC-004.** Routine logs, progress output, terminal diagnostics, and execution-error diagnostics MUST NOT contain record identifiers or source values. +- **REQ-SEC-005.** Evidence packages MAY contain record identifiers and source values only where required to identify or explain findings. +- **REQ-SEC-006.** Human-readable reports MUST state prominently that the evidence package may contain sensitive information. +- **REQ-SEC-007.** Absolute paths, raw command lines, environment variables, and secrets MUST NOT be included in results, reports, or routine logs. +- **REQ-SEC-008.** Input basenames MUST be treated as potentially sensitive evidence. +- **REQ-SEC-009.** If temporary files are used, they MUST NOT be made more accessible than the intended output. +- **REQ-SEC-010.** Temporary files MUST be removed after successful completion. +- **REQ-SEC-011.** Temporary files SHOULD be removed following an error where this can be done safely. +- **REQ-SEC-012.** Comparison-configuration YAML MUST be parsed using the safe-loader rules in Section 6.2.1. +- **REQ-SEC-013.** Spreadsheet-oriented exports MUST mitigate formula injection. +- **REQ-SEC-014.** The tool SHOULD document filesystem permission expectations. The laboratory is responsible for supplying appropriately pseudonymised input data where required and for controlling access to, retention of, and disposal of inputs and evidence packages. - Automated redaction, identifier pseudonymisation, keyed hashing, and field-level sensitivity policies are outside the scope of version 1.0. ## 12. Deployment, accessibility, and reliability -- all core comparison functionality MUST operate fully offline; -- the toolkit MUST NOT require a hosted service, account, licence server, or telemetry connection; -- an offline-installable artefact with integrity information SHOULD be provided; -- reports MUST use open formats and MUST NOT require proprietary software; -- errors and documentation SHOULD be usable without specialist Python knowledge; -- documentation and examples SHOULD be available offline; -- localisation SHOULD remain possible without changing result semantics. +- **REQ-DEP-001.** all core comparison functionality MUST operate fully offline; +- **REQ-DEP-002.** the toolkit MUST NOT require a hosted service, account, licence server, or telemetry connection; +- **REQ-DEP-003.** an offline-installable artefact with integrity information SHOULD be provided; +- **REQ-DEP-004.** reports MUST use open formats; +- **REQ-DEP-005.** reports MUST NOT require proprietary software; +- **REQ-DEP-006.** errors and documentation SHOULD be usable without specialist Python knowledge; +- **REQ-DEP-007.** documentation and examples SHOULD be available offline; +- **REQ-DEP-008.** localisation SHOULD remain possible without changing result semantics. + +**REQ-DEP-009.** Releases SHOULD state any enforced resource or input-size limits. -Releases SHOULD state any enforced resource or input-size limits. +**REQ-DEP-010.** Every valid comparison accepted for processing MUST attempt to assess all configured content. -Large inputs MUST either be processed within documented limits or stopped without presenting an incomplete assessment as complete. Resource exhaustion before construction of a complete and unambiguous record model MUST produce an error. Resource exhaustion after that model has been constructed MAY produce an `incomplete` run only when the evidence package identifies the unexamined content and otherwise satisfies Section 8.3. Interrupted runs MUST NOT leave apparently complete reports. Results MUST remain deterministic across supported Python versions and operating systems. Internal exceptions MUST be captured in an error report where safe, with concise diagnostics presented to the user. +**REQ-DEP-011.** Exceeding a documented resource or input-size limit MUST produce an error. Encountering resource exhaustion at any stage MUST produce an error. A resource-limit or resource-exhaustion error MUST NOT produce a partial comparison result. + +**REQ-DEP-012.** Interrupted runs MUST NOT leave apparently complete reports. + +**REQ-DEP-013.** Results MUST remain deterministic across supported Python versions and operating systems. + +**REQ-DEP-014.** Internal exceptions MUST be captured in an error report where safe. Concise diagnostics MUST be presented to the user. ## 13. Software quality and release requirements -### 13.1 Requirements traceability +### 13.1 Testing + +**REQ-TST-001.** The project MUST include: + +- **REQ-TST-009.** unit tests for parsing, typing, mapping, normalisation, and comparators; +- **REQ-TST-010.** behavioural contract tests for result semantics; +- **REQ-TST-011.** end-to-end CLI tests including exit codes; +- **REQ-TST-012.** schema validation tests for comparison configurations and canonical results; +- **REQ-TST-013.** installation tests from wheel and source distribution; +- **REQ-TST-014.** regression tests for every corrected material defect; +- **REQ-TST-015.** tests across all supported Python versions; +- **REQ-TST-016.** deterministic-output tests; +- **REQ-TST-017.** representative malformed and adversarial inputs; +- **REQ-TST-018.** tests for duplicate/composite keys and missing/extra content in both directions; +- **REQ-TST-019.** boundary tests for comparator tolerances. + +**REQ-TST-002.** Property-based testing SHOULD be used for parsing and comparison invariants. + +**REQ-TST-003.** Mutation testing SHOULD be used to assess whether the suite detects deliberately introduced comparison defects. + +### 13.2 Test data + +**REQ-TST-004.** Controlled test cases SHOULD record: -- Normative requirements intended for implementation MUST receive stable identifiers before the specification is approved. -- Tests MUST reference relevant requirement identifiers. -- A traceability report SHOULD show requirement-to-test and test-to-requirement coverage. +- **REQ-TST-020.** stable scenario identifier; +- **REQ-TST-021.** requirement(s) tested; +- **REQ-TST-022.** purpose and rationale; +- **REQ-TST-023.** input data and comparison configuration; +- **REQ-TST-024.** expected findings, counts, run status, per-phase statuses, and exit code; +- **REQ-TST-025.** fixture provenance or generation method; +- **REQ-TST-026.** fixture version and digest; +- **REQ-TST-027.** independent review status where used as formal validation evidence. -### 13.2 Testing +**REQ-TST-005.** Generated fixtures MUST be reproducible. -The project MUST include: +**REQ-TST-006.** Clinically derived fixtures MUST be appropriately de-identified and governed. -- unit tests for parsing, typing, mapping, normalisation, and comparators; -- behavioural contract tests for result semantics; -- end-to-end CLI tests including exit codes; -- schema validation tests for comparison configurations and reports; -- installation tests from wheel and source distribution; -- regression tests for every corrected material defect; -- tests across all supported Python versions; -- deterministic-output tests; -- representative malformed and adversarial inputs; -- tests for duplicate/composite keys and missing/extra content in both directions; -- boundary tests for comparator tolerances. +**REQ-TST-007.** Controlled test fixtures, including test comparison configurations, MUST be maintained by the development team under version control and its documented review process. -Property-based testing SHOULD be used for parsing and comparison invariants. Mutation testing SHOULD be used to assess whether the suite detects deliberately introduced comparison defects. +In operational use, each laboratory is responsible for creating, approving, maintaining, and retaining its own comparison configurations under its local quality-management arrangements. The AdvISO project does not centrally approve or manage laboratory configurations. -### 13.3 Test data +### 13.3 Continuous integration -Controlled test cases SHOULD record: +**REQ-TST-008.** CI MUST: -- stable scenario identifier; -- requirement(s) tested; -- purpose and rationale; -- input data and comparison configuration; -- expected findings, counts, run status, per-phase statuses, and exit code; -- fixture provenance or generation method; -- fixture version and digest; -- independent review status where used as formal validation evidence. +- **REQ-TST-028.** run on proposed changes and protected branches; +- **REQ-TST-029.** install the built package, not rely solely on the source checkout; +- **REQ-TST-030.** test the console entry point; +- **REQ-TST-031.** test the supported Python matrix; +- **REQ-TST-032.** fail on enforced lint, type, test, and build errors; +- **REQ-TST-033.** validate that generated schemas and examples are current; +- **REQ-TST-034.** avoid publishing releases when required checks fail. -Generated fixtures MUST be reproducible. Clinically derived fixtures MUST be appropriately de-identified and governed. Controlled test fixtures, including test comparison configurations, MUST be maintained by the development team under version control and its documented review process. In operational use, each laboratory is responsible for creating, approving, maintaining, and retaining its own comparison configurations under its local quality-management arrangements; the AdvISO project does not centrally approve or manage laboratory configurations. +### 13.4 User documentation -### 13.4 Continuous integration +**REQ-DOC-001.** Version 1 MUST include a version-controlled user guide or manual. The guide MUST be distributed with the software. It MUST be available offline. It MUST be written for users of the command-line tool and describe: -CI MUST: +- **REQ-DOC-006.** installation and verification of the installed executable; +- **REQ-DOC-007.** the complete command-line interface, including inputs, outputs, defaults, exit codes, overwrite behaviour, and worked invocations; +- **REQ-DOC-008.** the supported input formats and their parsing, encoding, record, field, and filename-extension requirements; +- **REQ-DOC-009.** how to use the bundled strict-exact comparison configuration and how to create, validate, and govern a custom comparison configuration; +- **REQ-DOC-010.** every property in the comparison-configuration schema, including its purpose, permitted values, required or optional status, default or omission semantics, constraints, and interactions with other properties; +- **REQ-DOC-011.** the comparison phases, processing order, comparators, mappings, normalisation operations, null handling, undeclared-field behaviour, and review priorities; +- **REQ-DOC-012.** the contents and interpretation of the evidence package; +- **REQ-DOC-013.** every property in the canonical results JSON schema, including finding codes, typed source and canonical values, counts, exclusions, run and phase statuses, and provenance fields; +- **REQ-DOC-014.** execution-error diagnostics and common corrective actions; +- **REQ-DOC-015.** the tool's intended use, non-claims, security and data-governance boundaries, and known limitations; and +- **REQ-DOC-016.** the toolkit, comparison-configuration-schema, and results-schema versions to which the documentation applies. -- run on proposed changes and protected branches; -- install the built package, not rely solely on the source checkout; -- test the console entry point; -- test the supported Python matrix; -- fail on enforced lint, type, test, and build errors; -- validate that generated schemas and examples are current; -- avoid publishing releases when required checks fail. +**REQ-DOC-002.** The guide MUST include complete, runnable examples covering the bundled configuration and representative custom configuration features. + +**REQ-DOC-003.** Configuration examples MUST validate against the published comparison-configuration schema. Results examples MUST validate against the published results schema. + +**REQ-DOC-004.** Documentation MUST distinguish machine-enforced schema constraints from explanatory guidance. It MUST accurately describe the behaviour of the documented toolkit version and MUST NOT contradict the machine-readable schemas it supports. Any material inconsistency is a release-blocking documentation defect. + +**REQ-DOC-005.** Command-line `--help` output MUST provide a concise description of every command and option. It MUST direct users to the full guide. ### 13.5 Releases and change control -Each release SHOULD provide: +**REQ-REL-001.** Each release SHOULD provide: + +- **REQ-REL-006.** an immutable semantic version; +- **REQ-REL-007.** release notes and known limitations; +- **REQ-REL-008.** compatibility statements for comparison-configuration and results schemas; +- **REQ-REL-009.** documented change-impact assessment; +- **REQ-REL-010.** signed or otherwise attestable source and build artefacts where practical; +- **REQ-REL-011.** a reproducible lockfile or equivalent dependency record; +- **REQ-REL-012.** evidence that required verification tests passed. + +**REQ-REL-002.** Comparison-configuration schema versions, results-schema versions, and toolkit versions MUST be independent. -- an immutable semantic version; -- release notes and known limitations; -- compatibility statements for comparison-configuration and report schemas; -- documented change-impact assessment; -- signed or otherwise attestable source and build artefacts where practical; -- a reproducible lockfile or equivalent dependency record; -- evidence that required verification tests passed. +**REQ-REL-003.** Each toolkit release MUST document the comparison-configuration schema versions it supports. It MUST document the results schema versions it supports. -Comparison-configuration schema versions and toolkit versions MUST be independent. Each toolkit release MUST document the comparison-configuration-schema and report-schema versions it supports. An unsupported schema version MUST be rejected clearly rather than interpreted approximately. The project MUST publish migration and compatibility rules before changing either schema and MUST provide a migration path or documented transition period for any incompatible change. +**REQ-REL-004.** An unsupported schema version MUST be rejected clearly rather than interpreted approximately. + +**REQ-REL-005.** The project MUST publish migration and compatibility rules before changing either schema. It MUST provide a migration path or documented transition period for any incompatible change. ## 14. Acceptance-test catalogue -This catalogue summarises minimum executable scenarios for the first release. It verifies, but does not independently define, the normative behaviour above. When normative requirement identifiers are assigned, every scenario MUST reference the requirements it verifies. +This catalogue summarises minimum executable scenarios for the first release. It verifies, but does not independently define, the normative behaviour above. Every scenario lists the requirements it verifies. ### 14.1 Input and contract scenarios - **AT-CON-001:** Identical typed datasets satisfy their contract. -- **AT-CON-002:** Required missing fields produce contract findings in both directions and across heterogeneous records; optional missing fields do not. Observed undeclared fields are handled according to each `fields.undeclared` mode and aligned only by identical input names under `compare`. -- **AT-CON-003:** Leading-zero identifiers remain strings unless explicitly transformed. -- **AT-CON-004:** Null, empty string, whitespace-only string, absent field, and configured missing tokens remain distinct according to policy. -- **AT-CON-005:** Unknown, conflicting, and incomplete mappings fail clearly; mapping to null differs from a missing map entry. -- **AT-CON-006:** Malformed syntax or structure, empty or duplicate column names, and irregular tabular rows receive their specified error classification without silent data loss. + - Verifies: `REQ-VAL-003`, `REQ-VAL-004`, `REQ-CMP-006`, `REQ-RUN-002` +- **AT-CON-002:** Required missing fields produce contract findings in both directions and across heterogeneous records; optional missing fields do not. Observed undeclared fields are handled according to each `fields.undeclared` mode and aligned only by identical input names under `compare`. This consolidated parameterized scenario also covers: `fields.undeclared` is required and accepts only `compare`, `exclude`, or `fail`; omission and unknown policies are rejected. Every declared field supplies non-empty `source.reference` and `source.query` names, including when the field is optional on either side; the `required` flags govern record-level presence rather than whether a correspondence is declared. + - Verifies: `REQ-CFG-030`, `REQ-CFG-031`, `REQ-CFG-037`, `REQ-CFG-038`, `REQ-CFG-039`, `REQ-CMP-006`, `REQ-CMP-007`, `REQ-CMP-034`, `REQ-CMP-035`, `REQ-CMP-036`, `REQ-FND-010`, `REQ-FND-011`, `REQ-VAL-002`, `REQ-CFG-035`, `REQ-CFG-036` +- **AT-CON-004:** Null, empty string, whitespace-only string, absent field, and configured missing tokens remain distinct according to policy. This consolidated parameterized scenario also covers: The bundled strict-exact configuration treats parser-native null as equal only to null and distinguishes null, absent fields, and empty strings. + - Verifies: `REQ-CFG-053`, `REQ-CFG-054`, `REQ-CFG-055`, `REQ-CFG-056`, `REQ-CFG-057`, `REQ-VAL-014`, `REQ-VAL-047`, `REQ-VAL-048`, `REQ-VAL-049`, `REQ-VAL-050`, `REQ-VAL-051`, `REQ-VAL-052`, `REQ-VAL-053`, `REQ-VAL-054`, `REQ-VAL-055`, `REQ-INP-028`, `REQ-CFG-113` +- **AT-CON-005:** Unknown, conflicting, and incomplete mappings fail clearly; mapping to null differs from a missing map entry. This consolidated parameterized scenario also covers: Source-value descriptors distinguish strings, numbers, booleans, and null; mapping to null remains distinct from an absent mapping entry. Parameterized mapping fixtures verify reference/query-to-canonical field mapping, required-field reporting, undeclared-field handling, declared source and destination types, permitted intentional many-to-one mappings, and rejection of accidental non-one-to-one mappings. + - Verifies: `REQ-CFG-065`, `REQ-CFG-071`, `REQ-CFG-072`, `REQ-CFG-073`, `REQ-CFG-074`, `REQ-VAL-058`, `REQ-VAL-065`, `REQ-VAL-067`, `REQ-CFG-058`, `REQ-CFG-059`, `REQ-VAL-068`, `REQ-VAL-056`, `REQ-VAL-057`, `REQ-VAL-059`, `REQ-VAL-060`, `REQ-VAL-064`, `REQ-VAL-066` +- **AT-CON-006:** Malformed syntax or structure, empty or duplicate column names, and irregular tabular rows receive their specified error classification without silent data loss. This consolidated parameterized scenario also covers: A table-driven parser suite verifies the fixed CSV and TSV dialects, UTF-8-only decoding, CRLF/LF equivalence, quoted delimiters, doubled quotes, embedded line breaks, source-row retention, and initial string preservation. + - Verifies: `REQ-INP-021`, `REQ-INP-022`, `REQ-INP-023`, `REQ-INP-024`, `REQ-INP-025`, `REQ-INP-034`, `REQ-INP-035`, `REQ-INP-036`, `REQ-INP-037`, `REQ-INP-038`, `REQ-CMP-028`, `REQ-CMP-029`, `REQ-PRN-002`, `REQ-PRN-008`, `REQ-INP-010`, `REQ-INP-011`, `REQ-INP-012`, `REQ-INP-014`, `REQ-INP-015`, `REQ-INP-016`, `REQ-INP-017`, `REQ-INP-018`, `REQ-INP-019`, `REQ-INP-020`, `REQ-INP-026`, `REQ-INP-027`, `REQ-INP-063`, `REQ-INP-064` - **AT-CON-007:** For two syntactically valid zero-record datasets, the records phase reports no missing or extra records and the values phase performs no comparisons under every configuration. Independently assessable contract requirements are still evaluated, and requirements that depend on examining a record are `not_assessable`. One-sided empty datasets produce all applicable missing or extra record findings. + - Verifies: `REQ-INP-052`, `REQ-INP-053`, `REQ-INP-054`, `REQ-INP-055`, `REQ-INP-056`, `REQ-INP-057`, `REQ-INP-058`, `REQ-INP-059`, `REQ-CMP-009`, `REQ-CMP-010`, `REQ-CFG-114` - **AT-CON-008:** Comparison-configuration YAML scalar resolution and cross-format typed dataset parsing are deterministic across supported environments. -- **AT-CON-009:** The bundled strict-exact configuration treats parser-native null as equal only to null and distinguishes null, absent fields, and empty strings. -- **AT-CON-010:** `fields.undeclared` is required and accepts only `compare`, `exclude`, or `fail`; omission and unknown policies are rejected. + - Verifies: `REQ-CFG-006`, `REQ-INP-039`, `REQ-INP-040`, `REQ-INP-041`, `REQ-INP-042`, `REQ-INP-043`, `REQ-INP-044`, `REQ-DEP-013` - **AT-CON-011:** A complete custom YAML configuration conforming to Section 6.2 validates; omitted optional processing properties and dataset sides expand to empty internal structures in the effective configuration, while explicitly empty placeholder lists or mappings are rejected. -- **AT-CON-012:** Duplicate or unknown YAML keys, unsafe YAML features, unsupported schema versions, invalid enumerations, and incompatible type/comparator combinations are rejected with precise configuration paths. -- **AT-CON-013:** Source-value descriptors distinguish strings, numbers, booleans, and null; mapping to null remains distinct from an absent mapping entry. -- **AT-CON-014:** The authoritative schema is present in the source tree and built distribution, can be loaded through the installed package-resource API, and validates every complete configuration example and controlled configuration fixture. + - Verifies: `REQ-CFG-003`, `REQ-CFG-005`, `REQ-CFG-014`, `REQ-CFG-060`, `REQ-CFG-061`, `REQ-CFG-062`, `REQ-CFG-063`, `REQ-CFG-069`, `REQ-CFG-070`, `REQ-CFG-075`, `REQ-CFG-076` +- **AT-CON-012:** Duplicate or unknown YAML keys, unsafe YAML features, unsupported schema versions, invalid enumerations, and incompatible type/comparator combinations are rejected with precise configuration paths. This consolidated parameterized scenario also covers: Safe-loader tests confirm that configuration parsing cannot construct arbitrary objects or execute tagged payloads. + - Verifies: `REQ-CFG-007`, `REQ-CFG-008`, `REQ-CFG-009`, `REQ-CFG-010`, `REQ-CFG-013`, `REQ-CFG-017`, `REQ-CFG-048`, `REQ-CFG-051`, `REQ-CFG-052`, `REQ-CFG-065`, `REQ-CFG-078`, `REQ-CFG-079`, `REQ-CFG-080`, `REQ-CFG-085`, `REQ-CFG-091`, `REQ-SEC-012` +- **AT-CON-014:** The authoritative schema is present in the source tree and built distribution, can be loaded through the installed package-resource API, and validates every complete configuration example and controlled configuration fixture. This consolidated parameterized scenario also covers: The packaged results schema resolves its configuration-schema reference, accepts representative complete and incomplete canonical results, rejects unknown properties and execution-error diagnostics, and requires exactly the documented top-level result properties. + - Verifies: `REQ-CFG-016`, `REQ-CFG-087`, `REQ-CFG-088`, `REQ-OUT-004`, `REQ-OUT-005`, `REQ-OUT-006`, `REQ-OUT-007`, `REQ-OUT-050`, `REQ-OUT-051`, `REQ-OUT-052`, `REQ-OUT-053`, `REQ-OUT-054`, `REQ-OUT-055`, `REQ-OUT-056`, `REQ-OUT-057`, `REQ-OUT-058`, `REQ-OUT-059`, `REQ-OUT-060` - **AT-CON-015:** Configurations that are structurally schema-valid but violate a semantic rule in Section 6.2.9 are rejected with the relevant configuration path and rule. + - Verifies: `REQ-CFG-090`, `REQ-CFG-091`, `REQ-CFG-104`, `REQ-CFG-105`, `REQ-CFG-106`, `REQ-CFG-107`, `REQ-CFG-108` - **AT-CON-016:** `review_priority` accepts only `low`, `normal`, or `high`; omission resolves to `normal` in the effective configuration. + - Verifies: `REQ-CFG-040` - **AT-CON-017:** `fields.record_key` is required, non-empty, ordered, and free of duplicate component fields; every entry names a field in `fields.definitions` that satisfies all key-field constraints. -- **AT-CON-018:** `metadata.id` and `metadata.revision` are required non-empty strings; title and description are optional, metadata does not alter comparison behaviour, and every metadata change alters the configuration digest. -- **AT-CON-019:** Every declared field supplies non-empty `source.reference` and `source.query` names, including when the field is optional on either side; the `required` flags govern record-level presence rather than whether a correspondence is declared. + - Verifies: `REQ-CFG-024`, `REQ-CFG-025`, `REQ-CFG-026`, `REQ-CFG-027`, `REQ-CFG-029`, `REQ-CFG-034`, `REQ-CFG-104`, `REQ-CFG-105` +- **AT-CON-018:** `metadata.id` and `metadata.revision` are required non-empty strings; title and description are optional, and metadata does not alter comparison behaviour. + - Verifies: `REQ-CFG-018`, `REQ-CFG-020`, `REQ-CFG-022` +- **AT-CON-021:** A table-driven JSON parser suite verifies the record-oriented root structure, preserved string-like tokens and numerical lexical forms, and the distinct finding, `not_assessable`, continuation, and execution-error outcomes for non-finite values in non-key fields and record keys. + - Verifies: `REQ-INP-029`, `REQ-INP-030`, `REQ-INP-033`, `REQ-INP-047`, `REQ-INP-048`, `REQ-INP-049`, `REQ-INP-050`, `REQ-INP-051`, `REQ-INP-065` +- **AT-CON-022:** Every comparison resolves one effective configuration whose schema version, identifier, revision, record key, field definitions, undeclared-field policy, and comparison rules are validated before either dataset is read; unknown keys fail with actionable locations, and the validated configuration is embedded in results. + - Verifies: `REQ-CFG-001`, `REQ-CFG-002`, `REQ-CFG-012`, `REQ-CFG-086`, `REQ-CFG-095`, `REQ-CFG-096`, `REQ-CFG-097`, `REQ-CFG-098`, `REQ-CFG-099`, `REQ-CFG-100`, `REQ-CFG-101`, `REQ-CFG-102`, `REQ-CFG-103` +- **AT-CON-023:** Parameterized configuration fixtures verify canonical field-name mapping, collection logical-type structure, legal omission of `logical_type` and operation parameters, placement and side-specific application of source and typed normalisation, and the required comparator declaration. + - Verifies: `REQ-CFG-028`, `REQ-CFG-032`, `REQ-CFG-033`, `REQ-CFG-049`, `REQ-CFG-050`, `REQ-CFG-064`, `REQ-CFG-066`, `REQ-CFG-067`, `REQ-CFG-077` +- **AT-CON-024:** The bundled configuration is schema-valid and implements the documented strict-exact behavior: `record_id` is the key on both sides, shared non-key fields require no declarations, reference-only and query-only fields become contract findings, and cross-format parser-native type differences remain distinguishable. This consolidated parameterized scenario also covers: `preserved_exact` follows the parser-native category table, including deterministic same-format and cross-format comparisons. + - Verifies: `REQ-CFG-093`, `REQ-CFG-094`, `REQ-CFG-109`, `REQ-CFG-110`, `REQ-CFG-115`, `REQ-INP-039`, `REQ-INP-046`, `REQ-CFG-111`, `REQ-CFG-112`, `REQ-CMP-032` ### 14.2 Record scenarios - **AT-REC-001:** Every reference-only record is reported as missing and every query-only record as extra, including when both occur together. + - Verifies: `REQ-INP-058`, `REQ-INP-059`, `REQ-CMP-011`, `REQ-CMP-012`, `REQ-FND-008`, `REQ-FND-009`, `REQ-PRN-003` - **AT-REC-002:** Duplicate single and composite keys produce execution errors without overwriting data; each error identifies the affected dataset, duplicate normalised key, and available source-record locations. + - Verifies: `REQ-REC-003`, `REQ-REC-009`, `REQ-REC-010`, `REQ-REC-011`, `REQ-REC-012`, `REQ-REC-013`, `REQ-REC-014` - **AT-REC-003:** Composite keys preserve multiple legitimate records belonging to one sample. -- **AT-REC-004:** Record order does not alter findings, metrics, or run and phase statuses. + - Verifies: `REQ-REC-001`, `REQ-REC-002`, `REQ-CFG-024` +- **AT-REC-005:** Parameterized key fixtures verify that key names, aliases, composites, and normalisation are configuration-defined; key values are not implicitly converted; no command-line key override exists; and findings retain enough source identity to distinguish records. + - Verifies: `REQ-REC-004`, `REQ-REC-005`, `REQ-REC-006`, `REQ-REC-007`, `REQ-REC-008`, `REQ-REC-015` ### 14.3 Typed value and comparator scenarios -- **AT-VAL-001:** Boolean `true`, integer `1`, decimal `1.0`, and textual `"1"` remain distinct under the applicable exact comparator. +- **AT-VAL-001:** Boolean `true`, integer `1`, decimal `1.0`, and textual `"1"` remain distinct under the applicable exact comparator. The parameterized suite also verifies declared parsing rather than inference and enumerates the stable comparator registry. + - Verifies: `REQ-VAL-009`, `REQ-VAL-027`, `REQ-CMP-017`, `REQ-VAL-001`, `REQ-VAL-005`, `REQ-VAL-007`, `REQ-VAL-008`, `REQ-CMP-015` - **AT-VAL-002:** Typed strings remain code-point-exact unless configured normalisations are applied in declared order. -- **AT-VAL-003:** Integer parsing accepts signed ASCII decimal integers and leading zeros but rejects decimal points, exponents, booleans, and invalid text. -- **AT-VAL-004:** Decimal parsing is exact, accepts scientific notation, and rejects `NaN` and infinities. + - Verifies: `REQ-INP-013`, `REQ-VAL-012`, `REQ-VAL-013`, `REQ-VAL-070`, `REQ-CMP-016` +- **AT-VAL-003:** Integer parsing accepts signed ASCII decimal integers and leading zeros but rejects decimal points, exponents, booleans, and invalid text. This consolidated parameterized scenario also covers: Leading-zero identifiers remain strings unless explicitly transformed. + - Verifies: `REQ-VAL-015`, `REQ-VAL-016`, `REQ-VAL-017`, `REQ-VAL-018` +- **AT-VAL-004:** Decimal parsing is exact, accepts scientific notation, and rejects `NaN` and infinities. Invalid attributable non-key decimals are also checked as contract findings that become `not_assessable`, not execution errors. + - Verifies: `REQ-INP-031`, `REQ-INP-032`, `REQ-INP-045`, `REQ-INP-046`, `REQ-VAL-019`, `REQ-VAL-020`, `REQ-VAL-021`, `REQ-VAL-022`, `REQ-VAL-023`, `REQ-VAL-011` - **AT-VAL-005:** Boolean parsing accepts native booleans and lowercase textual `true` or `false`; other representations require a mapping. + - Verifies: `REQ-VAL-024`, `REQ-VAL-025`, `REQ-VAL-026`, `REQ-VAL-027` - **AT-VAL-006:** Dates reject invalid or non-ISO full dates; datetimes require offsets and compare as instants. -- **AT-VAL-007:** Lists preserve order and duplicates, sets reject duplicates and ignore order, and multisets ignore order while preserving occurrence counts. -- **AT-VAL-008:** Typed transformations follow Section 6.8 and retain evidence of the attributable source value. -- **AT-VAL-009:** Numeric tolerance uses exact decimal arithmetic, a reference-relative component, and an inclusive boundary, including below, at, and above boundary cases. + - Verifies: `REQ-VAL-028`, `REQ-VAL-029`, `REQ-VAL-030`, `REQ-VAL-031`, `REQ-VAL-032`, `REQ-VAL-033`, `REQ-VAL-034`, `REQ-VAL-035`, `REQ-VAL-036`, `REQ-CMP-024` +- **AT-VAL-007:** Lists preserve order and duplicates, sets reject duplicates and ignore order, and multisets ignore order while preserving occurrence counts. This consolidated parameterized scenario also covers: Collection parsing accepts JSON arrays in JSON and JSON-array cell syntax in CSV/TSV, rejects ordinary delimited text, and rejects null collection elements with the specified finding and `not_assessable` evidence. The suite covers every supported typed collection kind and its required element declaration. + - Verifies: `REQ-VAL-037`, `REQ-VAL-038`, `REQ-VAL-039`, `REQ-VAL-040`, `REQ-VAL-041`, `REQ-VAL-042`, `REQ-CMP-025`, `REQ-VAL-043`, `REQ-VAL-044`, `REQ-VAL-045`, `REQ-VAL-046`, `REQ-VAL-006` +- **AT-VAL-008:** Typed transformations follow Section 6.8 and retain evidence of the attributable source value. This consolidated parameterized scenario also covers: Parameterized normalisation fixtures exercise every registered source-text and typed-value operation, including Unicode, whitespace, case, datetime offset, precision, and unordered-collection ordering, and verify that rounding or significant-figure changes occur only when explicitly configured. The suite also verifies retention of source lexical evidence. + - Verifies: `REQ-VAL-061`, `REQ-VAL-062`, `REQ-VAL-063`, `REQ-VAL-069`, `REQ-VAL-070`, `REQ-VAL-072`, `REQ-VAL-073`, `REQ-VAL-074`, `REQ-VAL-075`, `REQ-VAL-071`, `REQ-CMP-023`, `REQ-VAL-010` +- **AT-VAL-009:** Numeric tolerance accepts exactly one absolute or relative tolerance and rejects configurations containing neither or both. Each form uses exact decimal arithmetic and an inclusive boundary, with below, at, and above boundary cases; relative tolerance is reference-directed and permits no non-zero difference when the reference is zero. + - Verifies: `REQ-CFG-081`, `REQ-CFG-082`, `REQ-CFG-083`, `REQ-CFG-084`, `REQ-CMP-018`, `REQ-CMP-019`, `REQ-CMP-020`, `REQ-CMP-021`, `REQ-CMP-022` - **AT-VAL-010:** All assessable differences are collected in one run. + - Verifies: `REQ-CMP-026`, `REQ-CMP-027` - **AT-VAL-011:** Every defined field is included in comparison; under `fields.undeclared: "exclude"`, undeclared fields are `not_assessed` and represented by the required exclusion summary. -- **AT-VAL-012:** `preserved_exact` follows the parser-native category table, including deterministic same-format and cross-format comparisons. + - Verifies: `REQ-CFG-039`, `REQ-CMP-004`, `REQ-CMP-013`, `REQ-CMP-031`, `REQ-OUT-041` ### 14.4 Phase and run-status scenarios - **AT-OUT-001:** Every comparison performs the contract, records, and values phases in order and records a completion status for each. + - Verifies: `REQ-CMP-001`, `REQ-RUN-012`, `REQ-OUT-013` - **AT-OUT-002:** A finding or `not_assessable` item in an earlier phase does not prevent later assessment where prerequisites remain satisfied. -- **AT-OUT-003:** Content excluded deliberately is `not_assessed`; configured content blocked by a failed prerequisite is `not_assessable`. -- **AT-OUT-004:** A fully performed comparison is `complete` whether it finds no differences or any combination of contract findings, missing records, extra records, and value mismatches. -- **AT-OUT-005:** Complete, incomplete, invocation or input error, and unexpected internal error produce their documented exit behaviour. -- **AT-OUT-006:** Truncation produces `incomplete`, identifies unexamined content, and makes no claims about it. -- **AT-OUT-007:** Configured `not_assessable` content produces `incomplete`; deliberately `not_assessed` content does not. -- **AT-OUT-008:** Invalid configuration, malformed or unreadable input, and an incomplete or ambiguous record model produce `error`, never `incomplete`. + - Verifies: `REQ-CMP-002`, `REQ-CMP-003`, `REQ-RUN-007` +- **AT-OUT-005:** Complete, incomplete, invocation or input error, and unexpected internal error produce their documented exit behaviour. This consolidated parameterized scenario also covers: A fully performed comparison is `complete` whether it finds no differences or any combination of contract findings, missing records, extra records, and value mismatches. + - Verifies: `REQ-RUN-001`, `REQ-RUN-015`, `REQ-RUN-016`, `REQ-RUN-017`, `REQ-CLI-014`, `REQ-CLI-015`, `REQ-CLI-016`, `REQ-RUN-002` +- **AT-OUT-006:** Configured `not_assessable` content produces `incomplete`; deliberately `not_assessed` content does not. This consolidated parameterized scenario also covers: Content excluded deliberately is `not_assessed`; configured content blocked by a failed prerequisite is `not_assessable`. + - Verifies: `REQ-RUN-003`, `REQ-RUN-004`, `REQ-RUN-005`, `REQ-RUN-016`, `REQ-CMP-004` +- **AT-OUT-007:** Invalid configuration, malformed or unreadable input, an incomplete or ambiguous record model, an enforced resource limit, and resource exhaustion produce `error`, never `incomplete` or a partial comparison result. + - Verifies: `REQ-RUN-017`, `REQ-RUN-010`, `REQ-RUN-011`, `REQ-DEP-011` +- **AT-OUT-008:** An end-to-end mixed-defect comparison exercises every conceptual processing step, including invalid contract values, deliberate exclusions, failed prerequisites, mandatory key processing, and continued comparison of unaffected values. This consolidated parameterized scenario also covers: A large mixed-content fixture verifies that every configured value opportunity is either assessed or explicitly classified and that no valid accepted content is silently skipped. + - Verifies: `REQ-CMP-005`, `REQ-CMP-008`, `REQ-CMP-014`, `REQ-CMP-030`, `REQ-CMP-037`, `REQ-CMP-038`, `REQ-CMP-039`, `REQ-RUN-006`, `REQ-DEP-010` +- **AT-OUT-010:** A table-driven finding suite generates every finding code and verifies its structured fields, required contextual evidence, deterministic local ID, human-readable message, summary-only successful matches, and absence of message-only semantics. This consolidated parameterized scenario also covers: For every finding code, result fixtures enforce the code-specific inclusion and omission of source and canonical values, deterministic messages derived from structured data, deterministic sorting, and sequential report-local finding IDs. + - Verifies: `REQ-PRN-006`, `REQ-FND-001`, `REQ-FND-002`, `REQ-FND-003`, `REQ-FND-004`, `REQ-FND-005`, `REQ-FND-006`, `REQ-FND-012`, `REQ-FND-013`, `REQ-FND-014`, `REQ-FND-015`, `REQ-FND-016`, `REQ-FND-017`, `REQ-OUT-032`, `REQ-OUT-033`, `REQ-OUT-035`, `REQ-OUT-061`, `REQ-OUT-062`, `REQ-OUT-063`, `REQ-OUT-064`, `REQ-OUT-065`, `REQ-OUT-066`, `REQ-OUT-067`, `REQ-OUT-068` ### 14.5 Evidence, CLI, and packaging scenarios -- **AT-EVD-001:** Identical runs produce semantically identical canonical results except for explicitly variable run metadata. -- **AT-EVD-002:** Reports contain input and comparison-configuration digests without exposing absolute paths, raw command lines, environment variables, or secrets. +- **AT-EVD-001:** Identical runs produce semantically identical canonical results except for explicitly variable run metadata. This consolidated parameterized scenario also covers: Record order does not alter findings, metrics, or run and phase statuses. + - Verifies: `REQ-PRN-005`, `REQ-CMP-033`, `REQ-OUT-012`, `REQ-DEP-013`, `REQ-CMP-032` +- **AT-EVD-002:** Results contain exact-byte SHA-256 input digests and input basenames without exposing absolute paths, raw command lines, environment variables, or secrets; the complete validated comparison configuration is embedded without a separate configuration digest. + - Verifies: `REQ-CFG-023`, `REQ-OUT-017`, `REQ-OUT-018`, `REQ-OUT-019`, `REQ-OUT-022`, `REQ-OUT-023`, `REQ-SEC-007`, `REQ-SEC-008` - **AT-EVD-003:** If the optional CSV findings export is provided, findings containing commas, quotes, newlines, or formula prefixes are exported safely. -- **AT-EVD-004:** An output failure or interrupted write cannot leave an evidence package that appears complete. + - Verifies: `REQ-OUT-043`, `REQ-OUT-083`, `REQ-OUT-084`, `REQ-OUT-085`, `REQ-OUT-087`, `REQ-SEC-013` +- **AT-EVD-004:** An output failure or interrupted write cannot leave an evidence package that appears complete. This consolidated parameterized scenario also covers: Success and injected-failure runs verify that temporary files are no more accessible than the destination and are removed after success and, where safe, after errors. + - Verifies: `REQ-OUT-044`, `REQ-OUT-045`, `REQ-OUT-049`, `REQ-DEP-012`, `REQ-SEC-009`, `REQ-SEC-010`, `REQ-SEC-011` - **AT-EVD-005:** Every complete comparison exits with code 0 and produces its evidence package, including when differences are reported. -- **AT-EVD-006:** Omitting `--configuration` selects and records the bundled strict-exact configuration; a supplied value is treated only as a custom comparison-configuration path. -- **AT-EVD-007:** Explicit reference and query formats override extension detection; recognised extensions are detected and recorded; unknown or ambiguous extensions require explicit formats. -- **AT-EVD-008:** A complete or incomplete run creates the documented JSON and HTML evidence package, refuses an existing destination by default, and replaces it only with `--overwrite`. -- **AT-EVD-009:** When an output location can be established and an error report can be written, invocation, configuration, input, and output errors produce a clearly identified error report. -- **AT-EVD-010:** Routine logs, progress output, and terminal diagnostics omit record identifiers and source values. + - Verifies: `REQ-PRN-004`, `REQ-RUN-002`, `REQ-OUT-001`, `REQ-OUT-002`, `REQ-CLI-016` +- **AT-EVD-008:** A complete or incomplete run creates the documented JSON and HTML evidence package, refuses an existing destination by default, and replaces it only with `--overwrite`. This consolidated parameterized scenario also covers: Output-directory reporting and filename handling preserve the exact selected destination and manipulate suffixes structurally rather than by character stripping. + - Verifies: `REQ-OUT-001`, `REQ-OUT-002`, `REQ-OUT-046`, `REQ-CLI-011`, `REQ-CLI-012`, `REQ-OUT-047`, `REQ-OUT-048` +- **AT-EVD-009:** When an output location can be established and an error report can be written, invocation, configuration, input, resource, output, and internal errors produce a clearly identified diagnostic artefact with a stable execution error code. It contains no findings, finding codes, comparison summary metrics, or phase statuses and cannot be mistaken for a comparison result or evidence package. + - Verifies: `REQ-FND-007`, `REQ-RUN-008`, `REQ-RUN-009`, `REQ-RUN-010`, `REQ-RUN-011`, `REQ-OUT-008`, `REQ-DEP-014` +- **AT-EVD-010:** Routine logs, progress output, terminal diagnostics, and execution-error diagnostics omit record identifiers and source values. + - Verifies: `REQ-PRN-007`, `REQ-SEC-004`, `REQ-VAL-075` - **AT-EVD-011:** Evidence packages warn that they may contain sensitive information, and findings contain record identifiers or source values only where needed to identify or explain the finding. + - Verifies: `REQ-PRN-007`, `REQ-OUT-030`, `REQ-OUT-031`, `REQ-SEC-005`, `REQ-SEC-006` - **AT-EVD-012:** Core comparison completes without network access and makes no attempt to transmit inputs, comparison configurations, logs, or results. + - Verifies: `REQ-SEC-002`, `REQ-SEC-003`, `REQ-DEP-001`, `REQ-DEP-002` - **AT-EVD-013:** Findings attributable to declared fields inherit their resolved review priority; other findings do not. HTML presentation may distinguish priorities but does not rely on colour alone, and priority never changes findings, counts, status, or exit code. -- **AT-PKG-001:** The built distribution is named `adviso-validation`, contains an importable `adviso_validation` package and every required subpackage, installs the `adviso-validation` executable, and completes an end-to-end comparison through that executable. - -## 15. Compatibility with the current prototype - -The current `File_IO`, `FileCompare`, and `TableValidator` module names and APIs MUST NOT constrain the new core data model or public terminology. In particular, the new model MUST NOT use a single dictionary keyed only by sample ID as its sole record representation. - -Compatibility MAY be provided temporarily through an adapter, provided that: - -- legacy semantics are labelled clearly; -- unsafe behaviour such as silent duplicate overwrite is not preserved; -- the adapter has a documented removal policy; -- new evidence reports identify use of legacy compatibility mode. - -The current free-form result strings MUST NOT become the canonical result API. - -## 16. Decisions required before approval - -The following non-kernel decisions remain open. They do not prevent implementation of the comparison kernel, but MUST be resolved at the appropriate point before the specification and first release are approved. - -1. Which provenance fields are mandatory in local interactive use, CI, and formal retained evidence? -2. What compatibility period, if any, is required for the prototype CLI and outputs? - -## 17. Suggested approval criteria for this specification - -Before implementation begins, reviewers SHOULD confirm that: - -- the intended use and non-claims are accurate; -- terminology matches laboratory practice; -- the record model covers representative pipeline outputs; -- missing- and extra-content policies, and the handling of duplicate, null, and malformed data, are unambiguous; -- comparison evidence, run status, and the laboratory decision are clearly separated; -- data-handling and provenance requirements fit local governance; -- every first-milestone scenario has an agreed expected result; -- unresolved decisions have named owners and target dates; -- approved requirements have stable identifiers for test traceability. + - Verifies: `REQ-CFG-041`, `REQ-CFG-042`, `REQ-CFG-043`, `REQ-CFG-044`, `REQ-CFG-045`, `REQ-OUT-034` +- **AT-PKG-001:** The built distribution is named `adviso-validation`, contains an importable `adviso_validation` package and every required subpackage, installs the `adviso-validation` executable, and completes an end-to-end comparison through that executable. This consolidated parameterized scenario also covers: The release artefact can be installed and used without network access, contains offline documentation and integrity information, and supports an end-to-end comparison in an isolated environment. + - Verifies: `REQ-PRD-001`, `REQ-PRD-003`, `REQ-CFG-087`, `REQ-TST-013`, `REQ-TST-030`, `REQ-DEP-003`, `REQ-DEP-007` +- **AT-EVD-015:** Canonical results contain valid run, phase, software, runtime, and fixed-dialect metadata; timestamps are UTC RFC 3339 values; start time and duration are absent; and overall status reconciles with phase statuses. + - Verifies: `REQ-OUT-009`, `REQ-OUT-010`, `REQ-OUT-011`, `REQ-OUT-014`, `REQ-OUT-015`, `REQ-OUT-016`, `REQ-OUT-020` +- **AT-EVD-016:** Parameterized single and composite record identities verify ordered source and canonical key arrays, null identities for an absent side, presence-state distinctions, recursive parser-native typed values, canonical scalar and collection encodings, and deterministic object-member ordering. + - Verifies: `REQ-OUT-021`, `REQ-OUT-026`, `REQ-OUT-027`, `REQ-OUT-028`, `REQ-OUT-029` +- **AT-EVD-018:** Parameterized result fixtures verify fixed-shape zero-inclusive summaries, record-count reconciliation, finding totals by code and priority, and exactly one matched, mismatched, or not-assessable classification for every in-scope non-key value opportunity, including optional, required, invalid, and prohibited values. + - Verifies: `REQ-OUT-037`, `REQ-OUT-038`, `REQ-OUT-039`, `REQ-OUT-040`, `REQ-OUT-069`, `REQ-OUT-070`, `REQ-OUT-071`, `REQ-OUT-072`, `REQ-OUT-073` +- **AT-EVD-019:** HTML generated solely from canonical JSON uses an open non-proprietary format and presents the run purpose, phase and run statuses, input and configuration identities, counts, findings, discrepancies, limitations, exclusions, unassessed content, provenance, and no acceptance verdict; HTML is never the sole canonical evidence format. This consolidated parameterized scenario also covers: Complete and incomplete reports contain raw findings and metrics but no acceptance threshold, pass/fail verdict, or equivalent inferred laboratory decision. + - Verifies: `REQ-OUT-042`, `REQ-OUT-074`, `REQ-OUT-075`, `REQ-OUT-076`, `REQ-OUT-077`, `REQ-OUT-078`, `REQ-OUT-079`, `REQ-OUT-080`, `REQ-OUT-081`, `REQ-OUT-088`, `REQ-DEP-004`, `REQ-DEP-005`, `REQ-RUN-013` +- **AT-EVD-021:** Installed-CLI tests cover the required `adviso-validation compare` form, required reference, query, and output arguments, optional configuration, enumerated choices, invalid combinations before input access, normal summaries on standard output, and optional progress that never contaminates machine-readable output. This consolidated parameterized scenario also covers: Omitting `--configuration` selects and records the bundled strict-exact configuration; a supplied value is treated only as a custom comparison-configuration path. The reference and query formats are independently determined and recorded from recognised, case-insensitive filename extensions; a missing, unknown, or ambiguous extension causes an input error, and content that does not conform to the indicated format is rejected. Command-line `--help` describes every version 1 command and option and directs users to the full guide. + - Verifies: `REQ-PRD-002`, `REQ-INP-001`, `REQ-INP-002`, `REQ-INP-004`, `REQ-INP-060`, `REQ-INP-061`, `REQ-INP-062`, `REQ-CLI-001`, `REQ-CLI-002`, `REQ-CLI-003`, `REQ-CLI-006`, `REQ-CLI-007`, `REQ-CLI-008`, `REQ-INP-003`, `REQ-CFG-092`, `REQ-CLI-009`, `REQ-CLI-010`, `REQ-INP-005`, `REQ-INP-006`, `REQ-INP-007`, `REQ-INP-008`, `REQ-INP-009`, `REQ-CLI-013`, `REQ-CLI-004`, `REQ-DOC-005` + +### 14.6 Documentation scenarios + +- **AT-DOC-001:** The built distribution contains an offline user guide covering every topic required by Section 13.4, and the guide identifies the toolkit and schema versions to which it applies. This consolidated parameterized scenario also covers: Every complete comparison-configuration and canonical-results example in the guide validates against its corresponding packaged schema, and every documented command-line invocation uses the version 1 interface. + - Verifies: `REQ-DOC-001`, `REQ-DOC-006`, `REQ-DOC-007`, `REQ-DOC-008`, `REQ-DOC-009`, `REQ-DOC-010`, `REQ-DOC-011`, `REQ-DOC-012`, `REQ-DOC-013`, `REQ-DOC-014`, `REQ-DOC-015`, `REQ-DOC-016`, `REQ-DOC-002`, `REQ-DOC-003`, `REQ-DOC-004` + +### 14.7 Fixture and compatibility scenarios + +- **AT-TRC-002:** Regenerating every generated fixture from its recorded source or generator produces byte-identical or semantically identical controlled test data, as applicable. + - Verifies: `REQ-TST-005` +- **AT-TRC-003:** Every supported and unsupported comparison-configuration and results-schema version is exercised to confirm exact supported-version handling and clear rejection of unsupported versions. + - Verifies: `REQ-REL-004` + +## 15. Status of the current prototype + +**REQ-PRD-011.** The existing prototype was created only to stimulate requirements analysis and has never been used operationally. Its code, command-line interface, outputs, Python APIs, module structure, terminology, and behaviour impose no compatibility requirements on version 1. The version 1 implementation is not required to reuse any prototype code. Version 1 behaviour MUST NOT be derived implicitly from the prototype. diff --git a/src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json b/src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json index 9c59da4..f1302e6 100644 --- a/src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json +++ b/src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json @@ -617,6 +617,7 @@ "type": "object", "additionalProperties": false, "minProperties": 1, + "maxProperties": 1, "properties": { "absolute_tolerance": { "$ref": "#/$defs/nonNegativeDecimalString" diff --git a/src/adviso_validation/schemas/comparison-results-v1.0.schema.json b/src/adviso_validation/schemas/comparison-results-v1.0.schema.json new file mode 100644 index 0000000..050129b --- /dev/null +++ b/src/adviso_validation/schemas/comparison-results-v1.0.schema.json @@ -0,0 +1,1272 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "title": "AdvISO comparison results", + "description": "Normative schema for canonical comparison results schema version 1.0. Execution-error diagnostics are not comparison results and do not validate against this schema.", + "type": "object", + "additionalProperties": false, + "required": [ + "results_schema_version", + "run", + "phases", + "software", + "runtime", + "inputs", + "record_key_fields", + "comparison_configuration", + "summary", + "exclusions", + "findings" + ], + "properties": { + "results_schema_version": { + "const": "1.0" + }, + "run": { + "$ref": "#/$defs/run" + }, + "phases": { + "$ref": "#/$defs/phases" + }, + "software": { + "$ref": "#/$defs/software" + }, + "runtime": { + "$ref": "#/$defs/runtime" + }, + "inputs": { + "$ref": "#/$defs/inputs" + }, + "record_key_fields": { + "$ref": "#/$defs/recordKeyFields" + }, + "comparison_configuration": { + "$ref": "#/$defs/comparisonConfiguration" + }, + "summary": { + "$ref": "#/$defs/summary" + }, + "exclusions": { + "type": "array", + "items": { + "$ref": "#/$defs/exclusion" + } + }, + "findings": { + "type": "array", + "items": { + "$ref": "#/$defs/finding" + } + } + }, + "$defs": { + "nonEmptyString": { + "type": "string", + "minLength": 1 + }, + "nonNegativeInteger": { + "type": "integer", + "minimum": 0 + }, + "phaseStatus": { + "enum": [ + "complete", + "incomplete" + ] + }, + "run": { + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "status", + "completed_at" + ], + "properties": { + "id": { + "type": "string", + "format": "uuid" + }, + "status": { + "$ref": "#/$defs/phaseStatus" + }, + "completed_at": { + "type": "string", + "format": "date-time", + "pattern": "Z$" + } + } + }, + "phases": { + "type": "object", + "additionalProperties": false, + "required": [ + "contract", + "records", + "values" + ], + "properties": { + "contract": { + "$ref": "#/$defs/phaseStatus" + }, + "records": { + "$ref": "#/$defs/phaseStatus" + }, + "values": { + "$ref": "#/$defs/phaseStatus" + } + } + }, + "software": { + "type": "object", + "additionalProperties": false, + "required": [ + "name", + "version" + ], + "properties": { + "name": { + "const": "adviso-validation" + }, + "version": { + "$ref": "#/$defs/nonEmptyString" + }, + "build": { + "$ref": "#/$defs/nonEmptyString" + } + } + }, + "runtime": { + "type": "object", + "additionalProperties": false, + "required": [ + "python", + "platform", + "unicode_version" + ], + "properties": { + "python": { + "$ref": "#/$defs/nonEmptyString" + }, + "platform": { + "$ref": "#/$defs/nonEmptyString" + }, + "unicode_version": { + "$ref": "#/$defs/nonEmptyString" + } + } + }, + "sha256Digest": { + "type": "object", + "additionalProperties": false, + "required": [ + "algorithm", + "value" + ], + "properties": { + "algorithm": { + "const": "sha256" + }, + "value": { + "type": "string", + "pattern": "^[0-9a-f]{64}$" + } + } + }, + "input": { + "type": "object", + "additionalProperties": false, + "required": [ + "basename", + "format", + "encoding", + "size_bytes", + "digest" + ], + "properties": { + "basename": { + "$ref": "#/$defs/nonEmptyString" + }, + "format": { + "enum": [ + "csv", + "tsv", + "json" + ] + }, + "encoding": { + "const": "utf-8" + }, + "size_bytes": { + "$ref": "#/$defs/nonNegativeInteger" + }, + "digest": { + "$ref": "#/$defs/sha256Digest" + } + } + }, + "inputs": { + "type": "object", + "additionalProperties": false, + "required": [ + "reference", + "query" + ], + "properties": { + "reference": { + "$ref": "#/$defs/input" + }, + "query": { + "$ref": "#/$defs/input" + } + } + }, + "nonEmptyStringArray": { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { + "$ref": "#/$defs/nonEmptyString" + } + }, + "recordKeyFields": { + "type": "object", + "additionalProperties": false, + "required": [ + "reference", + "query", + "canonical" + ], + "properties": { + "reference": { + "$ref": "#/$defs/nonEmptyStringArray" + }, + "query": { + "$ref": "#/$defs/nonEmptyStringArray" + }, + "canonical": { + "$ref": "#/$defs/nonEmptyStringArray" + } + } + }, + "comparisonConfiguration": { + "type": "object", + "additionalProperties": false, + "required": [ + "document" + ], + "properties": { + "document": { + "$ref": "comparison-configuration-v1.0.schema.json" + } + } + }, + "recordSummary": { + "type": "object", + "additionalProperties": false, + "required": [ + "reference", + "query", + "matched", + "missing", + "extra" + ], + "properties": { + "reference": { + "$ref": "#/$defs/nonNegativeInteger" + }, + "query": { + "$ref": "#/$defs/nonNegativeInteger" + }, + "matched": { + "$ref": "#/$defs/nonNegativeInteger" + }, + "missing": { + "$ref": "#/$defs/nonNegativeInteger" + }, + "extra": { + "$ref": "#/$defs/nonNegativeInteger" + } + } + }, + "valueSummary": { + "type": "object", + "additionalProperties": false, + "required": [ + "matched", + "mismatched", + "not_assessable" + ], + "properties": { + "matched": { + "$ref": "#/$defs/nonNegativeInteger" + }, + "mismatched": { + "$ref": "#/$defs/nonNegativeInteger" + }, + "not_assessable": { + "$ref": "#/$defs/nonNegativeInteger" + } + } + }, + "findingCountsByCode": { + "type": "object", + "additionalProperties": false, + "required": [ + "missing_record", + "extra_record", + "missing_required_field", + "undeclared_field", + "type_mismatch", + "invalid_field_value", + "null_policy_violation", + "mapping_failure", + "value_mismatch", + "not_assessable" + ], + "properties": { + "missing_record": { + "$ref": "#/$defs/nonNegativeInteger" + }, + "extra_record": { + "$ref": "#/$defs/nonNegativeInteger" + }, + "missing_required_field": { + "$ref": "#/$defs/nonNegativeInteger" + }, + "undeclared_field": { + "$ref": "#/$defs/nonNegativeInteger" + }, + "type_mismatch": { + "$ref": "#/$defs/nonNegativeInteger" + }, + "invalid_field_value": { + "$ref": "#/$defs/nonNegativeInteger" + }, + "null_policy_violation": { + "$ref": "#/$defs/nonNegativeInteger" + }, + "mapping_failure": { + "$ref": "#/$defs/nonNegativeInteger" + }, + "value_mismatch": { + "$ref": "#/$defs/nonNegativeInteger" + }, + "not_assessable": { + "$ref": "#/$defs/nonNegativeInteger" + } + } + }, + "findingCountsByReviewPriority": { + "type": "object", + "additionalProperties": false, + "required": [ + "low", + "normal", + "high", + "unassigned" + ], + "properties": { + "low": { + "$ref": "#/$defs/nonNegativeInteger" + }, + "normal": { + "$ref": "#/$defs/nonNegativeInteger" + }, + "high": { + "$ref": "#/$defs/nonNegativeInteger" + }, + "unassigned": { + "$ref": "#/$defs/nonNegativeInteger" + } + } + }, + "findingSummary": { + "type": "object", + "additionalProperties": false, + "required": [ + "total", + "by_code", + "by_review_priority" + ], + "properties": { + "total": { + "$ref": "#/$defs/nonNegativeInteger" + }, + "by_code": { + "$ref": "#/$defs/findingCountsByCode" + }, + "by_review_priority": { + "$ref": "#/$defs/findingCountsByReviewPriority" + } + } + }, + "summary": { + "type": "object", + "additionalProperties": false, + "required": [ + "records", + "values", + "findings" + ], + "properties": { + "records": { + "$ref": "#/$defs/recordSummary" + }, + "values": { + "$ref": "#/$defs/valueSummary" + }, + "findings": { + "$ref": "#/$defs/findingSummary" + } + } + }, + "exclusion": { + "type": "object", + "additionalProperties": false, + "required": [ + "code", + "side", + "source_field", + "record_count" + ], + "properties": { + "code": { + "const": "undeclared_field_excluded" + }, + "side": { + "enum": [ + "reference", + "query" + ] + }, + "source_field": { + "$ref": "#/$defs/nonEmptyString" + }, + "record_count": { + "$ref": "#/$defs/nonNegativeInteger" + } + } + }, + "recordIdentityValues": { + "type": "array", + "minItems": 1, + "items": { + "type": "string" + } + }, + "nullableRecordIdentityValues": { + "oneOf": [ + { + "$ref": "#/$defs/recordIdentityValues" + }, + { + "type": "null" + } + ] + }, + "recordIdentity": { + "type": "object", + "additionalProperties": false, + "required": [ + "reference", + "query", + "canonical" + ], + "properties": { + "reference": { + "$ref": "#/$defs/nullableRecordIdentityValues" + }, + "query": { + "$ref": "#/$defs/nullableRecordIdentityValues" + }, + "canonical": { + "$ref": "#/$defs/recordIdentityValues" + } + } + }, + "tabularSourceLocation": { + "type": "object", + "additionalProperties": false, + "required": [ + "line_number" + ], + "properties": { + "record_number": { + "type": "integer", + "minimum": 1 + }, + "line_number": { + "type": "integer", + "minimum": 1 + }, + "field": { + "$ref": "#/$defs/nonEmptyString" + } + } + }, + "jsonSourceLocation": { + "type": "object", + "additionalProperties": false, + "required": [ + "json_pointer" + ], + "properties": { + "json_pointer": { + "type": "string", + "pattern": "^(?:/(?:[^~/]|~[01])*)*$" + } + } + }, + "sourceLocation": { + "oneOf": [ + { + "$ref": "#/$defs/tabularSourceLocation" + }, + { + "$ref": "#/$defs/jsonSourceLocation" + } + ] + }, + "nullableSourceLocation": { + "oneOf": [ + { + "$ref": "#/$defs/sourceLocation" + }, + { + "type": "null" + } + ] + }, + "sourceLocations": { + "type": "object", + "additionalProperties": false, + "required": [ + "reference", + "query" + ], + "properties": { + "reference": { + "$ref": "#/$defs/nullableSourceLocation" + }, + "query": { + "$ref": "#/$defs/nullableSourceLocation" + } + } + }, + "sourceValue": { + "oneOf": [ + { + "type": "object", + "additionalProperties": false, + "required": [ + "category", + "value" + ], + "properties": { + "category": { + "enum": [ + "string", + "integer", + "decimal" + ] + }, + "value": { + "type": "string" + } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "category", + "value" + ], + "properties": { + "category": { + "const": "boolean" + }, + "value": { + "type": "boolean" + } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "category" + ], + "properties": { + "category": { + "const": "null" + } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "category", + "value" + ], + "properties": { + "category": { + "const": "array" + }, + "value": { + "type": "array", + "items": { + "$ref": "#/$defs/sourceValue" + } + } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "category", + "value" + ], + "properties": { + "category": { + "const": "object" + }, + "value": { + "type": "object", + "additionalProperties": { + "$ref": "#/$defs/sourceValue" + } + } + } + } + ] + }, + "canonicalIntegerText": { + "type": "string", + "pattern": "^(?:0|-[1-9][0-9]*|[1-9][0-9]*)$" + }, + "canonicalDecimalText": { + "type": "string", + "pattern": "^(?:0|-?(?:[1-9][0-9]*)(?:\\.[0-9]*[1-9])?(?:e-?[1-9][0-9]*)?|-?0\\.[0-9]*[1-9](?:e-?[1-9][0-9]*)?)$" + }, + "canonicalValue": { + "oneOf": [ + { + "type": "object", + "additionalProperties": false, + "required": [ + "logical_type", + "value" + ], + "properties": { + "logical_type": { + "const": "string" + }, + "value": { + "type": "string" + } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "logical_type", + "value" + ], + "properties": { + "logical_type": { + "const": "integer" + }, + "value": { + "$ref": "#/$defs/canonicalIntegerText" + } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "logical_type", + "value" + ], + "properties": { + "logical_type": { + "const": "decimal" + }, + "value": { + "$ref": "#/$defs/canonicalDecimalText" + } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "logical_type", + "value" + ], + "properties": { + "logical_type": { + "const": "boolean" + }, + "value": { + "type": "boolean" + } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "logical_type", + "value" + ], + "properties": { + "logical_type": { + "const": "date" + }, + "value": { + "type": "string", + "format": "date" + } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "logical_type", + "value" + ], + "properties": { + "logical_type": { + "const": "datetime" + }, + "value": { + "type": "string", + "format": "date-time" + } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "logical_type", + "value" + ], + "properties": { + "logical_type": { + "enum": [ + "string", + "integer", + "decimal", + "boolean", + "date", + "datetime" + ] + }, + "value": { + "type": "null" + } + } + }, + { + "$ref": "#/$defs/canonicalCollectionValue" + } + ] + }, + "canonicalCollectionValue": { + "type": "object", + "additionalProperties": false, + "required": [ + "logical_type", + "element_type", + "value" + ], + "properties": { + "logical_type": { + "enum": [ + "list", + "set", + "multiset" + ] + }, + "element_type": { + "enum": [ + "string", + "integer", + "decimal", + "boolean", + "date", + "datetime" + ] + }, + "value": { + "oneOf": [ + { + "type": "array" + }, + { + "type": "null" + } + ] + } + }, + "allOf": [ + { + "if": { + "properties": { + "element_type": { + "const": "boolean" + } + }, + "required": [ + "element_type" + ] + }, + "then": { + "properties": { + "value": { + "items": { + "type": "boolean" + } + } + } + }, + "else": { + "properties": { + "value": { + "items": { + "type": "string" + } + } + } + } + }, + { + "if": { + "properties": { + "element_type": { + "const": "integer" + } + }, + "required": [ + "element_type" + ] + }, + "then": { + "properties": { + "value": { + "items": { + "$ref": "#/$defs/canonicalIntegerText" + } + } + } + } + }, + { + "if": { + "properties": { + "element_type": { + "const": "decimal" + } + }, + "required": [ + "element_type" + ] + }, + "then": { + "properties": { + "value": { + "items": { + "$ref": "#/$defs/canonicalDecimalText" + } + } + } + } + }, + { + "if": { + "properties": { + "element_type": { + "const": "date" + } + }, + "required": [ + "element_type" + ] + }, + "then": { + "properties": { + "value": { + "items": { + "type": "string", + "format": "date" + } + } + } + } + }, + { + "if": { + "properties": { + "element_type": { + "const": "datetime" + } + }, + "required": [ + "element_type" + ] + }, + "then": { + "properties": { + "value": { + "items": { + "type": "string", + "format": "date-time" + } + } + } + } + }, + { + "if": { + "properties": { + "logical_type": { + "const": "set" + }, + "value": { + "type": "array" + } + }, + "required": [ + "logical_type", + "value" + ] + }, + "then": { + "properties": { + "value": { + "uniqueItems": true + } + } + } + } + ] + }, + "fieldEvidence": { + "type": "object", + "additionalProperties": false, + "required": [ + "state" + ], + "properties": { + "state": { + "enum": [ + "present", + "absent" + ] + }, + "source": { + "$ref": "#/$defs/sourceValue" + }, + "canonical": { + "$ref": "#/$defs/canonicalValue" + } + }, + "allOf": [ + { + "if": { + "properties": { + "state": { + "const": "absent" + } + }, + "required": [ + "state" + ] + }, + "then": { + "properties": { + "source": false, + "canonical": false + } + } + } + ] + }, + "findingValues": { + "type": "object", + "additionalProperties": false, + "required": [ + "reference", + "query" + ], + "properties": { + "reference": { + "$ref": "#/$defs/fieldEvidence" + }, + "query": { + "$ref": "#/$defs/fieldEvidence" + } + } + }, + "finding": { + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "code", + "phase", + "message", + "affected_side" + ], + "properties": { + "id": { + "type": "string", + "pattern": "^F[0-9]{6,}$" + }, + "code": { + "enum": [ + "missing_record", + "extra_record", + "missing_required_field", + "undeclared_field", + "type_mismatch", + "invalid_field_value", + "null_policy_violation", + "mapping_failure", + "value_mismatch", + "not_assessable" + ] + }, + "phase": { + "enum": [ + "contract", + "records", + "values" + ] + }, + "message": { + "$ref": "#/$defs/nonEmptyString" + }, + "affected_side": { + "enum": [ + "reference", + "query", + "both" + ] + }, + "record_identity": { + "$ref": "#/$defs/recordIdentity" + }, + "source_locations": { + "$ref": "#/$defs/sourceLocations" + }, + "field": { + "$ref": "#/$defs/nonEmptyString" + }, + "source_field": { + "$ref": "#/$defs/nonEmptyString" + }, + "review_priority": { + "enum": [ + "low", + "normal", + "high" + ] + }, + "values": { + "$ref": "#/$defs/findingValues" + } + }, + "allOf": [ + { + "if": { + "properties": { + "code": { + "const": "missing_record" + } + }, + "required": [ + "code" + ] + }, + "then": { + "required": [ + "record_identity", + "source_locations" + ], + "properties": { + "phase": { + "const": "records" + }, + "affected_side": { + "const": "query" + }, + "record_identity": { + "properties": { + "reference": { + "$ref": "#/$defs/recordIdentityValues" + }, + "query": { + "type": "null" + } + } + } + } + } + }, + { + "if": { + "properties": { + "code": { + "const": "extra_record" + } + }, + "required": [ + "code" + ] + }, + "then": { + "required": [ + "record_identity", + "source_locations" + ], + "properties": { + "phase": { + "const": "records" + }, + "affected_side": { + "const": "reference" + }, + "record_identity": { + "properties": { + "reference": { + "type": "null" + }, + "query": { + "$ref": "#/$defs/recordIdentityValues" + } + } + } + } + } + }, + { + "if": { + "properties": { + "code": { + "enum": [ + "missing_required_field", + "type_mismatch", + "invalid_field_value", + "null_policy_violation", + "mapping_failure" + ] + } + }, + "required": [ + "code" + ] + }, + "then": { + "required": [ + "record_identity", + "source_locations", + "field", + "review_priority", + "values" + ], + "properties": { + "phase": { + "const": "contract" + }, + "affected_side": { + "enum": [ + "reference", + "query", + "both" + ] + } + } + } + }, + { + "if": { + "properties": { + "code": { + "const": "undeclared_field" + } + }, + "required": [ + "code" + ] + }, + "then": { + "required": [ + "source_field" + ], + "properties": { + "phase": { + "const": "contract" + }, + "affected_side": { + "enum": [ + "reference", + "query" + ] + } + } + } + }, + { + "if": { + "properties": { + "code": { + "const": "value_mismatch" + } + }, + "required": [ + "code" + ] + }, + "then": { + "required": [ + "record_identity", + "source_locations", + "field", + "review_priority", + "values" + ], + "properties": { + "phase": { + "const": "values" + }, + "affected_side": { + "const": "both" + } + } + } + }, + { + "if": { + "properties": { + "code": { + "const": "not_assessable" + } + }, + "required": [ + "code" + ] + }, + "then": {} + } + ] + } + } +} From 1d2e08b7c44df811a54c379c3c8d534cfbe2e7f8 Mon Sep 17 00:00:00 2001 From: Kevin Howe Date: Thu, 22 Oct 2026 16:02:42 +0100 Subject: [PATCH 05/45] More refinement of spec and prep for implementation --- AGENTS.md | 158 ++++++ SPECIFICATION.md | 472 ++++++++++++------ .../comparison-configuration-v1.0.schema.json | 159 +++++- .../comparison-results-v1.0.schema.json | 453 +++++++++++++++-- 4 files changed, 1052 insertions(+), 190 deletions(-) create mode 100644 AGENTS.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..749131a --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,158 @@ +# Agent guidance for `validation_toolkit` + +## Purpose and authority + +This repository is being reimplemented as `adviso-validation`, a tool that translates reference and query datasets from their source dialects into a declared canonical representation, compares them, and reports both post-translation differences and assessment issues. + +For version 1 behaviour, use this order of authority: + +1. `SPECIFICATION.md` — normative behaviour and requirement identifiers. +2. `src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json` — normative structural schema for comparison configurations. +3. `src/adviso_validation/schemas/comparison-results-v1.0.schema.json` — normative structural schema for canonical results. +4. Version 1 tests and controlled fixtures, once created — executable verification of the sources above. +5. User documentation — explanatory material that must agree with the normative sources. + +Do not infer version 1 behaviour from the prototype, its README, or its tests. If the specification and a schema appear inconsistent, stop and report the conflict. Do not silently choose one, weaken either, or invent compatibility behaviour. + +## Repository state + +The repository currently contains two generations of work: + +- `SPECIFICATION.md` and `src/adviso_validation/schemas/` define the intended version 1 product. +- `validation_toolkit/`, `tests/table_validator_tests/`, the current `README.md`, and parts of the packaging configuration describe an unused exploratory prototype. + +The prototype has no compatibility status. Its CLI, APIs, output formats, terminology, internal structure, and tests are not version 1 requirements. Do not copy prototype code merely to accelerate implementation. Reuse something only after confirming that it satisfies the version 1 specification and improves the new design. + +Do not delete or rewrite prototype material unless the assigned task or an approved implementation plan explicitly calls for it. Keeping old and new work separate during the transition is preferable to partially converting legacy modules. + +## Product identities + +Preserve these distinct names: + +- Distribution and executable: `adviso-validation` +- Python import package: `adviso_validation` +- Sole version 1 subcommand: `compare` + +New production code belongs under `src/adviso_validation/`. Do not add new production modules under the legacy `validation_toolkit/` package. + +## Working method + +Before implementing behaviour: + +1. Read the relevant specification sections in full, including referenced requirements in other sections. +2. Read both schemas when the work touches configuration, findings, results, or reporting. +3. Identify the requirement IDs affected by the change. +4. Add or update the smallest coherent set of tests that verifies those requirements. +5. Implement until those tests pass. +6. Run the proportionate wider test suite and static checks. + +Use test-driven development for the reimplementation. A failing test is useful only when it represents current specified behaviour. Do not add tests for discarded design ideas, prototype compatibility, renamed properties that never entered version 1, or other historical “fossils.” + +Tests should cite the requirement IDs they verify. Prefer a focused parameterized test that covers a coherent rule table over many near-duplicate tests. Do not merge unrelated behaviours merely to reduce the test count. Not every implementation, documentation, release-process, or quality requirement needs a direct automated test. + +## Change control + +- Do not change `SPECIFICATION.md`, either schema, or established requirement semantics unless the task explicitly authorizes a specification change. +- Behaviour changes must update all affected normative text, schemas, tests, fixtures, and user documentation together. +- Preserve existing requirement IDs. Add new IDs rather than renumbering requirements after insertion or deletion. +- Preserve schema version identifiers and canonical `$id` values unless a deliberate schema-version change has been approved. +- Treat unknown configuration and result properties as errors; do not add permissive extension points speculatively. +- Do not add CLI options, comparators, normalisers, file formats, report formats, plugin systems, compatibility shims, or acceptance logic outside the specified v1 scope. +- Keep commits and patches narrowly scoped. Do not mix behavioural work with unrelated restructuring or formatting. + +When a requirement is ambiguous, contradictory, or not implementable as written, present the exact conflict and its consequences to the repository owner. Do not resolve product decisions through undocumented assumptions. + +## Design guardrails + +- Keep the top-level CLI thin. Put comparison behaviour in independently testable internal modules. +- Keep parsing, configuration validation, canonicalisation, record alignment, comparison, result construction, semantic result validation, and rendering as distinct responsibilities. +- Prefer small explicit interfaces and ordinary modules. Version 1 does not need a general plugin framework. +- Keep the comparison kernel independent of CLI parsing and HTML presentation. +- Model the three processing phases (`contract`, `records`, and `values`) separately from finding classification (`comparison_difference` and `assessment_issue`). +- Treat the reference and query as having distinct roles where the specification does so, including directional relative tolerance. +- Never use binary floating-point arithmetic for specified decimal parsing, normalisation, tolerance calculation, or canonical rendering. +- Preserve source lexical evidence where required while keeping canonical typed values distinct from source values. +- Make ordering explicit. Never rely on dictionary, set, filesystem, locale, or hash iteration order for canonical output. +- Collect all safely assessable findings. Do not stop normal comparison at the first difference. +- Keep execution errors outside canonical comparison results and findings. +- Generate HTML only from validated canonical JSON and fixed renderer resources. + +## Validation boundaries + +Configuration processing has two mandatory layers: + +1. structural validation against the packaged comparison-configuration schema; +2. semantic validation of the cross-property rules in `SPECIFICATION.md`. + +Canonical-result validation likewise has two layers: + +1. structural validation against the packaged results schema, with Draft 2020-12 format assertion enabled; +2. semantic validation of reconciliation, ordering, classification, configuration agreement, and other cross-property invariants. + +Never publish an evidence package unless both result-validation layers pass. A generated-result validation failure is `internal.unexpected`, not a user finding. + +Load packaged schemas and the bundled configuration through the package-resource API. Core comparison must work without network access and must not resolve schemas by downloading their canonical HTTPS identifiers. + +## Python quality + +- Use type annotations for production interfaces and non-trivial internal functions. +- Prefer explicit domain types, enums, and immutable value objects where they prevent invalid states. +- Keep functions focused and make transformation stages easy to test in isolation. +- Raise or return domain-specific errors at architectural boundaries; do not use broad exception handling to hide defects. +- Convert unexpected exceptions into the specified CLI diagnostic only at the outer application boundary. +- Do not expose tracebacks during ordinary CLI operation. +- Avoid mutable global state and import-time side effects. +- Keep dependencies minimal, justified, pinned or bounded appropriately, and separated into runtime and development dependencies. +- Use safe YAML loading with the restrictions required by the specification. Never construct arbitrary Python objects from configuration input. +- Use secure temporary-file and atomic-publication patterns. Never broaden filesystem permissions. + +Do not introduce a formatter, linter, type checker, build backend, or supported-Python change as an incidental edit. Such project-wide tooling choices belong in the implementation plan and must be configured consistently before being enforced. + +## Testing guardrails + +The version 1 suite should be organized around public behaviour and stable internal contracts, not private implementation details. It must eventually include the test families specified in Section 14 of `SPECIFICATION.md`. + +In particular: + +- Use controlled, synthetic, non-sensitive fixtures. +- Keep fixtures small enough to review, except for tests specifically concerned with scale or resource behaviour. +- Record why generated fixtures exist and make their regeneration deterministic. +- Test both directions where reference/query symmetry is required. +- Test boundary values, malformed inputs, Unicode, composite keys, null versus absence, exact decimal behaviour, and deterministic ordering. +- Validate representative results against the packaged schema and the semantic result validator. +- Test the installed `adviso-validation` entry point, not only direct Python calls. +- Test the built wheel and source distribution rather than relying only on an editable source checkout. +- Do not update expected outputs blindly. Review semantic changes in canonical JSON fixtures field by field. + +Until a new v1 suite is established, do not treat passing `tests/table_validator_tests/` as evidence that v1 requirements are satisfied. + +## Security and clinical-data handling + +Assume inputs and evidence packages may contain sensitive clinical or laboratory information. + +- Never place real patient data, identifiers, credentials, absolute local paths, or secrets in source, tests, fixtures, snapshots, logs, or examples. +- Routine logs, progress output, terminal diagnostics, and execution-error diagnostics must not reproduce record identifiers or source values. +- Findings may contain identifiers or values only where the specification requires them to identify or explain a finding. +- Do not add telemetry, remote schema lookup, remote assets, or any other network transmission. +- Treat input basenames as potentially sensitive. +- Preserve the specification's overwrite, symlink, temporary-file, and atomic-publication protections. + +## Documentation + +The eventual user guide must describe the released behaviour, complete CLI, configuration schema, results schema, finding taxonomy, evidence package, and corrective guidance for execution errors. Documentation examples must validate against the packaged schemas. + +Do not update the current prototype README piecemeal so that it appears to describe a usable hybrid product. Replace it as a deliberate implementation milestone when the corresponding v1 interface is ready. + +## Before declaring work complete + +For the scope changed, confirm and report: + +- affected requirement IDs; +- tests added or updated and why they are sufficient; +- commands run and their results; +- schema validation performed, including format assertion where applicable; +- any checks not run and the reason; +- documentation or fixture changes; +- remaining assumptions, limitations, or follow-up work. + +Do not claim completion while relevant tests, schema checks, packaging checks, or documentation are knowingly inconsistent with the implementation. diff --git a/SPECIFICATION.md b/SPECIFICATION.md index e5960e5..085bf50 100644 --- a/SPECIFICATION.md +++ b/SPECIFICATION.md @@ -9,9 +9,11 @@ This document defines a specification for version 1.0 of the AdvISO Validation Toolkit: a command-line namespace for cohesive capabilities that support bioinformatics validation and verification activities associated with ISO 15189 accreditation. -The initial sole user-facing subcommand within the toolkit is `compare`. This subcommand compares a reference dataset with a query dataset according to an explicit comparison configuration and produces deterministic, machine-readable findings and a human-readable summary suitable for review and retention as one component of a bioinformatics pipeline validation or verification record. It is intended to answer this question: +The initial sole user-facing subcommand within the toolkit is `compare`. It is designed for two datasets that describe the same kinds of samples or observations but may express their fields and values using different names, types, vocabularies, or other source dialects. The declared comparison configuration translates those representations into a common canonical representation, after which the tool compares the canonical records and values. It produces deterministic, machine-readable findings and a human-readable summary suitable for review and retention as one component of a bioinformatics pipeline validation or verification record. -> Given these identified inputs and these declared comparison rules, which expected and observed records and values are equivalent, different, missing, extra, or not assessable? +The tool distinguishes differences that remain after successful translation from issues that prevented some intended translation or assessment. It is intended to answer these questions: + +> Given these identified inputs and declared translation and comparison rules, which expected and observed records and values are equivalent, different, missing, or extra after translation? Which input or configuration-related issues prevented an intended assessment? Given this narrow and precisely defined scope, the toolkit cannot be used _alone_ to assert that: @@ -96,9 +98,9 @@ The following principles govern all requirements: 1. **REQ-PRN-001.** **Explicit over inferred:** Comparison behaviour MUST be declared, not guessed from observed data. 2. **REQ-PRN-002.** **No silent data loss:** Input records, duplicate keys, unmapped values, and unsupported structures MUST NOT be silently discarded or overwritten. 3. **REQ-PRN-003.** **Symmetry:** Missing and extra content MUST be assessed in both directions. -4. **Separation of concerns:** Run status, comparison evidence, and the laboratory's acceptance decision are distinct concepts. +4. **REQ-PRN-009.** **Separation of concerns:** Run status, comparison evidence, and the laboratory's acceptance decision MUST remain distinct concepts. 5. **REQ-PRN-004.** **Complete evidence:** A normal comparison mismatch MUST still produce a complete report. -6. **REQ-PRN-005.** **Determinism:** Identical inputs, configuration, and tool version MUST produce semantically identical results. +6. **REQ-PRN-005.** **Determinism:** The same exact input bytes, input basenames, configuration document, toolkit version, and runtime identity MUST produce identical result content except for the explicitly variable run identifier and completion time. Comparison outcomes MUST additionally satisfy the invariance rules in Section 9.1. 7. **REQ-PRN-006.** **Traceability:** Findings MUST identify the rule, record, and field that produced them. 8. **REQ-PRN-007.** **Data minimisation:** Reports MUST include only the source identifiers and values necessary to explain and review the comparison. Routine logs, progress output, terminal diagnostics, and execution-error diagnostics MUST NOT reproduce them. 9. **REQ-PRN-008.** **Fail clearly:** Invalid or ambiguous input MUST produce an explicit error rather than an assumed interpretation. @@ -162,7 +164,7 @@ For each input: - **REQ-INP-011.** Invalid UTF-8 byte sequences MUST cause an input error. - **REQ-INP-012.** Invalid UTF-8 byte sequences MUST NOT be replaced silently. - **REQ-INP-013.** Field names and string values MUST be preserved exactly before declared normalisation. -- **REQ-INP-014.** Line endings MUST NOT affect comparison results. +- **REQ-INP-014.** Changing only accepted LF and CRLF line endings MUST NOT affect the comparison outcome defined in Section 9.1. Exact-byte size and digest evidence MAY differ. ### 5.4 Tabular parsing @@ -180,13 +182,13 @@ For CSV and TSV: - **REQ-INP-023.** delimiters, quoting, escaping, and embedded newlines MUST be handled by a standards-compliant parser; - **REQ-INP-024.** rows with too many or too few columns MUST be input errors; - **REQ-INP-025.** all physical input rows MUST be accounted for; -- **REQ-INP-026.** source row numbers SHOULD be retained for traceability; +- **REQ-INP-026.** the one-based data-record number and one-based starting physical line number of every record MUST be retained for traceability; the header is not a data record; - **REQ-INP-027.** Input values MUST initially remain strings unless a custom configuration declares a null representation or type conversion. - **REQ-INP-028.** In the bundled strict-exact configuration, an empty cell MUST be an empty string. ### 5.5 JSON parsing -- **REQ-INP-029.** JSON inputs MUST use a schema-validated list of record objects: +- **REQ-INP-029.** A JSON input MUST contain one root-level array of record objects. A dataset containing one record MUST use a one-element array; a root-level record object is not supported: ```json [ @@ -199,11 +201,12 @@ For CSV and TSV: - **REQ-INP-031.** JSON decimal numbers - i.e. numbers containing a fractional or exponent component - MUST be parsed using exact decimal arithmetic. - **REQ-INP-032.** JSON decimal numbers MUST NOT first be converted to binary floating-point values. - **REQ-INP-033.** The tool MUST retain the original text of each JSON number — for example, distinguishing the representations `0.100`, `0.1`, and `1e-1` — for inclusion in evidence when required. -- **REQ-INP-034.** A JSON input MUST contain either one object representing a single record or an array of objects representing multiple records. +- **REQ-INP-034.** A root-level JSON value that is not an array MUST cause an input error, including a root-level object that could otherwise represent one record. - **REQ-INP-035.** A root-level scalar MUST cause an input error. - **REQ-INP-036.** A root-level array containing any item that is not an object MUST cause an input error. - **REQ-INP-037.** An invalid-root error MUST identify the invalid root value or array item. - **REQ-INP-038.** An invalid-root error MUST explain the required structure. +- **REQ-INP-070.** Duplicate member names in any JSON object MUST cause an input error, including duplicates in record objects and nested object-valued fields. A parser MUST detect duplicates before constructing a mapping that could overwrite an earlier member. ### 5.6 Parser-native value model @@ -216,21 +219,25 @@ For CSV and TSV: - Parser-native values are equal only when they have the same category and recursively equal content. For array-valued fields, element order is significant; for object-valued fields, member names and values must match but member order is not significant. Top-level records are aligned by their declared record keys, and their input order does not affect comparison. - **REQ-INP-045.** Numerical values MUST be retained exactly as they appeared in the input for reporting and traceability. - **REQ-INP-046.** Two values in the same numerical category MUST compare as equal when they represent the same number; for example, decimal `1.0` and decimal `1.00` are equal. -- `NaN`, positive infinity, and negative infinity are not valid numerical values. -- **REQ-INP-047.** A non-finite value in an attributable non-key field MUST produce a contract finding. -- **REQ-INP-048.** A non-finite value in an attributable non-key field MUST make that value `not_assessable`. -- **REQ-INP-049.** Other assessable content MUST still be compared when a non-finite value occurs in an attributable non-key field. -- **REQ-INP-050.** A non-finite value in a record key MUST produce an execution error. -- **REQ-INP-051.** A non-finite value that prevents construction of an unambiguous record model MUST produce an execution error. +- `NaN`, positive infinity, and negative infinity are not valid JSON numerical values or valid declared decimals. +- **REQ-INP-047.** An unquoted non-finite token such as `NaN`, `Infinity`, or `-Infinity` in a JSON input MUST cause an input error because the document is not valid JSON, irrespective of the field in which it occurs. +- **REQ-INP-048.** A quoted representation such as `"NaN"` in JSON, CSV, or TSV MUST initially remain a parser-native string. +- **REQ-INP-049.** If such a string is parsed as a declared decimal in an attributable non-key field, it MUST produce an `invalid_field_value` finding, make that value opportunity `not_assessable`, and allow other assessable content to continue. +- **REQ-INP-050.** If such a string is parsed as a declared decimal record-key field, the invalid key MUST produce an execution error. +- **REQ-INP-051.** The JSON parser MUST reject non-standard non-finite tokens rather than enabling a permissive extension. Every accepted parser-native JSON number is therefore a finite exact integer or decimal. ### 5.7 Empty inputs -- **REQ-INP-052.** Empty files and datasets with zero records MUST be recognised explicitly. +- **REQ-INP-052.** Zero-byte files and datasets with zero records MUST be recognised explicitly and distinguished. A valid zero-record JSON dataset is `[]`; a valid zero-record CSV or TSV dataset contains a valid header and no data records. A zero-byte JSON file is malformed JSON, and a zero-byte CSV or TSV file is an input error because it lacks the required header. - **REQ-INP-053.** If both datasets are syntactically valid and contain no records, the records phase MUST report no missing records. - **REQ-INP-054.** If both datasets are syntactically valid and contain no records, the records phase MUST report no extra records. - **REQ-INP-055.** If both datasets are syntactically valid and contain no records, the values phase MUST perform no comparisons. - **REQ-INP-056.** Contract requirements that can be evaluated without records MUST still be checked. - **REQ-INP-057.** The tool MUST NOT infer that an empty dataset is unsuitable merely because it is empty. +- **REQ-INP-066.** Per-record requirements have no applicable opportunities in a zero-record dataset and MUST NOT produce findings or not-assessable classifications merely because no records exist. +- **REQ-INP-067.** Available dataset-level structure MUST still be assessed. A zero-record CSV or TSV header MAY establish declared, missing, or undeclared source columns. A JSON empty array exposes no field structure, so field presence or absence MUST NOT be inferred from it. +- **REQ-INP-068.** Two valid zero-record datasets MUST have complete records and values phases, zero matched, missing, and extra records, and zero value opportunities. Header-derived contract findings do not alter those phase statuses. +- **REQ-INP-069.** When exactly one dataset has records, every record on that side MUST be reported as missing or extra as applicable; because there are no aligned records, there are zero value opportunities and the records and values phases remain complete. - Any minimum-size acceptance criterion belongs to the external laboratory decision process. - **REQ-INP-058.** A non-empty reference and empty query MUST report every expected record as missing. - **REQ-INP-059.** An empty reference and non-empty query MUST report every observed record as extra. @@ -385,7 +392,7 @@ fields: ``` -**REQ-CFG-015.** The template is illustrative rather than an instruction to use these particular rules. Laboratories MUST choose identifiers, field definitions, and policies appropriate to the intended comparison. +The template is illustrative rather than a recommendation to use these particular rules. **REQ-CFG-016.** The published machine-readable schema MUST encode the complete version 1 comparison-configuration structure and constraints. @@ -397,9 +404,7 @@ The only permitted top-level keys are `schema_version`, `metadata`, and `fields` - **REQ-CFG-017.** `schema_version` MUST be a quoted string identifying a supported comparison-configuration schema version. - **REQ-CFG-018.** `metadata.id` MUST be a stable, non-empty identifier for the comparison configuration. -- **REQ-CFG-019.** `metadata.id` SHOULD use a collision-resistant organisational namespace. -- **REQ-CFG-020.** `metadata.revision` MUST be a non-empty revision string controlled by the configuration owner. -- **REQ-CFG-021.** Any behaviourally material change elsewhere in the configuration MUST produce a new revision. +- **REQ-CFG-020.** `metadata.revision` MUST be a non-empty string. - `metadata.title` and `metadata.description` are optional human-readable strings. - **REQ-CFG-022.** The `metadata` properties MUST NOT directly alter comparison behaviour. - **REQ-CFG-023.** The `metadata` properties MUST be preserved in the configuration embedded in the canonical result. @@ -417,11 +422,17 @@ The only permitted top-level keys are `schema_version`, `metadata`, and `fields` - Every field in `fields.definitions` is in scope. Field definitions are the sole mechanism for selecting explicitly configured fields in version 1. - **REQ-CFG-030.** `fields.undeclared` is required and MUST be one of `compare`, `exclude`, or `fail`. - - `compare` compares shared undeclared fields with `preserved_exact`; an undeclared field present on only one side creates a contract finding because no corresponding value can be compared. - - `exclude` does not compare undeclared fields and records them in the exclusion summary as `not_assessed`. - - `fail` creates a contract finding for every undeclared field. + - `compare` compares dataset-wide shared undeclared field names with `preserved_exact`; a name observed in only one dataset creates one dataset-level contract finding because no corresponding field can be established. + - `exclude` does not compare undeclared fields and records one entry per dataset side and source field in the exclusion summary as `not_assessed`. + - `fail` creates one dataset-level contract finding per dataset side and undeclared source field and does not compare that field. - **REQ-CFG-031.** Undeclared reference and query fields MUST be aligned only when their input field names are identical. - No name inference or source-name mapping is applied to undeclared fields. +- **REQ-CFG-116.** Undeclared source-field names MUST be discovered dataset-wide. For CSV and TSV, the header defines the observed names even when there are no records. For JSON, the observed names are the union of member names across all record objects. +- **REQ-CFG-117.** Under `compare`, an undeclared name observed in both datasets MUST become an implicit untyped comparison field whose reference, query, and canonical names are identical, whose comparator is `preserved_exact`, and whose review priority is unassigned. +- **REQ-CFG-118.** For a shared undeclared field under `compare`, each aligned record in which at least one side contains the field MUST contribute one value opportunity. Two present values are compared normally; one present and one absent value MUST produce one `value_mismatch` finding with presence-state evidence; and two absent values contribute no opportunity. +- **REQ-CFG-119.** Under `compare`, an undeclared name observed in only one dataset MUST produce one dataset-level `undeclared_field` finding for that side. Each aligned record containing that field MUST contribute one not-assessable value opportunity. Occurrences confined to missing or extra records do not create value opportunities. +- **REQ-CFG-120.** Under `fail`, each undeclared field MUST produce one dataset-level `undeclared_field` finding for each dataset side on which it is observed. Each aligned record in which the field is present on either side MUST contribute one not-assessable value opportunity, irrespective of whether the same source name occurs in both datasets. +- **REQ-CFG-121.** A dataset-level undeclared-field finding or exclusion entry MUST record the number of records on that dataset side that contain the source field. A field known only from a zero-record CSV or TSV header has a record count of zero. - **REQ-CFG-032.** Fields with different names MUST be declared explicitly under a common canonical field definition if they are intended to correspond. - **REQ-CFG-033.** `fields.definitions` MUST be a mapping keyed by unique canonical field name. - **REQ-CFG-034.** `fields.definitions` MUST contain a definition for every field referenced by `fields.record_key`. @@ -443,7 +454,7 @@ The only permitted top-level keys are `schema_version`, `metadata`, and `fields` **REQ-CFG-048.** For a scalar field, `logical_type` MUST be one of `string`, `integer`, `decimal`, `boolean`, `date`, or `datetime`. -**REQ-CFG-049.** `logical_type` MAY be omitted only when the comparator is `preserved_exact` and no typed parsing, mapping, or typed normalisation is configured. +**REQ-CFG-049.** `logical_type` MUST be omitted when, and only when, the comparator is `preserved_exact`. A `preserved_exact` field MUST also omit `source_normalisation`, `value_mapping`, `typed_normalisation`, and `nulls.missing_tokens`. Its requiredness and null policy still apply, and parser-native JSON null remains governed by that null policy, but no additional source representation is transformed or interpreted as null. **REQ-CFG-050.** For a collection field, `logical_type` MUST use this form: @@ -459,12 +470,18 @@ logical_type: **REQ-CFG-052.** The comparator named for the field itself MUST correspond to the collection kind: `list_exact`, `set_exact`, or `multiset_exact`. +**REQ-CFG-124.** A `set` or `multiset` element comparator MUST be `typed_exact`. + +**REQ-CFG-125.** A `list` element comparator MUST be `typed_exact` or `numeric_tolerance`. `numeric_tolerance` MUST be permitted only when `element_type` is `integer` or `decimal` and MUST use the same parameter rules as a field-level numeric-tolerance comparator. + +**REQ-CFG-126.** `preserved_exact`, `normalised_string`, `numeric_exact`, `date_exact`, `datetime_exact`, and the collection comparators MUST NOT be used as element comparators in version 1. Exact comparison of any declared scalar element type uses `typed_exact`. + **REQ-CFG-053.** Every declared field MUST contain a `nulls` mapping: - **REQ-CFG-054.** `allowed` and `nulls_equal` MUST be booleans. - **REQ-CFG-055.** `nulls_equal` MUST be `false` when `allowed` is `false`. - **REQ-CFG-056.** `missing_tokens` MAY be omitted; omission means that no additional source values are interpreted as null. Parser-native JSON null remains null independently of this property. -- **REQ-CFG-057.** When `missing_tokens` is present, it MUST contain `reference`, `query`, or both. Each present side MUST be a non-empty list of source-value descriptors. An omitted side has no configured textual missing tokens. A matched token is converted to null at processing step 3 in Section 6.8. +- **REQ-CFG-057.** When `missing_tokens` is present, it MUST contain `reference`, `query`, or both. Each present side MUST be a non-empty list of non-null source-value descriptors. An omitted side has no configured missing tokens. A matched token is converted to null at processing step 3 in Section 6.8. A source-value descriptor has one of these forms: @@ -486,6 +503,8 @@ category: "null" **REQ-CFG-059.** `value` MUST be present for every category except `null`. It MUST be a quoted lexical string for `string`, `integer`, and `decimal`. It MUST be `true` or `false` for `boolean`. The lexical value MUST be valid for its stated category. +**REQ-CFG-122.** A descriptor with `category: null` MUST be permitted only as the `to` destination of a value-mapping entry. It MUST NOT be permitted as a missing token or as a mapping `from` value, because parser-native null is recognised without configuration and null values bypass mapping. + Descriptors are matched using parser-native category and value equality under Section 5.6. The retained lexical form remains evidence but does not create a distinct numerical value. This explicit representation prevents YAML scalar resolution from changing mapping keys or missing tokens. #### 6.2.7 Normalisation and value mapping @@ -523,6 +542,8 @@ The first-release typed-value operation identifiers are: **REQ-CFG-066.** A source-text operation MUST appear only in `source_normalisation.reference` or `source_normalisation.query`. It applies only to the corresponding dataset. +**REQ-CFG-123.** A source-text operation MUST apply only when the current parser-native value is a string. A non-string value MUST pass through the source-normalisation stage unchanged; it MUST NOT be stringified, transformed, or rejected merely because a source-text operation is configured. + **REQ-CFG-067.** A typed-value operation MUST appear only in `typed_normalisation`. It applies identically to both datasets. **REQ-CFG-068.** Operation semantics MUST be published with the schema and recorded sufficiently for reproducibility. This information MUST include the Unicode-data version used by Unicode operations. @@ -540,9 +561,9 @@ The first-release typed-value operation identifiers are: value: "R" ``` -- **REQ-CFG-071.** `from` and `to` MUST be source-value descriptors as defined in Section 6.2.6. +- **REQ-CFG-071.** `from` MUST be a non-null source-value descriptor and `to` MUST be a source-value descriptor as defined in Section 6.2.6. - A `from` descriptor is matched against the source value after the applicable `source_normalisation` operations and missing-token recognition have been performed, in accordance with the processing order in Section 6.8. -- A `to` descriptor with `category: "null"` explicitly maps to null and is distinct from absence of a mapping entry. +- A `to` descriptor with `category: "null"` explicitly maps a non-null source value to null and is distinct from absence of a mapping entry. The null policy is then applied and typed parsing is bypassed. - **REQ-CFG-072.** Each `from` value MUST occur at most once within one side’s mapping. - **REQ-CFG-073.** Conflicting entries MUST invalidate the comparison configuration. - **REQ-CFG-074.** Unreachable entries MUST invalidate the comparison configuration. @@ -573,6 +594,8 @@ The first-release typed-value operation identifiers are: The normative JSON Schema Draft 2020-12 definition of comparison-configuration schema version `1.0` is [`src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json`](src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json). +**REQ-CFG-128.** The schema MUST use `https://adviso-project.github.io/validation_toolkit/schemas/comparison-configuration-v1.0.schema.json` as its canonical `$id`. + **REQ-CFG-086.** A YAML configuration MUST first be loaded using Section 6.2.1. The resulting JSON-compatible data structure MUST then validate against that schema before either input dataset is read. The standalone schema file is the authoritative machine-readable definition. @@ -585,9 +608,10 @@ The standalone schema file is the authoritative machine-readable definition. - **REQ-CFG-104.** every name in `fields.record_key` exists in `fields.definitions`; - **REQ-CFG-105.** every key field is required on both sides, non-nullable, scalar, and uses `typed_exact`; -- **REQ-CFG-106.** a field without `logical_type` uses `preserved_exact` and has no typed normalisation or value mapping; +- **REQ-CFG-106.** a field without `logical_type` uses `preserved_exact` and has no source normalisation, value mapping, typed normalisation, or configured missing tokens; conversely, a field using `preserved_exact` omits `logical_type` and all of those processing properties; - **REQ-CFG-107.** each comparator and normalisation operation is compatible with the field’s logical type and processing stage; - **REQ-CFG-108.** source-value descriptors contain lexically valid values for their declared categories, and each side’s value mapping has unique `from` values; +- **REQ-CFG-127.** the `source.reference` names are unique across canonical field definitions, and the `source.query` names are independently unique across canonical field definitions; **REQ-CFG-091.** A failure of any semantic check MUST be reported as an invalid comparison configuration with the relevant configuration path and violated rule. @@ -637,6 +661,19 @@ Normalisation, value mapping, and review priority are optional. An omitted revie Intended equivalence between differing source representations requires both sides to be parsed as the same declared destination logical type, with an explicit value mapping where the source representation cannot be parsed directly. +The accepted parser-native inputs for each declared logical type are fixed as follows: + +- **REQ-VAL-076.** A `string` field MUST accept only a parser-native string. No other parser-native category is converted to text implicitly. +- **REQ-VAL-077.** An `integer` field MUST accept a parser-native integer or a parser-native string satisfying the textual integer grammar in Section 6.5.2. A parser-native decimal MUST be rejected even when it represents a mathematically integral value. +- **REQ-VAL-078.** A `decimal` field MUST accept a parser-native integer, parser-native decimal, or parser-native string matching `[+-]?[0-9]+(?:\.[0-9]+)?(?:[eE][+-]?[0-9]+)?`. Conversion of a parser-native integer to a decimal is exact. Text such as `.5` and `1.` MUST be rejected. +- **REQ-VAL-079.** A `boolean` field MUST accept a parser-native boolean or a parser-native string accepted by Section 6.5.4. +- **REQ-VAL-080.** A `date` or `datetime` field MUST accept only a parser-native string, which MUST then satisfy the applicable grammar in Sections 6.5.5 and 6.5.6. +- **REQ-VAL-081.** A collection field MUST accept either a parser-native array or, for CSV and TSV input, a parser-native string containing one valid JSON array. No other parser-native category is accepted as a collection. +- **REQ-VAL-082.** After a collection has been obtained, each element MUST be parsed independently according to the declared scalar element type and the same source-category rules. +- **REQ-VAL-083.** A parser-native object MUST be invalid for every declared version 1 logical type. An untyped field using `preserved_exact` MAY still preserve and compare a parser-native object. +- **REQ-VAL-084.** A source representation not accepted directly by this matrix MUST require an explicit value mapping to an accepted representation. The implementation MUST NOT introduce any additional implicit conversion. +- **REQ-VAL-085.** Null recognition and null-policy handling MUST occur before typed parsing. Null is therefore not an accepted typed-parser input category. + **REQ-VAL-010.** The original source value or lexical representation MUST remain available to the comparison engine. It MAY be included in evidence when required to explain a finding. **REQ-VAL-011.** An invalid typed value in a non-key field that remains attributable to an unambiguous record MUST be a contract finding. It MUST make that value `not_assessable`. It MUST NOT be an execution error. @@ -681,7 +718,7 @@ Intended equivalence between differing source representations requires both side - **REQ-VAL-032.** Datetimes MUST use an RFC 3339-compatible representation containing a full date, time, and `Z` or numeric UTC offset. - **REQ-VAL-033.** Local datetimes without an offset MUST be invalid. - **REQ-VAL-034.** Equivalent instants expressed with different offsets MUST have the same canonical value. -- **REQ-VAL-035.** Fractional seconds MUST be preserved unless explicit precision normalisation is configured. +- **REQ-VAL-035.** Supplied fractional-second precision MUST remain available in the source lexical evidence. The canonical datetime representation MUST remove unnecessary trailing fractional-second zeros and omit a zero-valued fractional component. Any reduction that changes the represented instant, rather than only its redundant lexical form, MUST require explicit precision normalisation. - **REQ-VAL-036.** Leap-second representations MUST be invalid in the first release. #### 6.5.7 Collections @@ -689,7 +726,7 @@ Intended equivalence between differing source representations requires both side - **REQ-VAL-037.** Every collection MUST declare its element logical type and element comparator. - **REQ-VAL-038.** A `list` MUST preserve order and duplicates. - **REQ-VAL-039.** A `set` MUST be unordered. -- **REQ-VAL-040.** A `set` MUST reject duplicate source elements rather than silently collapse them. +- **REQ-VAL-040.** A `set` MUST reject duplicate elements after typed parsing, using typed-exact canonical equality, rather than silently collapse them. Distinct source representations that parse to the same canonical value, such as textual integers `"01"` and `"1"`, are duplicates. - **REQ-VAL-041.** A `multiset` MUST be unordered. - **REQ-VAL-042.** A `multiset` MUST preserve element occurrence counts. - **REQ-VAL-043.** JSON collections MUST use arrays. @@ -720,8 +757,8 @@ The field-level null policy governs the collection value as a whole and does not #### 6.7.1 Column mappings - **REQ-VAL-056.** Reference and query fields MAY map to a canonical field name. -- **REQ-VAL-057.** Mappings MUST be one-to-one unless an explicitly defined transformation supports another cardinality. -- **REQ-VAL-058.** Conflicting target mappings MUST be rejected. +- **REQ-VAL-057.** Column-name mappings MUST be one-to-one in version 1. One source field MUST NOT feed multiple canonical fields, and one canonical field MUST NOT receive multiple source fields from the same dataset side. +- **REQ-VAL-058.** Reuse of a source name by more than one canonical field on the same dataset side MUST invalidate the comparison configuration before either dataset is read. Reference and query namespaces are independent, so the same source name MAY occur once on each side. - **REQ-VAL-059.** Unknown source fields MUST follow the undeclared-field policy. - **REQ-VAL-060.** Required mapped fields that are absent MUST be reported. - **REQ-VAL-061.** All applied mappings MUST be identifiable in the evidence. @@ -737,21 +774,25 @@ The field-level null policy governs the collection value as a whole and does not - **REQ-VAL-068.** Values used in mappings MUST be represented using the typed source-value descriptor. The parser MUST NOT infer their intended category from YAML scalar syntax. - **REQ-VAL-069.** Applied mappings MUST be identifiable in the evidence. +**REQ-VAL-086.** Parser-native null and values converted to null by missing-token recognition MUST bypass value mapping and typed parsing. A non-null value explicitly mapped to null MUST bypass typed parsing after the null policy has been applied. Mapping from null is not supported. + ### 6.8 Normalisation and processing order **REQ-VAL-070.** Normalisation MUST be opt-in, field-specific, ordered deterministically, and recorded. Source-text normalisation is distinct from typed-value normalisation. -**REQ-VAL-071.** Source-text normalisation MAY include Unicode normalisation, leading or trailing whitespace trimming, and case folding. Typed-value normalisation MAY include datetime offset normalisation, declared precision reduction, and stable ordering of collections declared to be unordered. +**REQ-VAL-071.** Source-text normalisation MAY include Unicode normalisation, leading or trailing whitespace trimming, and case folding. Typed-value normalisation MAY include datetime offset normalisation and declared precision reduction. Deterministic serialization order for sets and multisets is canonical output handling, not a configured typed-normalisation operation. **REQ-VAL-072.** Typed processing MUST occur in this order: 1. preserve the source value and representation; -2. apply the source-text normalisation declared for the applicable dataset side; -3. recognise configured null or missing tokens; -4. apply a declared source-value mapping; -5. parse the declared logical type; -6. apply declared typed-value normalisation; -7. compare canonical values. +2. if the current value is a parser-native string, apply the source-text normalisation declared for the applicable dataset side; otherwise pass it through unchanged; +3. recognise parser-native null or a configured non-null missing token; +4. if the value is null, apply the null policy and bypass value mapping and typed parsing; +5. for a non-null value, apply a declared source-value mapping and produce a `mapping_failure` finding if a configured mapping has no matching entry; +6. if the mapping output is null, apply the null policy and bypass typed parsing; +7. otherwise parse the declared logical type; +8. apply declared typed-value normalisation; +9. compare canonical values. **REQ-VAL-073.** Column-name mapping MUST occur before field-value processing. @@ -773,7 +814,7 @@ The field-level null policy governs the collection value as a whole and does not - **REQ-CFG-114.** treat two syntactically valid zero-record datasets as having zero missing records, zero extra records, and zero value mismatches, while evaluating contract findings as required by Sections 5.7 and 7.2; - **REQ-CFG-115.** treat reference-only and query-only fields as contract findings. -**REQ-CFG-094.** Because parser-native types differ between formats, strict-exact comparison across different input formats MAY report type mismatches for lexically similar values. A custom typed configuration is required when cross-format logical equivalence is intended. +**REQ-CFG-094.** Because parser-native categories differ between formats, strict-exact comparison across different input formats MAY report `value_mismatch` findings for lexically similar values. Such a difference is not a `type_mismatch`, because `preserved_exact` declares no logical type contract. A custom typed configuration is required when cross-format logical equivalence is intended. ## 7. Comparison behaviour @@ -819,7 +860,7 @@ The field-level null policy governs the collection value as a whole and does not **REQ-CMP-009.** For a zero-record dataset, contract requirements that can be evaluated from available structural information, such as CSV or TSV headers, MUST still be assessed. -**REQ-CMP-010.** A requirement that depends on examining a record and cannot be evaluated because no records exist MUST be reported as `not_assessable`. It MUST NOT be treated as satisfied merely because no contradictory record was observed. +**REQ-CMP-010.** A per-record requirement with no applicable record opportunity MUST be treated as not applicable, not as satisfied, violated, `not_assessed`, or `not_assessable`. It MUST contribute no finding or value count. This rule does not suppress contract checks supported by available dataset-level structure. ### 7.3 Records comparison @@ -837,7 +878,7 @@ The field-level null policy governs the collection value as a whole and does not #### 7.4.1 `preserved_exact` -Used by the bundled strict-exact configuration. Values match according to the parser-native value model and equality rules in Section 5.6. No inference, coercion, mapping, normalisation, or tolerance is applied. Textual `"1"`, integer `1`, decimal `1.0`, and boolean `true` are distinct. +Used by the bundled strict-exact configuration. Values match according to the parser-native value model and equality rules in Section 5.6. No logical type, additional missing-token recognition, inference, coercion, mapping, normalisation, or tolerance is applied. Textual `"1"`, integer `1`, decimal `1.0`, and boolean `true` are distinct. #### 7.4.2 `typed_exact` @@ -875,17 +916,18 @@ difference = abs(reference - query) - **REQ-CMP-025.** Lists use ordered element-by-element comparison and their lengths MUST match. - Sets use order-independent comparison of unique elements; duplicate source elements are invalid under Section 6.5.7. - Multisets use order-independent comparison that preserves occurrence counts. -- Elements use the declared element comparator. +- **REQ-CMP-040.** List elements use their declared element comparator positionally: element `n` in the reference is compared only with element `n` in the query. Numeric tolerance is therefore unambiguous for lists. +- **REQ-CMP-041.** Set and multiset elements use typed-exact equality. Version 1 MUST NOT perform tolerance-based or other fuzzy matching within an unordered collection. ### 7.5 Completeness, subsets, and determinism -- **REQ-CMP-026.** Comparison findings MUST NOT terminate a normal run. +- **REQ-CMP-026.** Findings MUST NOT terminate a normal run. - **REQ-CMP-027.** All assessable differences MUST be collected. - **REQ-CMP-028.** A malformed record that prevents a complete and unambiguous record model MUST be an input error. - **REQ-CMP-029.** A malformed record MUST NOT disappear silently. - **REQ-CMP-030.** Key fields MUST always be processed. - **REQ-CMP-031.** Under `fields.undeclared: "exclude"`, undeclared fields MUST be represented by a summary rather than by an individual finding for every excluded occurrence. The canonical report MUST record a stable exclusion code, the policy, the affected input field names, and the number of excluded records or values where calculable. -- **REQ-CMP-032.** Record order MUST NOT affect results unless explicitly declared significant. +- **REQ-CMP-032.** Record order MUST NOT affect the comparison outcome defined in Section 9.1. Input digests, source locations, messages containing those locations, and report-local finding IDs MAY differ as permitted by the provenance and ordering rules. - **REQ-CMP-033.** Output ordering MUST be deterministic and independent of hash-map ordering and locale. ## 8. Findings and run status @@ -894,7 +936,7 @@ difference = abs(reference - query) **REQ-FND-001.** Every finding MUST follow the structured representation in Section 9.1.2 and the normative results schema. -**REQ-FND-002.** Every finding MUST contain a stable finding code, deterministic report-local ID, phase, affected side, and human-readable message. +**REQ-FND-002.** Every finding MUST contain a stable finding code, finding classification, deterministic report-local ID, phase, affected side, and human-readable message. **REQ-FND-003.** Record identity, source locations, canonical or source field name, review priority, and value evidence MUST be present exactly where required by the finding code and data-minimisation rules. @@ -902,7 +944,7 @@ difference = abs(reference - query) **REQ-FND-005.** Successful field matches MUST be represented in summary counts. Successful record matches MUST be represented in summary counts. Successful matches MUST NOT produce individual findings. -### 8.2 Minimum finding categories +### 8.2 Finding codes and classifications **REQ-FND-006.** Stable finding codes MUST exist for at least: @@ -910,12 +952,30 @@ difference = abs(reference - query) - **REQ-FND-009.** extra record; - **REQ-FND-010.** missing required field; - **REQ-FND-011.** undeclared field; -- **REQ-FND-012.** type mismatch; -- **REQ-FND-013.** invalid field value, including an attributable field-level parse or data-contract failure; +- **REQ-FND-012.** type mismatch: the parser-native category of a non-null source value is not accepted by the field's declared logical type under the matrix in Section 6.5; +- **REQ-FND-013.** invalid field value: the parser-native category is accepted by the declared logical type, but the value fails its lexical, domain, collection-content, or other field-level validity rule; - **REQ-FND-014.** null-policy violation; - **REQ-FND-015.** mapping failure; -- **REQ-FND-016.** value mismatch; -- **REQ-FND-017.** not assessable; +- **REQ-FND-016.** value mismatch: both values are valid and assessable, but the selected comparator determines that they differ; + +**REQ-FND-023.** Every finding MUST have exactly one of two classifications. Classification expresses the meaning of a finding, while phase records where in processing it was detected; neither replaces the other. + +- **REQ-FND-024.** `comparison_difference` means that translation succeeded sufficiently to compare the relevant records or values and a difference remained. `missing_record`, `extra_record`, and `value_mismatch` MUST have this classification. +- **REQ-FND-025.** `assessment_issue` means that the declared translation or assessment could not be applied cleanly to some input content. `missing_required_field`, `undeclared_field`, `type_mismatch`, `invalid_field_value`, `null_policy_violation`, and `mapping_failure` MUST have this classification. + +Finding classification does not assert scientific or clinical significance. In particular, an assessment issue such as an unrecognised mapped value may be important to investigate, but the tool cannot report it as a post-translation difference because translation did not succeed. + +**REQ-FND-018.** `type_mismatch`, `invalid_field_value`, and `value_mismatch` are mutually exclusive for one affected value or value opportunity. A parser-native category difference under `preserved_exact` MUST be a `value_mismatch`, not a `type_mismatch`, because no logical type contract has been declared. + +**REQ-FND-019.** A record-specific missing-required-field, type-mismatch, invalid-field-value, null-policy-violation, or mapping-failure finding MUST describe one record, one canonical field, and one affected dataset side. If reference and query independently violate the same field rule, the tool MUST emit two findings, one with `affected_side: reference` and one with `affected_side: query`. It MUST NOT combine the independent violations into one `both` finding. The dataset-level zero-record missing-field case is the explicit exception defined by `REQ-FND-022`. + +**REQ-FND-020.** A value-mismatch finding MUST describe one comparison between two individually valid, assessable values and MUST use `affected_side: both`. Missing-record and extra-record findings remain one finding per affected record key. Undeclared-field finding granularity is defined separately by the undeclared-field policy. + +**REQ-FND-021.** An `undeclared_field` finding MUST be dataset-level and identify one dataset side, one source field name, and the number of records on that side containing the field. It MUST NOT be repeated for individual record occurrences. A `value_mismatch` involving a shared undeclared field under `compare` is record-specific, identifies the shared source field name, has unassigned review priority, and is not accompanied by an `undeclared_field` finding. + +**REQ-FND-022.** If a zero-record CSV or TSV header omits a declared source column that is required on that dataset side, the tool MUST emit one dataset-level `missing_required_field` finding for that canonical field and side. It contains the canonical field and review priority but no record identity, source location, or value evidence. An empty JSON array cannot produce this finding because it exposes no field structure. + +**REQ-FND-017.** `not_assessable` MUST be a value-opportunity classification, not a finding code. Every not-assessable opportunity MUST be explained by its applicable causal finding, such as `missing_required_field`, `type_mismatch`, `invalid_field_value`, `null_policy_violation`, `mapping_failure`, or `undeclared_field`. The tool MUST NOT emit an additional finding merely to repeat that the affected opportunity is not assessable. **REQ-FND-007.** Execution errors MUST NOT be represented as findings or assigned finding codes. In particular, invalid invocation or configuration, malformed input structure, invalid record keys, resource failure, I/O failure, and internal failure are execution errors governed by Section 8.3. @@ -927,7 +987,7 @@ difference = abs(reference - query) - **REQ-RUN-016.** `incomplete`: the inputs and record model remain sufficiently trustworthy for a useful comparison, but one or more configured assessments could not be completed safely because a comparison prerequisite was not satisfied; - **REQ-RUN-017.** `error`: no trustworthy comparison result could be produced because of an invalid invocation or comparison configuration, malformed or unreadable input syntax or structure, a missing, null, invalid, or duplicate record key, an incomplete or ambiguous record model, an enforced resource limit, resource exhaustion, an input/output failure, or an unexpected internal failure. -**REQ-RUN-002.** Differences between valid, assessable reference and query content — including contract findings, missing or extra records, and value mismatches — MUST NOT by themselves change a run from `complete` to `incomplete` or `error`. The run status describes completeness and operational validity, not acceptability. +**REQ-RUN-002.** Findings, whether classified as `comparison_difference` or `assessment_issue`, MUST NOT by themselves change a run from `complete` to `incomplete` or `error`. A finding may identify a not-assessable opportunity that independently makes the run incomplete under `REQ-RUN-003`. Run status describes completeness and operational validity, not acceptability. **REQ-RUN-003.** Any configured content classified as `not_assessable` MUST make the run `incomplete`. @@ -935,17 +995,41 @@ difference = abs(reference - query) **REQ-RUN-005.** An incomplete report MUST identify the affected content. It MUST identify why the comparison prerequisite was not satisfied. -**REQ-RUN-006.** Content that is parseable and attributable to an unambiguous record but violates the declared data contract MUST be recorded as a comparison finding rather than an error. +**REQ-RUN-006.** Content that is parseable and attributable to an unambiguous record but violates the declared data contract MUST be recorded as an `assessment_issue` finding rather than an execution error. **REQ-RUN-007.** Comparisons that remain assessable MUST continue. -**REQ-RUN-008.** An error report SHOULD be written where possible. When written, it MUST be identified as an error report. +**REQ-RUN-008.** An errored run MUST write a concise diagnostic to standard error and MUST NOT publish an evidence-package directory or any filesystem report. + +**REQ-RUN-009.** The diagnostic MUST contain a stable execution-error code and category, a concise actionable message, and sufficient non-sensitive context to locate the cause. -An error report is a diagnostic artefact, not a comparison result or evidence package. +**REQ-RUN-010.** The diagnostic MUST NOT contain findings, finding codes, comparison summary metrics, phase statuses, record identifiers, or source values. -**REQ-RUN-009.** When an error report can be written safely, it MUST contain `run_status: error`, a stable execution error code and category, a concise actionable message, and sufficient non-sensitive context to locate the cause. +**REQ-RUN-024.** Version 1 execution errors MUST use exactly one of the following category and code combinations: -**REQ-RUN-010.** An error report MUST NOT contain findings, finding codes, comparison summary metrics, or phase statuses. +| Category | Code | Meaning | +| --- | --- | --- | +| `invocation` | `invalid_arguments` | Command-line arguments are missing, invalid, or mutually incompatible | +| `configuration` | `unreadable` | The selected comparison configuration cannot be read | +| `configuration` | `invalid_syntax` | The comparison configuration is not permitted YAML syntax | +| `configuration` | `schema_violation` | The parsed comparison configuration does not satisfy its JSON Schema | +| `configuration` | `semantic_violation` | The configuration is schema-valid but violates a semantic configuration rule | +| `configuration` | `unsupported_version` | The configuration names an unsupported schema version | +| `input` | `unreadable` | A reference or query input cannot be read | +| `input` | `unsupported_format` | An input has a missing, unknown, or ambiguous filename extension | +| `input` | `invalid_encoding` | An input is not valid UTF-8 | +| `input` | `invalid_syntax` | Input text is malformed for the format selected by its extension | +| `input` | `invalid_structure` | Parsed input does not satisfy the required dataset or record structure | +| `input` | `invalid_record_key` | A record key is missing, null, invalid, or duplicated, or cannot form an unambiguous record model | +| `resource` | `limit_exceeded` | A documented implementation safeguard stops assessment | +| `resource` | `exhausted` | An unplanned resource exhaustion prevents completion | +| `output` | `invalid_destination` | The selected output destination violates destination or overwrite rules | +| `output` | `write_failed` | The evidence package cannot be written or published | +| `internal` | `unexpected` | An unexpected implementation failure occurs, including generated-result self-validation failure | + +**REQ-RUN-025.** The execution-error code MUST be the dotted combination `.`. The first diagnostic line MUST have the exact form `ERROR [.]: `. The message MUST be non-empty and MUST NOT contain a line break. Additional non-sensitive context MAY follow on subsequent lines. + +**REQ-RUN-026.** Every execution-error code except `internal.unexpected` MUST return exit code 2. `internal.unexpected` MUST return exit code 3. **REQ-RUN-011.** Findings accumulated internally before the error occurred MUST NOT be published as a partial comparison result. @@ -953,6 +1037,18 @@ An error report is a diagnostic artefact, not a comparison result or evidence pa Overall status is `complete` only when all three phases are complete. +**REQ-RUN-018.** The contract phase MUST be `complete` when every applicable contract check was performed. Contract findings, including missing-field, type, invalid-value, null-policy, mapping, and undeclared-field findings, MUST NOT by themselves make the contract phase incomplete. + +**REQ-RUN-019.** The records phase MUST be `complete` when valid record keys were constructed and every record was aligned and classified. Missing-record and extra-record findings MUST NOT make the records phase incomplete. A key or record-model failure that prevents this assessment is an execution error and produces no comparison phase statuses. + +**REQ-RUN-020.** The values phase MUST be `complete` when every applicable in-scope value opportunity was compared or deliberately excluded. It MUST be `incomplete` when at least one such opportunity is `not_assessable`. + +**REQ-RUN-021.** A finding MUST NOT by itself make a phase incomplete. A phase is incomplete only when an intended assessment within that phase could not be performed safely. + +**REQ-RUN-022.** Content classified as `not_assessed` by an explicit exclusion policy MUST NOT make any phase incomplete. + +**REQ-RUN-023.** Overall run status MUST be `complete` if and only if the contract, records, and values phases are all `complete`; otherwise a non-error comparison run MUST be `incomplete`. + **REQ-RUN-013.** The tool MUST report raw findings and metrics without applying acceptance thresholds. It MUST NOT produce a pass/fail verdict. ### 8.4 Laboratory decision @@ -965,7 +1061,11 @@ The run status and reported differences do not constitute a laboratory validatio **REQ-OUT-001.** A non-error comparison, whether `complete` or `incomplete`, MUST create an evidence-package directory at the destination selected by `--output`. -**REQ-OUT-002.** The evidence package MUST contain the canonical JSON result. It MUST contain the HTML report. A CSV findings export MAY also be included. +**REQ-OUT-002.** The evidence package MUST contain the canonical JSON result and the HTML report. Version 1 MUST NOT produce any additional report or findings-export format. + +**REQ-OUT-115.** The canonical JSON result MUST be named `comparison-results.json`, and the HTML report MUST be named `comparison-report.html`. These names are fixed for evidence-package version 1. + +No separate package manifest is required in version 1; `comparison-results.json` contains the comparison and provenance metadata. **REQ-OUT-003.** Package member names MUST be stable and documented. Package layout MUST be stable and documented. @@ -973,6 +1073,10 @@ The run status and reported differences do not constitute a laboratory validatio **REQ-OUT-004.** JSON MUST be the canonical output format. Its normative JSON Schema Draft 2020-12 definition is [`src/adviso_validation/schemas/comparison-results-v1.0.schema.json`](src/adviso_validation/schemas/comparison-results-v1.0.schema.json). +**REQ-OUT-125.** `comparison-results.json` MUST be serialized as RFC 8259 JSON encoded as UTF-8 without a byte-order mark, using two-space indentation, LF line endings, and exactly one final newline. Non-ASCII characters MUST be emitted directly rather than escaped unless escaping is required by JSON syntax. Members of fixed-schema objects MUST be emitted in their documented schema order; members of dynamically named objects MUST be emitted in Unicode code-point order. + +**REQ-OUT-112.** The results schema MUST use `https://adviso-project.github.io/validation_toolkit/schemas/comparison-results-v1.0.schema.json` as its canonical `$id`. + **REQ-OUT-005.** The standalone results-schema file MUST be version controlled. It MUST be included as package data in `adviso_validation.schemas`. It MUST be resolvable together with the comparison-configuration schema that it references. The result-schema version and comparison-configuration-schema version are independent. @@ -997,13 +1101,21 @@ Every property is required, including empty `exclusions` and `findings` arrays. **REQ-OUT-008.** The schema accepts only non-error results with run status `complete` or `incomplete`. An execution-error diagnostic under Section 8.3 is not a canonical comparison result and MUST NOT validate against this schema. -**REQ-OUT-009.** The `run` object MUST contain a randomly generated UUID `id`, `status`, and `completed_at`. +**REQ-OUT-009.** The `run` object MUST contain `id`, `status`, and `completed_at`. `id` MUST be a randomly generated UUID version 4 in canonical lowercase hyphenated form. -**REQ-OUT-010.** `completed_at` MUST be an RFC 3339 UTC timestamp using `Z`. +**REQ-OUT-010.** `completed_at` MUST be an RFC 3339 UTC timestamp using `Z` and whole-second precision, in the exact form `YYYY-MM-DDTHH:MM:SSZ`. **REQ-OUT-011.** Start time MUST NOT be included. Elapsed duration MUST NOT be included. -**REQ-OUT-012.** Run ID and completion time are deliberately variable metadata. They MAY be excluded from semantic-determinism comparisons. +**REQ-OUT-012.** Run ID and completion time are deliberately variable metadata and MUST be excluded from repeat-determinism comparisons. + +**REQ-OUT-095.** For determinism requirements, the comparison outcome consists of run and phase statuses; summary counts; exclusions considered without input occurrence order; and each finding's classification, code, phase, affected side, record identity, canonical or source field identity, review priority, and value evidence. It excludes run ID, completion time, software-runtime provenance, input byte provenance, source locations, human-readable messages that reproduce variable locations, and report-local finding IDs. + +**REQ-OUT-096.** Given the same exact input bytes, input basenames, configuration document, toolkit version, and runtime identity, all canonical result content other than `run.id` and `run.completed_at` MUST be identical. + +**REQ-OUT-097.** Changing only input record order or accepted LF/CRLF line endings MUST leave the comparison outcome unchanged. Exact-byte input size and digest, source locations, location-dependent messages, and report-local finding IDs MAY change. The result for each particular ordered byte input MUST still satisfy all deterministic ordering rules. + +**REQ-OUT-098.** Across supported Python versions and operating systems, the comparison outcome MUST remain invariant for identical inputs and configuration. Recorded runtime provenance MAY differ. Release testing MUST verify that Unicode-data differences do not silently change comparison outcomes across the supported matrix. **REQ-OUT-013.** The `phases` object MUST contain `contract`, `records`, and `values`, each with status `complete` or `incomplete`. @@ -1013,6 +1125,14 @@ Every property is required, including empty `exclusions` and `findings` arrays. **REQ-OUT-016.** The `runtime` object MUST record Python version, a normalized non-identifying platform identifier, and the Unicode-data version. +**REQ-OUT-121.** `runtime.python` MUST have the form `-..`, using a lowercase implementation name and the full runtime version, for example `cpython-3.12.7`. A prerelease suffix MAY follow the micro version when applicable. + +**REQ-OUT-122.** `runtime.platform` MUST have the form `-` using lowercase normalized names, for example `linux-x86_64`, `linux-aarch64`, `windows-x86_64`, or `macos-aarch64`. `amd64` MUST be normalized to `x86_64`, and `arm64` MUST be normalized to `aarch64`. The supported release matrix determines the permitted combinations. + +**REQ-OUT-123.** `runtime.platform` MUST NOT include a distribution name, distribution version, kernel version, hostname, or other host-specific detail. + +**REQ-OUT-124.** `runtime.unicode_version` MUST be the exact runtime Unicode database version in `..` form. + **REQ-OUT-017.** Hostname, username, environment variables, working directory, absolute paths, raw command line, and a full dependency inventory MUST NOT be included. **REQ-OUT-018.** Each input object MUST contain its automatically captured basename, selected format, UTF-8 encoding, byte size, and SHA-256 digest of the exact input bytes. @@ -1023,17 +1143,15 @@ Basenames form part of the potentially sensitive evidence covered by Section 11. **REQ-OUT-020.** V1 MUST use one fixed documented interpretation for CSV. It MUST use one fixed documented interpretation for TSV. Because format uniquely determines that interpretation, no dialect identifier appears in the result. -**REQ-OUT-021.** `record_key_fields` MUST declare the ordered source key-field names for the reference and query and the ordered canonical key-field names. Findings identify records through corresponding ordered arrays of source lexical values and canonical textual values. A single-field key is represented by a one-element array. Each identity array MUST have the same length as its corresponding field-name array. The reference or query identity is JSON null when that record is absent on that side; the canonical identity is always present. +**REQ-OUT-021.** `record_key_fields` MUST declare the ordered source key-field names for the reference and query and the ordered canonical key-field names. Findings identify records through corresponding ordered arrays of source lexical values and canonical textual values. Canonical key components MUST use the scalar rendering rules in `REQ-OUT-029`. A single-field key is represented by a one-element array. Each identity array MUST have the same length as its corresponding field-name array. The reference or query identity is JSON null when that record is absent on that side; the canonical identity is always present. **REQ-OUT-022.** `comparison_configuration.document` MUST contain the complete validated JSON-compatible representation of the comparison configuration, including the bundled configuration when no custom file was supplied. **REQ-OUT-023.** The embedded document MUST validate against the referenced comparison-configuration schema. -No comparison-configuration digest or command-invocation object is included in version 1. - -**REQ-OUT-024.** Consumers MUST validate `results_schema_version` before interpreting a result. +**REQ-OUT-113.** A result with `results_schema_version: "1.0"` MUST embed a comparison configuration with `schema_version: "1.0"`. Support for a later comparison-configuration schema version requires a corresponding results-schema version that explicitly references that configuration-schema version. -**REQ-OUT-025.** Consumers MUST use structured properties and finding codes for program logic. They MUST NOT infer semantics by parsing human-readable finding messages. +No comparison-configuration digest or command-invocation object is included in version 1. #### 9.1.1 Value evidence @@ -1043,28 +1161,35 @@ No comparison-configuration digest or command-invocation object is included in v **REQ-OUT-028.** Object member order has no semantic significance. Object members MUST be serialized in Unicode code-point order. -**REQ-OUT-029.** Canonical scalar values MUST contain their logical type and value. Strings use JSON strings, booleans use JSON booleans, and dates and datetimes use canonical ISO text. Integers and decimals use normalized exact decimal text: no leading plus sign or unnecessary leading or trailing zeros, zero represented as `"0"`, and normalized lowercase scientific notation permitted where needed. Canonical collections contain the collection logical type, scalar element type, and an array of compact canonical element values. Lists preserve order; sets and multisets use deterministic canonical order; multisets retain duplicates. Null uses the declared logical type with a JSON null value. A `preserved_exact` field has no typed canonical value. +**REQ-OUT-029.** Canonical scalar values MUST contain their logical type and one uniquely rendered value: + +- **REQ-OUT-089.** A canonical integer MUST use ordinary base-10 text with no leading plus sign or unnecessary leading zeros; zero is `"0"`. +- **REQ-OUT-090.** A canonical decimal zero, including negative zero, MUST be `"0"`. Every non-zero canonical decimal MUST use normalized scientific form with exactly one non-zero digit before the optional decimal point, no trailing fractional zeros, lowercase `e`, no `e0`, no positive exponent sign, and no exponent leading zeros. For example, `1.00`, `1000`, `12.30`, `0.0010`, and `-0.0` become `"1"`, `"1e3"`, `"1.23e1"`, `"1e-3"`, and `"0"` respectively. +- **REQ-OUT-091.** A canonical date MUST use `YYYY-MM-DD`. +- **REQ-OUT-092.** A canonical datetime MUST be converted to UTC, use uppercase `Z`, remove unnecessary trailing fractional-second zeros, and omit a zero-valued fractional component. +- **REQ-OUT-093.** A canonical boolean MUST use a JSON boolean. A canonical string MUST contain its Unicode text after configured processing without any additional transformation. +- **REQ-OUT-094.** Canonical collections MUST contain the collection logical type, scalar element type, and an array of compact canonical element values. Lists preserve order. Sets and multisets MUST be serialized in locale-independent Unicode code-point order of the canonical element strings; multiset duplicates remain adjacent. Null uses the declared logical type with a JSON null value. A `preserved_exact` field has no typed canonical value. **REQ-OUT-030.** Values MUST be included only as follows: - **REQ-OUT-061.** missing and extra record findings contain no field values; -- **REQ-OUT-062.** missing-required-field findings contain presence states but do not reproduce the present side's value; -- **REQ-OUT-063.** undeclared-field findings contain no field value; +- **REQ-OUT-062.** record-specific missing-required-field findings contain presence states but do not reproduce the present side's value; the dataset-level zero-record form contains no value evidence; +- **REQ-OUT-063.** dataset-level undeclared-field findings contain no field value; - **REQ-OUT-064.** type-mismatch findings contain source values for affected sides and canonical values only where parsing succeeded; - **REQ-OUT-065.** invalid-field-value and mapping-failure findings contain the affected source value and no failed canonical value; - **REQ-OUT-066.** null-policy-violation findings contain the affected source and canonical null representation; -- **REQ-OUT-067.** value-mismatch findings contain source and canonical values for both sides; -- **REQ-OUT-068.** not-assessable findings contain only values necessary to explain the failed prerequisite. +- **REQ-OUT-067.** value-mismatch findings MUST state the presence state of both sides. Each present side MUST contain its source value and, for a declared typed field, its canonical value. An absent side contains neither value. A mismatch of two present untyped values contains no typed canonical values. +- **REQ-OUT-068.** A causal finding for a not-assessable value opportunity MUST contain only the values permitted for that finding code and necessary to explain the failed prerequisite. It MUST NOT contain separate or duplicated not-assessable evidence. **REQ-OUT-031.** Implementations MUST NOT add additional source values merely for convenience. #### 9.1.2 Findings -**REQ-OUT-032.** Every finding MUST contain a deterministic report-local ID, finding code, phase, human-readable English message, and affected side. +**REQ-OUT-032.** Every finding MUST contain a deterministic report-local ID, finding classification, finding code, phase, human-readable English message, and affected side. -**REQ-OUT-033.** A message is presentation text derived deterministically from structured properties. It MUST NOT add facts absent from those properties. Consumers MUST NOT parse it for program logic. +**REQ-OUT-033.** A message is presentation text derived deterministically from structured properties. It MUST NOT add facts absent from those properties, and every machine-significant fact in a message MUST also be represented by a structured property or finding code. -Specification-requirement identifiers and a separate finding category are not included in version 1. +Specification-requirement identifiers are not included in version 1. Finding classification is included explicitly and MUST agree with the finding code according to Section 8.2. The closed version 1 finding-code vocabulary is: @@ -1076,20 +1201,35 @@ The closed version 1 finding-code vocabulary is: - `invalid_field_value`; - `null_policy_violation`; - `mapping_failure`; -- `value_mismatch`; -- `not_assessable`. +- `value_mismatch`. + +**REQ-OUT-034.** `affected_side` MUST be `reference`, `query`, or `both` as required by the finding code. A finding attributable to a declared canonical field MUST contain that field name and its resolved review priority. A dataset-level undeclared-field finding instead contains the source field name and record count. A value mismatch involving a shared undeclared field contains its source field name and no review priority. Record-specific findings MUST contain record identity and source locations; dataset-level findings omit record identity and source locations. + +**REQ-OUT-117.** For CSV and TSV, every record-specific source location MUST contain the one-based data-record number, excluding the header, and the one-based physical line number on which the record begins. Embedded newlines in a quoted field do not change that record's starting line. A location for a present field MUST also contain its source field name; a record-level location or a location for an absent field MUST omit the field name. + +**REQ-OUT-118.** For JSON, a record-level source location MUST contain the RFC 6901 JSON Pointer to the array element, such as `/3`. A location for a present field MUST contain the full pointer to that member, such as `/3/result`, with `~` and `/` escaped as `~0` and `~1`. A location for an absent field MUST identify its containing record because the absent member has no source location. + +**REQ-OUT-119.** Reference and query locations MUST be represented independently. A location MUST be null when the corresponding record does not exist. If the record exists but the relevant field is absent, the location MUST identify the containing record as specified above. -**REQ-OUT-034.** `affected_side` MUST be `reference`, `query`, or `both`. A finding attributable to a declared canonical field MUST contain that field name and its resolved review priority. An undeclared-field finding instead contains the source field name. Record-specific findings MUST contain record identity and source locations; dataset-level findings omit record identity. +**REQ-OUT-035.** Findings MUST be sorted by this complete, locale-independent order: -For CSV and TSV, a source location records the one-based data-record number where applicable, the one-based starting physical line number, and the source field name where applicable. For JSON, it records an RFC 6901 JSON Pointer. The reference and query locations are represented independently; a location is null when the corresponding record or field does not exist. +1. classification: `comparison_difference`, `assessment_issue`; +2. phase: `contract`, `records`, `values`; +3. dataset-level findings before record-specific findings; +4. canonical record identity, compared component-by-component as Unicode strings; +5. field identity, using the canonical field name for a declared field and otherwise the source field name; +6. finding code in the fixed vocabulary order listed in Section 9.1.2; +7. affected side: `reference`, `query`, `both`; +8. source locations, comparing reference before query, with a null location before a present location; tabular locations compare by record number, physical line number, and field name, while JSON locations compare by JSON Pointer using Unicode code-point order; and +9. canonical JSON serialization of all remaining structured finding properties except `id` and `message`. -**REQ-OUT-035.** Findings MUST be sorted deterministically by phase, canonical record identity, canonical field name, finding code, affected side, and source location, using locale-independent Unicode code-point ordering. IDs `F000001`, `F000002`, and so on MUST be assigned after sorting. IDs are stable within one result but are not persistent identifiers across materially different comparisons. +**REQ-OUT-099.** Two candidate findings identical under the final ordering key represent the same finding and MUST be coalesced rather than emitted twice. IDs `F000001`, `F000002`, and so on MUST be assigned only after coalescence and sorting. IDs are stable within one result but are not persistent identifiers across materially different comparisons. **REQ-OUT-036.** Comparator parameters, mappings, and normalisation declarations MUST NOT be repeated in findings. Consumers and the HTML renderer obtain them from the embedded configuration using the canonical field name. Source and canonical values show their effect for the affected record. #### 9.1.3 Summary and exclusions -**REQ-OUT-037.** The summary MUST contain complete fixed-shape counts, including zero-valued entries. Record counts comprise reference records, query records, matched records, missing records, and extra records. Value counts comprise matched, mismatched, and not-assessable in-scope non-key value opportunities. Finding counts comprise total findings, every version 1 finding code, and review priorities `low`, `normal`, `high`, and `unassigned`. +**REQ-OUT-037.** The summary MUST contain complete fixed-shape counts, including zero-valued entries. Record counts comprise reference records, query records, matched records, missing records, and extra records. Value counts comprise matched, mismatched, and not-assessable in-scope non-key value opportunities. Finding counts comprise total findings, both finding classifications, every version 1 finding code, and review priorities `low`, `normal`, `high`, and `unassigned`. **REQ-OUT-038.** Record counts MUST reconcile as follows: @@ -1106,15 +1246,43 @@ query = matched + extra - **REQ-OUT-072.** absence on a required side is not assessable; - **REQ-OUT-073.** an invalid or prohibited value that prevents comparison is not assessable. -Under `fields.undeclared: compare`, undeclared non-key fields in aligned records are value opportunities: fields present on both sides use `preserved_exact`; one-sided fields create an undeclared-field finding and are not assessable. Under `exclude`, they are not value opportunities. Under `fail`, they create undeclared-field findings, are not compared, and are counted as not assessable. +Under `fields.undeclared: compare`, dataset-wide shared undeclared fields contribute value opportunities in aligned records where at least one side is present: two present values use `preserved_exact`, and one present and one absent value mismatch. A dataset-wide one-sided undeclared field creates one dataset-level `undeclared_field` finding and is not assessable in each aligned record containing it. Under `exclude`, undeclared fields are not value opportunities. Under `fail`, each undeclared field creates the dataset-level findings required by Section 6.2.5 and is not assessable in each aligned record where it occurs. Occurrences in missing or extra records do not contribute value opportunities. -**REQ-OUT-040.** `summary.findings.total` MUST equal the number of emitted findings. Counts by code and counts by review priority MUST each sum to the total. Findings not attributable to a declared field, including record and undeclared-field findings, count as `unassigned` review priority. +**REQ-OUT-040.** `summary.findings.total` MUST equal the number of emitted findings. Counts by classification, by code, and by review priority MUST each sum to the total. Findings not attributable to a declared field, including record and undeclared-field findings, count as `unassigned` review priority. -**REQ-OUT-041.** `exclusions` MUST always be an array. Under `fields.undeclared: exclude`, it contains one entry per source field and dataset side, recording code `undeclared_field_excluded`, side, source field name, and the number of records containing the field. Entries MUST be ordered by side and source field name. Exclusions are not findings, do not have review priorities, and do not contribute to value counts. A column known from a zero-record CSV or TSV header has record count zero. +**REQ-OUT-041.** `exclusions` MUST always be an array. Under `fields.undeclared: exclude`, it contains one entry per source field and dataset side, recording code `undeclared_field_excluded`, side, source field name, and the number of records containing the field. Entries MUST be ordered first by side (`reference`, then `query`) and then by source field name using Unicode code-point order. Exclusions are not findings, do not have review priorities, and do not contribute to value counts. A column known from a zero-record CSV or TSV header has record count zero. + +#### 9.1.4 Structural and semantic result validation + +**REQ-OUT-100.** The results schema MUST define each finding form as an exact alternative. For each finding code it MUST require the applicable classification and properties and reject properties that are not permitted for that form, including the declared-field and shared-undeclared-field forms of `value_mismatch` and the record-specific and dataset-level forms of `missing_required_field`. + +**REQ-OUT-101.** Schema validation alone is not sufficient to establish that a canonical result is internally coherent. The implementation MUST also apply semantic result validation. + +**REQ-OUT-102.** Semantic result validation MUST verify that overall run status agrees with all phase statuses. + +**REQ-OUT-103.** Semantic result validation MUST verify the record-count reconciliation equations in `REQ-OUT-038`. + +**REQ-OUT-104.** Semantic result validation MUST verify that every finding count agrees with the findings array and that counts by classification, code, and review priority each sum to the total. + +**REQ-OUT-105.** Semantic result validation MUST verify that every record-key field-name array and corresponding record-identity value array has the required common length and component order. + +**REQ-OUT-106.** Semantic result validation MUST verify that findings and exclusions use the required order, including classification-first finding order, that duplicate findings are absent, and that finding IDs are sequential after sorting. + +**REQ-OUT-107.** Semantic result validation MUST verify that each finding's classification, presence states, value evidence, locations, affected side, field identity, and review priority are consistent with its code, form, embedded configuration, and causal condition. + +**REQ-OUT-108.** Semantic result validation MUST verify that `record_key_fields`, finding field identities, review priorities, and other configuration-derived metadata agree with the embedded comparison configuration. + +**REQ-OUT-109.** Before publishing an evidence package, the tool MUST validate the generated canonical result against the results schema and every semantic result rule. + +**REQ-OUT-110.** Failure of generated-result validation MUST be treated as an unexpected internal error. The invalid canonical result and any HTML derived from it MUST NOT be published as an evidence package. + +**REQ-OUT-111.** The user guide MUST distinguish structural JSON Schema validation from semantic result validation and document the invariants checked by each layer. + +**REQ-OUT-114.** Generated-result schema validation MUST enable JSON Schema Draft 2020-12 format assertion. Invalid values for recognised formats, including UUIDs, dates, and date-times, MUST fail self-validation. The user guide's documented procedure for validating canonical results MUST instruct users to enable equivalent format assertion. ### 9.2 Human-readable output -**REQ-OUT-042.** The first release MUST produce an HTML human-readable report. The report SHOULD include: +**REQ-OUT-042.** The first release MUST produce an HTML human-readable report. The report MUST include: - **REQ-OUT-074.** purpose of the run and the three comparison phases performed; - **REQ-OUT-075.** prominent run status and per-phase completion status; @@ -1126,20 +1294,20 @@ Under `fields.undeclared: compare`, undeclared non-key fields in aligned records - **REQ-OUT-081.** provenance and reproduction information; - **REQ-OUT-082.** space or linkage for review and approval outside the tool. -**REQ-OUT-043.** CSV MAY be provided as an export of findings, but MUST: +**REQ-OUT-127.** The report's headline summary MUST present run completeness, total `comparison_difference` findings, and total `assessment_issue` findings as distinct measures. It MUST NOT describe either finding count as an acceptance or validation verdict. + +**REQ-OUT-128.** Detailed findings MUST be organized primarily by classification, with comparison differences presented before assessment issues. Phase MUST remain visible as secondary traceability information and MAY be used for subgrouping within each classification. + +**REQ-OUT-120.** `comparison-report.html` MUST be generated solely from the validated contents of `comparison-results.json` and fixed renderer resources. It MUST NOT contain additional comparison facts, findings, metrics, provenance, or run metadata unavailable in the canonical JSON. Fixed explanatory text and blank or generic external review and approval space MAY be added by the renderer. -- **REQ-OUT-083.** be produced using a conforming CSV writer; -- **REQ-OUT-084.** preserve record identifiers; -- **REQ-OUT-085.** quote delimiters, quotes, and newlines correctly; -- **REQ-OUT-086.** use a stable documented column schema; -- **REQ-OUT-087.** mitigate spreadsheet formula injection when intended for spreadsheet use; -- **REQ-OUT-088.** never be the only canonical evidence format. +**REQ-OUT-126.** `comparison-report.html` MUST be self-contained and fully usable offline. Required styling MUST be embedded. The report MUST NOT load external scripts, stylesheets, fonts, images, tracking resources, or any other network resource. Core content and navigation MUST remain usable with scripting disabled. Embedded scripting MAY provide non-essential navigation or filtering enhancements but MUST NOT change or supply comparison content. ### 9.3 Atomicity and overwrite protection - **REQ-OUT-044.** Output files MUST be written atomically where supported. - **REQ-OUT-045.** The complete evidence-package directory MUST be published atomically where the platform supports it. -- **REQ-OUT-046.** An existing destination MUST NOT be overwritten unless the user supplies `--overwrite`. +- **REQ-OUT-046.** Without `--overwrite`, any existing filesystem object at the selected destination MUST cause an output error. With `--overwrite`, the tool MAY replace only an existing regular directory that is recognisably a complete version 1 evidence package containing exactly `comparison-results.json` and `comparison-report.html`. It MUST reject an existing regular file, symbolic link, special filesystem object, empty directory, partial package, or directory containing any additional entry, even when `--overwrite` is supplied. +- **REQ-OUT-116.** When replacing an existing evidence package, the tool MUST leave the existing package intact until the replacement package has been fully generated and has passed structural and semantic self-validation. A failed replacement MUST preserve the existing package. - **REQ-OUT-047.** The actual selected output directory MUST be reported clearly. - **REQ-OUT-048.** File suffixes MUST be handled structurally, not with character stripping. - **REQ-OUT-049.** A failed or interrupted write MUST NOT publish a partial directory at the selected destination as if it were a complete evidence package. Temporary output MUST be identifiable as temporary and cleaned up where possible. @@ -1215,7 +1383,6 @@ Error classes represented by exit codes 2 and 3 have run status `error`. The dis - **REQ-SEC-010.** Temporary files MUST be removed after successful completion. - **REQ-SEC-011.** Temporary files SHOULD be removed following an error where this can be done safely. - **REQ-SEC-012.** Comparison-configuration YAML MUST be parsed using the safe-loader rules in Section 6.2.1. -- **REQ-SEC-013.** Spreadsheet-oriented exports MUST mitigate formula injection. - **REQ-SEC-014.** The tool SHOULD document filesystem permission expectations. The laboratory is responsible for supplying appropriately pseudonymised input data where required and for controlling access to, retention of, and disposal of inputs and evidence packages. - Automated redaction, identifier pseudonymisation, keyed hashing, and field-level sensitivity policies are outside the scope of version 1.0. @@ -1238,9 +1405,9 @@ Error classes represented by exit codes 2 and 3 have run status `error`. The dis **REQ-DEP-012.** Interrupted runs MUST NOT leave apparently complete reports. -**REQ-DEP-013.** Results MUST remain deterministic across supported Python versions and operating systems. +**REQ-DEP-013.** Comparison outcomes, as defined by `REQ-OUT-095`, MUST remain invariant across supported Python versions and operating systems. Runtime provenance MAY differ. -**REQ-DEP-014.** Internal exceptions MUST be captured in an error report where safe. Concise diagnostics MUST be presented to the user. +**REQ-DEP-014.** Internal exceptions MUST be converted into the standard-error diagnostic required by Section 8.3. A traceback MUST NOT be shown during ordinary CLI use. ## 13. Software quality and release requirements @@ -1306,9 +1473,9 @@ In operational use, each laboratory is responsible for creating, approving, main - **REQ-DOC-008.** the supported input formats and their parsing, encoding, record, field, and filename-extension requirements; - **REQ-DOC-009.** how to use the bundled strict-exact comparison configuration and how to create, validate, and govern a custom comparison configuration; - **REQ-DOC-010.** every property in the comparison-configuration schema, including its purpose, permitted values, required or optional status, default or omission semantics, constraints, and interactions with other properties; -- **REQ-DOC-011.** the comparison phases, processing order, comparators, mappings, normalisation operations, null handling, undeclared-field behaviour, and review priorities; +- **REQ-DOC-011.** the translation-to-canonical comparison model, finding classifications, comparison phases, processing order, comparators, mappings, normalisation operations, null handling, undeclared-field behaviour, and review priorities; - **REQ-DOC-012.** the contents and interpretation of the evidence package; -- **REQ-DOC-013.** every property in the canonical results JSON schema, including finding codes, typed source and canonical values, counts, exclusions, run and phase statuses, and provenance fields; +- **REQ-DOC-013.** every property in the canonical results JSON schema, including finding classifications and codes, typed source and canonical values, counts, exclusions, run and phase statuses, and provenance fields; the guide MUST explain that software consuming a result must first validate `results_schema_version` and must use structured properties, classifications, and finding codes rather than parsing presentation messages; - **REQ-DOC-014.** execution-error diagnostics and common corrective actions; - **REQ-DOC-015.** the tool's intended use, non-claims, security and data-governance boundaries, and known limitations; and - **REQ-DOC-016.** the toolkit, comparison-configuration-schema, and results-schema versions to which the documentation applies. @@ -1333,7 +1500,7 @@ In operational use, each laboratory is responsible for creating, approving, main - **REQ-REL-011.** a reproducible lockfile or equivalent dependency record; - **REQ-REL-012.** evidence that required verification tests passed. -**REQ-REL-002.** Comparison-configuration schema versions, results-schema versions, and toolkit versions MUST be independent. +**REQ-REL-002.** Comparison-configuration schemas, results schemas, and toolkit releases MUST each have their own version identifier and MUST NOT be assumed to advance in lockstep or to use matching version numbers. Each results-schema version nevertheless references the exact comparison-configuration schema version or versions permitted in that result, as required by `REQ-OUT-113`. **REQ-REL-003.** Each toolkit release MUST document the comparison-configuration schema versions it supports. It MUST document the results schema versions it supports. @@ -1349,40 +1516,41 @@ This catalogue summarises minimum executable scenarios for the first release. It - **AT-CON-001:** Identical typed datasets satisfy their contract. - Verifies: `REQ-VAL-003`, `REQ-VAL-004`, `REQ-CMP-006`, `REQ-RUN-002` -- **AT-CON-002:** Required missing fields produce contract findings in both directions and across heterogeneous records; optional missing fields do not. Observed undeclared fields are handled according to each `fields.undeclared` mode and aligned only by identical input names under `compare`. This consolidated parameterized scenario also covers: `fields.undeclared` is required and accepts only `compare`, `exclude`, or `fail`; omission and unknown policies are rejected. Every declared field supplies non-empty `source.reference` and `source.query` names, including when the field is optional on either side; the `required` flags govern record-level presence rather than whether a correspondence is declared. - - Verifies: `REQ-CFG-030`, `REQ-CFG-031`, `REQ-CFG-037`, `REQ-CFG-038`, `REQ-CFG-039`, `REQ-CMP-006`, `REQ-CMP-007`, `REQ-CMP-034`, `REQ-CMP-035`, `REQ-CMP-036`, `REQ-FND-010`, `REQ-FND-011`, `REQ-VAL-002`, `REQ-CFG-035`, `REQ-CFG-036` -- **AT-CON-004:** Null, empty string, whitespace-only string, absent field, and configured missing tokens remain distinct according to policy. This consolidated parameterized scenario also covers: The bundled strict-exact configuration treats parser-native null as equal only to null and distinguishes null, absent fields, and empty strings. - - Verifies: `REQ-CFG-053`, `REQ-CFG-054`, `REQ-CFG-055`, `REQ-CFG-056`, `REQ-CFG-057`, `REQ-VAL-014`, `REQ-VAL-047`, `REQ-VAL-048`, `REQ-VAL-049`, `REQ-VAL-050`, `REQ-VAL-051`, `REQ-VAL-052`, `REQ-VAL-053`, `REQ-VAL-054`, `REQ-VAL-055`, `REQ-INP-028`, `REQ-CFG-113` -- **AT-CON-005:** Unknown, conflicting, and incomplete mappings fail clearly; mapping to null differs from a missing map entry. This consolidated parameterized scenario also covers: Source-value descriptors distinguish strings, numbers, booleans, and null; mapping to null remains distinct from an absent mapping entry. Parameterized mapping fixtures verify reference/query-to-canonical field mapping, required-field reporting, undeclared-field handling, declared source and destination types, permitted intentional many-to-one mappings, and rejection of accidental non-one-to-one mappings. - - Verifies: `REQ-CFG-065`, `REQ-CFG-071`, `REQ-CFG-072`, `REQ-CFG-073`, `REQ-CFG-074`, `REQ-VAL-058`, `REQ-VAL-065`, `REQ-VAL-067`, `REQ-CFG-058`, `REQ-CFG-059`, `REQ-VAL-068`, `REQ-VAL-056`, `REQ-VAL-057`, `REQ-VAL-059`, `REQ-VAL-060`, `REQ-VAL-064`, `REQ-VAL-066` -- **AT-CON-006:** Malformed syntax or structure, empty or duplicate column names, and irregular tabular rows receive their specified error classification without silent data loss. This consolidated parameterized scenario also covers: A table-driven parser suite verifies the fixed CSV and TSV dialects, UTF-8-only decoding, CRLF/LF equivalence, quoted delimiters, doubled quotes, embedded line breaks, source-row retention, and initial string preservation. - - Verifies: `REQ-INP-021`, `REQ-INP-022`, `REQ-INP-023`, `REQ-INP-024`, `REQ-INP-025`, `REQ-INP-034`, `REQ-INP-035`, `REQ-INP-036`, `REQ-INP-037`, `REQ-INP-038`, `REQ-CMP-028`, `REQ-CMP-029`, `REQ-PRN-002`, `REQ-PRN-008`, `REQ-INP-010`, `REQ-INP-011`, `REQ-INP-012`, `REQ-INP-014`, `REQ-INP-015`, `REQ-INP-016`, `REQ-INP-017`, `REQ-INP-018`, `REQ-INP-019`, `REQ-INP-020`, `REQ-INP-026`, `REQ-INP-027`, `REQ-INP-063`, `REQ-INP-064` -- **AT-CON-007:** For two syntactically valid zero-record datasets, the records phase reports no missing or extra records and the values phase performs no comparisons under every configuration. Independently assessable contract requirements are still evaluated, and requirements that depend on examining a record are `not_assessable`. One-sided empty datasets produce all applicable missing or extra record findings. - - Verifies: `REQ-INP-052`, `REQ-INP-053`, `REQ-INP-054`, `REQ-INP-055`, `REQ-INP-056`, `REQ-INP-057`, `REQ-INP-058`, `REQ-INP-059`, `REQ-CMP-009`, `REQ-CMP-010`, `REQ-CFG-114` +- **AT-CON-002:** Required missing fields produce side-specific contract findings in both directions and across heterogeneous records; optional missing fields do not. Dataset-wide field discovery uses tabular headers and the union of JSON record members. Observed undeclared fields are handled according to each `fields.undeclared` mode and aligned only by identical input names under `compare`: shared names use implicit `preserved_exact` comparison with present/absent mismatches, while dataset-wide one-sided names and `fail` findings are consolidated by side and field with accurate record counts. This consolidated parameterized scenario also covers: `fields.undeclared` is required and accepts only `compare`, `exclude`, or `fail`; omission and unknown policies are rejected. Every declared field supplies non-empty `source.reference` and `source.query` names, including when the field is optional on either side; the `required` flags govern record-level presence rather than whether a correspondence is declared. + - Verifies: `REQ-CFG-030`, `REQ-CFG-031`, `REQ-CFG-037`, `REQ-CFG-038`, `REQ-CFG-039`, `REQ-CFG-116`, `REQ-CFG-117`, `REQ-CFG-118`, `REQ-CFG-119`, `REQ-CFG-120`, `REQ-CFG-121`, `REQ-CMP-006`, `REQ-CMP-007`, `REQ-CMP-034`, `REQ-CMP-035`, `REQ-CMP-036`, `REQ-FND-010`, `REQ-FND-011`, `REQ-FND-021`, `REQ-VAL-002`, `REQ-CFG-035`, `REQ-CFG-036` +- **AT-CON-004:** Null, empty string, whitespace-only string, absent field, and configured non-null missing tokens remain distinct according to policy. Parser-native null and recognised missing tokens bypass mapping and typed parsing; mapping to null applies the null policy and bypasses typed parsing; and null is rejected as a missing token or mapping source. This consolidated parameterized scenario also covers: The bundled strict-exact configuration treats parser-native null as equal only to null and distinguishes null, absent fields, and empty strings. + - Verifies: `REQ-CFG-053`, `REQ-CFG-054`, `REQ-CFG-055`, `REQ-CFG-056`, `REQ-CFG-057`, `REQ-CFG-122`, `REQ-VAL-014`, `REQ-VAL-047`, `REQ-VAL-048`, `REQ-VAL-049`, `REQ-VAL-050`, `REQ-VAL-051`, `REQ-VAL-052`, `REQ-VAL-053`, `REQ-VAL-054`, `REQ-VAL-055`, `REQ-VAL-086`, `REQ-INP-028`, `REQ-CFG-113` +- **AT-CON-005:** Unknown, conflicting, and incomplete value mappings fail clearly; mapping to null differs from a missing map entry, and mapping from null is rejected. This consolidated parameterized scenario also covers: Source-value descriptors distinguish strings, numbers, booleans, and null in their permitted positions; mapping to null remains distinct from an absent mapping entry. Parameterized mapping fixtures verify reference/query-to-canonical field mapping, required-field reporting, undeclared-field handling, declared source and destination types, permitted intentional many-to-one value mappings, and rejection of reused source column names or any other non-one-to-one column mapping within either dataset side. + - Verifies: `REQ-CFG-065`, `REQ-CFG-071`, `REQ-CFG-072`, `REQ-CFG-073`, `REQ-CFG-074`, `REQ-CFG-127`, `REQ-VAL-058`, `REQ-VAL-065`, `REQ-VAL-067`, `REQ-CFG-058`, `REQ-CFG-059`, `REQ-VAL-068`, `REQ-VAL-056`, `REQ-VAL-057`, `REQ-VAL-059`, `REQ-VAL-060`, `REQ-VAL-064`, `REQ-VAL-066` +- **AT-CON-006:** Malformed syntax or structure, non-array JSON roots (including a single record object), JSON arrays containing non-object items, duplicate JSON member names at any depth, empty or duplicate tabular column names, and irregular tabular rows receive their specified error classification without silent data loss. This consolidated parameterized scenario also covers: A table-driven parser suite verifies the fixed CSV and TSV dialects, UTF-8-only decoding, CRLF/LF equivalence, quoted delimiters, doubled quotes, embedded line breaks, retention of one-based data-record and starting physical line numbers, and initial string preservation. + - Verifies: `REQ-INP-021`, `REQ-INP-022`, `REQ-INP-023`, `REQ-INP-024`, `REQ-INP-025`, `REQ-INP-034`, `REQ-INP-035`, `REQ-INP-036`, `REQ-INP-037`, `REQ-INP-038`, `REQ-INP-070`, `REQ-CMP-028`, `REQ-CMP-029`, `REQ-PRN-002`, `REQ-PRN-008`, `REQ-INP-010`, `REQ-INP-011`, `REQ-INP-012`, `REQ-INP-014`, `REQ-INP-015`, `REQ-INP-016`, `REQ-INP-017`, `REQ-INP-018`, `REQ-INP-019`, `REQ-INP-020`, `REQ-INP-026`, `REQ-INP-027`, `REQ-INP-063`, `REQ-INP-064` +- **AT-CON-007:** JSON `[]` and header-only CSV/TSV are valid zero-record datasets, while zero-byte inputs receive their format-specific input errors. For two valid zero-record datasets, the records and values phases are complete with no record differences or value opportunities. Per-record requirements produce neither findings nor not-assessable classifications; available tabular header structure still produces applicable dataset-level contract findings, while JSON `[]` exposes no field structure. One-sided zero-record datasets produce all applicable missing or extra record findings, zero value opportunities, and complete records and values phases. + - Verifies: `REQ-INP-052`, `REQ-INP-053`, `REQ-INP-054`, `REQ-INP-055`, `REQ-INP-056`, `REQ-INP-057`, `REQ-INP-058`, `REQ-INP-059`, `REQ-INP-066`, `REQ-INP-067`, `REQ-INP-068`, `REQ-INP-069`, `REQ-CMP-009`, `REQ-CMP-010`, `REQ-CFG-114`, `REQ-FND-022` - **AT-CON-008:** Comparison-configuration YAML scalar resolution and cross-format typed dataset parsing are deterministic across supported environments. - Verifies: `REQ-CFG-006`, `REQ-INP-039`, `REQ-INP-040`, `REQ-INP-041`, `REQ-INP-042`, `REQ-INP-043`, `REQ-INP-044`, `REQ-DEP-013` - **AT-CON-011:** A complete custom YAML configuration conforming to Section 6.2 validates; omitted optional processing properties and dataset sides expand to empty internal structures in the effective configuration, while explicitly empty placeholder lists or mappings are rejected. - Verifies: `REQ-CFG-003`, `REQ-CFG-005`, `REQ-CFG-014`, `REQ-CFG-060`, `REQ-CFG-061`, `REQ-CFG-062`, `REQ-CFG-063`, `REQ-CFG-069`, `REQ-CFG-070`, `REQ-CFG-075`, `REQ-CFG-076` - **AT-CON-012:** Duplicate or unknown YAML keys, unsafe YAML features, unsupported schema versions, invalid enumerations, and incompatible type/comparator combinations are rejected with precise configuration paths. This consolidated parameterized scenario also covers: Safe-loader tests confirm that configuration parsing cannot construct arbitrary objects or execute tagged payloads. - Verifies: `REQ-CFG-007`, `REQ-CFG-008`, `REQ-CFG-009`, `REQ-CFG-010`, `REQ-CFG-013`, `REQ-CFG-017`, `REQ-CFG-048`, `REQ-CFG-051`, `REQ-CFG-052`, `REQ-CFG-065`, `REQ-CFG-078`, `REQ-CFG-079`, `REQ-CFG-080`, `REQ-CFG-085`, `REQ-CFG-091`, `REQ-SEC-012` -- **AT-CON-014:** The authoritative schema is present in the source tree and built distribution, can be loaded through the installed package-resource API, and validates every complete configuration example and controlled configuration fixture. This consolidated parameterized scenario also covers: The packaged results schema resolves its configuration-schema reference, accepts representative complete and incomplete canonical results, rejects unknown properties and execution-error diagnostics, and requires exactly the documented top-level result properties. - - Verifies: `REQ-CFG-016`, `REQ-CFG-087`, `REQ-CFG-088`, `REQ-OUT-004`, `REQ-OUT-005`, `REQ-OUT-006`, `REQ-OUT-007`, `REQ-OUT-050`, `REQ-OUT-051`, `REQ-OUT-052`, `REQ-OUT-053`, `REQ-OUT-054`, `REQ-OUT-055`, `REQ-OUT-056`, `REQ-OUT-057`, `REQ-OUT-058`, `REQ-OUT-059`, `REQ-OUT-060` -- **AT-CON-015:** Configurations that are structurally schema-valid but violate a semantic rule in Section 6.2.9 are rejected with the relevant configuration path and rule. - - Verifies: `REQ-CFG-090`, `REQ-CFG-091`, `REQ-CFG-104`, `REQ-CFG-105`, `REQ-CFG-106`, `REQ-CFG-107`, `REQ-CFG-108` +- **AT-CON-014:** Each authoritative schema has its specified canonical `$id`, is present in the source tree and built distribution, can be loaded through the installed package-resource API, and validates every complete configuration example and controlled configuration fixture. This consolidated parameterized scenario also covers: The packaged results schema resolves its configuration-schema reference, accepts representative complete and incomplete canonical results, rejects unknown properties and execution-error diagnostics, requires exactly the documented top-level result properties, and enforces the exact permitted shape of every finding form. + - Verifies: `REQ-CFG-128`, `REQ-OUT-004`, `REQ-OUT-005`, `REQ-OUT-100`, `REQ-OUT-112` + - Verifies: `REQ-CFG-016`, `REQ-CFG-087`, `REQ-CFG-088`, `REQ-OUT-004`, `REQ-OUT-005`, `REQ-OUT-006`, `REQ-OUT-007`, `REQ-OUT-050`, `REQ-OUT-051`, `REQ-OUT-052`, `REQ-OUT-053`, `REQ-OUT-054`, `REQ-OUT-055`, `REQ-OUT-056`, `REQ-OUT-057`, `REQ-OUT-058`, `REQ-OUT-059`, `REQ-OUT-060`, `REQ-OUT-100` +- **AT-CON-015:** Configurations that are structurally schema-valid but violate a semantic rule in Section 6.2.9, including reuse of a source field name within one dataset side, are rejected with the relevant configuration path and rule. + - Verifies: `REQ-CFG-090`, `REQ-CFG-091`, `REQ-CFG-104`, `REQ-CFG-105`, `REQ-CFG-106`, `REQ-CFG-107`, `REQ-CFG-108`, `REQ-CFG-127` - **AT-CON-016:** `review_priority` accepts only `low`, `normal`, or `high`; omission resolves to `normal` in the effective configuration. - Verifies: `REQ-CFG-040` - **AT-CON-017:** `fields.record_key` is required, non-empty, ordered, and free of duplicate component fields; every entry names a field in `fields.definitions` that satisfies all key-field constraints. - Verifies: `REQ-CFG-024`, `REQ-CFG-025`, `REQ-CFG-026`, `REQ-CFG-027`, `REQ-CFG-029`, `REQ-CFG-034`, `REQ-CFG-104`, `REQ-CFG-105` - **AT-CON-018:** `metadata.id` and `metadata.revision` are required non-empty strings; title and description are optional, and metadata does not alter comparison behaviour. - Verifies: `REQ-CFG-018`, `REQ-CFG-020`, `REQ-CFG-022` -- **AT-CON-021:** A table-driven JSON parser suite verifies the record-oriented root structure, preserved string-like tokens and numerical lexical forms, and the distinct finding, `not_assessable`, continuation, and execution-error outcomes for non-finite values in non-key fields and record keys. +- **AT-CON-021:** A table-driven JSON parser suite verifies the record-oriented root structure, preserved string-like tokens and numerical lexical forms, and rejection of unquoted `NaN`, `Infinity`, and `-Infinity` as malformed JSON in every location. Results-schema fixtures accept only valid JSON integer lexical forms for `integer` source evidence and valid JSON numerical forms containing a fraction or exponent for `decimal` source evidence. Quoted non-finite representations remain strings; when explicitly parsed as decimals they produce an `invalid_field_value` and not-assessable opportunity in an attributable non-key field, but an execution error in a record-key field. - Verifies: `REQ-INP-029`, `REQ-INP-030`, `REQ-INP-033`, `REQ-INP-047`, `REQ-INP-048`, `REQ-INP-049`, `REQ-INP-050`, `REQ-INP-051`, `REQ-INP-065` - **AT-CON-022:** Every comparison resolves one effective configuration whose schema version, identifier, revision, record key, field definitions, undeclared-field policy, and comparison rules are validated before either dataset is read; unknown keys fail with actionable locations, and the validated configuration is embedded in results. - Verifies: `REQ-CFG-001`, `REQ-CFG-002`, `REQ-CFG-012`, `REQ-CFG-086`, `REQ-CFG-095`, `REQ-CFG-096`, `REQ-CFG-097`, `REQ-CFG-098`, `REQ-CFG-099`, `REQ-CFG-100`, `REQ-CFG-101`, `REQ-CFG-102`, `REQ-CFG-103` -- **AT-CON-023:** Parameterized configuration fixtures verify canonical field-name mapping, collection logical-type structure, legal omission of `logical_type` and operation parameters, placement and side-specific application of source and typed normalisation, and the required comparator declaration. +- **AT-CON-023:** Parameterized configuration fixtures verify canonical field-name mapping, collection logical-type structure, the bidirectional relationship between omission of `logical_type` and `preserved_exact`, rejection of every processing property on a `preserved_exact` field, legal omission of operation parameters, placement and side-specific application of source and typed normalisation, and the required comparator declaration. - Verifies: `REQ-CFG-028`, `REQ-CFG-032`, `REQ-CFG-033`, `REQ-CFG-049`, `REQ-CFG-050`, `REQ-CFG-064`, `REQ-CFG-066`, `REQ-CFG-067`, `REQ-CFG-077` -- **AT-CON-024:** The bundled configuration is schema-valid and implements the documented strict-exact behavior: `record_id` is the key on both sides, shared non-key fields require no declarations, reference-only and query-only fields become contract findings, and cross-format parser-native type differences remain distinguishable. This consolidated parameterized scenario also covers: `preserved_exact` follows the parser-native category table, including deterministic same-format and cross-format comparisons. - - Verifies: `REQ-CFG-093`, `REQ-CFG-094`, `REQ-CFG-109`, `REQ-CFG-110`, `REQ-CFG-115`, `REQ-INP-039`, `REQ-INP-046`, `REQ-CFG-111`, `REQ-CFG-112`, `REQ-CMP-032` +- **AT-CON-024:** The bundled configuration is schema-valid and implements the documented strict-exact behavior: `record_id` is the key on both sides, shared non-key fields require no declarations, reference-only and query-only fields become contract findings, and cross-format parser-native category differences remain distinguishable as `value_mismatch` findings rather than `type_mismatch` findings. This consolidated parameterized scenario also covers: `preserved_exact` follows the parser-native category table, including deterministic same-format and cross-format comparisons. + - Verifies: `REQ-CFG-093`, `REQ-CFG-094`, `REQ-CFG-109`, `REQ-CFG-110`, `REQ-CFG-115`, `REQ-INP-039`, `REQ-INP-046`, `REQ-CFG-111`, `REQ-CFG-112`, `REQ-CMP-032`, `REQ-FND-018` ### 14.2 Record scenarios @@ -1397,22 +1565,22 @@ This catalogue summarises minimum executable scenarios for the first release. It ### 14.3 Typed value and comparator scenarios -- **AT-VAL-001:** Boolean `true`, integer `1`, decimal `1.0`, and textual `"1"` remain distinct under the applicable exact comparator. The parameterized suite also verifies declared parsing rather than inference and enumerates the stable comparator registry. - - Verifies: `REQ-VAL-009`, `REQ-VAL-027`, `REQ-CMP-017`, `REQ-VAL-001`, `REQ-VAL-005`, `REQ-VAL-007`, `REQ-VAL-008`, `REQ-CMP-015` +- **AT-VAL-001:** Boolean `true`, integer `1`, decimal `1.0`, and textual `"1"` remain distinct under the applicable exact comparator. A parameterized source-category matrix verifies every directly accepted category, exact integer-to-decimal widening, rejection of implicit narrowing and stringification, rejection of parser-native objects for typed fields, explicit mapping for representations outside the matrix, null handling before typed parsing, and element-wise application to collections. The suite also verifies declared parsing rather than inference and enumerates the stable comparator registry. + - Verifies: `REQ-VAL-009`, `REQ-VAL-027`, `REQ-CMP-017`, `REQ-VAL-001`, `REQ-VAL-005`, `REQ-VAL-007`, `REQ-VAL-008`, `REQ-VAL-076`, `REQ-VAL-077`, `REQ-VAL-078`, `REQ-VAL-079`, `REQ-VAL-080`, `REQ-VAL-081`, `REQ-VAL-082`, `REQ-VAL-083`, `REQ-VAL-084`, `REQ-VAL-085`, `REQ-CMP-015` - **AT-VAL-002:** Typed strings remain code-point-exact unless configured normalisations are applied in declared order. - Verifies: `REQ-INP-013`, `REQ-VAL-012`, `REQ-VAL-013`, `REQ-VAL-070`, `REQ-CMP-016` - **AT-VAL-003:** Integer parsing accepts signed ASCII decimal integers and leading zeros but rejects decimal points, exponents, booleans, and invalid text. This consolidated parameterized scenario also covers: Leading-zero identifiers remain strings unless explicitly transformed. - Verifies: `REQ-VAL-015`, `REQ-VAL-016`, `REQ-VAL-017`, `REQ-VAL-018` -- **AT-VAL-004:** Decimal parsing is exact, accepts scientific notation, and rejects `NaN` and infinities. Invalid attributable non-key decimals are also checked as contract findings that become `not_assessable`, not execution errors. - - Verifies: `REQ-INP-031`, `REQ-INP-032`, `REQ-INP-045`, `REQ-INP-046`, `REQ-VAL-019`, `REQ-VAL-020`, `REQ-VAL-021`, `REQ-VAL-022`, `REQ-VAL-023`, `REQ-VAL-011` +- **AT-VAL-004:** Decimal parsing is exact; accepts parser-native integers and decimals and textual forms such as `+1.2`, `01.2`, `1`, and `1E+2`; rejects `.5`, `1.`, `NaN`, and infinities; and never passes through binary floating-point. Invalid attributable non-key decimals are also checked as contract findings that become `not_assessable`, not execution errors. + - Verifies: `REQ-INP-031`, `REQ-INP-032`, `REQ-INP-045`, `REQ-INP-046`, `REQ-VAL-019`, `REQ-VAL-020`, `REQ-VAL-021`, `REQ-VAL-022`, `REQ-VAL-023`, `REQ-VAL-078`, `REQ-VAL-011` - **AT-VAL-005:** Boolean parsing accepts native booleans and lowercase textual `true` or `false`; other representations require a mapping. - Verifies: `REQ-VAL-024`, `REQ-VAL-025`, `REQ-VAL-026`, `REQ-VAL-027` - **AT-VAL-006:** Dates reject invalid or non-ISO full dates; datetimes require offsets and compare as instants. - Verifies: `REQ-VAL-028`, `REQ-VAL-029`, `REQ-VAL-030`, `REQ-VAL-031`, `REQ-VAL-032`, `REQ-VAL-033`, `REQ-VAL-034`, `REQ-VAL-035`, `REQ-VAL-036`, `REQ-CMP-024` -- **AT-VAL-007:** Lists preserve order and duplicates, sets reject duplicates and ignore order, and multisets ignore order while preserving occurrence counts. This consolidated parameterized scenario also covers: Collection parsing accepts JSON arrays in JSON and JSON-array cell syntax in CSV/TSV, rejects ordinary delimited text, and rejects null collection elements with the specified finding and `not_assessable` evidence. The suite covers every supported typed collection kind and its required element declaration. - - Verifies: `REQ-VAL-037`, `REQ-VAL-038`, `REQ-VAL-039`, `REQ-VAL-040`, `REQ-VAL-041`, `REQ-VAL-042`, `REQ-CMP-025`, `REQ-VAL-043`, `REQ-VAL-044`, `REQ-VAL-045`, `REQ-VAL-046`, `REQ-VAL-006` -- **AT-VAL-008:** Typed transformations follow Section 6.8 and retain evidence of the attributable source value. This consolidated parameterized scenario also covers: Parameterized normalisation fixtures exercise every registered source-text and typed-value operation, including Unicode, whitespace, case, datetime offset, precision, and unordered-collection ordering, and verify that rounding or significant-figure changes occur only when explicitly configured. The suite also verifies retention of source lexical evidence. - - Verifies: `REQ-VAL-061`, `REQ-VAL-062`, `REQ-VAL-063`, `REQ-VAL-069`, `REQ-VAL-070`, `REQ-VAL-072`, `REQ-VAL-073`, `REQ-VAL-074`, `REQ-VAL-075`, `REQ-VAL-071`, `REQ-CMP-023`, `REQ-VAL-010` +- **AT-VAL-007:** Lists preserve order and duplicates, sets reject typed-canonical duplicates and ignore order, and multisets ignore order while preserving occurrence counts. List elements support positional typed-exact comparison and numeric tolerance for numeric element types; sets and multisets accept only typed-exact element comparison, and every other element-comparator combination is rejected. This consolidated parameterized scenario also covers: Collection parsing accepts JSON arrays in JSON and JSON-array cell syntax in CSV/TSV, rejects ordinary delimited text, and rejects null collection elements with the specified finding and `not_assessable` evidence. The suite covers every supported typed collection kind and its required element declaration. + - Verifies: `REQ-CFG-124`, `REQ-CFG-125`, `REQ-CFG-126`, `REQ-VAL-037`, `REQ-VAL-038`, `REQ-VAL-039`, `REQ-VAL-040`, `REQ-VAL-041`, `REQ-VAL-042`, `REQ-CMP-025`, `REQ-CMP-040`, `REQ-CMP-041`, `REQ-VAL-043`, `REQ-VAL-044`, `REQ-VAL-045`, `REQ-VAL-046`, `REQ-VAL-006` +- **AT-VAL-008:** Typed transformations follow Section 6.8 and retain evidence of the attributable source value. This consolidated parameterized scenario also covers: Parameterized normalisation fixtures exercise every registered source-text and typed-value operation, verify that source-text operations affect strings but pass non-strings through unchanged, and cover Unicode normalization, whitespace trimming, case folding, decimal quantisation, and datetime truncation. Rounding or precision reduction occurs only when explicitly configured. Deterministic set and multiset output ordering is verified as canonical serialization rather than configured normalisation. The suite also verifies retention of source lexical evidence. + - Verifies: `REQ-CFG-123`, `REQ-VAL-061`, `REQ-VAL-062`, `REQ-VAL-063`, `REQ-VAL-069`, `REQ-VAL-070`, `REQ-VAL-072`, `REQ-VAL-073`, `REQ-VAL-074`, `REQ-VAL-075`, `REQ-VAL-071`, `REQ-CMP-023`, `REQ-VAL-010` - **AT-VAL-009:** Numeric tolerance accepts exactly one absolute or relative tolerance and rejects configurations containing neither or both. Each form uses exact decimal arithmetic and an inclusive boundary, with below, at, and above boundary cases; relative tolerance is reference-directed and permits no non-zero difference when the reference is zero. - Verifies: `REQ-CFG-081`, `REQ-CFG-082`, `REQ-CFG-083`, `REQ-CFG-084`, `REQ-CMP-018`, `REQ-CMP-019`, `REQ-CMP-020`, `REQ-CMP-021`, `REQ-CMP-022` - **AT-VAL-010:** All assessable differences are collected in one run. @@ -1422,37 +1590,35 @@ This catalogue summarises minimum executable scenarios for the first release. It ### 14.4 Phase and run-status scenarios -- **AT-OUT-001:** Every comparison performs the contract, records, and values phases in order and records a completion status for each. - - Verifies: `REQ-CMP-001`, `REQ-RUN-012`, `REQ-OUT-013` +- **AT-OUT-001:** Every comparison performs the contract, records, and values phases in order and records a completion status for each. Overall status is complete if and only if all three phase statuses are complete. + - Verifies: `REQ-CMP-001`, `REQ-RUN-012`, `REQ-RUN-023`, `REQ-OUT-013`, `REQ-OUT-014` - **AT-OUT-002:** A finding or `not_assessable` item in an earlier phase does not prevent later assessment where prerequisites remain satisfied. - Verifies: `REQ-CMP-002`, `REQ-CMP-003`, `REQ-RUN-007` -- **AT-OUT-005:** Complete, incomplete, invocation or input error, and unexpected internal error produce their documented exit behaviour. This consolidated parameterized scenario also covers: A fully performed comparison is `complete` whether it finds no differences or any combination of contract findings, missing records, extra records, and value mismatches. - - Verifies: `REQ-RUN-001`, `REQ-RUN-015`, `REQ-RUN-016`, `REQ-RUN-017`, `REQ-CLI-014`, `REQ-CLI-015`, `REQ-CLI-016`, `REQ-RUN-002` -- **AT-OUT-006:** Configured `not_assessable` content produces `incomplete`; deliberately `not_assessed` content does not. This consolidated parameterized scenario also covers: Content excluded deliberately is `not_assessed`; configured content blocked by a failed prerequisite is `not_assessable`. - - Verifies: `REQ-RUN-003`, `REQ-RUN-004`, `REQ-RUN-005`, `REQ-RUN-016`, `REQ-CMP-004` +- **AT-OUT-005:** Complete, incomplete, invocation or input error, and unexpected internal error produce their documented exit behaviour. This consolidated parameterized scenario also covers: Fully performed contract and record phases remain complete despite their findings, and a fully performed comparison is `complete` whether it finds no differences or any combination of contract findings, missing records, extra records, and value mismatches. + - Verifies: `REQ-RUN-001`, `REQ-RUN-015`, `REQ-RUN-016`, `REQ-RUN-017`, `REQ-RUN-018`, `REQ-RUN-019`, `REQ-RUN-021`, `REQ-CLI-014`, `REQ-CLI-015`, `REQ-CLI-016`, `REQ-RUN-002` +- **AT-OUT-006:** A not-assessable value opportunity makes the values phase and run incomplete while leaving a fully performed contract phase complete; deliberately `not_assessed` content does not make any phase incomplete. This consolidated parameterized scenario also covers: Content excluded deliberately is `not_assessed`; configured content blocked by a failed prerequisite is `not_assessable`. + - Verifies: `REQ-RUN-003`, `REQ-RUN-004`, `REQ-RUN-005`, `REQ-RUN-016`, `REQ-RUN-018`, `REQ-RUN-020`, `REQ-RUN-021`, `REQ-RUN-022`, `REQ-CMP-004` - **AT-OUT-007:** Invalid configuration, malformed or unreadable input, an incomplete or ambiguous record model, an enforced resource limit, and resource exhaustion produce `error`, never `incomplete` or a partial comparison result. - Verifies: `REQ-RUN-017`, `REQ-RUN-010`, `REQ-RUN-011`, `REQ-DEP-011` - **AT-OUT-008:** An end-to-end mixed-defect comparison exercises every conceptual processing step, including invalid contract values, deliberate exclusions, failed prerequisites, mandatory key processing, and continued comparison of unaffected values. This consolidated parameterized scenario also covers: A large mixed-content fixture verifies that every configured value opportunity is either assessed or explicitly classified and that no valid accepted content is silently skipped. - Verifies: `REQ-CMP-005`, `REQ-CMP-008`, `REQ-CMP-014`, `REQ-CMP-030`, `REQ-CMP-037`, `REQ-CMP-038`, `REQ-CMP-039`, `REQ-RUN-006`, `REQ-DEP-010` -- **AT-OUT-010:** A table-driven finding suite generates every finding code and verifies its structured fields, required contextual evidence, deterministic local ID, human-readable message, summary-only successful matches, and absence of message-only semantics. This consolidated parameterized scenario also covers: For every finding code, result fixtures enforce the code-specific inclusion and omission of source and canonical values, deterministic messages derived from structured data, deterministic sorting, and sequential report-local finding IDs. - - Verifies: `REQ-PRN-006`, `REQ-FND-001`, `REQ-FND-002`, `REQ-FND-003`, `REQ-FND-004`, `REQ-FND-005`, `REQ-FND-006`, `REQ-FND-012`, `REQ-FND-013`, `REQ-FND-014`, `REQ-FND-015`, `REQ-FND-016`, `REQ-FND-017`, `REQ-OUT-032`, `REQ-OUT-033`, `REQ-OUT-035`, `REQ-OUT-061`, `REQ-OUT-062`, `REQ-OUT-063`, `REQ-OUT-064`, `REQ-OUT-065`, `REQ-OUT-066`, `REQ-OUT-067`, `REQ-OUT-068` +- **AT-OUT-010:** A table-driven finding suite generates every finding code and verifies its fixed classification, structured fields, required contextual evidence, deterministic local ID, human-readable message, summary-only successful matches, and absence of message-only semantics. `missing_record`, `extra_record`, and `value_mismatch` are always `comparison_difference`; every other finding code is always `assessment_issue`; and phase remains independently correct. This consolidated parameterized scenario also covers: For every finding code, result fixtures enforce the code-specific inclusion and omission of source and canonical values, deterministic messages derived from structured data, classification-first total finding order, duplicate coalescence, and sequential post-sort report-local IDs. Category incompatibility, invalid content within an accepted category, and comparison of two valid unequal values produce mutually exclusive `type_mismatch`, `invalid_field_value`, and `value_mismatch` findings. Independent reference and query contract violations produce separate side-specific findings, while comparison of two valid unequal values produces one `both` finding. Each not-assessable value opportunity has one applicable causal finding and does not generate a separate `not_assessable` finding. + - Verifies: `REQ-PRN-006`, `REQ-FND-001`, `REQ-FND-002`, `REQ-FND-003`, `REQ-FND-004`, `REQ-FND-005`, `REQ-FND-006`, `REQ-FND-012`, `REQ-FND-013`, `REQ-FND-014`, `REQ-FND-015`, `REQ-FND-016`, `REQ-FND-017`, `REQ-FND-018`, `REQ-FND-019`, `REQ-FND-020`, `REQ-FND-023`, `REQ-FND-024`, `REQ-FND-025`, `REQ-OUT-032`, `REQ-OUT-033`, `REQ-OUT-034`, `REQ-OUT-035`, `REQ-OUT-099`, `REQ-OUT-061`, `REQ-OUT-062`, `REQ-OUT-063`, `REQ-OUT-064`, `REQ-OUT-065`, `REQ-OUT-066`, `REQ-OUT-067`, `REQ-OUT-068` ### 14.5 Evidence, CLI, and packaging scenarios -- **AT-EVD-001:** Identical runs produce semantically identical canonical results except for explicitly variable run metadata. This consolidated parameterized scenario also covers: Record order does not alter findings, metrics, or run and phase statuses. - - Verifies: `REQ-PRN-005`, `REQ-CMP-033`, `REQ-OUT-012`, `REQ-DEP-013`, `REQ-CMP-032` +- **AT-EVD-001:** Runs with identical exact bytes, basenames, configuration, toolkit version, and runtime identity produce byte-identical canonical JSON after normalizing only run ID and completion time. Serialization fixtures verify UTF-8 without a byte-order mark, two-space indentation, LF line endings, exactly one final newline, direct non-ASCII output, fixed-schema member order, and Unicode code-point order for dynamic members. Parameterized record-order and LF/CRLF variants preserve the defined comparison outcome while permitting only the documented byte-provenance, source-location, location-dependent message, and report-local ID differences. The comparison outcome also remains invariant across every supported Python and operating-system environment, including Unicode-sensitive fixtures. + - Verifies: `REQ-PRN-005`, `REQ-INP-014`, `REQ-CMP-033`, `REQ-OUT-012`, `REQ-OUT-095`, `REQ-OUT-096`, `REQ-OUT-097`, `REQ-OUT-098`, `REQ-OUT-125`, `REQ-DEP-013`, `REQ-CMP-032` - **AT-EVD-002:** Results contain exact-byte SHA-256 input digests and input basenames without exposing absolute paths, raw command lines, environment variables, or secrets; the complete validated comparison configuration is embedded without a separate configuration digest. - Verifies: `REQ-CFG-023`, `REQ-OUT-017`, `REQ-OUT-018`, `REQ-OUT-019`, `REQ-OUT-022`, `REQ-OUT-023`, `REQ-SEC-007`, `REQ-SEC-008` -- **AT-EVD-003:** If the optional CSV findings export is provided, findings containing commas, quotes, newlines, or formula prefixes are exported safely. - - Verifies: `REQ-OUT-043`, `REQ-OUT-083`, `REQ-OUT-084`, `REQ-OUT-085`, `REQ-OUT-087`, `REQ-SEC-013` -- **AT-EVD-004:** An output failure or interrupted write cannot leave an evidence package that appears complete. This consolidated parameterized scenario also covers: Success and injected-failure runs verify that temporary files are no more accessible than the destination and are removed after success and, where safe, after errors. - - Verifies: `REQ-OUT-044`, `REQ-OUT-045`, `REQ-OUT-049`, `REQ-DEP-012`, `REQ-SEC-009`, `REQ-SEC-010`, `REQ-SEC-011` +- **AT-EVD-004:** An output failure or interrupted write cannot leave an evidence package that appears complete. This consolidated parameterized scenario also covers: Success and injected-failure runs verify that temporary files are no more accessible than the destination and are removed after success and, where safe, after errors; a failed attempted replacement preserves the existing evidence package byte for byte. + - Verifies: `REQ-OUT-044`, `REQ-OUT-045`, `REQ-OUT-049`, `REQ-OUT-116`, `REQ-DEP-012`, `REQ-SEC-009`, `REQ-SEC-010`, `REQ-SEC-011` - **AT-EVD-005:** Every complete comparison exits with code 0 and produces its evidence package, including when differences are reported. - Verifies: `REQ-PRN-004`, `REQ-RUN-002`, `REQ-OUT-001`, `REQ-OUT-002`, `REQ-CLI-016` -- **AT-EVD-008:** A complete or incomplete run creates the documented JSON and HTML evidence package, refuses an existing destination by default, and replaces it only with `--overwrite`. This consolidated parameterized scenario also covers: Output-directory reporting and filename handling preserve the exact selected destination and manipulate suffixes structurally rather than by character stripping. - - Verifies: `REQ-OUT-001`, `REQ-OUT-002`, `REQ-OUT-046`, `REQ-CLI-011`, `REQ-CLI-012`, `REQ-OUT-047`, `REQ-OUT-048` -- **AT-EVD-009:** When an output location can be established and an error report can be written, invocation, configuration, input, resource, output, and internal errors produce a clearly identified diagnostic artefact with a stable execution error code. It contains no findings, finding codes, comparison summary metrics, or phase statuses and cannot be mistaken for a comparison result or evidence package. - - Verifies: `REQ-FND-007`, `REQ-RUN-008`, `REQ-RUN-009`, `REQ-RUN-010`, `REQ-RUN-011`, `REQ-OUT-008`, `REQ-DEP-014` +- **AT-EVD-008:** A complete or incomplete run creates the documented two-file evidence package containing `comparison-results.json` and `comparison-report.html`, with no separate manifest or additional report or findings-export format. It refuses every existing destination by default. With `--overwrite`, it replaces a complete two-file v1 evidence-package directory but rejects regular files, symbolic links, special filesystem objects, empty directories, partial packages, and directories with additional entries. This consolidated parameterized scenario also covers: Output-directory reporting and filename handling preserve the exact selected destination and manipulate suffixes structurally rather than by character stripping. + - Verifies: `REQ-OUT-001`, `REQ-OUT-002`, `REQ-OUT-044`, `REQ-OUT-045`, `REQ-OUT-046`, `REQ-OUT-047`, `REQ-OUT-048`, `REQ-OUT-115`, `REQ-CLI-011`, `REQ-CLI-012` +- **AT-EVD-009:** A parameterized suite produces every version 1 invocation, configuration, input, resource, output, and internal execution-error code. Each case writes the exact `ERROR [.]: ` first-line form and only permitted non-sensitive continuation context, returns exit code 2 except for `internal.unexpected`, which returns 3, and publishes no evidence-package directory or filesystem report. Diagnostics contain no findings, finding codes, comparison summary metrics, phase statuses, record identifiers, source values, or ordinary-use traceback. Representative failures at each classification boundary verify that the most specific applicable code is selected. + - Verifies: `REQ-FND-007`, `REQ-RUN-008`, `REQ-RUN-009`, `REQ-RUN-010`, `REQ-RUN-011`, `REQ-RUN-024`, `REQ-RUN-025`, `REQ-RUN-026`, `REQ-OUT-008`, `REQ-DEP-014` - **AT-EVD-010:** Routine logs, progress output, terminal diagnostics, and execution-error diagnostics omit record identifiers and source values. - Verifies: `REQ-PRN-007`, `REQ-SEC-004`, `REQ-VAL-075` - **AT-EVD-011:** Evidence packages warn that they may contain sensitive information, and findings contain record identifiers or source values only where needed to identify or explain the finding. @@ -1463,28 +1629,30 @@ This catalogue summarises minimum executable scenarios for the first release. It - Verifies: `REQ-CFG-041`, `REQ-CFG-042`, `REQ-CFG-043`, `REQ-CFG-044`, `REQ-CFG-045`, `REQ-OUT-034` - **AT-PKG-001:** The built distribution is named `adviso-validation`, contains an importable `adviso_validation` package and every required subpackage, installs the `adviso-validation` executable, and completes an end-to-end comparison through that executable. This consolidated parameterized scenario also covers: The release artefact can be installed and used without network access, contains offline documentation and integrity information, and supports an end-to-end comparison in an isolated environment. - Verifies: `REQ-PRD-001`, `REQ-PRD-003`, `REQ-CFG-087`, `REQ-TST-013`, `REQ-TST-030`, `REQ-DEP-003`, `REQ-DEP-007` -- **AT-EVD-015:** Canonical results contain valid run, phase, software, runtime, and fixed-dialect metadata; timestamps are UTC RFC 3339 values; start time and duration are absent; and overall status reconciles with phase statuses. - - Verifies: `REQ-OUT-009`, `REQ-OUT-010`, `REQ-OUT-011`, `REQ-OUT-014`, `REQ-OUT-015`, `REQ-OUT-016`, `REQ-OUT-020` -- **AT-EVD-016:** Parameterized single and composite record identities verify ordered source and canonical key arrays, null identities for an absent side, presence-state distinctions, recursive parser-native typed values, canonical scalar and collection encodings, and deterministic object-member ordering. - - Verifies: `REQ-OUT-021`, `REQ-OUT-026`, `REQ-OUT-027`, `REQ-OUT-028`, `REQ-OUT-029` -- **AT-EVD-018:** Parameterized result fixtures verify fixed-shape zero-inclusive summaries, record-count reconciliation, finding totals by code and priority, and exactly one matched, mismatched, or not-assessable classification for every in-scope non-key value opportunity, including optional, required, invalid, and prohibited values. - - Verifies: `REQ-OUT-037`, `REQ-OUT-038`, `REQ-OUT-039`, `REQ-OUT-040`, `REQ-OUT-069`, `REQ-OUT-070`, `REQ-OUT-071`, `REQ-OUT-072`, `REQ-OUT-073` -- **AT-EVD-019:** HTML generated solely from canonical JSON uses an open non-proprietary format and presents the run purpose, phase and run statuses, input and configuration identities, counts, findings, discrepancies, limitations, exclusions, unassessed content, provenance, and no acceptance verdict; HTML is never the sole canonical evidence format. This consolidated parameterized scenario also covers: Complete and incomplete reports contain raw findings and metrics but no acceptance threshold, pass/fail verdict, or equivalent inferred laboratory decision. - - Verifies: `REQ-OUT-042`, `REQ-OUT-074`, `REQ-OUT-075`, `REQ-OUT-076`, `REQ-OUT-077`, `REQ-OUT-078`, `REQ-OUT-079`, `REQ-OUT-080`, `REQ-OUT-081`, `REQ-OUT-088`, `REQ-DEP-004`, `REQ-DEP-005`, `REQ-RUN-013` +- **AT-EVD-015:** Canonical results contain valid run, phase, software, runtime, and fixed-dialect metadata; run IDs are canonical lowercase UUIDv4 values; completion timestamps are UTC RFC 3339 values using `Z` and whole-second precision; Python implementation/version, normalized platform, and Unicode database version use their prescribed forms without identifying host details; start time and duration are absent; and overall status reconciles with phase statuses. + - Verifies: `REQ-OUT-009`, `REQ-OUT-010`, `REQ-OUT-011`, `REQ-OUT-014`, `REQ-OUT-015`, `REQ-OUT-016`, `REQ-OUT-020`, `REQ-OUT-121`, `REQ-OUT-122`, `REQ-OUT-123`, `REQ-OUT-124` +- **AT-EVD-016:** Parameterized single and composite record identities verify ordered source and canonical key arrays, null identities for an absent side, presence-state distinctions, recursive parser-native typed values, unique canonical rendering of every scalar type, UTC datetime rendering, canonical key-component rendering, deterministic set and multiset element order, and deterministic object-member ordering. Source-location fixtures verify tabular data-record and starting-line numbers across multiline records, field-name inclusion only for present fields, record-level locations for absent fields, JSON record and field pointers, RFC 6901 escaping, and null locations for absent records. + - Verifies: `REQ-OUT-021`, `REQ-OUT-026`, `REQ-OUT-027`, `REQ-OUT-028`, `REQ-OUT-029`, `REQ-OUT-089`, `REQ-OUT-090`, `REQ-OUT-091`, `REQ-OUT-092`, `REQ-OUT-093`, `REQ-OUT-094`, `REQ-OUT-117`, `REQ-OUT-118`, `REQ-OUT-119` +- **AT-EVD-018:** Parameterized result fixtures verify fixed-shape zero-inclusive summaries, record-count reconciliation, finding totals by classification, code, and priority, and exactly one matched, mismatched, or not-assessable classification for every in-scope non-key value opportunity, including optional, required, invalid, and prohibited values. + - Verifies: `REQ-OUT-037`, `REQ-OUT-038`, `REQ-OUT-039`, `REQ-OUT-040`, `REQ-OUT-069`, `REQ-OUT-070`, `REQ-OUT-071`, `REQ-OUT-072`, `REQ-OUT-073`, `REQ-OUT-101`, `REQ-OUT-102`, `REQ-OUT-103`, `REQ-OUT-104`, `REQ-OUT-105`, `REQ-OUT-106`, `REQ-OUT-107`, `REQ-OUT-108` +- **AT-EVD-022:** Before publication, representative generated results pass both structural and semantic self-validation with Draft 2020-12 format assertion enabled. Injected schema and semantic inconsistencies, including malformed UUIDs, dates, and date-times, produce an unexpected internal error and publish no canonical result, HTML report, or evidence-package directory. + - Verifies: `REQ-OUT-109`, `REQ-OUT-110`, `REQ-OUT-114` +- **AT-EVD-019:** HTML generated solely from canonical JSON uses an open non-proprietary format and presents the run purpose, phase and run statuses, input and configuration identities, counts, findings, discrepancies, limitations, exclusions, unassessed content, provenance, and no acceptance verdict; HTML is never the sole canonical evidence format. Its headline distinguishes run completeness, comparison-difference count, and assessment-issue count. Detailed findings are organized primarily by classification with comparison differences first and phase retained as secondary traceability information. This consolidated parameterized scenario also covers: Complete and incomplete reports contain raw findings and metrics but no acceptance threshold, pass/fail verdict, or equivalent inferred laboratory decision. + - Verifies: `REQ-PRN-009`, `REQ-OUT-042`, `REQ-OUT-074`, `REQ-OUT-075`, `REQ-OUT-076`, `REQ-OUT-077`, `REQ-OUT-078`, `REQ-OUT-079`, `REQ-OUT-080`, `REQ-OUT-081`, `REQ-OUT-120`, `REQ-OUT-126`, `REQ-OUT-127`, `REQ-OUT-128`, `REQ-DEP-004`, `REQ-DEP-005`, `REQ-RUN-013` - **AT-EVD-021:** Installed-CLI tests cover the required `adviso-validation compare` form, required reference, query, and output arguments, optional configuration, enumerated choices, invalid combinations before input access, normal summaries on standard output, and optional progress that never contaminates machine-readable output. This consolidated parameterized scenario also covers: Omitting `--configuration` selects and records the bundled strict-exact configuration; a supplied value is treated only as a custom comparison-configuration path. The reference and query formats are independently determined and recorded from recognised, case-insensitive filename extensions; a missing, unknown, or ambiguous extension causes an input error, and content that does not conform to the indicated format is rejected. Command-line `--help` describes every version 1 command and option and directs users to the full guide. - Verifies: `REQ-PRD-002`, `REQ-INP-001`, `REQ-INP-002`, `REQ-INP-004`, `REQ-INP-060`, `REQ-INP-061`, `REQ-INP-062`, `REQ-CLI-001`, `REQ-CLI-002`, `REQ-CLI-003`, `REQ-CLI-006`, `REQ-CLI-007`, `REQ-CLI-008`, `REQ-INP-003`, `REQ-CFG-092`, `REQ-CLI-009`, `REQ-CLI-010`, `REQ-INP-005`, `REQ-INP-006`, `REQ-INP-007`, `REQ-INP-008`, `REQ-INP-009`, `REQ-CLI-013`, `REQ-CLI-004`, `REQ-DOC-005` ### 14.6 Documentation scenarios - **AT-DOC-001:** The built distribution contains an offline user guide covering every topic required by Section 13.4, and the guide identifies the toolkit and schema versions to which it applies. This consolidated parameterized scenario also covers: Every complete comparison-configuration and canonical-results example in the guide validates against its corresponding packaged schema, and every documented command-line invocation uses the version 1 interface. - - Verifies: `REQ-DOC-001`, `REQ-DOC-006`, `REQ-DOC-007`, `REQ-DOC-008`, `REQ-DOC-009`, `REQ-DOC-010`, `REQ-DOC-011`, `REQ-DOC-012`, `REQ-DOC-013`, `REQ-DOC-014`, `REQ-DOC-015`, `REQ-DOC-016`, `REQ-DOC-002`, `REQ-DOC-003`, `REQ-DOC-004` + - Verifies: `REQ-DOC-001`, `REQ-DOC-006`, `REQ-DOC-007`, `REQ-DOC-008`, `REQ-DOC-009`, `REQ-DOC-010`, `REQ-DOC-011`, `REQ-DOC-012`, `REQ-DOC-013`, `REQ-DOC-014`, `REQ-DOC-015`, `REQ-DOC-016`, `REQ-DOC-002`, `REQ-DOC-003`, `REQ-DOC-004`, `REQ-OUT-111` ### 14.7 Fixture and compatibility scenarios - **AT-TRC-002:** Regenerating every generated fixture from its recorded source or generator produces byte-identical or semantically identical controlled test data, as applicable. - Verifies: `REQ-TST-005` -- **AT-TRC-003:** Every supported and unsupported comparison-configuration and results-schema version is exercised to confirm exact supported-version handling and clear rejection of unsupported versions. - - Verifies: `REQ-REL-004` +- **AT-TRC-003:** Every supported and unsupported comparison-configuration and results-schema version is exercised to confirm exact supported-version handling and clear rejection of unsupported versions. A version 1.0 result accepts only an embedded version 1.0 comparison configuration. + - Verifies: `REQ-REL-002`, `REQ-REL-004`, `REQ-OUT-113` ## 15. Status of the current prototype diff --git a/src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json b/src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json index f1302e6..5133346 100644 --- a/src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json +++ b/src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json @@ -1,5 +1,6 @@ { "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://adviso-project.github.io/validation_toolkit/schemas/comparison-configuration-v1.0.schema.json", "title": "AdvISO comparison configuration", "description": "Normative structural schema for comparison-configuration schema version 1.0.", "type": "object", @@ -211,6 +212,63 @@ ] } } + }, + { + "if": { + "properties": { + "comparator": { + "properties": { + "name": { + "const": "preserved_exact" + } + }, + "required": [ + "name" + ] + } + }, + "required": [ + "comparator" + ] + }, + "then": { + "not": { + "anyOf": [ + { + "required": [ + "logical_type" + ] + }, + { + "required": [ + "source_normalisation" + ] + }, + { + "required": [ + "value_mapping" + ] + }, + { + "required": [ + "typed_normalisation" + ] + }, + { + "properties": { + "nulls": { + "required": [ + "missing_tokens" + ] + } + }, + "required": [ + "nulls" + ] + } + ] + } + } } ] }, @@ -251,7 +309,81 @@ "element_comparator": { "$ref": "#/$defs/comparator" } - } + }, + "allOf": [ + { + "properties": { + "element_comparator": { + "properties": { + "name": { + "enum": [ + "typed_exact", + "numeric_tolerance" + ] + } + } + } + } + }, + { + "if": { + "properties": { + "name": { + "enum": [ + "set", + "multiset" + ] + } + }, + "required": [ + "name" + ] + }, + "then": { + "properties": { + "element_comparator": { + "properties": { + "name": { + "const": "typed_exact" + } + } + } + } + } + }, + { + "if": { + "properties": { + "element_comparator": { + "properties": { + "name": { + "const": "numeric_tolerance" + } + }, + "required": [ + "name" + ] + } + }, + "required": [ + "element_comparator" + ] + }, + "then": { + "properties": { + "name": { + "const": "list" + }, + "element_type": { + "enum": [ + "integer", + "decimal" + ] + } + } + } + } + ] } ] }, @@ -278,14 +410,14 @@ "type": "array", "minItems": 1, "items": { - "$ref": "#/$defs/sourceValueDescriptor" + "$ref": "#/$defs/nonNullSourceValueDescriptor" } }, "query": { "type": "array", "minItems": 1, "items": { - "$ref": "#/$defs/sourceValueDescriptor" + "$ref": "#/$defs/nonNullSourceValueDescriptor" } } } @@ -395,6 +527,25 @@ } ] }, + "nonNullSourceValueDescriptor": { + "allOf": [ + { + "$ref": "#/$defs/sourceValueDescriptor" + }, + { + "not": { + "properties": { + "category": { + "const": "null" + } + }, + "required": [ + "category" + ] + } + } + ] + }, "sourceNormalisation": { "oneOf": [ { @@ -571,7 +722,7 @@ ], "properties": { "from": { - "$ref": "#/$defs/sourceValueDescriptor" + "$ref": "#/$defs/nonNullSourceValueDescriptor" }, "to": { "$ref": "#/$defs/sourceValueDescriptor" diff --git a/src/adviso_validation/schemas/comparison-results-v1.0.schema.json b/src/adviso_validation/schemas/comparison-results-v1.0.schema.json index 050129b..2862624 100644 --- a/src/adviso_validation/schemas/comparison-results-v1.0.schema.json +++ b/src/adviso_validation/schemas/comparison-results-v1.0.schema.json @@ -1,5 +1,6 @@ { "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://adviso-project.github.io/validation_toolkit/schemas/comparison-results-v1.0.schema.json", "title": "AdvISO comparison results", "description": "Normative schema for canonical comparison results schema version 1.0. Execution-error diagnostics are not comparison results and do not validate against this schema.", "type": "object", @@ -84,7 +85,8 @@ "properties": { "id": { "type": "string", - "format": "uuid" + "format": "uuid", + "pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$" }, "status": { "$ref": "#/$defs/phaseStatus" @@ -92,7 +94,7 @@ "completed_at": { "type": "string", "format": "date-time", - "pattern": "Z$" + "pattern": "^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$" } } }, @@ -145,13 +147,16 @@ ], "properties": { "python": { - "$ref": "#/$defs/nonEmptyString" + "type": "string", + "pattern": "^[a-z][a-z0-9_]*-[0-9]+\\.[0-9]+\\.[0-9]+(?:[a-z]+[0-9]+)?$" }, "platform": { - "$ref": "#/$defs/nonEmptyString" + "type": "string", + "pattern": "^[a-z][a-z0-9_]*-[a-z0-9_]+$" }, "unicode_version": { - "$ref": "#/$defs/nonEmptyString" + "type": "string", + "pattern": "^[0-9]+\\.[0-9]+\\.[0-9]+$" } } }, @@ -320,8 +325,7 @@ "invalid_field_value", "null_policy_violation", "mapping_failure", - "value_mismatch", - "not_assessable" + "value_mismatch" ], "properties": { "missing_record": { @@ -350,8 +354,21 @@ }, "value_mismatch": { "$ref": "#/$defs/nonNegativeInteger" + } + } + }, + "findingCountsByClassification": { + "type": "object", + "additionalProperties": false, + "required": [ + "comparison_difference", + "assessment_issue" + ], + "properties": { + "comparison_difference": { + "$ref": "#/$defs/nonNegativeInteger" }, - "not_assessable": { + "assessment_issue": { "$ref": "#/$defs/nonNegativeInteger" } } @@ -385,6 +402,7 @@ "additionalProperties": false, "required": [ "total", + "by_classification", "by_code", "by_review_priority" ], @@ -392,6 +410,9 @@ "total": { "$ref": "#/$defs/nonNegativeInteger" }, + "by_classification": { + "$ref": "#/$defs/findingCountsByClassification" + }, "by_code": { "$ref": "#/$defs/findingCountsByCode" }, @@ -488,6 +509,7 @@ "type": "object", "additionalProperties": false, "required": [ + "record_number", "line_number" ], "properties": { @@ -564,17 +586,47 @@ ], "properties": { "category": { - "enum": [ - "string", - "integer", - "decimal" - ] + "const": "string" }, "value": { "type": "string" } } }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "category", + "value" + ], + "properties": { + "category": { + "const": "integer" + }, + "value": { + "type": "string", + "pattern": "^-?(?:0|[1-9][0-9]*)$" + } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "category", + "value" + ], + "properties": { + "category": { + "const": "decimal" + }, + "value": { + "type": "string", + "pattern": "^-?(?:0|[1-9][0-9]*)(?:(?:\\.[0-9]+)(?:[eE][+-]?[0-9]+)?|[eE][+-]?[0-9]+)$" + } + } + }, { "type": "object", "additionalProperties": false, @@ -649,7 +701,7 @@ }, "canonicalDecimalText": { "type": "string", - "pattern": "^(?:0|-?(?:[1-9][0-9]*)(?:\\.[0-9]*[1-9])?(?:e-?[1-9][0-9]*)?|-?0\\.[0-9]*[1-9](?:e-?[1-9][0-9]*)?)$" + "pattern": "^(?:0|-?[1-9](?:\\.[0-9]*[1-9])?(?:e-?[1-9][0-9]*)?)$" }, "canonicalValue": { "oneOf": [ @@ -747,7 +799,8 @@ }, "value": { "type": "string", - "format": "date-time" + "format": "date-time", + "pattern": "^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}(?:\\.[0-9]*[1-9])?Z$" } } }, @@ -927,7 +980,8 @@ "value": { "items": { "type": "string", - "format": "date-time" + "format": "date-time", + "pattern": "^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}(?:\\.[0-9]*[1-9])?Z$" } } } @@ -1016,10 +1070,240 @@ } }, "finding": { + "oneOf": [ + { + "allOf": [ + { + "$ref": "#/$defs/findingShape" + }, + { + "properties": { + "code": { + "const": "missing_record" + } + }, + "required": [ + "code" + ], + "not": { + "anyOf": [ + { "required": ["field"] }, + { "required": ["source_field"] }, + { "required": ["record_count"] }, + { "required": ["review_priority"] }, + { "required": ["values"] } + ] + } + } + ] + }, + { + "allOf": [ + { + "$ref": "#/$defs/findingShape" + }, + { + "properties": { + "code": { + "const": "extra_record" + } + }, + "required": [ + "code" + ], + "not": { + "anyOf": [ + { "required": ["field"] }, + { "required": ["source_field"] }, + { "required": ["record_count"] }, + { "required": ["review_priority"] }, + { "required": ["values"] } + ] + } + } + ] + }, + { + "allOf": [ + { + "$ref": "#/$defs/findingShape" + }, + { + "properties": { + "code": { + "const": "missing_required_field" + } + }, + "required": [ + "code", + "record_identity", + "source_locations", + "field", + "review_priority", + "values" + ], + "not": { + "anyOf": [ + { "required": ["source_field"] }, + { "required": ["record_count"] } + ] + } + } + ] + }, + { + "allOf": [ + { + "$ref": "#/$defs/findingShape" + }, + { + "properties": { + "code": { + "const": "missing_required_field" + } + }, + "required": [ + "code", + "field", + "review_priority" + ], + "not": { + "anyOf": [ + { "required": ["record_identity"] }, + { "required": ["source_locations"] }, + { "required": ["source_field"] }, + { "required": ["record_count"] }, + { "required": ["values"] } + ] + } + } + ] + }, + { + "allOf": [ + { + "$ref": "#/$defs/findingShape" + }, + { + "properties": { + "code": { + "enum": [ + "type_mismatch", + "invalid_field_value", + "null_policy_violation", + "mapping_failure" + ] + } + }, + "required": [ + "code", + "record_identity", + "source_locations", + "field", + "review_priority", + "values" + ], + "not": { + "anyOf": [ + { "required": ["source_field"] }, + { "required": ["record_count"] } + ] + } + } + ] + }, + { + "allOf": [ + { + "$ref": "#/$defs/findingShape" + }, + { + "properties": { + "code": { + "const": "undeclared_field" + } + }, + "required": [ + "code", + "source_field", + "record_count" + ], + "not": { + "anyOf": [ + { "required": ["record_identity"] }, + { "required": ["source_locations"] }, + { "required": ["field"] }, + { "required": ["review_priority"] }, + { "required": ["values"] } + ] + } + } + ] + }, + { + "allOf": [ + { + "$ref": "#/$defs/findingShape" + }, + { + "properties": { + "code": { + "const": "value_mismatch" + } + }, + "required": [ + "code", + "record_identity", + "source_locations", + "field", + "review_priority", + "values" + ], + "not": { + "anyOf": [ + { "required": ["source_field"] }, + { "required": ["record_count"] } + ] + } + } + ] + }, + { + "allOf": [ + { + "$ref": "#/$defs/findingShape" + }, + { + "properties": { + "code": { + "const": "value_mismatch" + } + }, + "required": [ + "code", + "record_identity", + "source_locations", + "source_field", + "values" + ], + "not": { + "anyOf": [ + { "required": ["field"] }, + { "required": ["record_count"] }, + { "required": ["review_priority"] } + ] + } + } + ] + } + ] + }, + "findingShape": { "type": "object", "additionalProperties": false, "required": [ "id", + "classification", "code", "phase", "message", @@ -1030,6 +1314,12 @@ "type": "string", "pattern": "^F[0-9]{6,}$" }, + "classification": { + "enum": [ + "comparison_difference", + "assessment_issue" + ] + }, "code": { "enum": [ "missing_record", @@ -1040,8 +1330,7 @@ "invalid_field_value", "null_policy_violation", "mapping_failure", - "value_mismatch", - "not_assessable" + "value_mismatch" ] }, "phase": { @@ -1073,6 +1362,9 @@ "source_field": { "$ref": "#/$defs/nonEmptyString" }, + "record_count": { + "$ref": "#/$defs/nonNegativeInteger" + }, "review_priority": { "enum": [ "low", @@ -1102,6 +1394,9 @@ "source_locations" ], "properties": { + "classification": { + "const": "comparison_difference" + }, "phase": { "const": "records" }, @@ -1138,6 +1433,9 @@ "source_locations" ], "properties": { + "classification": { + "const": "comparison_difference" + }, "phase": { "const": "records" }, @@ -1162,7 +1460,6 @@ "properties": { "code": { "enum": [ - "missing_required_field", "type_mismatch", "invalid_field_value", "null_policy_violation", @@ -1183,14 +1480,16 @@ "values" ], "properties": { + "classification": { + "const": "assessment_issue" + }, "phase": { "const": "contract" }, "affected_side": { "enum": [ "reference", - "query", - "both" + "query" ] } } @@ -1200,7 +1499,7 @@ "if": { "properties": { "code": { - "const": "undeclared_field" + "const": "missing_required_field" } }, "required": [ @@ -1209,9 +1508,13 @@ }, "then": { "required": [ - "source_field" + "field", + "review_priority" ], "properties": { + "classification": { + "const": "assessment_issue" + }, "phase": { "const": "contract" }, @@ -1221,14 +1524,44 @@ "query" ] } - } + }, + "oneOf": [ + { + "required": [ + "record_identity", + "source_locations", + "values" + ] + }, + { + "not": { + "anyOf": [ + { + "required": [ + "record_identity" + ] + }, + { + "required": [ + "source_locations" + ] + }, + { + "required": [ + "values" + ] + } + ] + } + } + ] } }, { "if": { "properties": { "code": { - "const": "value_mismatch" + "const": "undeclared_field" } }, "required": [ @@ -1237,18 +1570,21 @@ }, "then": { "required": [ - "record_identity", - "source_locations", - "field", - "review_priority", - "values" + "source_field", + "record_count" ], "properties": { + "classification": { + "const": "assessment_issue" + }, "phase": { - "const": "values" + "const": "contract" }, "affected_side": { - "const": "both" + "enum": [ + "reference", + "query" + ] } } } @@ -1257,14 +1593,63 @@ "if": { "properties": { "code": { - "const": "not_assessable" + "const": "value_mismatch" } }, "required": [ "code" ] }, - "then": {} + "then": { + "required": [ + "record_identity", + "source_locations", + "values" + ], + "oneOf": [ + { + "required": [ + "field", + "review_priority" + ], + "not": { + "required": [ + "source_field" + ] + } + }, + { + "required": [ + "source_field" + ], + "not": { + "anyOf": [ + { + "required": [ + "field" + ] + }, + { + "required": [ + "review_priority" + ] + } + ] + } + } + ], + "properties": { + "classification": { + "const": "comparison_difference" + }, + "phase": { + "const": "values" + }, + "affected_side": { + "const": "both" + } + } + } } ] } From 237e0bea0d9439c9f1ac36bc87f1030561a9bdff Mon Sep 17 00:00:00 2001 From: Kevin Howe Date: Thu, 22 Oct 2026 16:03:15 +0100 Subject: [PATCH 06/45] Moved specification to docs --- SPECIFICATION.md => docs/SPECIFICATION.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename SPECIFICATION.md => docs/SPECIFICATION.md (100%) diff --git a/SPECIFICATION.md b/docs/SPECIFICATION.md similarity index 100% rename from SPECIFICATION.md rename to docs/SPECIFICATION.md From 716e55cf2087ac9146449570f2c8fc4481d90628 Mon Sep 17 00:00:00 2001 From: Kevin Howe Date: Thu, 22 Oct 2026 16:03:46 +0100 Subject: [PATCH 07/45] Deactiviated CI; will reinstate for v1 --- .github/workflows/ci.yml | 36 ------------------------------------ 1 file changed, 36 deletions(-) delete mode 100644 .github/workflows/ci.yml diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml deleted file mode 100644 index 44746e4..0000000 --- a/.github/workflows/ci.yml +++ /dev/null @@ -1,36 +0,0 @@ -name: Run PyTest on Push - -on: - push: - branches: [ "*" ] - -jobs: - build: - runs-on: ubuntu-latest - - steps: - - name: Checkout code - uses: actions/checkout@v4 - - - name: Set up Python 3.12 - uses: actions/setup-python@v5 - with: - python-version: '3.12' - - - name: Install dependencies - run: | - python -m pip install --upgrade pip - pip install pytest flake8 importlib-resources pyyaml - - - name: Run linting - shell: bash -l {0} - run: | - # stop the build if there are Python syntax errors or undefined names - # exit-zero treats all errors as warnings. - flake8 . --count --select=E9,F63,F7 --show-source --statistics --exit-zero - # Check function comlexity and line length. The GitHub editor is 127 chars wide - # exit-zero treats all errors as warnings. - flake8 . --count --max-complexity=10 --max-line-length=127 --statistics --exit-zero - - - name: Run pytest - run: pytest -v \ No newline at end of file From c8219947ac98977236c9c4bfa5deeb2776daaced Mon Sep 17 00:00:00 2001 From: Kevin Howe Date: Thu, 22 Oct 2026 16:06:51 +0100 Subject: [PATCH 08/45] Remove prototype in preparation for v1 implementation --- tests/table_validator_tests/__init__.py | 0 .../test_data/data_map_test_left.csv | 6 - .../test_data/data_map_test_right.csv | 6 - .../test_data/pytest_out_all_match.json | 70 ---- .../pytest_out_all_match_schema_check.csv | 8 - .../pytest_out_all_match_value_check.csv | 8 - .../test_data/pytest_out_value_mismatch.json | 70 ---- .../query_toy_json_schemamismatch.json | 32 -- .../test_data/query_toy_json_valmismatch.json | 32 -- .../test_data/ref_toy_csv.csv | 6 - .../test_data/ref_toy_csv.txt | 6 - .../test_data/ref_toy_json.json | 32 -- .../test_data/ref_toy_tsv.tsv | 6 - .../test_data/ref_toy_tsv.txt | 6 - .../test_data/ref_toy_yaml.yaml | 25 -- .../test_data/sample_config.yaml | 15 - .../test_data/test_true_txt_complex.txt | 2 - .../test_data/test_true_txt_simple.txt | 1 - .../test_data/toy_data.csv | 6 - .../test_file_compare.py | 174 ---------- tests/table_validator_tests/test_file_io.py | 77 ----- validation_toolkit/TableValidator/__init__.py | 0 .../TableValidator/custom_exceptions.py | 22 -- .../TableValidator/file_compare.py | 226 ------------ validation_toolkit/TableValidator/file_io.py | 322 ------------------ validation_toolkit/__init__.py | 0 validation_toolkit/cli.py | 148 -------- 27 files changed, 1306 deletions(-) delete mode 100644 tests/table_validator_tests/__init__.py delete mode 100644 tests/table_validator_tests/test_data/data_map_test_left.csv delete mode 100644 tests/table_validator_tests/test_data/data_map_test_right.csv delete mode 100644 tests/table_validator_tests/test_data/pytest_out_all_match.json delete mode 100644 tests/table_validator_tests/test_data/pytest_out_all_match_schema_check.csv delete mode 100644 tests/table_validator_tests/test_data/pytest_out_all_match_value_check.csv delete mode 100644 tests/table_validator_tests/test_data/pytest_out_value_mismatch.json delete mode 100644 tests/table_validator_tests/test_data/query_toy_json_schemamismatch.json delete mode 100644 tests/table_validator_tests/test_data/query_toy_json_valmismatch.json delete mode 100644 tests/table_validator_tests/test_data/ref_toy_csv.csv delete mode 100644 tests/table_validator_tests/test_data/ref_toy_csv.txt delete mode 100644 tests/table_validator_tests/test_data/ref_toy_json.json delete mode 100644 tests/table_validator_tests/test_data/ref_toy_tsv.tsv delete mode 100644 tests/table_validator_tests/test_data/ref_toy_tsv.txt delete mode 100644 tests/table_validator_tests/test_data/ref_toy_yaml.yaml delete mode 100644 tests/table_validator_tests/test_data/sample_config.yaml delete mode 100644 tests/table_validator_tests/test_data/test_true_txt_complex.txt delete mode 100644 tests/table_validator_tests/test_data/test_true_txt_simple.txt delete mode 100644 tests/table_validator_tests/test_data/toy_data.csv delete mode 100644 tests/table_validator_tests/test_file_compare.py delete mode 100644 tests/table_validator_tests/test_file_io.py delete mode 100644 validation_toolkit/TableValidator/__init__.py delete mode 100644 validation_toolkit/TableValidator/custom_exceptions.py delete mode 100644 validation_toolkit/TableValidator/file_compare.py delete mode 100644 validation_toolkit/TableValidator/file_io.py delete mode 100644 validation_toolkit/__init__.py delete mode 100644 validation_toolkit/cli.py diff --git a/tests/table_validator_tests/__init__.py b/tests/table_validator_tests/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/tests/table_validator_tests/test_data/data_map_test_left.csv b/tests/table_validator_tests/test_data/data_map_test_left.csv deleted file mode 100644 index bbf95a5..0000000 --- a/tests/table_validator_tests/test_data/data_map_test_left.csv +++ /dev/null @@ -1,6 +0,0 @@ -col_left_1,col_left_2,col_left_3 -sample1,True,resistant -sample2,False,resistant -sample3,False,unknown -sample4,False,sensitive -sample5,True,sensitive \ No newline at end of file diff --git a/tests/table_validator_tests/test_data/data_map_test_right.csv b/tests/table_validator_tests/test_data/data_map_test_right.csv deleted file mode 100644 index 220329d..0000000 --- a/tests/table_validator_tests/test_data/data_map_test_right.csv +++ /dev/null @@ -1,6 +0,0 @@ -col_right_1,col_right_2,col_right_3 -sample1,1,1 -sample2,0,1 -sample3,0,2 -sample4,0,0 -sample5,1,0 \ No newline at end of file diff --git a/tests/table_validator_tests/test_data/pytest_out_all_match.json b/tests/table_validator_tests/test_data/pytest_out_all_match.json deleted file mode 100644 index 87a7252..0000000 --- a/tests/table_validator_tests/test_data/pytest_out_all_match.json +++ /dev/null @@ -1,70 +0,0 @@ -{ - "metadata": { - "prog": "TableValidator_pytest", - "prog_version": "0.0.1dev" - }, - "schema_check": { - "sample_1": { - "sample_id": "SchemaMatch", - "field1": "SchemaMatch", - "field2": "SchemaMatch", - "field3": "SchemaMatch" - }, - "sample_2": { - "sample_id": "SchemaMatch", - "field1": "SchemaMatch", - "field2": "SchemaMatch", - "field3": "SchemaMatch" - }, - "sample_3": { - "sample_id": "SchemaMatch", - "field1": "SchemaMatch", - "field2": "SchemaMatch", - "field3": "SchemaMatch" - }, - "sample_4": { - "sample_id": "SchemaMatch", - "field1": "SchemaMatch", - "field2": "SchemaMatch", - "field3": "SchemaMatch" - }, - "sample_5": { - "sample_id": "SchemaMatch", - "field1": "SchemaMatch", - "field2": "SchemaMatch", - "field3": "SchemaMatch" - } - }, - "value_check": { - "sample_1": { - "sample_id": "ValueMatch", - "field1": "ValueMatch", - "field2": "ValueMatch", - "field3": "ValueMatch" - }, - "sample_2": { - "sample_id": "ValueMatch", - "field1": "ValueMatch", - "field2": "ValueMatch", - "field3": "ValueMatch" - }, - "sample_3": { - "sample_id": "ValueMatch", - "field1": "ValueMatch", - "field2": "ValueMatch", - "field3": "ValueMatch" - }, - "sample_4": { - "sample_id": "ValueMatch", - "field1": "ValueMatch", - "field2": "ValueMatch", - "field3": "ValueMatch" - }, - "sample_5": { - "sample_id": "ValueMatch", - "field1": "ValueMatch", - "field2": "ValueMatch", - "field3": "ValueMatch" - } - } -} \ No newline at end of file diff --git a/tests/table_validator_tests/test_data/pytest_out_all_match_schema_check.csv b/tests/table_validator_tests/test_data/pytest_out_all_match_schema_check.csv deleted file mode 100644 index ee544f5..0000000 --- a/tests/table_validator_tests/test_data/pytest_out_all_match_schema_check.csv +++ /dev/null @@ -1,8 +0,0 @@ -## prog: TableValidator_pytest -## prog_version: 0.0.1dev -sample_id,field1,field2,field3 -SchemaMatch,SchemaMatch,SchemaMatch,SchemaMatch -SchemaMatch,SchemaMatch,SchemaMatch,SchemaMatch -SchemaMatch,SchemaMatch,SchemaMatch,SchemaMatch -SchemaMatch,SchemaMatch,SchemaMatch,SchemaMatch -SchemaMatch,SchemaMatch,SchemaMatch,SchemaMatch diff --git a/tests/table_validator_tests/test_data/pytest_out_all_match_value_check.csv b/tests/table_validator_tests/test_data/pytest_out_all_match_value_check.csv deleted file mode 100644 index bbc8268..0000000 --- a/tests/table_validator_tests/test_data/pytest_out_all_match_value_check.csv +++ /dev/null @@ -1,8 +0,0 @@ -## prog: TableValidator_pytest -## prog_version: 0.0.1dev -sample_id,field1,field2,field3 -ValueMatch,ValueMatch,ValueMatch,ValueMatch -ValueMatch,ValueMatch,ValueMatch,ValueMatch -ValueMatch,ValueMatch,ValueMatch,ValueMatch -ValueMatch,ValueMatch,ValueMatch,ValueMatch -ValueMatch,ValueMatch,ValueMatch,ValueMatch diff --git a/tests/table_validator_tests/test_data/pytest_out_value_mismatch.json b/tests/table_validator_tests/test_data/pytest_out_value_mismatch.json deleted file mode 100644 index 2da9970..0000000 --- a/tests/table_validator_tests/test_data/pytest_out_value_mismatch.json +++ /dev/null @@ -1,70 +0,0 @@ -{ - "metadata": { - "prog": "TableValidator_pytest", - "prog_version": "0.0.1dev" - }, - "schema_check": { - "sample_1": { - "sample_id": "SchemaMatch", - "field1": "SchemaMatch", - "field2": "SchemaMatch", - "field3": "SchemaMatch" - }, - "sample_2": { - "sample_id": "SchemaMatch", - "field1": "SchemaMatch", - "field2": "SchemaMatch", - "field3": "SchemaMatch" - }, - "sample_3": { - "sample_id": "SchemaMatch", - "field1": "SchemaMatch", - "field2": "SchemaMatch", - "field3": "SchemaMatch" - }, - "sample_4": { - "sample_id": "SchemaMatch", - "field1": "SchemaMatch", - "field2": "SchemaMatch", - "field3": "SchemaMatch" - }, - "sample_5": { - "sample_id": "SchemaMatch", - "field1": "SchemaMatch", - "field2": "SchemaMatch", - "field3": "SchemaMatch" - } - }, - "value_check": { - "sample_1": { - "sample_id": "ValueMatch", - "field1": "ValueMatch", - "field2": "ValueMatch", - "field3": "ValueMatch" - }, - "sample_2": { - "sample_id": "ValueMatch", - "field1": "ValueMatch", - "field2": "ValueMatch", - "field3": "ValueMatch" - }, - "sample_3": { - "sample_id": "ValueMatch", - "field1": "ValueMatch", - "field2": "ValueMatch", - "field3": "ValueMatch" - }, - "sample_4": { - "sample_id": "ValueMatch", - "field1": "ValueMatch", - "field2": "ValueMismatch. Expected 42 Found fourtwo", - "field3": "ValueMatch" - }, - "sample_5": { - "sample_id": "ValueMatch", - "field1": "ValueMatch", - "field2": "ValueMatch", - "field3": "ValueMatch" - } - } -} \ No newline at end of file diff --git a/tests/table_validator_tests/test_data/query_toy_json_schemamismatch.json b/tests/table_validator_tests/test_data/query_toy_json_schemamismatch.json deleted file mode 100644 index e04393a..0000000 --- a/tests/table_validator_tests/test_data/query_toy_json_schemamismatch.json +++ /dev/null @@ -1,32 +0,0 @@ -{ - "sample_1": { - "sample_id": "sample_1", - "field1": "oneone", - "field2": "onetwo", - "field3": "onethree" - }, - "sample_2": { - "sample_id": "sample_2", - "field1": "twoone", - "field2": "twotwo", - "field3": "twothree" - }, - "sample_3": { - "sample_id": "sample_3", - "field1": "threeone", - "field2": "threetwo", - "field3": "threethree" - }, - "sample_4": { - "sample_id": "sample_4", - "field1": "fourone", - "field2": false, - "field3": "fourthree" - }, - "sample_5": { - "sample_id": "sample_5", - "field1": "fiveone", - "field2": "fivetwo", - "field3": true - } -} \ No newline at end of file diff --git a/tests/table_validator_tests/test_data/query_toy_json_valmismatch.json b/tests/table_validator_tests/test_data/query_toy_json_valmismatch.json deleted file mode 100644 index 2796c4f..0000000 --- a/tests/table_validator_tests/test_data/query_toy_json_valmismatch.json +++ /dev/null @@ -1,32 +0,0 @@ -{ - "sample_1": { - "sample_id": "sample_1", - "field1": "oneone", - "field2": "onetwo", - "field3": "onethree" - }, - "sample_2": { - "sample_id": "sample_2", - "field1": "twoone", - "field2": "twotwo", - "field3": "twothree" - }, - "sample_3": { - "sample_id": "sample_3", - "field1": "threeone", - "field2": "threetwo", - "field3": "threethree" - }, - "sample_4": { - "sample_id": "sample_4", - "field1": "fourone", - "field2": "fourtwo", - "field3": "fourthree" - }, - "sample_5": { - "sample_id": "sample_5", - "field1": "fiveone", - "field2": "fivetwo", - "field3": true - } -} \ No newline at end of file diff --git a/tests/table_validator_tests/test_data/ref_toy_csv.csv b/tests/table_validator_tests/test_data/ref_toy_csv.csv deleted file mode 100644 index 017a768..0000000 --- a/tests/table_validator_tests/test_data/ref_toy_csv.csv +++ /dev/null @@ -1,6 +0,0 @@ -sample_id,field1,field2,field3 -sample_1,oneone,onetwo,onethree -sample_2,twoone,twotwo,twothree -sample_3,threeone,threetwo,threethree -sample_4,fourone,42,fourthree -sample_5,fiveone,fivetwo,True diff --git a/tests/table_validator_tests/test_data/ref_toy_csv.txt b/tests/table_validator_tests/test_data/ref_toy_csv.txt deleted file mode 100644 index 017a768..0000000 --- a/tests/table_validator_tests/test_data/ref_toy_csv.txt +++ /dev/null @@ -1,6 +0,0 @@ -sample_id,field1,field2,field3 -sample_1,oneone,onetwo,onethree -sample_2,twoone,twotwo,twothree -sample_3,threeone,threetwo,threethree -sample_4,fourone,42,fourthree -sample_5,fiveone,fivetwo,True diff --git a/tests/table_validator_tests/test_data/ref_toy_json.json b/tests/table_validator_tests/test_data/ref_toy_json.json deleted file mode 100644 index 841fb53..0000000 --- a/tests/table_validator_tests/test_data/ref_toy_json.json +++ /dev/null @@ -1,32 +0,0 @@ -{ - "sample_1": { - "sample_id": "sample_1", - "field1": "oneone", - "field2": "onetwo", - "field3": "onethree" - }, - "sample_2": { - "sample_id": "sample_2", - "field1": "twoone", - "field2": "twotwo", - "field3": "twothree" - }, - "sample_3": { - "sample_id": "sample_3", - "field1": "threeone", - "field2": "threetwo", - "field3": "threethree" - }, - "sample_4": { - "sample_id": "sample_4", - "field1": "fourone", - "field2": 42, - "field3": "fourthree" - }, - "sample_5": { - "sample_id": "sample_5", - "field1": "fiveone", - "field2": "fivetwo", - "field3": true - } -} \ No newline at end of file diff --git a/tests/table_validator_tests/test_data/ref_toy_tsv.tsv b/tests/table_validator_tests/test_data/ref_toy_tsv.tsv deleted file mode 100644 index 45284e6..0000000 --- a/tests/table_validator_tests/test_data/ref_toy_tsv.tsv +++ /dev/null @@ -1,6 +0,0 @@ -sample_id field1 field2 field3 -sample_1 oneone onetwo onethree -sample_2 twoone twotwo twothree -sample_3 threeone threetwo threethree -sample_4 fourone 42 fourthree -sample_5 fiveone fivetwo True diff --git a/tests/table_validator_tests/test_data/ref_toy_tsv.txt b/tests/table_validator_tests/test_data/ref_toy_tsv.txt deleted file mode 100644 index 45284e6..0000000 --- a/tests/table_validator_tests/test_data/ref_toy_tsv.txt +++ /dev/null @@ -1,6 +0,0 @@ -sample_id field1 field2 field3 -sample_1 oneone onetwo onethree -sample_2 twoone twotwo twothree -sample_3 threeone threetwo threethree -sample_4 fourone 42 fourthree -sample_5 fiveone fivetwo True diff --git a/tests/table_validator_tests/test_data/ref_toy_yaml.yaml b/tests/table_validator_tests/test_data/ref_toy_yaml.yaml deleted file mode 100644 index 2b3df55..0000000 --- a/tests/table_validator_tests/test_data/ref_toy_yaml.yaml +++ /dev/null @@ -1,25 +0,0 @@ -sample_1: - field1: oneone - field2: onetwo - field3: onethree - sample_id: sample_1 -sample_2: - field1: twoone - field2: twotwo - field3: twothree - sample_id: sample_2 -sample_3: - field1: threeone - field2: threetwo - field3: threethree - sample_id: sample_3 -sample_4: - field1: fourone - field2: 42 - field3: fourthree - sample_id: sample_4 -sample_5: - field1: fiveone - field2: fivetwo - field3: true - sample_id: sample_5 diff --git a/tests/table_validator_tests/test_data/sample_config.yaml b/tests/table_validator_tests/test_data/sample_config.yaml deleted file mode 100644 index 07f48a1..0000000 --- a/tests/table_validator_tests/test_data/sample_config.yaml +++ /dev/null @@ -1,15 +0,0 @@ -col_left_1: - col_name: col_right_1 - value_map: null -col_left_2: - col_name: col_right_2 - value_map: - true: 1 - false: 0 - unknown: null -col_left_3: - col_name: col_right_3 - value_map: - resistant: 1 - sensitive: 0 - unknown: 2 diff --git a/tests/table_validator_tests/test_data/test_true_txt_complex.txt b/tests/table_validator_tests/test_data/test_true_txt_complex.txt deleted file mode 100644 index 3cc8fb9..0000000 --- a/tests/table_validator_tests/test_data/test_true_txt_complex.txt +++ /dev/null @@ -1,2 +0,0 @@ -Hello, world! This is a sample text file. -Basically, I want to challenge the program with actual commas. \ No newline at end of file diff --git a/tests/table_validator_tests/test_data/test_true_txt_simple.txt b/tests/table_validator_tests/test_data/test_true_txt_simple.txt deleted file mode 100644 index 5844139..0000000 --- a/tests/table_validator_tests/test_data/test_true_txt_simple.txt +++ /dev/null @@ -1 +0,0 @@ -Hello, world! This is a sample text file. \ No newline at end of file diff --git a/tests/table_validator_tests/test_data/toy_data.csv b/tests/table_validator_tests/test_data/toy_data.csv deleted file mode 100644 index 44d796b..0000000 --- a/tests/table_validator_tests/test_data/toy_data.csv +++ /dev/null @@ -1,6 +0,0 @@ -sample_id,field1,field2,field3 -sample_1,oneone,onetwo,onethree -sample_2,twoone,twotwo,twothree -sample_3,threeone,threetwo,threethree -sample_4,fourone,42,fourthree -sample_5,fiveone,fivetwo,True \ No newline at end of file diff --git a/tests/table_validator_tests/test_file_compare.py b/tests/table_validator_tests/test_file_compare.py deleted file mode 100644 index c1c2ace..0000000 --- a/tests/table_validator_tests/test_file_compare.py +++ /dev/null @@ -1,174 +0,0 @@ -import pytest, warnings -import difflib -from importlib_resources import files - -from validation_toolkit.TableValidator.file_io import File_IO -from validation_toolkit.TableValidator.file_compare import FileCompare -from validation_toolkit.TableValidator.custom_exceptions import SchemaMismatchWarning, ValueMismatchWarning - -from pathlib import Path - -TEST_DATA_DIR = files('tests.table_validator_tests.test_data') - -@pytest.fixture -def test_init(): - left_data = File_IO(input_file= TEST_DATA_DIR / "ref_toy_csv.csv", sample_id_key="sample_id") - right_data = File_IO(input_file= TEST_DATA_DIR / "ref_toy_json.json", sample_id_key="sample_id") - - test_object = FileCompare(left_data=left_data.data_reader, right_data=right_data.data_reader) - assert test_object, "A test file_compare object should be built" - return test_object - -@pytest.fixture -def test_init_config(): - my_ref_fileio_object = File_IO(input_file= TEST_DATA_DIR / "data_map_test_left.csv", sample_id_key="col_left_1", config= TEST_DATA_DIR / "sample_config.yaml") - my_ref_fileio_object_no_config = File_IO(input_file= TEST_DATA_DIR / "data_map_test_left.csv", sample_id_key="col_left_1") - my_query_fileio_object = File_IO(input_file= TEST_DATA_DIR / "data_map_test_right.csv", sample_id_key="col_right_1") - - test_object_no_config = FileCompare(left_data=my_ref_fileio_object_no_config.data_reader, right_data=my_query_fileio_object.data_reader) - test_object_w_config = FileCompare(left_data=my_ref_fileio_object.data_reader, right_data=my_query_fileio_object.data_reader) - - return test_object_no_config, test_object_w_config - -def test_create_schema(test_init): - test_object = test_init - test_object.create_schema() - - assert len(test_object.sample_schema.keys()) == 4, "There should be 4 fields in the schema" - assert test_object.sample_schema["sample_id"]["valid_types"] == test_object.sample_schema["field1"]["valid_types"] == set([str]) - -def test_confirm_schema(test_init): - test_object = test_init - test_object.create_schema() - test_object.confirm_schema() - assert test_object.schema_errors["sample_1"] == {"sample_id": "SchemaMatch", "field1": "SchemaMatch", "field2": "SchemaMatch", "field3": "SchemaMatch"} - -def test_compare_values(test_init): - test_object = test_init - test_object.create_schema() - test_object.compare_values() - assert test_object.value_mismatches["sample_1"] == {"sample_id": "ValueMatch", "field1": "ValueMatch", "field2": "ValueMatch", "field3": "ValueMatch"} - -def test_confirm_schema_fail(): - left_data = File_IO(input_file= TEST_DATA_DIR / "ref_toy_csv.csv", sample_id_key="sample_id") - right_data = File_IO(input_file= TEST_DATA_DIR / "query_toy_json_schemamismatch.json", sample_id_key="sample_id") - test_object_lenient = FileCompare(left_data=left_data.data_reader, right_data=right_data.data_reader) - test_object_lenient.create_schema() - - ## there should be a schema check failure here - with pytest.warns(SchemaMismatchWarning): - test_object_lenient.confirm_schema() - - ## using strict mode, this warning should now be an exception -- pytest.raises needed to catch it - test_object_strict = FileCompare(left_data=left_data.data_reader, right_data=right_data.data_reader, strict=True) - test_object_strict.create_schema() - with pytest.raises(SchemaMismatchWarning) as error_info: - test_object_strict.confirm_schema() - -def test_compare_values_fail(): - left_data = File_IO(input_file= TEST_DATA_DIR / "ref_toy_csv.csv", sample_id_key="sample_id") - right_data = File_IO(input_file= TEST_DATA_DIR / "query_toy_json_valmismatch.json", sample_id_key="sample_id") - test_object = FileCompare(left_data=left_data.data_reader, right_data=right_data.data_reader) - test_object.create_schema() - test_object.confirm_schema() - ## there should be a value mismatch flagged here - with pytest.warns(ValueMismatchWarning, match="Mismatch in value of field field2 for sample sample_4. Expected: 42 Found: fourtwo"): - test_object.compare_values() - -def test_compare_values_specific_fields(): - left_data = File_IO(input_file= TEST_DATA_DIR / "ref_toy_csv.csv", sample_id_key="sample_id") - right_data = File_IO(input_file= TEST_DATA_DIR / "query_toy_json_valmismatch.json", sample_id_key="sample_id") - test_object = FileCompare(left_data=left_data.data_reader, right_data=right_data.data_reader) - test_object.create_schema() - test_object.confirm_schema() - ## even though this test uses a value_mismatch test fixture - ## there should be no warnings as the field in question is not included in the provided list of field names to check - test_object.compare_values(field_names=["sample_id", "field1", "field3"]) - assert test_object.value_mismatches["sample_1"] == {"sample_id": "ValueMatch", "field1": "ValueMatch", "field2": "Field not checked", "field3": "ValueMatch"} - -def test_compare_fail_with_no_config(test_init_config): - test_object_no_config, _ = test_init_config - test_object_no_config.create_schema() - with pytest.raises(Exception): - test_object_no_config.confirm_schema() - -def test_compare_values_config(test_init_config): - _, test_object_w_config = test_init_config - test_object_w_config.create_schema() - test_object_w_config.confirm_schema() - test_object_w_config.compare_values() - assert test_object_w_config.schema_errors["sample1"] == {"sample_id": "sample1", "col_right_1": "SchemaMatch", "col_right_2": "SchemaMatch", "col_right_3": "SchemaMatch"} - assert test_object_w_config.value_mismatches["sample1"] == {"sample_id": "sample1", "col_right_1": "ValueMatch", "col_right_2": "ValueMatch", "col_right_3": "ValueMatch"} - -def _files_are_identical(file1, file2): - with open(file1, "r") as f1: - with open(file2, "r") as f2: - diff = difflib.unified_diff( - f1.readlines(), - f2.readlines(), - fromfile="f1", - tofile="f2", - ) - diff = [i for i in diff] - - if diff: - return False, diff - else: - return True, [] - -def test_output_files_json(test_init, tmp_path): - test_object = test_init - test_object.create_schema() - test_object.confirm_schema() - test_object.compare_values() - metadata = { - "prog": "TableValidator_pytest", - "prog_version": "0.0.1dev", - } - test_object.write_results(metadata=metadata, output=f"{tmp_path}/test_out_all_match.json", out_format="json") - - assert _files_are_identical(file1= TEST_DATA_DIR / "pytest_out_all_match.json", file2=f"{tmp_path}/test_out_all_match.json")[0], "Files should be identical!" - -def test_output_files_csv(test_init, tmp_path): - test_object = test_init - test_object.create_schema() - test_object.confirm_schema() - test_object.compare_values() - metadata = { - "prog": "TableValidator_pytest", - "prog_version": "0.0.1dev", - } - test_object.write_results(metadata=metadata, output=f"{tmp_path}/test_out_all_match.csv", out_format="csv") - - assert _files_are_identical(file1= TEST_DATA_DIR / "pytest_out_all_match_schema_check.csv", file2=f"{tmp_path}/test_out_all_match_schema_check.csv")[0], "Files should be identical!" - assert _files_are_identical(file1= TEST_DATA_DIR / "pytest_out_all_match_value_check.csv", file2=f"{tmp_path}/test_out_all_match_value_check.csv")[0], "Files should be identical!" - -def test_output_files_schema_mismatch(): - left_data = File_IO(input_file= TEST_DATA_DIR / "ref_toy_csv.csv", sample_id_key="sample_id") - right_data = File_IO(input_file= TEST_DATA_DIR / "query_toy_json_schemamismatch.json", sample_id_key="sample_id") - test_object_lenient = FileCompare(left_data=left_data.data_reader, right_data=right_data.data_reader) - test_object_lenient.create_schema() - - ## there should be a schema check failure here - with pytest.warns(SchemaMismatchWarning): - test_object_lenient.confirm_schema() - - assert test_object_lenient.schema_errors["sample_4"]["field2"] in ["SchemaMismatch. Expected {, } Found ", "SchemaMismatch. Expected {, } Found "] - -def test_output_files_value_mismatch(tmp_path): - left_data = File_IO(input_file= TEST_DATA_DIR / "ref_toy_csv.csv", sample_id_key="sample_id") - right_data = File_IO(input_file= TEST_DATA_DIR / "query_toy_json_valmismatch.json", sample_id_key="sample_id") - test_object = FileCompare(left_data=left_data.data_reader, right_data=right_data.data_reader) - test_object.create_schema() - test_object.confirm_schema() - ## there should be a value mismatch flagged here - with pytest.warns(ValueMismatchWarning, match="Mismatch in value of field field2 for sample sample_4. Expected: 42 Found: fourtwo"): - test_object.compare_values() - - metadata = { - "prog": "TableValidator_pytest", - "prog_version": "0.0.1dev", - } - test_object.write_results(metadata=metadata, output=f"{tmp_path}/test_out_value_mismatch.json", out_format="json") - - assert _files_are_identical(file1= TEST_DATA_DIR / "pytest_out_value_mismatch.json", file2=f"{tmp_path}/test_out_value_mismatch.json")[0], "Files should be identical!" diff --git a/tests/table_validator_tests/test_file_io.py b/tests/table_validator_tests/test_file_io.py deleted file mode 100644 index 1fb995a..0000000 --- a/tests/table_validator_tests/test_file_io.py +++ /dev/null @@ -1,77 +0,0 @@ -import pytest -from importlib_resources import files -from pathlib import Path - -from validation_toolkit.TableValidator.file_io import File_IO -from validation_toolkit.TableValidator.custom_exceptions import UnsupportedFileFormatError - -TEST_DATA_DIR = files('tests.table_validator_tests.test_data') - -def test_init(): - my_fileio_object = File_IO(input_file= TEST_DATA_DIR / "ref_toy_csv.csv", sample_id_key="sample_id") - assert my_fileio_object.input_file_type == "csv", "Wrong file type found" - assert len(my_fileio_object.data_reader.keys()) == 5, "Expected 5 sample_ids in the test data" - assert len(my_fileio_object.fields) == 4, f"Expected 4 columns/fields in data" - assert sorted(my_fileio_object.fields) == sorted(["sample_id", "field1", "field2", "field3"]), "Did not find all expected fields" - -def test_read_from_tsv(): - my_fileio_object = File_IO(input_file= TEST_DATA_DIR / "ref_toy_tsv.tsv", sample_id_key="sample_id") - assert my_fileio_object.input_file_type == "tsv", "Wrong file type found" - assert len(my_fileio_object.data_reader.keys()) == 5, "Expected 5 sample_ids in the test data" - assert len(my_fileio_object.fields) == 4, f"Expected 4 columns/fields in data" - assert sorted(my_fileio_object.fields) == sorted(["sample_id", "field1", "field2", "field3"]), "Did not find all expected fields" - -def test_read_from_json(): - my_fileio_object = File_IO(input_file= TEST_DATA_DIR / "ref_toy_json.json", sample_id_key="sample_id") - assert my_fileio_object.input_file_type == "json", "Wrong file type found" - assert len(my_fileio_object.data_reader.keys()) == 5, "Expected 5 sample_ids in the test data" - assert len(my_fileio_object.fields) == 4, f"Expected 4 columns/fields in data" - assert sorted(my_fileio_object.fields) == sorted(["sample_id", "field1", "field2", "field3"]), "Did not find all expected fields" - -def test_read_from_yaml(): - my_fileio_object = File_IO(input_file= TEST_DATA_DIR / "ref_toy_yaml.yaml", sample_id_key="sample_id") - assert my_fileio_object.input_file_type == "yaml", "Wrong file type found" - assert len(my_fileio_object.data_reader.keys()) == 5, "Expected 5 sample_ids in the test data" - assert len(my_fileio_object.fields) == 4, f"Expected 4 columns/fields in data" - assert sorted(my_fileio_object.fields) == sorted(["sample_id", "field1", "field2", "field3"]), "Did not find all expected fields" - -def test_equality_from_different_input_formats(): - my_fileio_object_csv = File_IO(input_file= TEST_DATA_DIR / "ref_toy_csv.csv", sample_id_key="sample_id") - my_fileio_object_tsv = File_IO(input_file= TEST_DATA_DIR / "ref_toy_tsv.tsv", sample_id_key="sample_id") - my_fileio_object_json = File_IO(input_file= TEST_DATA_DIR / "ref_toy_json.json", sample_id_key="sample_id") - my_fileio_object_yaml = File_IO(input_file= TEST_DATA_DIR / "ref_toy_yaml.yaml", sample_id_key="sample_id") - - assert sorted(my_fileio_object_csv.fields) == sorted(my_fileio_object_tsv.fields) == sorted(my_fileio_object_json.fields) == sorted(my_fileio_object_yaml.fields), \ - "Fields from all input types should be the same" - -def test_format_inference(): - my_fileio_object_csv = File_IO(input_file= TEST_DATA_DIR / "ref_toy_csv.txt", sample_id_key="sample_id") - assert my_fileio_object_csv.input_file_type == "csv", "Expected to infer as csv from txt file" - - my_fileio_object_tsv = File_IO(input_file= TEST_DATA_DIR / "ref_toy_tsv.txt", sample_id_key="sample_id") - assert my_fileio_object_tsv.input_file_type == "tsv", "Expected to infer as tsv from txt file" - -def test_format_inference_error(): - with pytest.raises(UnsupportedFileFormatError) as error1_info: - my_fileio_object_faulty_simple = File_IO(input_file= TEST_DATA_DIR / "test_true_txt_simple.txt", sample_id_key="sample_id") - assert str(error1_info.value) == "Given file does not appear to be in an accepted format. Accepted formats are:\n{self.SUPPORTED_FILE_TYPES}" - with pytest.raises(UnsupportedFileFormatError) as error2_info: - my_fileio_object_faulty_complex = File_IO(input_file= TEST_DATA_DIR / "test_true_txt_complex.txt", sample_id_key="sample_id") - assert str(error2_info.value) == "Given file does not appear to be in an accepted format. Accepted formats are:\n{self.SUPPORTED_FILE_TYPES}" - -def test_infer_types(): - my_fileio_object = File_IO(input_file= TEST_DATA_DIR / "ref_toy_csv.csv", sample_id_key="sample_id") - assert my_fileio_object._infer_type(var="42") == 42, "Integer inference fails" - assert my_fileio_object._infer_type(var="True") == True, "Bool inference fails" - assert my_fileio_object._infer_type(var="fAlse") == False, "Bool inference fails" - assert my_fileio_object._infer_type(var="3.14") == 3.14, "Float inference fails" - assert my_fileio_object._infer_type(var="simple_string") == "simple_string", "Inference incorrectly applied to string" - -def test_data_mapping(): - my_ref_fileio_object = File_IO(input_file= TEST_DATA_DIR / "data_map_test_left.csv", sample_id_key="col_left_1", config= TEST_DATA_DIR / "sample_config.yaml") - my_ref_fileio_object_no_config = File_IO(input_file= TEST_DATA_DIR / "data_map_test_left.csv", sample_id_key="col_left_1") - my_query_fileio_object = File_IO(input_file= TEST_DATA_DIR / "data_map_test_right.csv", sample_id_key="col_right_1") - - assert list(my_ref_fileio_object.data_reader.keys()) == list(my_query_fileio_object.data_reader.keys()), "Config mapping erroneously altered sample_ids" - assert sorted(my_ref_fileio_object.fields) == sorted(my_query_fileio_object.fields), "Config mapping on column names went wrong" - assert not sorted(my_ref_fileio_object_no_config.fields) == sorted(my_query_fileio_object.fields), "Column names should not match when config is not used" diff --git a/validation_toolkit/TableValidator/__init__.py b/validation_toolkit/TableValidator/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/validation_toolkit/TableValidator/custom_exceptions.py b/validation_toolkit/TableValidator/custom_exceptions.py deleted file mode 100644 index f12059c..0000000 --- a/validation_toolkit/TableValidator/custom_exceptions.py +++ /dev/null @@ -1,22 +0,0 @@ - -class UnsupportedFileFormatError(Exception): - pass - -class UnexpectedFileFormatError(Exception): - pass - -## warnings -class KeyMismatchWarning(Warning): - pass - -class FieldMismatchWarning(Warning): - pass - -class SchemaMismatchWarning(Warning): - pass - -class ValueMismatchWarning(Warning): - pass - -class MissingSampleWarning(Warning): - pass diff --git a/validation_toolkit/TableValidator/file_compare.py b/validation_toolkit/TableValidator/file_compare.py deleted file mode 100644 index 40976a3..0000000 --- a/validation_toolkit/TableValidator/file_compare.py +++ /dev/null @@ -1,226 +0,0 @@ -import warnings -import json -from collections import defaultdict -from validation_toolkit.TableValidator.custom_exceptions import KeyMismatchWarning, SchemaMismatchWarning, FieldMismatchWarning, ValueMismatchWarning, MissingSampleWarning - -class FileCompare(): - def __init__(self, left_data: dict, right_data: dict = None, strict: bool = False): - """Initialise a file comparison object with reference and query datasets - - Args: - left_data (dict): Data formatted by FileIO from the reference table - right_data (dict, optional): Data formatted by FileIO from the query/subject table - strict (bool, optional): Whether to elevate warnings to exceptions and terminate the program. Defaults to False. - """ - self.ref_table = left_data - self.query_table = right_data - self.schema_errors = defaultdict(dict) - self.value_mismatches = defaultdict(dict) - - ## if strict, ask warnings to be elevated to terminating exceptions - if strict: - warnings.filterwarnings("error") - - def create_schema(self, required_fields: list = None): - """Using the reference data, infer a "schema": expected field names and expected data types for each field - - Args: - required_fields (list, optional): List of field names to mark as required. Defaults to None. - """ - self.sample_schema = {} - example_id = min(sample_id for sample_id in self.ref_table.keys()) - example_data = self.ref_table[example_id] - ## set up a first pass schema using data from the first arbitrary sample ID - ## here schema looks like {field_name: {"valid_types": set(type(field_value))}} - for k, v in example_data.items(): - self.sample_schema[k] = {"valid_types": set([type(v)])} - - ## now iterate over all values and update the schema if field X is seen to have other data types - for data_values in self.ref_table.values(): - for k, v in data_values.items(): - ## print message if new field found and add it to schema. - ## practically this should never happen - if k not in self.sample_schema.keys(): - print(f"Did not find field {k} in data from example record, updating schema.") - self.sample_schema[k]["valid_types"] = [type(v)] - ## check if the current value for the current value is of type that is already seen - ## if not, add the new data type to the schema for that field - elif not type(v) in self.sample_schema[k]: - self.sample_schema[k]["valid_types"].add(type(v)) - - ## now, add another property per-field in the schema - ## after this schema looks like: {field_name: {"valid_types": set(type(field_value)), "required": bool}} - ## if list of required fields is given, set those fields to required = True - if required_fields: - for k in self.sample_schema.keys(): - if k in required_fields: - self.sample_schema[k].update({"required": True}) - else: - self.sample_schema[k].update({"required": False}) - ## if list of required fields is NOT given, set required = True for all fields - else: - for k in self.sample_schema.keys(): - self.sample_schema[k].update({"required": True}) - - def confirm_schema(self): - """Function to check data types in the query/subject table to ensure it matches the schema inferred from the reference table - """ - - col_name_warnings = 0 - for sample_id, data_value in self.query_table.items(): - self.schema_errors[sample_id] = {"sample_id": sample_id} - for k, v in data_value.items(): - ## first check if a given field exists in the schema - ## throw a warning if not - field_features = self.sample_schema.get(k, {}) - if not field_features: - warnings.warn(FieldMismatchWarning(f"Field {k} found in query table but not in schema!")) - col_name_warnings += 1 - ## if the number of column mismatches equals the number of columns, this use case likely needs to employ the config. - if col_name_warnings == len(data_value.keys()): - raise Exception(f"No column names seem to match between the schema and the query table. You must provide a config file when column names differ between the reference and query tables.") - continue - - ## explicitly collect expected types from schema and whether the field is required - expected_types = field_features["valid_types"] - field_required = field_features["required"] - - ## also collect the type of the data in the query/subject - ## and check if that type is expected - type_found = type(v) - if type_found in expected_types: - self.schema_errors[sample_id].update({k: "SchemaMatch"}) - else: - if field_required: - warnings.warn(SchemaMismatchWarning(f"Field {k} is of type {type_found}, expected types are {expected_types}")) - self.schema_errors[sample_id].update({k: f"SchemaMismatch. Expected {expected_types} Found {type_found}"}) - continue - else: - self.schema_errors[sample_id].append({k: f"SchemaMismatch (optional field). Expected {expected_types} Found {type_found}"}) - continue - - def compare_values(self, field_names: list = None, col_maps: dict = None): - """Function to actually compare field values between reference and query/subject datasets - - Args: - field_names (list, optional): List of field names to compare values for. Defaults to None and all fields are checked. - col_maps (dict, optional): Mapping of column names between reference and query datasets if the schemas are not identical. Defaults to None. - """ - if col_maps: - raise NotImplementedError("Operation mode that uses this argument is not yet implemented.") - - ## first check if reference and query tables have the same set of sample IDs, raise warning if not - if not sorted(list(self.ref_table.keys())) == sorted(list(self.query_table.keys())): - warnings.warn(KeyMismatchWarning("Sample IDs do not match between the two given datasets.")) - - ## now on a per sample basis, compare values for each field - for sample_id in self.ref_table: - self.value_mismatches[sample_id] = {"sample_id": sample_id} - ref_data_dict = self.ref_table.get(sample_id, {}) - query_data_dict = self.query_table.get(sample_id, {}) - - ## make sure that the program can retrieve sample data form both datasets - ## this is a redundant check and given the previous try/assert block, should never evaluate to the else condition - if ref_data_dict and query_data_dict: - for k, v in ref_data_dict.items(): - ## only check fields specified in field_names IF this arg is provided - if field_names and k not in field_names: - self.value_mismatches[sample_id].update({k: f"Field not checked"}) - continue - else: - if v == query_data_dict[k]: - self.value_mismatches[sample_id].update({k: "ValueMatch"}) - else: - self.value_mismatches[sample_id].update({k: f"ValueMismatch. Expected {v} Found {query_data_dict[k]}"}) - warnings.warn(ValueMismatchWarning(f"Mismatch in value of field {k} for sample {sample_id}. Expected: {v} Found: {query_data_dict[k]}")) - continue - ## if for given sample ID, data from either reference or query/subject is not found, throw a warning - ## practically this code should never be accessed if using strict mode as there will be a failure in the sample_id matching try/assert block above - else: - warnings.warn(MissingSampleWarning(f"Sample ID {sample_id} not found in query table.")) - self.value_mismatches[sample_id].append({k: f"Missing field"}) - - def _format_csv_output(self, metadata: dict, output: str): - """Format output data into CSV format. When CSV is used as the output file type, - two files are produced, using the basename of the output file provided: - {basename}_schema_check.csv: contains results of schema comparison - {basename}_value_check.csv: contains results of value comparison - NB: metadata is written as comments starting with "##" to both of the above files. - - Args: - metadata (dict): Metadata dict containing info on comparison run - output (str): Path to output file path - """ - metadata_lines = [] - schema_lines = [] - value_lines = [] - - for k, v in metadata.items(): - metadata_lines.append(f"## {k}: {v}\n") - - example_key = min([k for k in self.schema_errors.keys()]) - example_val = self.schema_errors[example_key] - header = ",".join([k for k in example_val.keys()])+"\n" - - schema_lines.append(header) - for k, v in self.schema_errors.items(): - line = [] - for val_v in v.values(): - line.append(val_v) - schema_lines.append(",".join(line)+"\n") - - value_lines.append(header) - for k, v in self.value_mismatches.items(): - line = [] - for val_v in v.values(): - line.append(val_v) - value_lines.append(",".join(line)+"\n") - - ## set up output files - schema_out = output.strip(".csv") + "_schema_check" + ".csv" - value_out = output.strip(".csv") + "_value_check" + ".csv" - - with open(schema_out, "w") as schema_outhandle: - for line in metadata_lines: - schema_outhandle.write(line) - for line in schema_lines: - schema_outhandle.write(line) - - schema_outhandle.close() - - with open(value_out, "w") as value_outhandle: - for line in metadata_lines: - value_outhandle.write(line) - for line in value_lines: - value_outhandle.write(line) - - value_outhandle.close() - - def write_results(self, metadata: dict, output: str, out_format: str = "json"): - """Function to write comparison outputs to file - - Args: - metadata (dict): Dict containing metadata of the current comparison run - output (str): Path to output file - out_format (str, optional): Format to write outputs to ['json', 'csv']. Defaults to "json". - if 'csv' is provided, two files are produced, - {output_basename}_schema_check.csv: contains results of schema comparison - {output_basename}_value_check.csv: contains results of value comparison - NB: metadata is written as comments starting with "##" to both of the above files. - """ - ## if output format is json, format all dicts in a single one and write to file - if out_format == "json": - out_dict = { - "metadata": metadata, - "schema_check": self.schema_errors, - "value_check": self.value_mismatches - } - - with open(output, "w") as outhandle: - json.dump(out_dict, outhandle, indent=4) - - ## if output format is csv, convert data from all 3 dicts to lines - ## write metadata and schema results to {output_basename}_schema_check.csv - ## write metadata (again) and schema results to {output_basename}_value_check.csv - elif out_format == "csv": - self._format_csv_output(metadata=metadata, output=output) \ No newline at end of file diff --git a/validation_toolkit/TableValidator/file_io.py b/validation_toolkit/TableValidator/file_io.py deleted file mode 100644 index 9e7ded2..0000000 --- a/validation_toolkit/TableValidator/file_io.py +++ /dev/null @@ -1,322 +0,0 @@ -import os, sys -import csv -import json -import yaml -from validation_toolkit.TableValidator.custom_exceptions import UnsupportedFileFormatError, UnexpectedFileFormatError - -class File_IO(): - """Read, write and intercovert between a range of file types. - Currently supports csv, tsv, json, and yaml - """ - def __init__(self, input_file: str, sample_id_key: str, config: str = None, data_reader: dict = None): - """Initialising a File_IO object - - Args: - input_file (str/pathlike): File to read from. - data_reader (dict, optional): Dictionary formatted as {sample_id: {field1: val1, field2: val2...}}. Defaults to None. - This is the output/internal structure of data in File_IO - File_IO can alternatively start with such a dict - sample_id_key (str, optional): The name of the field or column in the input file corresponding to sample IDs. - - """ - ## raise error if sample_id column not provided - ## TODO: add code to possibly try and infer sample_id column - if not sample_id_key: - raise NotImplementedError("Code to handle this must be written and does not yet exist.") - - ## set up a list of accepted file formats - self.SUPPORTED_FILE_TYPES = [".csv", ".tsv", ".json", ".yaml"] - ## set up map of file format to corresponding read functions for ease of use - read_functions = { - "csv": self._read_table, - "tsv": self._read_table, - "json": self._read_serialized, - "yaml": self._read_serialized, - } - - ## if data_reader is provided, use that to initialise - ## if input file is ALSO provided, it is ignored - if data_reader or data_reader and input_file: - self.data_reader = data_reader - - ## if ONLY input file provided, read it in - elif input_file and not data_reader: - self.full_path_input_file = os.path.abspath(input_file) - - ## get file type of input file from its extension - self.input_file_type = self._get_file_type(self.full_path_input_file) - - ## if csv or tsv, pass to tabular file reader function - if self.input_file_type in ["csv", "tsv"]: - tmp_data_reader = read_functions[self.input_file_type](self.full_path_input_file, sample_id_key) - self.data_reader = self._recover_types_in_data(tmp_data_reader) - - ## if json or yaml, pass to serialized file reader function - else: - self.data_reader = read_functions[self.input_file_type](self.full_path_input_file) - - ##TODO: Add this method - if config: - self.data_reader = self.map_data(data_reader_dict=self.data_reader, config=config) - - ## collect a list of fields/column names from the input - self.fields = [] - example_key = list(self.data_reader.keys())[0] - self.fields.extend(list(self.data_reader[example_key].keys())) - - def _get_file_type(self, input_file: str): - """Use file extension to infer file type - - Args: - input_file (str/pathlike): File to read from - - Returns: - file_type (str): The inferred file format. Currently support formats are: csv, tsv, json, yaml - """ - file_type = None - - ## get file extension - file_tuple = os.path.splitext(input_file) - extension = file_tuple[1] - - ## if extension not in known formats but it is txt, try to check if it is csv or tsv - if extension not in self.SUPPORTED_FILE_TYPES: - if extension == ".txt": - sys.stderr.write("Found a .txt file, checking if it is a csv or tsv...\n") - - ## use helper function to check delimiters - try: - file_type = "csv" if self._check_delimiter(input_file=input_file, sep_to_check=",") else "tsv" if self._check_delimiter(input_file=input_file, sep_to_check="\t") else None - print(f"found: {file_type}") - except StopIteration: - file_type = None - print(f"Bad file! Found: {file_type}") - - ## at this point, if at all possible, file_type should have a value - if not file_type: - raise UnsupportedFileFormatError(f"Given file does not appear to be in an accepted format. Accepted formats are:\n{self.SUPPORTED_FILE_TYPES}") - - else: - file_type = extension.strip(".") - - return file_type - - def _check_delimiter(self, input_file:str, sep_to_check:str): - """Check if given file has a delimiter that the program supports checking of ([",", "\t"]) - - Args: - input_file (str): Path to input file to check - sep_to_check (str): Field separator to check for - - Returns: - bool: Whether the given input file is delimited using the given sep_to_check - """ - with open(input_file, "rt") as in_file_handle: - ## use given sep to load file - reader = csv.DictReader(in_file_handle, delimiter=f"{sep_to_check}") - try: - ## try getting the first line, likely the header - ## this invocation of next will fail for invalid files - ## return False if it does - header = next(reader) - except StopIteration: - return False - - ## if this does not fail, also get the putative first "row" - row = next(reader) - num_cols = len(header) - in_file_handle.close() - - ## if only one "column" is found, this is likely not an appropriately formatted table, return False - if num_cols == 1: - return False - ## if >1 "column" found, check the same for the first "row" - ## if cols >1 and len(row) == 1, this is not a valid table - else: - if len(row) == 1: - return False - - ## if all of the above pass, this file probably IS delimited by the given sep_to_check, return True - return True - - def _read_table(self, csv_or_tsv_file: str, sample_id_key: str): - """Read in a tabular file (csv/tsv) and format the data into the required internal representation - - Args: - csv_or_tsv_file (str): Path to data file - sample_id_key (str): Column name containing sample ID info - - Returns: - data (dict): Dictionary representation of the data, formatted as follows: - key: sample_id - value: a dict where - key: field/column name - value: field/column value - """ - ## first make sure the file_type is csv or tsv - ## this is a redundant check and should never evaluate to False - if not self.input_file_type in ["csv", "tsv"]: - raise UnexpectedFileFormatError("Found unexpected file format. Supported formats are: [csv, tsv]") - - ## use the file type to read file in with the correct delimiter - with open(csv_or_tsv_file, "rt") as in_csv_or_tsv_handle: - if self.input_file_type == "csv": - file_reader = csv.DictReader(in_csv_or_tsv_handle, delimiter=",") - else: - file_reader = csv.DictReader(in_csv_or_tsv_handle, delimiter="\t") - - ## populate the data dictionary - data = {} - for row in file_reader: - sample_id = row.get(sample_id_key, "") - if not sample_id: - raise KeyError(f"Specified sample ID key: {sample_id_key} not found in table.") - sample_data = {k: v for k, v in row.items()} - data[sample_id] = sample_data - in_csv_or_tsv_handle.close() - - return data - - def _read_serialized(self, json_or_yaml_file: str): - """Read in from a serialized file format (json/yaml) - - Args: - json_or_yaml_file (str): Path to input file - - Returns: - serial_reader (dict): The serialized data, which is already expected to be in the required structure - """ - ## first make sure the file_type is json or yaml - ## this is a redundant check and should never evaluate to False - if not self.input_file_type in ["json", "yaml"]: - raise UnexpectedFileFormatError("Found unexpected file format. Supported formats are: [json, yaml]") - - with open(json_or_yaml_file, "rt") as in_json_to_yaml_handle: - if self.input_file_type == "json": - serial_reader = json.load(in_json_to_yaml_handle) - if self.input_file_type == "yaml": - serial_reader = yaml.safe_load(in_json_to_yaml_handle) - - return serial_reader - - def _recover_types_in_data(self, data_dict: dict): - """Wrapper function that applies fn _infer_type to a formatted data dictionary. Only applied if the input file was tabular. - - Args: - data_dict (dict): Dictionary representation of the data, formatted as follows: - key: sample_id - value: a dict where - key: field/column name - value: field/column value - - Returns: - updated_data (dict): Updated data dict where data types are inferred and non string data is converted appropriately - """ - updated_data = {} - for sample_id, row_data in data_dict.items(): - new_row_data = {} - for k, v in row_data.items(): - ## iterating over field data in the data dict - ## infer (and convert if needed) data types - converted_value = self._infer_type(v) - new_row_data.update({k: converted_value}) - updated_data.update({sample_id: new_row_data}) - return updated_data - - def _infer_type(self, var: str): - """Function to take string values and convert to other primitive types if applicable - - Args: - var (str): Input value in strring form, as it will be found in tabular files - - Returns: - str | int | float | bool: The var input, converted to int/float/bool as applicable - """ - bool_map = {"true": True, "false": False} - ## first check if the string is numeric - ## if yes, return the int - if var.isnumeric(): - return int(var) - ## next check if string might contain a bool expression - ## if so, return the correct bool - elif var.lower() in ["true", "false"]: - return bool_map[var.lower()] - else: - try: - ## next try casting it to float - ## this only works if the string format can be resolved to str(float) - return float(var) - except ValueError: - ## if the str really cannot be cast to anything else, return as is - return var - - def write_to_csv(self, output_file: str): - """Function to write data to csv - - Args: - output_file (str): Path to output file - """ - rows_to_write = [] - for sample_id in self.data_reader.keys(): - row = {"sample_id": sample_id} - row.update(self.data_reader[sample_id]) - rows_to_write.append(row) - with open(output_file, "w") as out_handle: - writer = csv.DictWriter(out_handle, fieldnames=self.fields, delimiter=",") - writer.writeheader() - for row in rows_to_write: - writer.writerow(row) - - def write_to_tsv(self, output_file: str): - """Function to write data to tsv - - Args: - output_file (str): Path to output file - """ - rows_to_write = [] - for sample_id in self.data_reader.keys(): - row = {"sample_id": sample_id} - row.update(self.data_reader[sample_id]) - rows_to_write.append(row) - with open(output_file, "w") as out_handle: - writer = csv.DictWriter(out_handle, fieldnames=self.fields, delimiter="\t") - writer.writeheader() - for row in rows_to_write: - writer.writerow(row) - - def write_to_json(self, output_file: str): - """Function to write data to JSON - - Args: - output_file (str): Path to output file - """ - with open(output_file, "w") as outhandle: - json.dump(self.data_reader, outhandle, indent=4) - - def write_to_yaml(self, output_file: str): - """Function to write data to yaml - - Args: - output_file (str): Path to output file - """ - with open(output_file, "w") as outhandle: - yaml.dump(self.data_reader, outhandle, indent=4) - - def map_data(self, data_reader_dict, config): - with open(config, "rt") as config_handle: - config_dict = yaml.safe_load(config_handle) - - col_mappings = {k: v["col_name"] for k, v in config_dict.items()} - val_mappings = {k: v["value_map"] for k, v in config_dict.items()} - - mapped = {} - for k, v in self.data_reader.items(): - mapped_vals = {} - for val_k, val_v in v.items(): - if val_mappings[val_k]: - mapped_vals[col_mappings[val_k]] = val_mappings[val_k][val_v] - else: - mapped_vals[col_mappings[val_k]] = val_v - mapped[k] = mapped_vals - - return mapped \ No newline at end of file diff --git a/validation_toolkit/__init__.py b/validation_toolkit/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/validation_toolkit/cli.py b/validation_toolkit/cli.py deleted file mode 100644 index fc38ff8..0000000 --- a/validation_toolkit/cli.py +++ /dev/null @@ -1,148 +0,0 @@ -import os, sys, argparse, datetime -from validation_toolkit.TableValidator.file_io import File_IO -from validation_toolkit.TableValidator.file_compare import FileCompare - -try: - from validation_toolkit.version import __version__ -except ImportError: - __version__ = "0.0.1dev" - version_tuple = (0, 0, "1dev") - -def collect_args(): - parser = argparse.ArgumentParser(description="A tool to compare tabular data against specified reference data.") - parser.add_argument("-v", "--version", action="version", version=f"TableValidator: {__version__}") - - subparsers = parser.add_subparsers(title="subcommands", help="TableValidator sub-commands", dest="run_mode") - - file_compare_mode = subparsers.add_parser("compare") - file_compare_mode.add_argument("-r", "--ref_table", type=str, required=True, help="Reference tabular dataset.") - file_compare_mode.add_argument("-q", "--query_table", type=str, required=True, help="Subject tabular dataset to compare.") - file_compare_mode.add_argument("-rs", "--ref_sampleid_key", type=str, required=True, help="The column/field name that contains sample IDs in the reference table.") - file_compare_mode.add_argument("-qs", "--query_sampleid_key", type=str, required=False, help="The column/field name that contains sample IDs in the query table.") - file_compare_mode.add_argument("-e", "--strict", action = "store_true", required=False, help="Whether to raise warnings as exceptions or not.") - file_compare_mode.add_argument("-m", "--mode", type=str, required=False, default="quick_compare", help="How to execute comparison. [quick_compare, compare_subset]") - file_compare_mode.add_argument("-l", "--field_list", type=str, required=False, help="Comma-separated list of column names/fields to compare, required and used ONLY if using mode 'compare_subset'.") - file_compare_mode.add_argument("-c", "--custom_config", type=str, required=False, help="Custom config file containing col_name and value mappings in .yaml format.") - file_compare_mode.add_argument("-o", "--output_filename", type=str, required=True, help="Path to output file to be written.") - file_compare_mode.add_argument("-f", "--write_format", type=str, required=False, default="json", help="File format to write comparison output to ['json', 'csv'].") - - format_convert_mode = subparsers.add_parser("convert_format") - format_convert_mode.add_argument("-i", "--input_filepath", type=str, required=False, help="File path to read data from for conversion to new format.") - ## code duplication, consider alternatives - format_convert_mode.add_argument("-s", "--sampleid_key", type=str, required=True, help="The column/field name that contains sample IDs.") - format_convert_mode.add_argument("-o", "--output_filename", type=str, required=False, help="File name to write converted data to.") - format_convert_mode.add_argument("-d", "--output_dir", type=str, required=False, default=".", help="Directory path to write converted data to.") - format_convert_mode.add_argument("-f", "--write_format", type=str, required=False, help="File format to convert input data to.") - - args = parser.parse_args() - return args - -def main(): - ## collect basic run metadata - NOW = f"{datetime.datetime.now():%Y-%m-%d %H:%M:%S}" - PROG = __name__.split(".")[0] - metadata = { - "prog": f"{PROG}", - "prog_version": f"{__version__}", - "datetime": f"{NOW}", - "cli": " ".join(sys.argv), - } - - args = collect_args() - - ## confirm accepted mode - if args.mode not in ["quick_compare", "compare_subset"]: - raise ValueError("args.mode should be in [quick_compare, compare_subset]") - - ## set up output file - ## if file with given name exists, use timestamp to prevent overwriting - if args.write_format == "json": - if not os.path.exists(args.output_filename): - validator_output_file = args.output_filename - else: - print(f"Path: {args.output_filename} already exists. Using timestamp to prevent overwriting.") - validator_output_file = args.output_filename.strip(f".json") + "_" + ".".join(NOW.split(" ")[1].split(":")) + ".json" - - ## if output format is csv, 2 files are generated, so two filenames need to be checked - elif args.write_format == "csv": - base = args.output_filename.strip(f".csv") - files_to_check = [f"{base}_schema_check.csv", f"{base}_value_check.csv"] - files_exist = any(os.path.exists(file) for file in files_to_check) - if not files_exist: - validator_output_file = args.output_filename - else: - print(f"Paths using basename: {base} already exist. Using timestamp to prevent overwriting.") - validator_output_file = f"{base}_" + ".".join(NOW.split(" ")[1].split(":")) + ".csv" - print(validator_output_file) - - else: - raise ValueError(f"Unsupported output file format! Supported formats: csv, json. You provided: {args.write_format}") - - ## when run_mode is to compare, collect ref and query filepaths - if args.run_mode == "compare": - r_file = os.path.abspath(args.ref_table) - q_file = os.path.abspath(args.query_table) - - ## add file paths to run metadata - metadata["ref"] = r_file - metadata["query"] = q_file - - ## if only ref sample_id_key provided, assume it is the same in query - r_key = args.ref_sampleid_key - if r_key and not args.query_sampleid_key: - q_key = r_key - else: - q_key = args.query_sampleid_key - - - print("Reading data from files...") - ## if custom config given, use it to map colnames and values onto reference file - ## colnames and values in ref should now match representations in the query - if args.custom_config: - print("Custom config found...") - r_table = File_IO(input_file=r_file, sample_id_key=r_key, config=args.custom_config) - q_table = File_IO(input_file=q_file, sample_id_key=q_key) - - ## initialise validator object - validator = FileCompare(left_data=r_table.data_reader, right_data=q_table.data_reader, strict=args.strict) - - print("Setting up and validating schema...") - ## create and confirm schema - validator.create_schema() - validator.confirm_schema() - - ## in quick_compare, nothing else needs to be done before comparing values - if args.mode == "quick_compare": - print("Running quick_compare...") - validator.compare_values() - print(f"Writing output to: {validator_output_file}") - validator.write_results(metadata=metadata, output=validator_output_file, out_format=args.write_format) - - ## if subset of colnames to be compared, make sure user-provided list of this subset exists - elif args.mode == "compare_subset": - print("Running compare_subset...") - if not args.field_list: - raise ValueError("Cannot run subset comparison without a list of fields to compare.") - fields_to_compare = args.field_list.split(",") - validator.compare_values(field_names=fields_to_compare) - print(f"Writing output to: {validator_output_file}") - validator.write_results(metadata=metadata, output=validator_output_file, out_format=args.write_format) - - print("Success!") - - ## when run_mode is "convert_format" - ## simply read data in, and write out to provided format - elif args.run_mode == "convert_format": - ## admin - input_file = os.path.abspath(args.input_filepath) - output_dir = os.path.abspath(args.output_dir) - output_file = os.path.join(output_dir, args.output_filepath) - reader_object = File_IO(input_file=input_file, sample_id_key=args.sampleid_key) - match args.write_format: - case "csv": reader_object.write_to_csv(output_file=output_file) - case "tsv": reader_object.write_to_tsv(output_file=output_file) - case "json": reader_object.write_to_json(output_file=output_file) - case "yaml": reader_object.write_to_yaml(output_file=output_file) - -if __name__ == "main": - main() From 52e79f7ed4c9dbf6bb6981cb68799c824afcf8c8 Mon Sep 17 00:00:00 2001 From: Kevin Howe Date: Thu, 22 Oct 2026 16:22:03 +0100 Subject: [PATCH 09/45] More prep for implementation --- AGENTS.md | 31 ++++----- README.md | 143 +++--------------------------------------- docs/SPECIFICATION.md | 8 +-- pyproject.toml | 89 +++++++++++++++----------- requirements.txt | 2 - setup.cfg | 6 -- 6 files changed, 78 insertions(+), 201 deletions(-) delete mode 100644 requirements.txt delete mode 100644 setup.cfg diff --git a/AGENTS.md b/AGENTS.md index 749131a..8043ab2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,29 +1,24 @@ -# Agent guidance for `validation_toolkit` +# Agent guidance for `adviso-validation` ## Purpose and authority -This repository is being reimplemented as `adviso-validation`, a tool that translates reference and query datasets from their source dialects into a declared canonical representation, compares them, and reports both post-translation differences and assessment issues. +This repository is being implemented as `adviso-validation`, a tool that translates reference and query datasets from their source dialects into a declared canonical representation, compares them, and reports both post-translation differences and assessment issues. For version 1 behaviour, use this order of authority: -1. `SPECIFICATION.md` — normative behaviour and requirement identifiers. +1. `docs/SPECIFICATION.md` — normative behaviour and requirement identifiers. 2. `src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json` — normative structural schema for comparison configurations. 3. `src/adviso_validation/schemas/comparison-results-v1.0.schema.json` — normative structural schema for canonical results. 4. Version 1 tests and controlled fixtures, once created — executable verification of the sources above. 5. User documentation — explanatory material that must agree with the normative sources. -Do not infer version 1 behaviour from the prototype, its README, or its tests. If the specification and a schema appear inconsistent, stop and report the conflict. Do not silently choose one, weaken either, or invent compatibility behaviour. +If the specification and a schema appear inconsistent, stop and report the conflict. Do not silently choose one, weaken either, or invent compatibility behaviour. ## Repository state -The repository currently contains two generations of work: +This is a clean-slate version 1 implementation. Do not recover behaviour, interfaces, tests, fixtures, terminology, or implementation patterns from Git history unless the repository owner explicitly requests historical investigation. -- `SPECIFICATION.md` and `src/adviso_validation/schemas/` define the intended version 1 product. -- `validation_toolkit/`, `tests/table_validator_tests/`, the current `README.md`, and parts of the packaging configuration describe an unused exploratory prototype. - -The prototype has no compatibility status. Its CLI, APIs, output formats, terminology, internal structure, and tests are not version 1 requirements. Do not copy prototype code merely to accelerate implementation. Reuse something only after confirming that it satisfies the version 1 specification and improves the new design. - -Do not delete or rewrite prototype material unless the assigned task or an approved implementation plan explicitly calls for it. Keeping old and new work separate during the transition is preferable to partially converting legacy modules. +The behaviour specification and schemas are substantially developed; production code and the version 1 test suite are not. Do not describe the package as operational until the specified implementation, verification, documentation, and packaging work is complete. ## Product identities @@ -33,7 +28,7 @@ Preserve these distinct names: - Python import package: `adviso_validation` - Sole version 1 subcommand: `compare` -New production code belongs under `src/adviso_validation/`. Do not add new production modules under the legacy `validation_toolkit/` package. +Production code belongs under `src/adviso_validation/`. ## Working method @@ -46,13 +41,13 @@ Before implementing behaviour: 5. Implement until those tests pass. 6. Run the proportionate wider test suite and static checks. -Use test-driven development for the reimplementation. A failing test is useful only when it represents current specified behaviour. Do not add tests for discarded design ideas, prototype compatibility, renamed properties that never entered version 1, or other historical “fossils.” +Use test-driven development for the implementation. A failing test is useful only when it represents current specified behaviour. Do not add tests for discarded design ideas, renamed properties that never entered version 1, or other historical “fossils.” Tests should cite the requirement IDs they verify. Prefer a focused parameterized test that covers a coherent rule table over many near-duplicate tests. Do not merge unrelated behaviours merely to reduce the test count. Not every implementation, documentation, release-process, or quality requirement needs a direct automated test. ## Change control -- Do not change `SPECIFICATION.md`, either schema, or established requirement semantics unless the task explicitly authorizes a specification change. +- Do not change `docs/SPECIFICATION.md`, either schema, or established requirement semantics unless the task explicitly authorizes a specification change. - Behaviour changes must update all affected normative text, schemas, tests, fixtures, and user documentation together. - Preserve existing requirement IDs. Add new IDs rather than renumbering requirements after insertion or deletion. - Preserve schema version identifiers and canonical `$id` values unless a deliberate schema-version change has been approved. @@ -82,7 +77,7 @@ When a requirement is ambiguous, contradictory, or not implementable as written, Configuration processing has two mandatory layers: 1. structural validation against the packaged comparison-configuration schema; -2. semantic validation of the cross-property rules in `SPECIFICATION.md`. +2. semantic validation of the cross-property rules in `docs/SPECIFICATION.md`. Canonical-result validation likewise has two layers: @@ -110,7 +105,7 @@ Do not introduce a formatter, linter, type checker, build backend, or supported- ## Testing guardrails -The version 1 suite should be organized around public behaviour and stable internal contracts, not private implementation details. It must eventually include the test families specified in Section 14 of `SPECIFICATION.md`. +The version 1 suite should be organized around public behaviour and stable internal contracts, not private implementation details. It must eventually include the test families specified in Section 14 of `docs/SPECIFICATION.md`. In particular: @@ -124,8 +119,6 @@ In particular: - Test the built wheel and source distribution rather than relying only on an editable source checkout. - Do not update expected outputs blindly. Review semantic changes in canonical JSON fixtures field by field. -Until a new v1 suite is established, do not treat passing `tests/table_validator_tests/` as evidence that v1 requirements are satisfied. - ## Security and clinical-data handling Assume inputs and evidence packages may contain sensitive clinical or laboratory information. @@ -141,7 +134,7 @@ Assume inputs and evidence packages may contain sensitive clinical or laboratory The eventual user guide must describe the released behaviour, complete CLI, configuration schema, results schema, finding taxonomy, evidence package, and corrective guidance for execution errors. Documentation examples must validate against the packaged schemas. -Do not update the current prototype README piecemeal so that it appears to describe a usable hybrid product. Replace it as a deliberate implementation milestone when the corresponding v1 interface is ready. +Keep the root README concise and truthful about implementation status. Expand installation and usage instructions only when the corresponding v1 interface is implemented and verified. ## Before declaring work complete diff --git a/README.md b/README.md index 4ecdfe4..398cbdf 100644 --- a/README.md +++ b/README.md @@ -1,140 +1,17 @@ -# Validation Toolkit +# AdvISO Validation -A Python package to compare tabular datasets and record discrepancies. +`adviso-validation` is a command-line tool for comparing a reference dataset with a query dataset. It uses explicit configuration rules to translate different source representations into a common canonical representation, then reports post-translation differences separately from issues that prevented assessment. -# Install +The project is intended to support bioinformatics validation and verification activities associated with ISO 15189 accreditation. It produces comparison evidence; it does not determine whether a pipeline is valid, acceptable, or compliant. -From source +## Project status -```bash -git clone path/to/repo -cd path/to/repo -pip install . -``` +Version 1 is currently specified but not yet implemented. The repository is being prepared for a test-driven implementation. -With pip +The current normative artefacts are: -```bash -pip install validation_toolkit@git+path/to/repo -``` +- [Behaviour specification](docs/SPECIFICATION.md) +- [Comparison-configuration schema](src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json) +- [Canonical-results schema](src/adviso_validation/schemas/comparison-results-v1.0.schema.json) -# Usage -Using this package, you can compare a known "reference" table to a new "query" table. When verifying that a change in your data pipeline did not change the output table, the older iteration of the table is the reference and the new one generated after your change is the query; when comparing the outputs of your pipeline to those of another existing one, the outputs of the existing pipeline can be considered the reference. Depending on your use case, you may want to compare two tables that have identical schemas; or compare two such tables but only on a subset of columns; or else compare two tables that have entirely different schemas -- the CLI for each case is shown below. - -To compare two tables that have the same column names and data vocabularies: - -```bash -validation_toolkit compare \ - -r path/to/reference/data_table \ - -q path/to/query/data_table \ - -rs sample_id_colname \ - -o path/to/output/file \ - -f output_file_format -``` - -To compare **a subset of columns** from two tables that have the same column names and data vocabularies: -```bash -validation_toolkit compare \ - -r path/to/reference/data_table \ - -q path/to/query/data_table \ - -m compare_subset \ - -l col_name_1,col_name_2,col_name3 \ - -rs sample_id_colname \ - -o path/to/output/file \ - -f output_file_format -``` - -To compare two tables that have **different** column names and/or data vocabularies: -```bash -validation_toolkit compare \ - -r path/to/reference/data_table \ - -q path/to/query/data_table \ - -l col_name_1,col_name_2,col_name3 \ - -rs sample_id_colname_in_ref \ - -qs sample_id_colname_in_query \ - -c path/to/custom_config.yaml \ - -o path/to/output/file \ - -f output_file_format -``` - -# Using Custom Configs - -If comparing tabular data that may have been produced by different data pipelines, a custom configuration file in YAML format can be provided using the `-c` flag as shown above. When such a config file is used, the internal representation of the **reference** table is mapped to the provided mappings in the config, and the data in the query table is represented as-is. - -The custom config file has the following structure: - -```yaml -col_1_ref: - col_name: col_1_query - value_map: null -col_2_ref: - col_name: col_2_query - value_map: - true: 1 - false: 0 - unknown: null -col_3_ref: - col_name: col_3_query - value_map: - resistant: 1 - sensitive: 0 - unknown: 2 -``` - -The example config above says: the column `col_1_ref` in the reference table maps to the column called `col_1_query` in the query dataset. In terms of data vocabularies, however, this first column is the same across both datasets. - -This is not true for the column `col_2_ref` in the reference table: this column has a different name in the query table (`col_2_query`) but also, a `boolean true` in the reference table is represented as a `int 1` in the query, and so on. - -# Output Files -This progam can produce outputs in either JSON or CSV formats. If using JSON, the output file will contain the following data: - -```json -{ - "metadata": { - "prog": "validation_toolkit", - "prog_version": "0.0.1dev", - "datetime": "2025-03-25 15:52:19", - "cli": "path/to/validation_toolkit compare -r tests/test_data/data_map_test_left.csv -q tests/test_data/data_map_test_right.csv -rs col_left_1 -qs col_right_1 -c tests/test_data/sample_config.yaml -m compare_subset -l col_right_2,col_right_3 -o test_out.json -f json", - "ref": "path/to/ref_table.csv", - "query": "path/to/query_table.csv" - }, - "schema_check": { - "sample1": { - "sample_id": "sample1", - "col_right_1": "SchemaMatch", - "col_right_2": "SchemaMatch", - "col_right_3": "SchemaMatch" - }, - ... - }, - "value_check": { - "sample1": { - "sample_id": "sample1", - "col_right_1": "Field not checked", - "col_right_2": "ValueMatch", - "col_right_3": "ValueMatch" - }, - ... - } -} -``` -The three pieces of data in this JSON are the `metadata`, which stores information on the features of the comparison run you just executed; the `schema_check`, which stores information on whether the structure of the two tables (after applying config-based mapping if this is applicable) matches or not; and the `value_check`, which stores information on whether the corresponiding values across the tables match or not. - -Alternatively, the user can output this information in CSV format, whereby two files are written (assuming the output file provided by the user is `output_basename.csv`): -- `{output_basename}_schema_check.csv` contains `schema_check` data -- `{output_basename}_value_check.csv` contains `value_check` data - -Metadata lines are formatted as comments starting with two hashes; these are written to the top of both of the above CSV files. - -# List of Options -- `-h`, `--help`: show this help message and exit -- `-r`, `--ref_table`: Reference tabular dataset. -- `-q`, `--query_table`: Subject tabular dataset to compare. -- `-rs`, `--ref_sampleid_key`: The column/field name that contains sample IDs in the reference table. -- `-qs`, `--query_sampleid_key`: The column/field name that contains sample IDs in the query table. -- `-e`, `--strict`: Whether to raise warnings as exceptions or not. **NB** Using this option will cause the program to quit upon encountering differences in the datasets and consequenctly, no output files will be generated. Ordinarily, this flag should never be used. -- `-m`, `--mode`: How to execute comparison. [quick_compare, compare_subset] -- `-l`, `--field_list`: Comma-separated list of column names/fields to compare, required and used ONLY if using mode 'compare_subset'. -- `-c`, `--custom_config`: Custom config file containing col_name and value mappings in .yaml format. -- `-o`, `--output_filename`: Path to output file to be written. -- `-f`, `--write_format`: File format to write comparison output to ['json', 'csv']. +The planned distribution and executable are named `adviso-validation`; the Python import package is `adviso_validation`. diff --git a/docs/SPECIFICATION.md b/docs/SPECIFICATION.md index 085bf50..425cb60 100644 --- a/docs/SPECIFICATION.md +++ b/docs/SPECIFICATION.md @@ -592,7 +592,7 @@ The first-release typed-value operation identifiers are: #### 6.2.9 Normative machine-readable schema and semantic validation -The normative JSON Schema Draft 2020-12 definition of comparison-configuration schema version `1.0` is [`src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json`](src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json). +The normative JSON Schema Draft 2020-12 definition of comparison-configuration schema version `1.0` is [`src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json`](../src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json). **REQ-CFG-128.** The schema MUST use `https://adviso-project.github.io/validation_toolkit/schemas/comparison-configuration-v1.0.schema.json` as its canonical `$id`. @@ -1071,7 +1071,7 @@ No separate package manifest is required in version 1; `comparison-results.json` ### 9.1 Machine-readable output -**REQ-OUT-004.** JSON MUST be the canonical output format. Its normative JSON Schema Draft 2020-12 definition is [`src/adviso_validation/schemas/comparison-results-v1.0.schema.json`](src/adviso_validation/schemas/comparison-results-v1.0.schema.json). +**REQ-OUT-004.** JSON MUST be the canonical output format. Its normative JSON Schema Draft 2020-12 definition is [`src/adviso_validation/schemas/comparison-results-v1.0.schema.json`](../src/adviso_validation/schemas/comparison-results-v1.0.schema.json). **REQ-OUT-125.** `comparison-results.json` MUST be serialized as RFC 8259 JSON encoded as UTF-8 without a byte-order mark, using two-space indentation, LF line endings, and exactly one final newline. Non-ASCII characters MUST be emitted directly rather than escaped unless escaping is required by JSON syntax. Members of fixed-schema objects MUST be emitted in their documented schema order; members of dynamically named objects MUST be emitted in Unicode code-point order. @@ -1653,7 +1653,3 @@ This catalogue summarises minimum executable scenarios for the first release. It - Verifies: `REQ-TST-005` - **AT-TRC-003:** Every supported and unsupported comparison-configuration and results-schema version is exercised to confirm exact supported-version handling and clear rejection of unsupported versions. A version 1.0 result accepts only an embedded version 1.0 comparison configuration. - Verifies: `REQ-REL-002`, `REQ-REL-004`, `REQ-OUT-113` - -## 15. Status of the current prototype - -**REQ-PRD-011.** The existing prototype was created only to stimulate requirements analysis and has never been used operationally. Its code, command-line interface, outputs, Python APIs, module structure, terminology, and behaviour impose no compatibility requirements on version 1. The version 1 implementation is not required to reuse any prototype code. Version 1 behaviour MUST NOT be derived implicitly from the prototype. diff --git a/pyproject.toml b/pyproject.toml index 4dcfe21..86a5fcd 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,48 +1,67 @@ [build-system] -requires = ["setuptools >= 61.0", - "setuptools_scm[toml] >= 4, <6", - "setuptools_scm_git_archive", - "wheel >= 0.29.0" - ] +requires = [ + "setuptools>=69", + "setuptools-scm>=8", + "wheel", +] build-backend = "setuptools.build_meta" -[tool.setuptools_scm] -# setuptools_scm obtains the package version number from Git tags so we don't -# have to have a version number hardcoded in scripts in the package -# upon installation, the version is written (from git tag) to this file -write_to = "validation_toolkit/version.py" -git_describe_command = "git describe --dirty --tags --long --match v* --first-parent" - -[tool.setuptools.packages.find] -where = ["."] -include = ["validation_toolkit"] -exclude = [] -namespaces = false - [project] -name = "validation_toolkit" -authors = [ - {name = "Bruhad Dave", email = "bd8@sanger.ac.uk"}, -] -description = "Compare (verify/validate) analysis outputs against ISO-in-a-Box standard references." +name = "adviso-validation" +dynamic = ["version"] +description = "Translate, compare, and report differences between structured bioinformatics datasets." readme = "README.md" requires-python = ">=3.10" -keywords = [""] +authors = [ + { name = "Bruhad Dave", email = "bd8@sanger.ac.uk" }, +] +keywords = [ + "bioinformatics", + "data validation", + "ISO 15189", +] classifiers = [ - "Programming Language :: Python :: 3.11", - "Operating System :: OS Independent" - + "Development Status :: 2 - Pre-Alpha", + "Environment :: Console", + "Intended Audience :: Healthcare Industry", + "Intended Audience :: Science/Research", + "Operating System :: OS Independent", + "Programming Language :: Python :: 3", ] dependencies = [ - ##TODO: Add as needed - "flake8>=7.0.0", - "pytest>=6.2.2", - "pyyaml", - "importlib-resources>=5.1.0" + "jsonschema[format-nongpl]>=4.23,<5", + "PyYAML>=6.0,<7", ] -dynamic = ["version"] +[project.optional-dependencies] +dev = [ + "build>=1.2,<2", + "pytest>=8,<9", +] [project.scripts] -# create an executable for the CLI -validation_toolkit = "validation_toolkit.cli:main" \ No newline at end of file +adviso-validation = "adviso_validation.cli:main" + +[tool.setuptools] +package-dir = { "" = "src" } +include-package-data = true + +[tool.setuptools.packages.find] +where = ["src"] +include = ["adviso_validation*"] +namespaces = false + +[tool.setuptools.package-data] +"adviso_validation.schemas" = ["*.json"] + +[tool.setuptools_scm] +write_to = "src/adviso_validation/_version.py" +fallback_version = "0.0.0" + +[tool.pytest.ini_options] +addopts = [ + "--strict-config", + "--strict-markers", +] +testpaths = ["tests"] +tmp_path_retention_policy = "failed" diff --git a/requirements.txt b/requirements.txt deleted file mode 100644 index e22f84e..0000000 --- a/requirements.txt +++ /dev/null @@ -1,2 +0,0 @@ -- pyyaml -- importlib_resources \ No newline at end of file diff --git a/setup.cfg b/setup.cfg deleted file mode 100644 index 5f29dac..0000000 --- a/setup.cfg +++ /dev/null @@ -1,6 +0,0 @@ -[flake8] -extend-ignore = - E251, E262, E266 - -[tool:pytest] -tmp_path_retention_policy = failed \ No newline at end of file From bb3e5635ccef0ba6a0f9b773f8b0803ed45db67f Mon Sep 17 00:00:00 2001 From: Kevin Howe Date: Thu, 22 Oct 2026 16:30:54 +0100 Subject: [PATCH 10/45] Tightened AGENTS.md --- AGENTS.md | 192 ++++++++++++++++++++++-------------------------------- 1 file changed, 79 insertions(+), 113 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 8043ab2..6452f18 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,151 +1,117 @@ # Agent guidance for `adviso-validation` -## Purpose and authority +## Purpose of this file -This repository is being implemented as `adviso-validation`, a tool that translates reference and query datasets from their source dialects into a declared canonical representation, compares them, and reports both post-translation differences and assessment issues. +This file governs how agents work in this repository. It does not define product behaviour. Product requirements belong in `docs/SPECIFICATION.md` and the normative schemas; do not duplicate them here. -For version 1 behaviour, use this order of authority: +## Sources of truth -1. `docs/SPECIFICATION.md` — normative behaviour and requirement identifiers. -2. `src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json` — normative structural schema for comparison configurations. -3. `src/adviso_validation/schemas/comparison-results-v1.0.schema.json` — normative structural schema for canonical results. -4. Version 1 tests and controlled fixtures, once created — executable verification of the sources above. -5. User documentation — explanatory material that must agree with the normative sources. - -If the specification and a schema appear inconsistent, stop and report the conflict. Do not silently choose one, weaken either, or invent compatibility behaviour. - -## Repository state +Use this order of authority: -This is a clean-slate version 1 implementation. Do not recover behaviour, interfaces, tests, fixtures, terminology, or implementation patterns from Git history unless the repository owner explicitly requests historical investigation. - -The behaviour specification and schemas are substantially developed; production code and the version 1 test suite are not. Do not describe the package as operational until the specified implementation, verification, documentation, and packaging work is complete. +1. `docs/SPECIFICATION.md` — normative behaviour and requirement identifiers. +2. `src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json` — normative comparison-configuration structure. +3. `src/adviso_validation/schemas/comparison-results-v1.0.schema.json` — normative canonical-results structure. +4. Version 1 tests and controlled fixtures — executable verification of the normative sources. +5. User documentation — explanatory material that must agree with the sources above. -## Product identities +If these sources appear inconsistent, stop and report the exact conflict and its consequences. Do not silently choose an interpretation, weaken a requirement, or invent behaviour. -Preserve these distinct names: +## Repository layout -- Distribution and executable: `adviso-validation` -- Python import package: `adviso_validation` -- Sole version 1 subcommand: `compare` +- Put production code under `src/adviso_validation/`. +- Put automated tests under `tests/`. +- Put maintained user and developer documentation under `docs/`. +- Keep runtime package resources inside `src/adviso_validation/`. +- Treat generated version files, build outputs, caches, coverage output, and virtual environments as generated artefacts rather than source. -Production code belongs under `src/adviso_validation/`. +Do not create new top-level directories or reorganize established modules without a concrete need in the approved implementation plan. ## Working method -Before implementing behaviour: +Before changing behaviour: -1. Read the relevant specification sections in full, including referenced requirements in other sections. -2. Read both schemas when the work touches configuration, findings, results, or reporting. -3. Identify the requirement IDs affected by the change. +1. Read the relevant specification sections in full, including referenced requirements elsewhere in the document. +2. Read the relevant schema definitions. +3. Identify the requirement IDs affected by the work. 4. Add or update the smallest coherent set of tests that verifies those requirements. 5. Implement until those tests pass. -6. Run the proportionate wider test suite and static checks. - -Use test-driven development for the implementation. A failing test is useful only when it represents current specified behaviour. Do not add tests for discarded design ideas, renamed properties that never entered version 1, or other historical “fossils.” +6. Run proportionate wider tests, static checks, schema checks, and packaging checks. +7. Update affected documentation and controlled fixtures. -Tests should cite the requirement IDs they verify. Prefer a focused parameterized test that covers a coherent rule table over many near-duplicate tests. Do not merge unrelated behaviours merely to reduce the test count. Not every implementation, documentation, release-process, or quality requirement needs a direct automated test. +Use test-driven development for the version 1 implementation. A failing test is useful only when it represents current specified behaviour. Do not create tests for discarded ideas, historical decisions that never entered version 1, or renamed and removed properties. ## Change control -- Do not change `docs/SPECIFICATION.md`, either schema, or established requirement semantics unless the task explicitly authorizes a specification change. -- Behaviour changes must update all affected normative text, schemas, tests, fixtures, and user documentation together. -- Preserve existing requirement IDs. Add new IDs rather than renumbering requirements after insertion or deletion. -- Preserve schema version identifiers and canonical `$id` values unless a deliberate schema-version change has been approved. -- Treat unknown configuration and result properties as errors; do not add permissive extension points speculatively. -- Do not add CLI options, comparators, normalisers, file formats, report formats, plugin systems, compatibility shims, or acceptance logic outside the specified v1 scope. -- Keep commits and patches narrowly scoped. Do not mix behavioural work with unrelated restructuring or formatting. - -When a requirement is ambiguous, contradictory, or not implementable as written, present the exact conflict and its consequences to the repository owner. Do not resolve product decisions through undocumented assumptions. - -## Design guardrails - -- Keep the top-level CLI thin. Put comparison behaviour in independently testable internal modules. -- Keep parsing, configuration validation, canonicalisation, record alignment, comparison, result construction, semantic result validation, and rendering as distinct responsibilities. -- Prefer small explicit interfaces and ordinary modules. Version 1 does not need a general plugin framework. -- Keep the comparison kernel independent of CLI parsing and HTML presentation. -- Model the three processing phases (`contract`, `records`, and `values`) separately from finding classification (`comparison_difference` and `assessment_issue`). -- Treat the reference and query as having distinct roles where the specification does so, including directional relative tolerance. -- Never use binary floating-point arithmetic for specified decimal parsing, normalisation, tolerance calculation, or canonical rendering. -- Preserve source lexical evidence where required while keeping canonical typed values distinct from source values. -- Make ordering explicit. Never rely on dictionary, set, filesystem, locale, or hash iteration order for canonical output. -- Collect all safely assessable findings. Do not stop normal comparison at the first difference. -- Keep execution errors outside canonical comparison results and findings. -- Generate HTML only from validated canonical JSON and fixed renderer resources. - -## Validation boundaries - -Configuration processing has two mandatory layers: - -1. structural validation against the packaged comparison-configuration schema; -2. semantic validation of the cross-property rules in `docs/SPECIFICATION.md`. - -Canonical-result validation likewise has two layers: - -1. structural validation against the packaged results schema, with Draft 2020-12 format assertion enabled; -2. semantic validation of reconciliation, ordering, classification, configuration agreement, and other cross-property invariants. - -Never publish an evidence package unless both result-validation layers pass. A generated-result validation failure is `internal.unexpected`, not a user finding. - -Load packaged schemas and the bundled configuration through the package-resource API. Core comparison must work without network access and must not resolve schemas by downloading their canonical HTTPS identifiers. - -## Python quality - +- Do not change `docs/SPECIFICATION.md`, either normative schema, or established requirement semantics unless the task explicitly authorizes a specification change. +- Preserve existing requirement IDs. Add new IDs rather than renumbering requirements. +- Preserve schema version identifiers and canonical `$id` values unless a deliberate version change has been approved. +- When behaviour changes are authorized, update all affected normative text, schemas, tests, fixtures, and user documentation together. +- Do not add speculative features, extension points, compatibility layers, dependencies, or abstractions. +- Keep patches narrowly scoped. Do not mix behavioural work with unrelated restructuring or formatting. +- Preserve unrelated work already present in the worktree. + +When a requirement is ambiguous, contradictory, or not implementable as written, ask for a decision rather than encoding an undocumented assumption. + +## Code quality + +- Follow PEP 8 and write idiomatic Python compatible with the project's supported Python versions. +- Follow PEP 257 for public-module, class, function, and method docstrings; document intent and constraints rather than restating the code. +- Where configured automated formatting, linting, or type-checking rules differ from general style guidance, the project configuration is authoritative. +- Avoid unrelated formatting changes that obscure the substantive content of a patch. +- Keep the command-line layer thin and keep domain logic independently testable. +- Separate I/O, domain logic, validation, and presentation responsibilities. +- Prefer small explicit interfaces and ordinary modules over framework-like abstraction. - Use type annotations for production interfaces and non-trivial internal functions. -- Prefer explicit domain types, enums, and immutable value objects where they prevent invalid states. -- Keep functions focused and make transformation stages easy to test in isolation. -- Raise or return domain-specific errors at architectural boundaries; do not use broad exception handling to hide defects. -- Convert unexpected exceptions into the specified CLI diagnostic only at the outer application boundary. -- Do not expose tracebacks during ordinary CLI operation. +- Prefer explicit domain types and immutable value objects where they prevent invalid states. +- Keep functions focused and transformation stages testable in isolation. - Avoid mutable global state and import-time side effects. -- Keep dependencies minimal, justified, pinned or bounded appropriately, and separated into runtime and development dependencies. -- Use safe YAML loading with the restrictions required by the specification. Never construct arbitrary Python objects from configuration input. -- Use secure temporary-file and atomic-publication patterns. Never broaden filesystem permissions. +- Use domain-specific exceptions or result types at architectural boundaries. +- Do not use broad exception handling to conceal defects. +- Keep dependencies minimal and justified, with runtime and development dependencies separated. +- Write deterministic code; do not depend accidentally on unordered collections, filesystem order, locale, or hash iteration. -Do not introduce a formatter, linter, type checker, build backend, or supported-Python change as an incidental edit. Such project-wide tooling choices belong in the implementation plan and must be configured consistently before being enforced. +Do not introduce or replace a formatter, linter, type checker, build backend, test framework, dependency manager, or supported-Python policy as an incidental edit. Project-wide tooling choices require deliberate configuration and approval. -## Testing guardrails +## Testing practice -The version 1 suite should be organized around public behaviour and stable internal contracts, not private implementation details. It must eventually include the test families specified in Section 14 of `docs/SPECIFICATION.md`. +- Cite the requirement IDs verified by each behavioural test or coherent parameterized scenario. +- Prefer parameterization when it expresses one rule table clearly; do not merge unrelated behaviours merely to reduce test count. +- Test public behaviour and stable internal contracts rather than private implementation details. +- Use controlled, synthetic, non-sensitive fixtures. +- Keep fixtures small enough for meaningful review unless scale is the subject of the test. +- Document generated fixtures and make their regeneration deterministic. +- Test both directions where the specification requires symmetry. +- Exercise boundary values, malformed inputs, and failure paths, not only successful examples. +- Validate representative configuration and result documents against their packaged schemas. +- Include installed-entry-point and built-distribution tests where required by the specification. +- Do not update expected outputs blindly; review semantic changes field by field. -In particular: +Not every implementation, documentation, release-process, or quality requirement needs a direct automated test. Add automated coverage where it provides a reliable and maintainable verification of observable behaviour. -- Use controlled, synthetic, non-sensitive fixtures. -- Keep fixtures small enough to review, except for tests specifically concerned with scale or resource behaviour. -- Record why generated fixtures exist and make their regeneration deterministic. -- Test both directions where reference/query symmetry is required. -- Test boundary values, malformed inputs, Unicode, composite keys, null versus absence, exact decimal behaviour, and deterministic ordering. -- Validate representative results against the packaged schema and the semantic result validator. -- Test the installed `adviso-validation` entry point, not only direct Python calls. -- Test the built wheel and source distribution rather than relying only on an editable source checkout. -- Do not update expected outputs blindly. Review semantic changes in canonical JSON fixtures field by field. - -## Security and clinical-data handling - -Assume inputs and evidence packages may contain sensitive clinical or laboratory information. - -- Never place real patient data, identifiers, credentials, absolute local paths, or secrets in source, tests, fixtures, snapshots, logs, or examples. -- Routine logs, progress output, terminal diagnostics, and execution-error diagnostics must not reproduce record identifiers or source values. -- Findings may contain identifiers or values only where the specification requires them to identify or explain a finding. -- Do not add telemetry, remote schema lookup, remote assets, or any other network transmission. -- Treat input basenames as potentially sensitive. -- Preserve the specification's overwrite, symlink, temporary-file, and atomic-publication protections. +## Sensitive data and repository safety -## Documentation +- Never place real patient data, clinical identifiers, credentials, secrets, or identifying local paths in source, tests, fixtures, snapshots, logs, examples, or documentation. +- Do not use destructive repository or filesystem operations unless the user has explicitly authorized them and the exact target has been verified. +- Never overwrite unrelated user changes to make a check pass. -The eventual user guide must describe the released behaviour, complete CLI, configuration schema, results schema, finding taxonomy, evidence package, and corrective guidance for execution errors. Documentation examples must validate against the packaged schemas. +Refer to the specification for the software's normative security and data-governance behaviour. + +## Documentation -Keep the root README concise and truthful about implementation status. Expand installation and usage instructions only when the corresponding v1 interface is implemented and verified. +- Keep the README and user documentation accurate for the implemented version. +- Do not document planned behaviour as already available. +- Ensure complete examples validate against the corresponding packaged schema. +- Link to normative material instead of copying substantial requirements into explanatory documentation. ## Before declaring work complete -For the scope changed, confirm and report: +Report: -- affected requirement IDs; +- the requirement IDs affected; - tests added or updated and why they are sufficient; - commands run and their results; -- schema validation performed, including format assertion where applicable; -- any checks not run and the reason; -- documentation or fixture changes; +- schema, static, packaging, and documentation checks performed; +- checks not run and the reason; - remaining assumptions, limitations, or follow-up work. -Do not claim completion while relevant tests, schema checks, packaging checks, or documentation are knowingly inconsistent with the implementation. +Do not claim completion while relevant tests, schemas, documentation, or packaging metadata are knowingly inconsistent with the implementation. From 5ac14b280927d3177f8b029620d57741930ee4e2 Mon Sep 17 00:00:00 2001 From: Kevin Howe Date: Fri, 23 Oct 2026 17:00:43 +0100 Subject: [PATCH 11/45] More refinement of specification and schemas --- docs/SPECIFICATION.md | 363 +++++++++++------- .../comparison-configuration-v1.0.schema.json | 183 ++++++++- .../comparison-results-v1.0.schema.json | 229 +++-------- 3 files changed, 440 insertions(+), 335 deletions(-) diff --git a/docs/SPECIFICATION.md b/docs/SPECIFICATION.md index 425cb60..2c78022 100644 --- a/docs/SPECIFICATION.md +++ b/docs/SPECIFICATION.md @@ -68,7 +68,7 @@ adviso-validation compare ... The first-release scope of `compare` is: - comparison of structured records represented in CSV, TSV or JSON; -- explicit single-field or composite record keys; +- configuration-defined record keys and conservative single-field key selection in automatic mode; - detection of matching, differing, missing, and extra records and fields; - field-specific comparison rules; - controlled column-name and value mappings; @@ -95,12 +95,12 @@ The `compare` command does not: The following principles govern all requirements: -1. **REQ-PRN-001.** **Explicit over inferred:** Comparison behaviour MUST be declared, not guessed from observed data. +1. **REQ-PRN-001.** **Explicit over inferred:** A custom comparison configuration MUST declare comparison behaviour rather than relying on observed-data inference. The automatic strict-exact mode defined in Section 6.9 is the sole version 1 exception: it MAY infer same-name field correspondence and a single record key only by the fixed rules in that section, and MUST NOT infer types, conversions, mappings, normalisation, tolerance, or biological equivalence. 2. **REQ-PRN-002.** **No silent data loss:** Input records, duplicate keys, unmapped values, and unsupported structures MUST NOT be silently discarded or overwritten. 3. **REQ-PRN-003.** **Symmetry:** Missing and extra content MUST be assessed in both directions. 4. **REQ-PRN-009.** **Separation of concerns:** Run status, comparison evidence, and the laboratory's acceptance decision MUST remain distinct concepts. 5. **REQ-PRN-004.** **Complete evidence:** A normal comparison mismatch MUST still produce a complete report. -6. **REQ-PRN-005.** **Determinism:** The same exact input bytes, input basenames, configuration document, toolkit version, and runtime identity MUST produce identical result content except for the explicitly variable run identifier and completion time. Comparison outcomes MUST additionally satisfy the invariance rules in Section 9.1. +6. **REQ-PRN-005.** **Determinism:** The same exact input bytes, input basenames, configuration selection, toolkit version, and runtime identity MUST produce identical result content except for the explicitly variable run identifier and completion time. Configuration selection includes omission of `--configuration`, in which case automatic construction MUST itself be deterministic. Comparison outcomes MUST additionally satisfy the invariance rules in Section 9.1. 7. **REQ-PRN-006.** **Traceability:** Findings MUST identify the rule, record, and field that produced them. 8. **REQ-PRN-007.** **Data minimisation:** Reports MUST include only the source identifiers and values necessary to explain and review the comparison. Routine logs, progress output, terminal diagnostics, and execution-error diagnostics MUST NOT reproduce them. 9. **REQ-PRN-008.** **Fail clearly:** Invalid or ambiguous input MUST produce an explicit error rather than an assumed interpretation. @@ -109,8 +109,10 @@ The following principles govern all requirements: - **Reference dataset:** The dataset containing the approved expected or comparator records. “Reference” does not imply intrinsic truth. - **Query dataset:** The dataset being evaluated against the reference. -- **Comparison configuration:** The versioned YAML configuration declaring the data contract, normalisation, and comparison behaviour. Its permitted structure and semantics are defined by the comparison-configuration schema and this behaviour specification. -- **Not assessed:** Deliberately excluded by an explicit field policy. +- **Custom comparison configuration:** A versioned YAML document supplied by the user to declare the data contract, normalisation, and comparison behaviour. +- **Effective comparison configuration:** The schema-valid configuration used for a run: either the validated custom configuration or the deterministic configuration generated by automatic strict-exact mode. +- **Automatic strict-exact mode:** The configuration-free mode used when `--configuration` is omitted. It constructs an effective comparison configuration from the inputs using only the rules in Section 6.9. +- **Not assessed:** Outside the canonical fields selected by `fields.in_scope`. - **Not assessable:** Included in the configured comparison, but unable to be compared safely because a prerequisite was not satisfied. - **Contract comparison:** Assessment of whether inputs satisfy the declared data contract, including fields, logical types, key definitions, nullability, mappings, and configured constraints. - **Records comparison:** Alignment and assessment of valid record identities, including matching, missing, and extra records. Invalid, missing, null, or duplicate record keys are execution errors rather than record-comparison findings. @@ -140,7 +142,7 @@ The following principles govern all requirements: **REQ-INP-002.** A custom comparison configuration MAY be supplied. -**REQ-INP-003.** If a custom comparison configuration is omitted, the bundled strict-exact configuration defined in Section 6.9 MUST be used. +**REQ-INP-003.** If a custom comparison configuration is omitted, the tool MUST use the automatic strict-exact mode defined in Section 6.9. ### 5.2 Supported formats @@ -184,7 +186,7 @@ For CSV and TSV: - **REQ-INP-025.** all physical input rows MUST be accounted for; - **REQ-INP-026.** the one-based data-record number and one-based starting physical line number of every record MUST be retained for traceability; the header is not a data record; - **REQ-INP-027.** Input values MUST initially remain strings unless a custom configuration declares a null representation or type conversion. -- **REQ-INP-028.** In the bundled strict-exact configuration, an empty cell MUST be an empty string. +- **REQ-INP-028.** CSV and TSV cells, including empty cells, MUST initially remain strings before any configured field processing. ### 5.5 JSON parsing @@ -229,18 +231,19 @@ For CSV and TSV: ### 5.7 Empty inputs - **REQ-INP-052.** Zero-byte files and datasets with zero records MUST be recognised explicitly and distinguished. A valid zero-record JSON dataset is `[]`; a valid zero-record CSV or TSV dataset contains a valid header and no data records. A zero-byte JSON file is malformed JSON, and a zero-byte CSV or TSV file is an input error because it lacks the required header. -- **REQ-INP-053.** If both datasets are syntactically valid and contain no records, the records phase MUST report no missing records. -- **REQ-INP-054.** If both datasets are syntactically valid and contain no records, the records phase MUST report no extra records. -- **REQ-INP-055.** If both datasets are syntactically valid and contain no records, the values phase MUST perform no comparisons. +- **REQ-INP-053.** If both datasets are syntactically valid, contain no records, and provide or permit construction of a valid effective comparison configuration and record key, the records phase MUST report no missing records. +- **REQ-INP-054.** Under the conditions in `REQ-INP-053`, the records phase MUST report no extra records. +- **REQ-INP-055.** Under the conditions in `REQ-INP-053`, the values phase MUST perform no comparisons. - **REQ-INP-056.** Contract requirements that can be evaluated without records MUST still be checked. - **REQ-INP-057.** The tool MUST NOT infer that an empty dataset is unsuitable merely because it is empty. - **REQ-INP-066.** Per-record requirements have no applicable opportunities in a zero-record dataset and MUST NOT produce findings or not-assessable classifications merely because no records exist. -- **REQ-INP-067.** Available dataset-level structure MUST still be assessed. A zero-record CSV or TSV header MAY establish declared, missing, or undeclared source columns. A JSON empty array exposes no field structure, so field presence or absence MUST NOT be inferred from it. -- **REQ-INP-068.** Two valid zero-record datasets MUST have complete records and values phases, zero matched, missing, and extra records, and zero value opportunities. Header-derived contract findings do not alter those phase statuses. -- **REQ-INP-069.** When exactly one dataset has records, every record on that side MUST be reported as missing or extra as applicable; because there are no aligned records, there are zero value opportunities and the records and values phases remain complete. +- **REQ-INP-067.** Available dataset-level structure MUST still be assessed. A zero-record CSV or TSV header MAY establish configured, missing, or otherwise excluded source columns. A JSON empty array exposes no field structure, so field presence or absence MUST NOT be inferred from it. +- **REQ-INP-068.** When the conditions in `REQ-INP-053` are satisfied, two valid zero-record datasets MUST have complete records and values phases, zero matched, missing, and extra records, and zero value opportunities. Header-derived contract findings do not alter those phase statuses. +- **REQ-INP-069.** When exactly one dataset has records and a valid effective comparison configuration and record key are available, every record on that side MUST be reported as missing or extra as applicable; because there are no aligned records, there are zero value opportunities and the records and values phases remain complete. +- **REQ-INP-071.** In automatic strict-exact mode, two empty JSON arrays do not expose a field from which a record key can be selected and MUST produce an `input.invalid_record_key` execution error. A valid custom configuration can compare the same inputs. - Any minimum-size acceptance criterion belongs to the external laboratory decision process. -- **REQ-INP-058.** A non-empty reference and empty query MUST report every expected record as missing. -- **REQ-INP-059.** An empty reference and non-empty query MUST report every observed record as extra. +- **REQ-INP-058.** When a valid effective comparison configuration and record key are available, a non-empty reference and empty query MUST report every expected record as missing. +- **REQ-INP-059.** When a valid effective comparison configuration and record key are available, an empty reference and non-empty query MUST report every observed record as extra. ## 6. Comparison configurations @@ -255,10 +258,12 @@ For CSV and TSV: - **REQ-CFG-099.** provide actionable validation errors with configuration locations; - **REQ-CFG-100.** be embedded in the canonical comparison result. +An automatically generated effective configuration is constructed only after both datasets have been parsed and their dataset-wide field names have been discovered. It MUST satisfy the comparison-configuration schema and semantic rules before any contract, record, or value comparison is performed, and it MUST be embedded in the canonical comparison result. + **REQ-CFG-002.** A valid comparison configuration MUST declare at least: - **REQ-CFG-101.** record key fields; -- **REQ-CFG-102.** field definitions and an explicit policy for undeclared fields; +- **REQ-CFG-102.** field definitions and an explicit list of fields in scope for comparison; - **REQ-CFG-103.** comparison rules. ### 6.2 Custom comparison-configuration YAML format @@ -295,30 +300,34 @@ metadata: fields: record_key: - "record_id" - undeclared: "fail" + in_scope: + - "record_id" + - "field_example_1" + - "field_example_2" + - "field_example_3" definitions: record_id: - source: + source_fields: reference: "sample_id" query: "specimen_id" required: reference: true query: true logical_type: "string" - nulls: + null_policy: allowed: false nulls_equal: false comparator: name: "typed_exact" field_example_1: - source: + source_fields: reference: "expected_result" query: "reported_result" required: reference: true query: true logical_type: "decimal" - nulls: + null_policy: allowed: false nulls_equal: false typed_normalisation: @@ -332,14 +341,14 @@ fields: absolute_tolerance: "0.01" review_priority: "high" # optional; default: normal field_example_2: - source: + source_fields: reference: "expected_interpretation" query: "reported_interpretation" required: reference: true query: true logical_type: "string" - nulls: + null_policy: allowed: true nulls_equal: true missing_tokens: @@ -389,6 +398,45 @@ fields: comparator: name: "typed_exact" review_priority: "low" # optional; default: normal + field_example_3: + source_fields: + reference: "expected_markers" + query: "reported_markers" + required: + reference: true + query: true + logical_type: + collection_type: "set" + element_type: "string" + element_comparator: + name: "typed_exact" + null_policy: + allowed: false + nulls_equal: false + collection_representation: + query: + format: "delimited_text" + delimiter: "/" + element_normalisation: + - operation: "trim" + parameters: + side: "both" + - operation: "case_fold" + element_mapping: + - from: + category: "string" + value: "wild_type" + to: + category: "string" + value: "WT" + - from: + category: "string" + value: "mutant" + to: + category: "string" + value: "MUT" + comparator: + name: "set_exact" ``` @@ -396,7 +444,7 @@ The template is illustrative rather than a recommendation to use these particula **REQ-CFG-016.** The published machine-readable schema MUST encode the complete version 1 comparison-configuration structure and constraints. -The three field definitions illustrate different configuration roles: `record_id` is a renamed exact key; `field_example_1` is parsed as an exact decimal and quantised before tolerance comparison; and `field_example_2` demonstrates source-text normalisation, side-specific missing tokens, and mapping different source vocabularies to common canonical string values. The deliberately generic names make clear that these are user-defined canonical field names rather than reserved configuration keywords. These example rules are not domain recommendations. +The four field definitions illustrate different configuration roles: `record_id` is a renamed exact key; `field_example_1` is parsed as an exact decimal and quantised before tolerance comparison; `field_example_2` demonstrates source-text normalisation, side-specific missing tokens, and mapping different source vocabularies to common canonical string values; and `field_example_3` compares a reference JSON array with a slash-delimited query string after element-level normalisation and mapping. The deliberately generic names make clear that these are user-defined canonical field names rather than reserved configuration keywords. These example rules are not domain recommendations. #### 6.2.3 Top-level keys and identity @@ -420,27 +468,23 @@ The only permitted top-level keys are `schema_version`, `metadata`, and `fields` #### 6.2.5 Field selection, declaration, and source names -- Every field in `fields.definitions` is in scope. Field definitions are the sole mechanism for selecting explicitly configured fields in version 1. -- **REQ-CFG-030.** `fields.undeclared` is required and MUST be one of `compare`, `exclude`, or `fail`. - - `compare` compares dataset-wide shared undeclared field names with `preserved_exact`; a name observed in only one dataset creates one dataset-level contract finding because no corresponding field can be established. - - `exclude` does not compare undeclared fields and records one entry per dataset side and source field in the exclusion summary as `not_assessed`. - - `fail` creates one dataset-level contract finding per dataset side and undeclared source field and does not compare that field. -- **REQ-CFG-031.** Undeclared reference and query fields MUST be aligned only when their input field names are identical. -- No name inference or source-name mapping is applied to undeclared fields. -- **REQ-CFG-116.** Undeclared source-field names MUST be discovered dataset-wide. For CSV and TSV, the header defines the observed names even when there are no records. For JSON, the observed names are the union of member names across all record objects. -- **REQ-CFG-117.** Under `compare`, an undeclared name observed in both datasets MUST become an implicit untyped comparison field whose reference, query, and canonical names are identical, whose comparator is `preserved_exact`, and whose review priority is unassigned. -- **REQ-CFG-118.** For a shared undeclared field under `compare`, each aligned record in which at least one side contains the field MUST contribute one value opportunity. Two present values are compared normally; one present and one absent value MUST produce one `value_mismatch` finding with presence-state evidence; and two absent values contribute no opportunity. -- **REQ-CFG-119.** Under `compare`, an undeclared name observed in only one dataset MUST produce one dataset-level `undeclared_field` finding for that side. Each aligned record containing that field MUST contribute one not-assessable value opportunity. Occurrences confined to missing or extra records do not create value opportunities. -- **REQ-CFG-120.** Under `fail`, each undeclared field MUST produce one dataset-level `undeclared_field` finding for each dataset side on which it is observed. Each aligned record in which the field is present on either side MUST contribute one not-assessable value opportunity, irrespective of whether the same source name occurs in both datasets. -- **REQ-CFG-121.** A dataset-level undeclared-field finding or exclusion entry MUST record the number of records on that dataset side that contain the source field. A field known only from a zero-record CSV or TSV header has a record count of zero. +- `fields.definitions` declares canonical fields and their reference-to-query correspondence. Declaration alone does not select a field for assessment. +- **REQ-CFG-030.** `fields.in_scope` is required and MUST be a non-empty list of unique canonical field names. Every listed name MUST exist in `fields.definitions`. +- **REQ-CFG-031.** The order of names in `fields.in_scope` MUST NOT affect the comparison outcome or the ordering of findings and exclusions. The embedded configuration preserves the effective list as provenance. +- **REQ-CFG-116.** Source-field names MUST be discovered dataset-wide. For CSV and TSV, the header defines the observed names even when there are no records. For JSON, the observed names are the union of member names across all record objects. +- **REQ-CFG-117.** A field present in `fields.definitions` but absent from `fields.in_scope` is deliberately excluded from assessment. It MUST produce exactly one `configured_field_excluded` entry naming its canonical field, regardless of whether either source field is observed in the inputs. +- **REQ-CFG-118.** A source field observed on a dataset side but not named by any field definition for that side is outside scope. It MUST produce one `undeclared_field_excluded` entry for that dataset side and source field. +- **REQ-CFG-119.** Excluded fields MUST NOT be compared, create findings, create value opportunities, or affect run or phase status. +- **REQ-CFG-120.** Exclusions MUST be summarised rather than repeated for individual records and MUST use the exact forms and deterministic ordering defined in Section 9.1.3. +- **REQ-CFG-121.** An `undeclared_field_excluded` entry MUST record the number of records on that dataset side that contain the source field. A field known only from a zero-record CSV or TSV header has a record count of zero. - **REQ-CFG-032.** Fields with different names MUST be declared explicitly under a common canonical field definition if they are intended to correspond. - **REQ-CFG-033.** `fields.definitions` MUST be a mapping keyed by unique canonical field name. -- **REQ-CFG-034.** `fields.definitions` MUST contain a definition for every field referenced by `fields.record_key`. -- **REQ-CFG-035.** `source.reference` and `source.query` MUST each contain a non-empty source field name. A declared field always defines correspondence between one reference field and one query field, regardless of whether either field is required to be present in every record. +- **REQ-CFG-034.** `fields.definitions` MUST contain a definition for every field referenced by `fields.record_key`, and every record-key field MUST also occur in `fields.in_scope`. +- **REQ-CFG-035.** `source_fields.reference` and `source_fields.query` MUST each contain a non-empty source field name. A declared field always defines correspondence between one reference field and one query field, regardless of whether either field is required to be present in every record. - **REQ-CFG-036.** `required.reference` and `required.query` MUST be booleans. - **REQ-CFG-037.** Absence of a field required on a particular side MUST create a contract finding. - **REQ-CFG-038.** Absence of an optional field MUST NOT create a missing-field finding. -- **REQ-CFG-039.** A field observed in an input but absent from `fields.definitions` is an undeclared field and MUST be handled only according to `fields.undeclared`. +- **REQ-CFG-039.** A source field observed in an input but not represented by a field definition for that dataset side MUST be excluded as specified by `REQ-CFG-118`. It MUST NOT be aligned or compared implicitly, even when the same source-field name occurs on both sides. - **REQ-CFG-040.** `review_priority`, when present, MUST be `low`, `normal`, or `high`. When omitted, it defaults to `normal` in the effective configuration. - **REQ-CFG-041.** A declared field's resolved `review_priority` MUST be attached to every finding attributable to that canonical field, including missing-field, null-policy, type, parsing, mapping, and value-mismatch findings. - **REQ-CFG-042.** Review priority MUST NOT be inherited by findings that cannot be attributed to a declared field, such as missing or extra records. @@ -454,13 +498,13 @@ The only permitted top-level keys are `schema_version`, `metadata`, and `fields` **REQ-CFG-048.** For a scalar field, `logical_type` MUST be one of `string`, `integer`, `decimal`, `boolean`, `date`, or `datetime`. -**REQ-CFG-049.** `logical_type` MUST be omitted when, and only when, the comparator is `preserved_exact`. A `preserved_exact` field MUST also omit `source_normalisation`, `value_mapping`, `typed_normalisation`, and `nulls.missing_tokens`. Its requiredness and null policy still apply, and parser-native JSON null remains governed by that null policy, but no additional source representation is transformed or interpreted as null. +**REQ-CFG-049.** `logical_type` MUST be omitted when, and only when, the comparator is `preserved_exact`. A `preserved_exact` field MUST also omit `source_normalisation`, `collection_representation`, `value_mapping`, `typed_normalisation`, and `null_policy.missing_tokens`. Its requiredness and null policy still apply, and parser-native JSON null remains governed by that null policy, but no additional source representation is transformed or interpreted as null. **REQ-CFG-050.** For a collection field, `logical_type` MUST use this form: ```yaml logical_type: - name: "list" # list, set, or multiset + collection_type: "list" # list, set, or multiset element_type: "string" # scalar logical type element_comparator: name: "typed_exact" @@ -476,7 +520,7 @@ logical_type: **REQ-CFG-126.** `preserved_exact`, `normalised_string`, `numeric_exact`, `date_exact`, `datetime_exact`, and the collection comparators MUST NOT be used as element comparators in version 1. Exact comparison of any declared scalar element type uses `typed_exact`. -**REQ-CFG-053.** Every declared field MUST contain a `nulls` mapping: +**REQ-CFG-053.** Every declared field MUST contain a `null_policy` mapping: - **REQ-CFG-054.** `allowed` and `nulls_equal` MUST be booleans. - **REQ-CFG-055.** `nulls_equal` MUST be `false` when `allowed` is `false`. @@ -507,7 +551,7 @@ category: "null" Descriptors are matched using parser-native category and value equality under Section 5.6. The retained lexical form remains evidence but does not create a distinct numerical value. This explicit representation prevents YAML scalar resolution from changing mapping keys or missing tokens. -#### 6.2.7 Normalisation and value mapping +#### 6.2.7 Normalisation, collection representation, and value mapping **REQ-CFG-060.** `source_normalisation` MAY be omitted. Omission means that no source-text normalisation is applied. @@ -518,9 +562,10 @@ Descriptors are matched using parser-native category and value equality under Se **REQ-CFG-063.** When present, `typed_normalisation` MUST be a non-empty ordered list applied to the canonical typed values on both sides. Each operation entry has this form: ```yaml -- operation: "trim" +- operation: "decimal_quantise" parameters: - side: "both" + places: 3 + rounding: "half_even" ``` **REQ-CFG-064.** `parameters` MAY be omitted only when the registered operation has no parameters. @@ -540,7 +585,7 @@ The first-release typed-value operation identifiers are: - `decimal_quantise`, requiring a non-negative integer `places` and `rounding` with one of `half_even`, `half_up`, or `down`; and - `datetime_truncate`, requiring `precision` with one of `second`, `millisecond`, or `microsecond`. -**REQ-CFG-066.** A source-text operation MUST appear only in `source_normalisation.reference` or `source_normalisation.query`. It applies only to the corresponding dataset. +**REQ-CFG-066.** A source-text operation MUST appear only in `source_normalisation.reference`, `source_normalisation.query`, or a side's `collection_representation..element_normalisation`. It applies only to values or decoded elements, respectively, from the corresponding dataset side. **REQ-CFG-123.** A source-text operation MUST apply only when the current parser-native value is a string. A non-string value MUST pass through the source-normalisation stage unchanged; it MUST NOT be stringified, transformed, or rejected merely because a source-text operation is configured. @@ -548,6 +593,25 @@ The first-release typed-value operation identifiers are: **REQ-CFG-068.** Operation semantics MUST be published with the schema and recorded sufficiently for reproducibility. This information MUST include the Unicode-data version used by Unicode operations. +**REQ-CFG-137.** `collection_representation` MAY be present only for a collection logical type. It MAY contain `reference`, `query`, or both; each present side declares how that dataset side represents the collection and MAY declare element-level source processing. An omitted side uses `json_array` with no element normalisation or element mapping. + +**REQ-CFG-138.** Each present side of `collection_representation` MUST contain `format`, which MUST be `json_array` or `delimited_text`. + +- `json_array` means a parser-native array in JSON or a parser-native string containing one valid JSON array in CSV or TSV. +- `delimited_text` means a parser-native string split using an explicitly configured delimiter. It does not mean a CSV or TSV row-level dialect. + +**REQ-CFG-139.** A `delimited_text` declaration MUST contain a non-empty string `delimiter`. A `json_array` declaration MUST omit `delimiter`. + +**REQ-CFG-140.** `element_normalisation`, when present, MUST be a non-empty ordered list of the source-text operations defined in this section. The operations apply only to string elements from the declared dataset side. Non-string elements pass through unchanged. + +**REQ-CFG-141.** `element_mapping`, when present, MUST be a non-empty ordered list using the same `from` and `to` structure as `value_mapping`. Both descriptors MUST be non-null. Each `from` value MUST occur at most once within one side's element mapping; conflicting or unreachable entries MUST invalidate the comparison configuration. Many-to-one element mappings are permitted when declared explicitly. + +**REQ-CFG-142.** When an element mapping is present, an element with no matching entry MUST produce a `mapping_failure` contract finding, prevent construction of that field's canonical collection value, and make that value opportunity `not_assessable`. Unmapped elements MUST NOT pass through implicitly. + +**REQ-CFG-143.** An empty optional `collection_representation`, `element_normalisation`, or `element_mapping` mapping or list MUST be rejected. Absence MUST be represented by omission. + +**REQ-CFG-145.** A collection field MUST omit field-level `source_normalisation`, `value_mapping`, and `typed_normalisation`. Source processing of collection elements MUST be declared through the applicable side of `collection_representation`. The field-level null policy and missing-token recognition continue to apply to the collection value as a whole before representation decoding. + **REQ-CFG-069.** `value_mapping` MAY be omitted. Omission means that no value mapping is applied. **REQ-CFG-070.** When present, `value_mapping` MUST contain `reference`, `query`, or both. Each present side MUST be a non-empty ordered list of entries with the following form. An omitted side has no value mapping: @@ -567,7 +631,7 @@ The first-release typed-value operation identifiers are: - **REQ-CFG-072.** Each `from` value MUST occur at most once within one side’s mapping. - **REQ-CFG-073.** Conflicting entries MUST invalidate the comparison configuration. - **REQ-CFG-074.** Unreachable entries MUST invalidate the comparison configuration. -- A value mapping requires a scalar `logical_type`; value mapping of collection fields is not supported in the first release. Its `to` values are parsed as the destination type after mapping. +- A field-level `value_mapping` requires a scalar `logical_type`; whole-collection value mapping is not supported in the first release. Collection elements may instead use `collection_representation..element_mapping`. A field-level mapping's `to` values are parsed as the destination type after mapping. - When a side has a value mapping, a source value with no mapping entry creates a contract finding. Pass-through of unmapped values is not supported in the first release. **REQ-CFG-075.** Before comparison, the tool MAY expand omitted optional processing properties and omitted dataset sides into explicit empty internal structures for internal use. @@ -607,11 +671,14 @@ The standalone schema file is the authoritative machine-readable definition. **REQ-CFG-090.** JSON Schema cannot reliably express every relationship involving user-defined mapping keys, uniqueness across separate arrays, or references between distant parts of a document. After schema validation, the implementation MUST therefore apply the following semantic checks before reading either dataset: - **REQ-CFG-104.** every name in `fields.record_key` exists in `fields.definitions`; +- **REQ-CFG-129.** every name in `fields.in_scope` exists in `fields.definitions`, and every name in `fields.record_key` also occurs in `fields.in_scope`; - **REQ-CFG-105.** every key field is required on both sides, non-nullable, scalar, and uses `typed_exact`; -- **REQ-CFG-106.** a field without `logical_type` uses `preserved_exact` and has no source normalisation, value mapping, typed normalisation, or configured missing tokens; conversely, a field using `preserved_exact` omits `logical_type` and all of those processing properties; +- **REQ-CFG-106.** a field without `logical_type` uses `preserved_exact` and has no source normalisation, collection representation, value mapping, typed normalisation, or configured missing tokens; conversely, a field using `preserved_exact` omits `logical_type` and all of those processing properties; - **REQ-CFG-107.** each comparator and normalisation operation is compatible with the field’s logical type and processing stage; - **REQ-CFG-108.** source-value descriptors contain lexically valid values for their declared categories, and each side’s value mapping has unique `from` values; -- **REQ-CFG-127.** the `source.reference` names are unique across canonical field definitions, and the `source.query` names are independently unique across canonical field definitions; +- **REQ-CFG-144.** each `collection_representation` is used only with a collection logical type, and each side's element mapping has unique `from` values and contains no conflicting or unreachable entries; +- **REQ-CFG-127.** the `source_fields.reference` names are unique across canonical field definitions, and the `source_fields.query` names are independently unique across canonical field definitions; +- **REQ-CFG-136.** a custom configuration does not use the reserved automatic-mode identifier `org.adviso.automatic-strict-exact`; **REQ-CFG-091.** A failure of any semantic check MUST be reported as an invalid comparison configuration with the relevant configuration path and violated rule. @@ -622,13 +689,13 @@ The standalone schema file is the authoritative machine-readable definition. - **REQ-REC-003.** Key fields MUST be present, valid under their declared field definitions, and non-null. - **REQ-REC-004.** Key values MUST NOT undergo implicit numeric or boolean conversion. - **REQ-REC-005.** Normalisation of key values MUST be explicitly configured. -- **REQ-REC-006.** Alternative key names and composite keys MUST be declared in a custom configuration. -- **REQ-REC-007.** Key names MUST NOT be inferred. +- **REQ-REC-006.** Composite keys MUST be declared in a custom configuration. A non-`record_id` single-field key MAY be selected without a custom configuration only by the automatic discovery rules in Section 6.9. +- **REQ-REC-007.** Key names MUST NOT be inferred except by the automatic strict-exact mode defined in Section 6.9. - **REQ-REC-008.** Command-line key overrides MUST NOT be supported in the first release. - **REQ-REC-009.** The normalised key MUST uniquely identify one record within its dataset. -- **REQ-REC-010.** A missing, null, invalid, or duplicate record key prevents construction of an unambiguous record model and MUST be an input error in every comparison. -- **REQ-REC-011.** The resulting execution error MUST identify the affected dataset. -- **REQ-REC-012.** The resulting execution error MUST identify the key or key field involved. +- **REQ-REC-010.** A missing, null, invalid, duplicate, undiscoverable, or ambiguous record key prevents construction of an unambiguous record model and MUST be an input error in every comparison. +- **REQ-REC-011.** The resulting execution error MUST identify the affected dataset when the defect is attributable to one side. An automatic discovery failure or ambiguity MUST state that both inputs were considered. +- **REQ-REC-012.** The resulting execution error MUST identify the selected key or candidate field names where they exist. A no-candidate error MUST state that no field satisfied the automatic key criteria. - **REQ-REC-013.** Where available, the resulting execution error MUST identify the relevant source-record locations. - **REQ-REC-014.** Duplicate records MUST never overwrite one another silently. - **REQ-REC-015.** Findings and reports MUST retain enough information to distinguish every source record. @@ -639,7 +706,7 @@ The standalone schema file is the authoritative machine-readable definition. A type declaration is required for a field that uses typed parsing, normalisation, mapping, or a type-specific comparator. -**REQ-VAL-002.** Every custom configuration MUST explicitly declare how undeclared fields are treated: `compare`, `exclude`, or `fail`. There MUST be no implicit policy. An `exclude` policy MUST be represented as an assessment exclusion in accordance with Section 7.5. +**REQ-VAL-002.** Every custom configuration MUST explicitly list the canonical fields selected for assessment in `fields.in_scope`. Defined fields omitted from that list and observed source fields not represented by a definition are outside scope and MUST be represented as exclusions in accordance with Section 7.5. **REQ-VAL-003.** Every declared field MUST define its canonical name, reference source name, query source name, required status on each side, null policy, and comparator. @@ -668,8 +735,8 @@ The accepted parser-native inputs for each declared logical type are fixed as fo - **REQ-VAL-078.** A `decimal` field MUST accept a parser-native integer, parser-native decimal, or parser-native string matching `[+-]?[0-9]+(?:\.[0-9]+)?(?:[eE][+-]?[0-9]+)?`. Conversion of a parser-native integer to a decimal is exact. Text such as `.5` and `1.` MUST be rejected. - **REQ-VAL-079.** A `boolean` field MUST accept a parser-native boolean or a parser-native string accepted by Section 6.5.4. - **REQ-VAL-080.** A `date` or `datetime` field MUST accept only a parser-native string, which MUST then satisfy the applicable grammar in Sections 6.5.5 and 6.5.6. -- **REQ-VAL-081.** A collection field MUST accept either a parser-native array or, for CSV and TSV input, a parser-native string containing one valid JSON array. No other parser-native category is accepted as a collection. -- **REQ-VAL-082.** After a collection has been obtained, each element MUST be parsed independently according to the declared scalar element type and the same source-category rules. +- **REQ-VAL-081.** A collection side using the default or explicitly declared `json_array` representation MUST accept a parser-native array from JSON or a parser-native string containing one valid JSON array from CSV or TSV. No other parser-native category is accepted for that representation. +- **REQ-VAL-082.** After a collection has been decoded and any configured element processing has been applied, each element MUST be parsed independently according to the declared scalar element type and the same source-category rules. - **REQ-VAL-083.** A parser-native object MUST be invalid for every declared version 1 logical type. An untyped field using `preserved_exact` MAY still preserve and compare a parser-native object. - **REQ-VAL-084.** A source representation not accepted directly by this matrix MUST require an explicit value mapping to an accepted representation. The implementation MUST NOT introduce any additional implicit conversion. - **REQ-VAL-085.** Null recognition and null-policy handling MUST occur before typed parsing. Null is therefore not an accepted typed-parser input category. @@ -729,10 +796,15 @@ The accepted parser-native inputs for each declared logical type are fixed as fo - **REQ-VAL-040.** A `set` MUST reject duplicate elements after typed parsing, using typed-exact canonical equality, rather than silently collapse them. Distinct source representations that parse to the same canonical value, such as textual integers `"01"` and `"1"`, are duplicates. - **REQ-VAL-041.** A `multiset` MUST be unordered. - **REQ-VAL-042.** A `multiset` MUST preserve element occurrence counts. -- **REQ-VAL-043.** JSON collections MUST use arrays. -- **REQ-VAL-044.** A CSV or TSV collection cell MUST use valid JSON-array syntax. -- **REQ-VAL-045.** Ordinary delimited text MUST NOT be inferred to be a collection. +- **REQ-VAL-043.** A JSON collection using `json_array` representation MUST be a parser-native array. +- **REQ-VAL-044.** A CSV or TSV collection cell using `json_array` representation MUST use valid JSON-array syntax. +- **REQ-VAL-045.** Ordinary delimited text MUST NOT be inferred to be a collection. It is accepted only for a dataset side explicitly configured with `format: "delimited_text"`. - **REQ-VAL-046.** Collection elements MUST be non-null in version 1. A parser-native null element MUST produce an `invalid_field_value` finding, prevent construction of a canonical collection value, and make that field comparison `not_assessable`. +- **REQ-VAL-087.** A `delimited_text` representation MUST accept only a parser-native string. The configured delimiter MUST be matched literally, not as a regular expression, and splitting MUST proceed deterministically from left to right using non-overlapping occurrences. +- **REQ-VAL-088.** CSV or TSV quoting and escaping MUST be resolved by the dataset parser before collection splitting. Delimited collection text has no additional quoting or escaping syntax in version 1; delimiter text within an element cannot be represented by that encoding. +- **REQ-VAL-089.** An empty delimited string MUST decode to an empty collection. Otherwise, leading, trailing, or consecutive delimiters MUST produce empty-string elements, which then undergo the configured element processing and typed parsing normally. Whitespace MUST NOT be trimmed implicitly. +- **REQ-VAL-090.** For each decoded collection element, configured `element_normalisation` operations MUST run in declared order before configured `element_mapping`; the resulting non-null value MUST then undergo scalar element-type parsing. No whole-field missing-token or null-policy rule is reapplied to individual elements. +- **REQ-VAL-091.** Element mappings MUST use parser-native category and value equality after element normalisation. A configured element mapping is closed: every non-null element must match an entry. Applied element mappings MUST remain identifiable through the embedded comparison configuration and the source and canonical evidence required for a finding. - Nested collections are outside the first-release scope. ### 6.6 Null, blank, and absent values @@ -748,7 +820,7 @@ The accepted parser-native inputs for each declared logical type are fixed as fo - **REQ-VAL-054.** A prohibited null MUST be a contract finding. - **REQ-VAL-055.** A prohibited null MUST make the affected value `not_assessable` unless another treatment is explicitly configured. -The bundled strict-exact configuration permits parser-native nulls. Under that configuration, null matches only null and remains distinct from an absent field, an empty string, and every non-null value. +For a nullable configured field, null matches only null when `nulls_equal` is true and remains distinct from an absent field, an empty string, and every non-null value. The field-level null policy governs the collection value as a whole and does not permit null collection elements. Version 1 has no separate element-null policy. @@ -759,12 +831,14 @@ The field-level null policy governs the collection value as a whole and does not - **REQ-VAL-056.** Reference and query fields MAY map to a canonical field name. - **REQ-VAL-057.** Column-name mappings MUST be one-to-one in version 1. One source field MUST NOT feed multiple canonical fields, and one canonical field MUST NOT receive multiple source fields from the same dataset side. - **REQ-VAL-058.** Reuse of a source name by more than one canonical field on the same dataset side MUST invalidate the comparison configuration before either dataset is read. Reference and query namespaces are independent, so the same source name MAY occur once on each side. -- **REQ-VAL-059.** Unknown source fields MUST follow the undeclared-field policy. +- **REQ-VAL-059.** Unknown source fields MUST be excluded from assessment and summarised in the canonical result. They MUST NOT be inferred, mapped, or compared implicitly. - **REQ-VAL-060.** Required mapped fields that are absent MUST be reported. - **REQ-VAL-061.** All applied mappings MUST be identifiable in the evidence. #### 6.7.2 Value mappings +Requirements in this subsection describe field-level `value_mapping` unless they explicitly mention element mapping. Collection-element mapping has the additional constraints and processing order in Sections 6.2.7 and 6.5.7. + - **REQ-VAL-062.** A value mapping MUST operate on a preserved source value before typed parsing. - **REQ-VAL-063.** The mapping output MUST be parsed as the declared destination logical type. - **REQ-VAL-064.** A map MUST declare its source representation and destination logical type. @@ -790,7 +864,7 @@ The field-level null policy governs the collection value as a whole and does not 4. if the value is null, apply the null policy and bypass value mapping and typed parsing; 5. for a non-null value, apply a declared source-value mapping and produce a `mapping_failure` finding if a configured mapping has no matching entry; 6. if the mapping output is null, apply the null policy and bypass typed parsing; -7. otherwise parse the declared logical type; +7. otherwise parse the declared logical type; for a collection, first decode the side's declared or default collection representation, then apply element normalisation, element mapping, and scalar element-type parsing to each element in that order; 8. apply declared typed-value normalisation; 9. compare canonical values. @@ -800,21 +874,37 @@ The field-level null policy governs the collection value as a whole and does not **REQ-VAL-075.** The source value MUST be included in the evidence package only when required to explain the failure. It MUST NOT be reproduced in routine logs, progress output, terminal diagnostics, or execution-error diagnostics. -### 6.9 Bundled strict-exact configuration +### 6.9 Automatic strict-exact mode -**REQ-CFG-092.** If `--configuration` is omitted, the tool MUST select the bundled strict-exact configuration. It MUST be distributed as a documented template from which laboratories can develop controlled custom configurations, and the selected effective configuration MUST be recorded in provenance. Supplying `--configuration` MUST select the named custom configuration. +**REQ-CFG-092.** If `--configuration` is omitted, the tool MUST construct an effective comparison configuration deterministically from the parsed inputs using this section. Supplying `--configuration` MUST disable automatic construction and select only the named custom configuration. -**REQ-CFG-093.** The bundled default configuration MUST: +**REQ-CFG-093.** Automatic strict-exact mode MUST discover source-field names dataset-wide using `REQ-CFG-116`. Field-name matching MUST be exact and case-sensitive. It MUST NOT apply aliases, approximate matching, or Unicode, case, or whitespace normalisation to field names. -- **REQ-CFG-109.** declare one key whose canonical, reference, and query field names are all `record_id`; -- **REQ-CFG-110.** include every shared non-key field without requiring field-specific declarations; -- **REQ-CFG-111.** preserve parser-native types and use `preserved_exact` comparison; -- **REQ-CFG-112.** perform no type inference, implicit coercion, mapping, normalisation, or tolerance; -- **REQ-CFG-113.** permit parser-native null values and compare them according to Section 6.6; -- **REQ-CFG-114.** treat two syntactically valid zero-record datasets as having zero missing records, zero extra records, and zero value mismatches, while evaluating contract findings as required by Sections 5.7 and 7.2; -- **REQ-CFG-115.** treat reference-only and query-only fields as contract findings. +#### 6.9.1 Automatic record-key selection -**REQ-CFG-094.** Because parser-native categories differ between formats, strict-exact comparison across different input formats MAY report `value_mismatch` findings for lexically similar values. Such a difference is not a `type_mismatch`, because `preserved_exact` declares no logical type contract. A custom typed configuration is required when cross-format logical equivalence is intended. +- **REQ-CFG-109.** If the exact field name `record_id` is observed on both dataset sides, it MUST be selected as the single-field record key. +- **REQ-CFG-110.** An automatically selected `record_id` MUST be present in every record, non-null, parser-native string-valued, and unique within each dataset. It MAY be selected from matching CSV or TSV headers when both datasets contain zero records. +- **REQ-CFG-111.** If an automatically selected `record_id` fails any key requirement, the run MUST produce an `input.invalid_record_key` execution error. The tool MUST NOT fall back to another candidate key. +- **REQ-CFG-112.** If `record_id` is not observed on both sides, an automatic key candidate MUST satisfy all of the following conditions: + 1. the exact same field name is observed on both sides; + 2. the field is present in every record on both sides; + 3. every value is a non-null parser-native string; + 4. values are unique within each dataset; and + 5. the reference and query value sets have at least one exactly equal value in common. +- **REQ-CFG-113.** Candidate assessment MUST use preserved source strings without conversion or normalisation. It MUST NOT rank candidates by column position, name patterns, preferred-name lists, value-set overlap, or any other heuristic. +- **REQ-CFG-114.** If exactly one field satisfies `REQ-CFG-112`, it MUST be selected as the single-field record key. If none qualifies, the run MUST produce an `input.invalid_record_key` execution error explaining that no automatic key could be established. If more than one qualifies, the same error code MUST identify the candidate field names and explain that key selection is ambiguous. Source key values MUST NOT be reproduced in the diagnostic. +- **REQ-CFG-115.** Automatic mode MUST NOT construct a composite key. A composite key requires a custom comparison configuration. + +#### 6.9.2 Generated field definitions + +- **REQ-CFG-130.** The generated effective configuration MUST use `schema_version: "1.0"`, `metadata.id: "org.adviso.automatic-strict-exact"`, and `metadata.revision: "1.0"`. The identifier is reserved for tool-generated automatic configurations so that result consumers can identify the configuration's origin without an additional results property. +- **REQ-CFG-131.** The selected key field's canonical, reference source, and query source names MUST be identical. It MUST be declared as required and non-nullable on both sides, with logical type `string` and comparator `typed_exact`. +- **REQ-CFG-132.** Every non-key source-field name observed on both sides MUST become one generated canonical field whose canonical, reference source, and query source names are identical. It MUST be included in `fields.in_scope`, be optional on both sides, permit null with null equal to null, use `preserved_exact`, and have no logical type, missing tokens, mapping, normalisation, coercion, or tolerance. +- **REQ-CFG-133.** A source field observed on only one side MUST remain undefined in the generated configuration and MUST be represented by an `undeclared_field_excluded` entry under Section 9.1.3. It MUST NOT produce a finding or value opportunity. +- **REQ-CFG-134.** The generated `fields.record_key` MUST contain only the selected key. `fields.in_scope` and `fields.definitions` MUST contain the selected key and every same-name field observed on both sides, ordered by Unicode code point with the key first. +- **REQ-CFG-135.** The generated document MUST validate against the comparison-configuration schema and all applicable semantic configuration rules before the contract phase begins. It MUST be embedded unchanged in the canonical result as the effective comparison configuration. + +**REQ-CFG-094.** Automatic mode MUST perform no type inference, implicit coercion, value mapping, normalisation, or tolerance. Non-key values compare according to `preserved_exact`, so lexically or scientifically equivalent representations can differ, particularly across input formats. The HTML report MUST identify automatic mode prominently, explain that only identically named fields observed on both sides were compared, and direct the user to review exclusions before interpreting the findings. A custom configuration is required whenever record identity, field correspondence, or value equivalence requires domain knowledge. ## 7. Comparison behaviour @@ -830,18 +920,19 @@ The field-level null policy governs the collection value as a whole and does not **REQ-CMP-003.** Later comparison MUST continue wherever its prerequisites remain satisfied. -**REQ-CMP-004.** Configured content that cannot be compared safely MUST be `not_assessable`. Content deliberately excluded by field policy MUST be `not_assessed`. +**REQ-CMP-004.** In-scope content that cannot be compared safely MUST be `not_assessable`. Content outside `fields.in_scope` MUST be `not_assessed`. **REQ-CMP-005.** A run MUST conceptually: -1. validate the effective comparison configuration; +1. validate a supplied custom comparison configuration, if any; 2. identify and hash inputs; 3. parse inputs without silent loss; -4. perform the contract phase; -5. align and compare records; -6. compare values for safely aligned records; -7. determine whether every configured assessment was completed; -8. write results and provenance atomically. +4. when no custom configuration was supplied, construct and validate the automatic effective comparison configuration; +5. perform the contract phase; +6. align and compare records; +7. compare values for safely aligned records; +8. determine whether every configured assessment was completed; +9. write results and provenance atomically. ### 7.2 Contract comparison @@ -849,7 +940,7 @@ The field-level null policy governs the collection value as a whole and does not - **REQ-CMP-034.** field present and valid; - **REQ-CMP-035.** required field missing from the reference or query; -- **REQ-CMP-036.** observed undeclared field and the treatment required by `fields.undeclared`; +- **REQ-CMP-036.** configured in-scope field or content excluded from assessment; - **REQ-CMP-037.** invalid logical type, null state, mapping, or controlled value; - **REQ-CMP-038.** field deliberately `not_assessed`; - **REQ-CMP-039.** field `not_assessable` because a prerequisite failed. @@ -870,7 +961,7 @@ The field-level null policy governs the collection value as a whole and does not ### 7.4 Values comparison and comparators -**REQ-CMP-013.** The values phase MUST compare every configured, valid field in safely aligned records. +**REQ-CMP-013.** The values phase MUST compare every valid non-key field listed in `fields.in_scope` in safely aligned records. **REQ-CMP-014.** A value whose contract or alignment prerequisite failed MUST be `not_assessable`. Other values MUST continue to be compared. @@ -878,7 +969,7 @@ The field-level null policy governs the collection value as a whole and does not #### 7.4.1 `preserved_exact` -Used by the bundled strict-exact configuration. Values match according to the parser-native value model and equality rules in Section 5.6. No logical type, additional missing-token recognition, inference, coercion, mapping, normalisation, or tolerance is applied. Textual `"1"`, integer `1`, decimal `1.0`, and boolean `true` are distinct. +Values match according to the parser-native value model and equality rules in Section 5.6. No logical type, additional missing-token recognition, inference, coercion, mapping, normalisation, or tolerance is applied. Textual `"1"`, integer `1`, decimal `1.0`, and boolean `true` are distinct. #### 7.4.2 `typed_exact` @@ -926,7 +1017,7 @@ difference = abs(reference - query) - **REQ-CMP-028.** A malformed record that prevents a complete and unambiguous record model MUST be an input error. - **REQ-CMP-029.** A malformed record MUST NOT disappear silently. - **REQ-CMP-030.** Key fields MUST always be processed. -- **REQ-CMP-031.** Under `fields.undeclared: "exclude"`, undeclared fields MUST be represented by a summary rather than by an individual finding for every excluded occurrence. The canonical report MUST record a stable exclusion code, the policy, the affected input field names, and the number of excluded records or values where calculable. +- **REQ-CMP-031.** Out-of-scope content MUST be represented by exclusion summaries rather than individual findings or per-record exclusions. The canonical report MUST distinguish fields deliberately omitted from `fields.in_scope` from observed source fields not represented in `fields.definitions`. - **REQ-CMP-032.** Record order MUST NOT affect the comparison outcome defined in Section 9.1. Input digests, source locations, messages containing those locations, and report-local finding IDs MAY differ as permitted by the provenance and ordering rules. - **REQ-CMP-033.** Output ordering MUST be deterministic and independent of hash-map ordering and locale. @@ -951,7 +1042,6 @@ difference = abs(reference - query) - **REQ-FND-008.** missing record; - **REQ-FND-009.** extra record; - **REQ-FND-010.** missing required field; -- **REQ-FND-011.** undeclared field; - **REQ-FND-012.** type mismatch: the parser-native category of a non-null source value is not accepted by the field's declared logical type under the matrix in Section 6.5; - **REQ-FND-013.** invalid field value: the parser-native category is accepted by the declared logical type, but the value fails its lexical, domain, collection-content, or other field-level validity rule; - **REQ-FND-014.** null-policy violation; @@ -961,7 +1051,7 @@ difference = abs(reference - query) **REQ-FND-023.** Every finding MUST have exactly one of two classifications. Classification expresses the meaning of a finding, while phase records where in processing it was detected; neither replaces the other. - **REQ-FND-024.** `comparison_difference` means that translation succeeded sufficiently to compare the relevant records or values and a difference remained. `missing_record`, `extra_record`, and `value_mismatch` MUST have this classification. -- **REQ-FND-025.** `assessment_issue` means that the declared translation or assessment could not be applied cleanly to some input content. `missing_required_field`, `undeclared_field`, `type_mismatch`, `invalid_field_value`, `null_policy_violation`, and `mapping_failure` MUST have this classification. +- **REQ-FND-025.** `assessment_issue` means that the declared translation or assessment could not be applied cleanly to some input content. `missing_required_field`, `type_mismatch`, `invalid_field_value`, `null_policy_violation`, and `mapping_failure` MUST have this classification. Finding classification does not assert scientific or clinical significance. In particular, an assessment issue such as an unrecognised mapped value may be important to investigate, but the tool cannot report it as a post-translation difference because translation did not succeed. @@ -969,13 +1059,11 @@ Finding classification does not assert scientific or clinical significance. In p **REQ-FND-019.** A record-specific missing-required-field, type-mismatch, invalid-field-value, null-policy-violation, or mapping-failure finding MUST describe one record, one canonical field, and one affected dataset side. If reference and query independently violate the same field rule, the tool MUST emit two findings, one with `affected_side: reference` and one with `affected_side: query`. It MUST NOT combine the independent violations into one `both` finding. The dataset-level zero-record missing-field case is the explicit exception defined by `REQ-FND-022`. -**REQ-FND-020.** A value-mismatch finding MUST describe one comparison between two individually valid, assessable values and MUST use `affected_side: both`. Missing-record and extra-record findings remain one finding per affected record key. Undeclared-field finding granularity is defined separately by the undeclared-field policy. - -**REQ-FND-021.** An `undeclared_field` finding MUST be dataset-level and identify one dataset side, one source field name, and the number of records on that side containing the field. It MUST NOT be repeated for individual record occurrences. A `value_mismatch` involving a shared undeclared field under `compare` is record-specific, identifies the shared source field name, has unassigned review priority, and is not accompanied by an `undeclared_field` finding. +**REQ-FND-020.** A value-mismatch finding MUST describe one comparison between two individually valid, assessable values and MUST use `affected_side: both`. Missing-record and extra-record findings remain one finding per affected record key. **REQ-FND-022.** If a zero-record CSV or TSV header omits a declared source column that is required on that dataset side, the tool MUST emit one dataset-level `missing_required_field` finding for that canonical field and side. It contains the canonical field and review priority but no record identity, source location, or value evidence. An empty JSON array cannot produce this finding because it exposes no field structure. -**REQ-FND-017.** `not_assessable` MUST be a value-opportunity classification, not a finding code. Every not-assessable opportunity MUST be explained by its applicable causal finding, such as `missing_required_field`, `type_mismatch`, `invalid_field_value`, `null_policy_violation`, `mapping_failure`, or `undeclared_field`. The tool MUST NOT emit an additional finding merely to repeat that the affected opportunity is not assessable. +**REQ-FND-017.** `not_assessable` MUST be a value-opportunity classification, not a finding code. Every not-assessable opportunity MUST be explained by its applicable causal finding, such as `missing_required_field`, `type_mismatch`, `invalid_field_value`, `null_policy_violation`, or `mapping_failure`. The tool MUST NOT emit an additional finding merely to repeat that the affected opportunity is not assessable. **REQ-FND-007.** Execution errors MUST NOT be represented as findings or assigned finding codes. In particular, invalid invocation or configuration, malformed input structure, invalid record keys, resource failure, I/O failure, and internal failure are execution errors governed by Section 8.3. @@ -1020,7 +1108,7 @@ Finding classification does not assert scientific or clinical significance. In p | `input` | `invalid_encoding` | An input is not valid UTF-8 | | `input` | `invalid_syntax` | Input text is malformed for the format selected by its extension | | `input` | `invalid_structure` | Parsed input does not satisfy the required dataset or record structure | -| `input` | `invalid_record_key` | A record key is missing, null, invalid, or duplicated, or cannot form an unambiguous record model | +| `input` | `invalid_record_key` | A record key is missing, null, invalid, duplicated, undiscoverable, ambiguous, or otherwise unable to form an unambiguous record model | | `resource` | `limit_exceeded` | A documented implementation safeguard stops assessment | | `resource` | `exhausted` | An unplanned resource exhaustion prevents completion | | `output` | `invalid_destination` | The selected output destination violates destination or overwrite rules | @@ -1037,15 +1125,15 @@ Finding classification does not assert scientific or clinical significance. In p Overall status is `complete` only when all three phases are complete. -**REQ-RUN-018.** The contract phase MUST be `complete` when every applicable contract check was performed. Contract findings, including missing-field, type, invalid-value, null-policy, mapping, and undeclared-field findings, MUST NOT by themselves make the contract phase incomplete. +**REQ-RUN-018.** The contract phase MUST be `complete` when every applicable contract check was performed. Contract findings, including missing-field, type, invalid-value, null-policy, and mapping findings, MUST NOT by themselves make the contract phase incomplete. **REQ-RUN-019.** The records phase MUST be `complete` when valid record keys were constructed and every record was aligned and classified. Missing-record and extra-record findings MUST NOT make the records phase incomplete. A key or record-model failure that prevents this assessment is an execution error and produces no comparison phase statuses. -**REQ-RUN-020.** The values phase MUST be `complete` when every applicable in-scope value opportunity was compared or deliberately excluded. It MUST be `incomplete` when at least one such opportunity is `not_assessable`. +**REQ-RUN-020.** The values phase MUST be `complete` when every applicable in-scope value opportunity was compared. It MUST be `incomplete` when at least one such opportunity is `not_assessable`. **REQ-RUN-021.** A finding MUST NOT by itself make a phase incomplete. A phase is incomplete only when an intended assessment within that phase could not be performed safely. -**REQ-RUN-022.** Content classified as `not_assessed` by an explicit exclusion policy MUST NOT make any phase incomplete. +**REQ-RUN-022.** Content classified as `not_assessed` because it is outside `fields.in_scope` MUST NOT make any phase incomplete. **REQ-RUN-023.** Overall run status MUST be `complete` if and only if the contract, records, and values phases are all `complete`; otherwise a non-error comparison run MUST be `incomplete`. @@ -1111,11 +1199,11 @@ Every property is required, including empty `exclusions` and `findings` arrays. **REQ-OUT-095.** For determinism requirements, the comparison outcome consists of run and phase statuses; summary counts; exclusions considered without input occurrence order; and each finding's classification, code, phase, affected side, record identity, canonical or source field identity, review priority, and value evidence. It excludes run ID, completion time, software-runtime provenance, input byte provenance, source locations, human-readable messages that reproduce variable locations, and report-local finding IDs. -**REQ-OUT-096.** Given the same exact input bytes, input basenames, configuration document, toolkit version, and runtime identity, all canonical result content other than `run.id` and `run.completed_at` MUST be identical. +**REQ-OUT-096.** Given the same exact input bytes, input basenames, configuration selection, toolkit version, and runtime identity, all canonical result content other than `run.id` and `run.completed_at` MUST be identical. Repeated automatic-mode runs therefore MUST generate byte-identical effective configuration documents. **REQ-OUT-097.** Changing only input record order or accepted LF/CRLF line endings MUST leave the comparison outcome unchanged. Exact-byte input size and digest, source locations, location-dependent messages, and report-local finding IDs MAY change. The result for each particular ordered byte input MUST still satisfy all deterministic ordering rules. -**REQ-OUT-098.** Across supported Python versions and operating systems, the comparison outcome MUST remain invariant for identical inputs and configuration. Recorded runtime provenance MAY differ. Release testing MUST verify that Unicode-data differences do not silently change comparison outcomes across the supported matrix. +**REQ-OUT-098.** Across supported Python versions and operating systems, the comparison outcome MUST remain invariant for identical inputs and configuration selection. Recorded runtime provenance MAY differ. Release testing MUST verify that Unicode-data differences do not silently change comparison outcomes across the supported matrix. **REQ-OUT-013.** The `phases` object MUST contain `contract`, `records`, and `values`, each with status `complete` or `incomplete`. @@ -1145,7 +1233,7 @@ Basenames form part of the potentially sensitive evidence covered by Section 11. **REQ-OUT-021.** `record_key_fields` MUST declare the ordered source key-field names for the reference and query and the ordered canonical key-field names. Findings identify records through corresponding ordered arrays of source lexical values and canonical textual values. Canonical key components MUST use the scalar rendering rules in `REQ-OUT-029`. A single-field key is represented by a one-element array. Each identity array MUST have the same length as its corresponding field-name array. The reference or query identity is JSON null when that record is absent on that side; the canonical identity is always present. -**REQ-OUT-022.** `comparison_configuration.document` MUST contain the complete validated JSON-compatible representation of the comparison configuration, including the bundled configuration when no custom file was supplied. +**REQ-OUT-022.** `comparison_configuration.document` MUST contain the complete validated JSON-compatible effective comparison configuration. In automatic mode, this MUST be the generated document defined in Section 6.9. **REQ-OUT-023.** The embedded document MUST validate against the referenced comparison-configuration schema. @@ -1174,7 +1262,6 @@ No comparison-configuration digest or command-invocation object is included in v - **REQ-OUT-061.** missing and extra record findings contain no field values; - **REQ-OUT-062.** record-specific missing-required-field findings contain presence states but do not reproduce the present side's value; the dataset-level zero-record form contains no value evidence; -- **REQ-OUT-063.** dataset-level undeclared-field findings contain no field value; - **REQ-OUT-064.** type-mismatch findings contain source values for affected sides and canonical values only where parsing succeeded; - **REQ-OUT-065.** invalid-field-value and mapping-failure findings contain the affected source value and no failed canonical value; - **REQ-OUT-066.** null-policy-violation findings contain the affected source and canonical null representation; @@ -1196,14 +1283,13 @@ The closed version 1 finding-code vocabulary is: - `missing_record`; - `extra_record`; - `missing_required_field`; -- `undeclared_field`; - `type_mismatch`; - `invalid_field_value`; - `null_policy_violation`; - `mapping_failure`; - `value_mismatch`. -**REQ-OUT-034.** `affected_side` MUST be `reference`, `query`, or `both` as required by the finding code. A finding attributable to a declared canonical field MUST contain that field name and its resolved review priority. A dataset-level undeclared-field finding instead contains the source field name and record count. A value mismatch involving a shared undeclared field contains its source field name and no review priority. Record-specific findings MUST contain record identity and source locations; dataset-level findings omit record identity and source locations. +**REQ-OUT-034.** `affected_side` MUST be `reference`, `query`, or `both` as required by the finding code. A finding attributable to a declared canonical field MUST contain that field name and its resolved review priority. Record-specific findings MUST contain record identity and source locations; dataset-level missing-required-field findings omit record identity and source locations. **REQ-OUT-117.** For CSV and TSV, every record-specific source location MUST contain the one-based data-record number, excluding the header, and the one-based physical line number on which the record begins. Embedded newlines in a quoted field do not change that record's starting line. A location for a present field MUST also contain its source field name; a record-level location or a location for an absent field MUST omit the field name. @@ -1217,7 +1303,7 @@ The closed version 1 finding-code vocabulary is: 2. phase: `contract`, `records`, `values`; 3. dataset-level findings before record-specific findings; 4. canonical record identity, compared component-by-component as Unicode strings; -5. field identity, using the canonical field name for a declared field and otherwise the source field name; +5. canonical field name, with no field sorting component for record findings; 6. finding code in the fixed vocabulary order listed in Section 9.1.2; 7. affected side: `reference`, `query`, `both`; 8. source locations, comparing reference before query, with a null location before a present location; tabular locations compare by record number, physical line number, and field name, while JSON locations compare by JSON Pointer using Unicode code-point order; and @@ -1246,15 +1332,20 @@ query = matched + extra - **REQ-OUT-072.** absence on a required side is not assessable; - **REQ-OUT-073.** an invalid or prohibited value that prevents comparison is not assessable. -Under `fields.undeclared: compare`, dataset-wide shared undeclared fields contribute value opportunities in aligned records where at least one side is present: two present values use `preserved_exact`, and one present and one absent value mismatch. A dataset-wide one-sided undeclared field creates one dataset-level `undeclared_field` finding and is not assessable in each aligned record containing it. Under `exclude`, undeclared fields are not value opportunities. Under `fail`, each undeclared field creates the dataset-level findings required by Section 6.2.5 and is not assessable in each aligned record where it occurs. Occurrences in missing or extra records do not contribute value opportunities. +Fields absent from `fields.in_scope`, including observed source fields not represented by any field definition, do not contribute value opportunities. + +**REQ-OUT-040.** `summary.findings.total` MUST equal the number of emitted findings. Counts by classification, by code, and by review priority MUST each sum to the total. Record findings, which are not attributable to a declared non-key field, count as `unassigned` review priority. + +**REQ-OUT-041.** `exclusions` MUST always be an array containing exactly these two forms: -**REQ-OUT-040.** `summary.findings.total` MUST equal the number of emitted findings. Counts by classification, by code, and by review priority MUST each sum to the total. Findings not attributable to a declared field, including record and undeclared-field findings, count as `unassigned` review priority. +- one `configured_field_excluded` entry for every canonical field in `fields.definitions` but absent from `fields.in_scope`; each such entry contains only `code` and `canonical_field`; +- one `undeclared_field_excluded` entry for every observed source field not represented by a field definition on that dataset side; each such entry contains `code`, `side`, `source_field`, and the number of records containing the field. -**REQ-OUT-041.** `exclusions` MUST always be an array. Under `fields.undeclared: exclude`, it contains one entry per source field and dataset side, recording code `undeclared_field_excluded`, side, source field name, and the number of records containing the field. Entries MUST be ordered first by side (`reference`, then `query`) and then by source field name using Unicode code-point order. Exclusions are not findings, do not have review priorities, and do not contribute to value counts. A column known from a zero-record CSV or TSV header has record count zero. +Configured-field exclusions MUST be ordered first by `canonical_field` using Unicode code-point order. Undeclared-field exclusions follow, ordered first by side (`reference`, then `query`) and then by `source_field` using Unicode code-point order. Exclusions are not findings, do not have review priorities, and do not contribute to value counts. A source field known only from a zero-record CSV or TSV header has record count zero. #### 9.1.4 Structural and semantic result validation -**REQ-OUT-100.** The results schema MUST define each finding form as an exact alternative. For each finding code it MUST require the applicable classification and properties and reject properties that are not permitted for that form, including the declared-field and shared-undeclared-field forms of `value_mismatch` and the record-specific and dataset-level forms of `missing_required_field`. +**REQ-OUT-100.** The results schema MUST define each finding form as an exact alternative. For each finding code it MUST require the applicable classification and properties and reject properties that are not permitted for that form, including the record-specific and dataset-level forms of `missing_required_field`. **REQ-OUT-101.** Schema validation alone is not sufficient to establish that a canonical result is internally coherent. The implementation MUST also apply semantic result validation. @@ -1270,7 +1361,7 @@ Under `fields.undeclared: compare`, dataset-wide shared undeclared fields contri **REQ-OUT-107.** Semantic result validation MUST verify that each finding's classification, presence states, value evidence, locations, affected side, field identity, and review priority are consistent with its code, form, embedded configuration, and causal condition. -**REQ-OUT-108.** Semantic result validation MUST verify that `record_key_fields`, finding field identities, review priorities, and other configuration-derived metadata agree with the embedded comparison configuration. +**REQ-OUT-108.** Semantic result validation MUST verify that `record_key_fields`, finding field identities, review priorities, configured-field exclusions, and other configuration-derived metadata agree with the embedded comparison configuration. It MUST also verify that undeclared-field exclusions agree with the source fields observed on each dataset side. **REQ-OUT-109.** Before publishing an evidence package, the tool MUST validate the generated canonical result against the results schema and every semantic result rule. @@ -1340,7 +1431,7 @@ adviso-validation compare \ **REQ-CLI-009.** `--configuration` MUST be optional. -**REQ-CLI-010.** Omission of `--configuration` MUST select the bundled strict-exact configuration. When `--configuration` is present, its value MUST be treated as the path to a custom comparison-configuration file. +**REQ-CLI-010.** Omission of `--configuration` MUST select automatic strict-exact mode. When `--configuration` is present, its value MUST be treated only as the path to a custom comparison-configuration file. **REQ-CLI-011.** `--output` MUST name the evidence-package directory described in Section 9. @@ -1471,16 +1562,16 @@ In operational use, each laboratory is responsible for creating, approving, main - **REQ-DOC-006.** installation and verification of the installed executable; - **REQ-DOC-007.** the complete command-line interface, including inputs, outputs, defaults, exit codes, overwrite behaviour, and worked invocations; - **REQ-DOC-008.** the supported input formats and their parsing, encoding, record, field, and filename-extension requirements; -- **REQ-DOC-009.** how to use the bundled strict-exact comparison configuration and how to create, validate, and govern a custom comparison configuration; +- **REQ-DOC-009.** how automatic strict-exact mode selects a key, generates its effective configuration, compares shared fields, handles exclusions, and can fail when key selection is impossible or ambiguous, together with how to create, validate, and govern a custom comparison configuration; - **REQ-DOC-010.** every property in the comparison-configuration schema, including its purpose, permitted values, required or optional status, default or omission semantics, constraints, and interactions with other properties; -- **REQ-DOC-011.** the translation-to-canonical comparison model, finding classifications, comparison phases, processing order, comparators, mappings, normalisation operations, null handling, undeclared-field behaviour, and review priorities; +- **REQ-DOC-011.** the translation-to-canonical comparison model, field-scope and exclusion behaviour, finding classifications, comparison phases, processing order, comparators, mappings, normalisation operations, null handling, and review priorities; - **REQ-DOC-012.** the contents and interpretation of the evidence package; - **REQ-DOC-013.** every property in the canonical results JSON schema, including finding classifications and codes, typed source and canonical values, counts, exclusions, run and phase statuses, and provenance fields; the guide MUST explain that software consuming a result must first validate `results_schema_version` and must use structured properties, classifications, and finding codes rather than parsing presentation messages; - **REQ-DOC-014.** execution-error diagnostics and common corrective actions; - **REQ-DOC-015.** the tool's intended use, non-claims, security and data-governance boundaries, and known limitations; and - **REQ-DOC-016.** the toolkit, comparison-configuration-schema, and results-schema versions to which the documentation applies. -**REQ-DOC-002.** The guide MUST include complete, runnable examples covering the bundled configuration and representative custom configuration features. +**REQ-DOC-002.** The guide MUST include complete, runnable examples covering automatic strict-exact mode, including automatic key discovery, and representative custom configuration features. **REQ-DOC-003.** Configuration examples MUST validate against the published comparison-configuration schema. Results examples MUST validate against the published results schema. @@ -1516,27 +1607,27 @@ This catalogue summarises minimum executable scenarios for the first release. It - **AT-CON-001:** Identical typed datasets satisfy their contract. - Verifies: `REQ-VAL-003`, `REQ-VAL-004`, `REQ-CMP-006`, `REQ-RUN-002` -- **AT-CON-002:** Required missing fields produce side-specific contract findings in both directions and across heterogeneous records; optional missing fields do not. Dataset-wide field discovery uses tabular headers and the union of JSON record members. Observed undeclared fields are handled according to each `fields.undeclared` mode and aligned only by identical input names under `compare`: shared names use implicit `preserved_exact` comparison with present/absent mismatches, while dataset-wide one-sided names and `fail` findings are consolidated by side and field with accurate record counts. This consolidated parameterized scenario also covers: `fields.undeclared` is required and accepts only `compare`, `exclude`, or `fail`; omission and unknown policies are rejected. Every declared field supplies non-empty `source.reference` and `source.query` names, including when the field is optional on either side; the `required` flags govern record-level presence rather than whether a correspondence is declared. - - Verifies: `REQ-CFG-030`, `REQ-CFG-031`, `REQ-CFG-037`, `REQ-CFG-038`, `REQ-CFG-039`, `REQ-CFG-116`, `REQ-CFG-117`, `REQ-CFG-118`, `REQ-CFG-119`, `REQ-CFG-120`, `REQ-CFG-121`, `REQ-CMP-006`, `REQ-CMP-007`, `REQ-CMP-034`, `REQ-CMP-035`, `REQ-CMP-036`, `REQ-FND-010`, `REQ-FND-011`, `REQ-FND-021`, `REQ-VAL-002`, `REQ-CFG-035`, `REQ-CFG-036` -- **AT-CON-004:** Null, empty string, whitespace-only string, absent field, and configured non-null missing tokens remain distinct according to policy. Parser-native null and recognised missing tokens bypass mapping and typed parsing; mapping to null applies the null policy and bypasses typed parsing; and null is rejected as a missing token or mapping source. This consolidated parameterized scenario also covers: The bundled strict-exact configuration treats parser-native null as equal only to null and distinguishes null, absent fields, and empty strings. - - Verifies: `REQ-CFG-053`, `REQ-CFG-054`, `REQ-CFG-055`, `REQ-CFG-056`, `REQ-CFG-057`, `REQ-CFG-122`, `REQ-VAL-014`, `REQ-VAL-047`, `REQ-VAL-048`, `REQ-VAL-049`, `REQ-VAL-050`, `REQ-VAL-051`, `REQ-VAL-052`, `REQ-VAL-053`, `REQ-VAL-054`, `REQ-VAL-055`, `REQ-VAL-086`, `REQ-INP-028`, `REQ-CFG-113` -- **AT-CON-005:** Unknown, conflicting, and incomplete value mappings fail clearly; mapping to null differs from a missing map entry, and mapping from null is rejected. This consolidated parameterized scenario also covers: Source-value descriptors distinguish strings, numbers, booleans, and null in their permitted positions; mapping to null remains distinct from an absent mapping entry. Parameterized mapping fixtures verify reference/query-to-canonical field mapping, required-field reporting, undeclared-field handling, declared source and destination types, permitted intentional many-to-one value mappings, and rejection of reused source column names or any other non-one-to-one column mapping within either dataset side. +- **AT-CON-002:** Required missing fields produce side-specific contract findings in both directions and across heterogeneous records; optional missing fields do not. Dataset-wide field discovery uses tabular headers and the union of JSON record members. `fields.in_scope` is required, non-empty, unique, and contains only defined canonical fields; list order does not affect behaviour. Defined fields omitted from it and observed source fields not represented by a definition produce the two exact exclusion forms, never findings or value opportunities. Undeclared exclusions are consolidated by side and source field with accurate record counts. Every declared field supplies non-empty `source_fields.reference` and `source_fields.query` names, including when the field is optional on either side; the `required` flags govern record-level presence rather than whether a correspondence is declared. + - Verifies: `REQ-CFG-030`, `REQ-CFG-031`, `REQ-CFG-037`, `REQ-CFG-038`, `REQ-CFG-039`, `REQ-CFG-116`, `REQ-CFG-117`, `REQ-CFG-118`, `REQ-CFG-119`, `REQ-CFG-120`, `REQ-CFG-121`, `REQ-CMP-006`, `REQ-CMP-007`, `REQ-CMP-034`, `REQ-CMP-035`, `REQ-CMP-036`, `REQ-FND-010`, `REQ-VAL-002`, `REQ-CFG-035`, `REQ-CFG-036`, `REQ-OUT-041` +- **AT-CON-004:** Null, empty string, whitespace-only string, absent field, and configured non-null missing tokens remain distinct according to policy. Parser-native null and recognised missing tokens bypass mapping and typed parsing; mapping to null applies the null policy and bypasses typed parsing; and null is rejected as a missing token or mapping source. + - Verifies: `REQ-CFG-053`, `REQ-CFG-054`, `REQ-CFG-055`, `REQ-CFG-056`, `REQ-CFG-057`, `REQ-CFG-122`, `REQ-VAL-014`, `REQ-VAL-047`, `REQ-VAL-048`, `REQ-VAL-049`, `REQ-VAL-050`, `REQ-VAL-051`, `REQ-VAL-052`, `REQ-VAL-053`, `REQ-VAL-054`, `REQ-VAL-055`, `REQ-VAL-086`, `REQ-INP-028` +- **AT-CON-005:** Unknown, conflicting, and incomplete value mappings fail clearly; mapping to null differs from a missing map entry, and mapping from null is rejected. This consolidated parameterized scenario also covers: Source-value descriptors distinguish strings, numbers, booleans, and null in their permitted positions; mapping to null remains distinct from an absent mapping entry. Parameterized mapping fixtures verify reference/query-to-canonical field mapping, required-field reporting, exclusion of unknown source fields, declared source and destination types, permitted intentional many-to-one value mappings, and rejection of reused source column names or any other non-one-to-one column mapping within either dataset side. - Verifies: `REQ-CFG-065`, `REQ-CFG-071`, `REQ-CFG-072`, `REQ-CFG-073`, `REQ-CFG-074`, `REQ-CFG-127`, `REQ-VAL-058`, `REQ-VAL-065`, `REQ-VAL-067`, `REQ-CFG-058`, `REQ-CFG-059`, `REQ-VAL-068`, `REQ-VAL-056`, `REQ-VAL-057`, `REQ-VAL-059`, `REQ-VAL-060`, `REQ-VAL-064`, `REQ-VAL-066` - **AT-CON-006:** Malformed syntax or structure, non-array JSON roots (including a single record object), JSON arrays containing non-object items, duplicate JSON member names at any depth, empty or duplicate tabular column names, and irregular tabular rows receive their specified error classification without silent data loss. This consolidated parameterized scenario also covers: A table-driven parser suite verifies the fixed CSV and TSV dialects, UTF-8-only decoding, CRLF/LF equivalence, quoted delimiters, doubled quotes, embedded line breaks, retention of one-based data-record and starting physical line numbers, and initial string preservation. - Verifies: `REQ-INP-021`, `REQ-INP-022`, `REQ-INP-023`, `REQ-INP-024`, `REQ-INP-025`, `REQ-INP-034`, `REQ-INP-035`, `REQ-INP-036`, `REQ-INP-037`, `REQ-INP-038`, `REQ-INP-070`, `REQ-CMP-028`, `REQ-CMP-029`, `REQ-PRN-002`, `REQ-PRN-008`, `REQ-INP-010`, `REQ-INP-011`, `REQ-INP-012`, `REQ-INP-014`, `REQ-INP-015`, `REQ-INP-016`, `REQ-INP-017`, `REQ-INP-018`, `REQ-INP-019`, `REQ-INP-020`, `REQ-INP-026`, `REQ-INP-027`, `REQ-INP-063`, `REQ-INP-064` -- **AT-CON-007:** JSON `[]` and header-only CSV/TSV are valid zero-record datasets, while zero-byte inputs receive their format-specific input errors. For two valid zero-record datasets, the records and values phases are complete with no record differences or value opportunities. Per-record requirements produce neither findings nor not-assessable classifications; available tabular header structure still produces applicable dataset-level contract findings, while JSON `[]` exposes no field structure. One-sided zero-record datasets produce all applicable missing or extra record findings, zero value opportunities, and complete records and values phases. - - Verifies: `REQ-INP-052`, `REQ-INP-053`, `REQ-INP-054`, `REQ-INP-055`, `REQ-INP-056`, `REQ-INP-057`, `REQ-INP-058`, `REQ-INP-059`, `REQ-INP-066`, `REQ-INP-067`, `REQ-INP-068`, `REQ-INP-069`, `REQ-CMP-009`, `REQ-CMP-010`, `REQ-CFG-114`, `REQ-FND-022` +- **AT-CON-007:** JSON `[]` and header-only CSV/TSV are valid zero-record datasets, while zero-byte inputs receive their format-specific input errors. With a custom configuration, two valid zero-record datasets have complete records and values phases with no record differences or value opportunities. In automatic mode, two header-only tabular datasets containing `record_id` behave likewise, while two empty JSON arrays produce `input.invalid_record_key` because they expose no selectable key. Per-record requirements produce neither findings nor not-assessable classifications; available tabular header structure still produces applicable dataset-level contract findings. A one-sided zero-record comparison reports every record as missing or extra when a valid key is available and otherwise receives the specified automatic-key error. + - Verifies: `REQ-INP-052`, `REQ-INP-053`, `REQ-INP-054`, `REQ-INP-055`, `REQ-INP-056`, `REQ-INP-057`, `REQ-INP-058`, `REQ-INP-059`, `REQ-INP-066`, `REQ-INP-067`, `REQ-INP-068`, `REQ-INP-069`, `REQ-INP-071`, `REQ-CMP-009`, `REQ-CMP-010`, `REQ-FND-022` - **AT-CON-008:** Comparison-configuration YAML scalar resolution and cross-format typed dataset parsing are deterministic across supported environments. - Verifies: `REQ-CFG-006`, `REQ-INP-039`, `REQ-INP-040`, `REQ-INP-041`, `REQ-INP-042`, `REQ-INP-043`, `REQ-INP-044`, `REQ-DEP-013` -- **AT-CON-011:** A complete custom YAML configuration conforming to Section 6.2 validates; omitted optional processing properties and dataset sides expand to empty internal structures in the effective configuration, while explicitly empty placeholder lists or mappings are rejected. - - Verifies: `REQ-CFG-003`, `REQ-CFG-005`, `REQ-CFG-014`, `REQ-CFG-060`, `REQ-CFG-061`, `REQ-CFG-062`, `REQ-CFG-063`, `REQ-CFG-069`, `REQ-CFG-070`, `REQ-CFG-075`, `REQ-CFG-076` -- **AT-CON-012:** Duplicate or unknown YAML keys, unsafe YAML features, unsupported schema versions, invalid enumerations, and incompatible type/comparator combinations are rejected with precise configuration paths. This consolidated parameterized scenario also covers: Safe-loader tests confirm that configuration parsing cannot construct arbitrary objects or execute tagged payloads. - - Verifies: `REQ-CFG-007`, `REQ-CFG-008`, `REQ-CFG-009`, `REQ-CFG-010`, `REQ-CFG-013`, `REQ-CFG-017`, `REQ-CFG-048`, `REQ-CFG-051`, `REQ-CFG-052`, `REQ-CFG-065`, `REQ-CFG-078`, `REQ-CFG-079`, `REQ-CFG-080`, `REQ-CFG-085`, `REQ-CFG-091`, `REQ-SEC-012` +- **AT-CON-011:** A complete custom YAML configuration conforming to Section 6.2 validates; omitted optional processing properties, collection representations, and dataset sides expand to their specified default or empty internal structures, while explicitly empty placeholder lists or mappings are rejected. + - Verifies: `REQ-CFG-003`, `REQ-CFG-005`, `REQ-CFG-014`, `REQ-CFG-060`, `REQ-CFG-061`, `REQ-CFG-062`, `REQ-CFG-063`, `REQ-CFG-069`, `REQ-CFG-070`, `REQ-CFG-075`, `REQ-CFG-076`, `REQ-CFG-137`, `REQ-CFG-138`, `REQ-CFG-139`, `REQ-CFG-140`, `REQ-CFG-141`, `REQ-CFG-143` +- **AT-CON-012:** Duplicate or unknown YAML keys, unsafe YAML features, unsupported schema versions, invalid enumerations, incompatible type/comparator combinations, and collection fields using field-level processing properties are rejected with precise configuration paths. This consolidated parameterized scenario also covers: Safe-loader tests confirm that configuration parsing cannot construct arbitrary objects or execute tagged payloads. + - Verifies: `REQ-CFG-007`, `REQ-CFG-008`, `REQ-CFG-009`, `REQ-CFG-010`, `REQ-CFG-013`, `REQ-CFG-017`, `REQ-CFG-048`, `REQ-CFG-051`, `REQ-CFG-052`, `REQ-CFG-065`, `REQ-CFG-078`, `REQ-CFG-079`, `REQ-CFG-080`, `REQ-CFG-085`, `REQ-CFG-091`, `REQ-CFG-145`, `REQ-SEC-012` - **AT-CON-014:** Each authoritative schema has its specified canonical `$id`, is present in the source tree and built distribution, can be loaded through the installed package-resource API, and validates every complete configuration example and controlled configuration fixture. This consolidated parameterized scenario also covers: The packaged results schema resolves its configuration-schema reference, accepts representative complete and incomplete canonical results, rejects unknown properties and execution-error diagnostics, requires exactly the documented top-level result properties, and enforces the exact permitted shape of every finding form. - Verifies: `REQ-CFG-128`, `REQ-OUT-004`, `REQ-OUT-005`, `REQ-OUT-100`, `REQ-OUT-112` - Verifies: `REQ-CFG-016`, `REQ-CFG-087`, `REQ-CFG-088`, `REQ-OUT-004`, `REQ-OUT-005`, `REQ-OUT-006`, `REQ-OUT-007`, `REQ-OUT-050`, `REQ-OUT-051`, `REQ-OUT-052`, `REQ-OUT-053`, `REQ-OUT-054`, `REQ-OUT-055`, `REQ-OUT-056`, `REQ-OUT-057`, `REQ-OUT-058`, `REQ-OUT-059`, `REQ-OUT-060`, `REQ-OUT-100` -- **AT-CON-015:** Configurations that are structurally schema-valid but violate a semantic rule in Section 6.2.9, including reuse of a source field name within one dataset side, are rejected with the relevant configuration path and rule. - - Verifies: `REQ-CFG-090`, `REQ-CFG-091`, `REQ-CFG-104`, `REQ-CFG-105`, `REQ-CFG-106`, `REQ-CFG-107`, `REQ-CFG-108`, `REQ-CFG-127` +- **AT-CON-015:** Configurations that are structurally schema-valid but violate a semantic rule in Section 6.2.9, including an in-scope or key name absent from the appropriate list or definitions, reuse of a source field name within one dataset side, invalid collection-representation placement or element mappings, and use of the reserved automatic-mode identifier by a custom configuration, are rejected with the relevant configuration path and rule. + - Verifies: `REQ-CFG-090`, `REQ-CFG-091`, `REQ-CFG-104`, `REQ-CFG-105`, `REQ-CFG-106`, `REQ-CFG-107`, `REQ-CFG-108`, `REQ-CFG-127`, `REQ-CFG-129`, `REQ-CFG-136`, `REQ-CFG-144` - **AT-CON-016:** `review_priority` accepts only `low`, `normal`, or `high`; omission resolves to `normal` in the effective configuration. - Verifies: `REQ-CFG-040` - **AT-CON-017:** `fields.record_key` is required, non-empty, ordered, and free of duplicate component fields; every entry names a field in `fields.definitions` that satisfies all key-field constraints. @@ -1545,12 +1636,12 @@ This catalogue summarises minimum executable scenarios for the first release. It - Verifies: `REQ-CFG-018`, `REQ-CFG-020`, `REQ-CFG-022` - **AT-CON-021:** A table-driven JSON parser suite verifies the record-oriented root structure, preserved string-like tokens and numerical lexical forms, and rejection of unquoted `NaN`, `Infinity`, and `-Infinity` as malformed JSON in every location. Results-schema fixtures accept only valid JSON integer lexical forms for `integer` source evidence and valid JSON numerical forms containing a fraction or exponent for `decimal` source evidence. Quoted non-finite representations remain strings; when explicitly parsed as decimals they produce an `invalid_field_value` and not-assessable opportunity in an attributable non-key field, but an execution error in a record-key field. - Verifies: `REQ-INP-029`, `REQ-INP-030`, `REQ-INP-033`, `REQ-INP-047`, `REQ-INP-048`, `REQ-INP-049`, `REQ-INP-050`, `REQ-INP-051`, `REQ-INP-065` -- **AT-CON-022:** Every comparison resolves one effective configuration whose schema version, identifier, revision, record key, field definitions, undeclared-field policy, and comparison rules are validated before either dataset is read; unknown keys fail with actionable locations, and the validated configuration is embedded in results. +- **AT-CON-022:** Every comparison resolves one effective configuration whose schema version, identifier, revision, record key, in-scope fields, field definitions, and comparison rules are validated and embedded in results. A supplied custom configuration is validated before either dataset is read; an automatically generated configuration is constructed after input parsing and field discovery but validated before comparison begins. Unknown custom keys fail with actionable locations. - Verifies: `REQ-CFG-001`, `REQ-CFG-002`, `REQ-CFG-012`, `REQ-CFG-086`, `REQ-CFG-095`, `REQ-CFG-096`, `REQ-CFG-097`, `REQ-CFG-098`, `REQ-CFG-099`, `REQ-CFG-100`, `REQ-CFG-101`, `REQ-CFG-102`, `REQ-CFG-103` -- **AT-CON-023:** Parameterized configuration fixtures verify canonical field-name mapping, collection logical-type structure, the bidirectional relationship between omission of `logical_type` and `preserved_exact`, rejection of every processing property on a `preserved_exact` field, legal omission of operation parameters, placement and side-specific application of source and typed normalisation, and the required comparator declaration. +- **AT-CON-023:** Parameterized configuration fixtures verify canonical field-name mapping, collection logical-type structure, the bidirectional relationship between omission of `logical_type` and `preserved_exact`, rejection of every processing property on a `preserved_exact` field, legal omission of operation parameters, placement and side-specific application of source and typed normalisation, and the required comparator declaration. They use the public names `source_fields`, `null_policy`, and `logical_type.collection_type`. - Verifies: `REQ-CFG-028`, `REQ-CFG-032`, `REQ-CFG-033`, `REQ-CFG-049`, `REQ-CFG-050`, `REQ-CFG-064`, `REQ-CFG-066`, `REQ-CFG-067`, `REQ-CFG-077` -- **AT-CON-024:** The bundled configuration is schema-valid and implements the documented strict-exact behavior: `record_id` is the key on both sides, shared non-key fields require no declarations, reference-only and query-only fields become contract findings, and cross-format parser-native category differences remain distinguishable as `value_mismatch` findings rather than `type_mismatch` findings. This consolidated parameterized scenario also covers: `preserved_exact` follows the parser-native category table, including deterministic same-format and cross-format comparisons. - - Verifies: `REQ-CFG-093`, `REQ-CFG-094`, `REQ-CFG-109`, `REQ-CFG-110`, `REQ-CFG-115`, `REQ-INP-039`, `REQ-INP-046`, `REQ-CFG-111`, `REQ-CFG-112`, `REQ-CMP-032`, `REQ-FND-018` +- **AT-CON-024:** Automatic strict-exact fixtures verify exact dataset-wide field discovery and deterministic generation of a schema-valid embedded effective configuration. `record_id` is preferred when present on both sides and, if invalid, produces `input.invalid_record_key` without fallback. Without a shared `record_id`, exactly one common, complete, non-null, unique string field with at least one shared value is selected; zero candidates and multiple candidates produce actionable key errors, the latter listing candidate field names but no values. Tests reject numeric, null, missing, duplicate, heuristic, positional, ranked, and composite automatic keys. Every same-name non-key field is compared with `preserved_exact`; one-sided fields are exclusions; and no mapping, normalisation, coercion, tolerance, or type inference occurs. The HTML report presents the required automatic-mode warning. + - Verifies: `REQ-PRN-001`, `REQ-CFG-092`, `REQ-CFG-093`, `REQ-CFG-094`, `REQ-CFG-109`, `REQ-CFG-110`, `REQ-CFG-111`, `REQ-CFG-112`, `REQ-CFG-113`, `REQ-CFG-114`, `REQ-CFG-115`, `REQ-CFG-130`, `REQ-CFG-131`, `REQ-CFG-132`, `REQ-CFG-133`, `REQ-CFG-134`, `REQ-CFG-135`, `REQ-CFG-136`, `REQ-REC-006`, `REQ-REC-007`, `REQ-REC-008`, `REQ-REC-010` ### 14.2 Record scenarios @@ -1560,7 +1651,7 @@ This catalogue summarises minimum executable scenarios for the first release. It - Verifies: `REQ-REC-003`, `REQ-REC-009`, `REQ-REC-010`, `REQ-REC-011`, `REQ-REC-012`, `REQ-REC-013`, `REQ-REC-014` - **AT-REC-003:** Composite keys preserve multiple legitimate records belonging to one sample. - Verifies: `REQ-REC-001`, `REQ-REC-002`, `REQ-CFG-024` -- **AT-REC-005:** Parameterized key fixtures verify that key names, aliases, composites, and normalisation are configuration-defined; key values are not implicitly converted; no command-line key override exists; and findings retain enough source identity to distinguish records. +- **AT-REC-005:** Parameterized key fixtures verify that custom key names, aliases, composites, and normalisation are configuration-defined; the sole automatic-key exception follows Section 6.9; key values are not implicitly converted; no command-line key override exists; and findings retain enough source identity to distinguish records. - Verifies: `REQ-REC-004`, `REQ-REC-005`, `REQ-REC-006`, `REQ-REC-007`, `REQ-REC-008`, `REQ-REC-015` ### 14.3 Typed value and comparator scenarios @@ -1577,15 +1668,15 @@ This catalogue summarises minimum executable scenarios for the first release. It - Verifies: `REQ-VAL-024`, `REQ-VAL-025`, `REQ-VAL-026`, `REQ-VAL-027` - **AT-VAL-006:** Dates reject invalid or non-ISO full dates; datetimes require offsets and compare as instants. - Verifies: `REQ-VAL-028`, `REQ-VAL-029`, `REQ-VAL-030`, `REQ-VAL-031`, `REQ-VAL-032`, `REQ-VAL-033`, `REQ-VAL-034`, `REQ-VAL-035`, `REQ-VAL-036`, `REQ-CMP-024` -- **AT-VAL-007:** Lists preserve order and duplicates, sets reject typed-canonical duplicates and ignore order, and multisets ignore order while preserving occurrence counts. List elements support positional typed-exact comparison and numeric tolerance for numeric element types; sets and multisets accept only typed-exact element comparison, and every other element-comparator combination is rejected. This consolidated parameterized scenario also covers: Collection parsing accepts JSON arrays in JSON and JSON-array cell syntax in CSV/TSV, rejects ordinary delimited text, and rejects null collection elements with the specified finding and `not_assessable` evidence. The suite covers every supported typed collection kind and its required element declaration. - - Verifies: `REQ-CFG-124`, `REQ-CFG-125`, `REQ-CFG-126`, `REQ-VAL-037`, `REQ-VAL-038`, `REQ-VAL-039`, `REQ-VAL-040`, `REQ-VAL-041`, `REQ-VAL-042`, `REQ-CMP-025`, `REQ-CMP-040`, `REQ-CMP-041`, `REQ-VAL-043`, `REQ-VAL-044`, `REQ-VAL-045`, `REQ-VAL-046`, `REQ-VAL-006` +- **AT-VAL-007:** Lists preserve order and duplicates, sets reject typed-canonical duplicates and ignore order, and multisets ignore order while preserving occurrence counts. List elements support positional typed-exact comparison and numeric tolerance for numeric element types; sets and multisets accept only typed-exact element comparison, and every other element-comparator combination is rejected. This consolidated parameterized scenario also covers: the default and explicit `json_array` representations accept JSON arrays in JSON and JSON-array cell syntax in CSV/TSV; ordinary delimited text is rejected unless `delimited_text` is explicitly configured for that side; literal single- and multi-character delimiters, empty input, empty elements, lack of collection-level escaping, element normalisation, closed element mapping, and element typed parsing follow the defined order; and null collection elements produce the specified finding and `not_assessable` evidence. The suite covers every supported typed collection kind and its required element declaration. + - Verifies: `REQ-CFG-124`, `REQ-CFG-125`, `REQ-CFG-126`, `REQ-CFG-137`, `REQ-CFG-138`, `REQ-CFG-139`, `REQ-CFG-140`, `REQ-CFG-141`, `REQ-CFG-142`, `REQ-VAL-037`, `REQ-VAL-038`, `REQ-VAL-039`, `REQ-VAL-040`, `REQ-VAL-041`, `REQ-VAL-042`, `REQ-CMP-025`, `REQ-CMP-040`, `REQ-CMP-041`, `REQ-VAL-043`, `REQ-VAL-044`, `REQ-VAL-045`, `REQ-VAL-046`, `REQ-VAL-006`, `REQ-VAL-081`, `REQ-VAL-082`, `REQ-VAL-087`, `REQ-VAL-088`, `REQ-VAL-089`, `REQ-VAL-090`, `REQ-VAL-091` - **AT-VAL-008:** Typed transformations follow Section 6.8 and retain evidence of the attributable source value. This consolidated parameterized scenario also covers: Parameterized normalisation fixtures exercise every registered source-text and typed-value operation, verify that source-text operations affect strings but pass non-strings through unchanged, and cover Unicode normalization, whitespace trimming, case folding, decimal quantisation, and datetime truncation. Rounding or precision reduction occurs only when explicitly configured. Deterministic set and multiset output ordering is verified as canonical serialization rather than configured normalisation. The suite also verifies retention of source lexical evidence. - Verifies: `REQ-CFG-123`, `REQ-VAL-061`, `REQ-VAL-062`, `REQ-VAL-063`, `REQ-VAL-069`, `REQ-VAL-070`, `REQ-VAL-072`, `REQ-VAL-073`, `REQ-VAL-074`, `REQ-VAL-075`, `REQ-VAL-071`, `REQ-CMP-023`, `REQ-VAL-010` - **AT-VAL-009:** Numeric tolerance accepts exactly one absolute or relative tolerance and rejects configurations containing neither or both. Each form uses exact decimal arithmetic and an inclusive boundary, with below, at, and above boundary cases; relative tolerance is reference-directed and permits no non-zero difference when the reference is zero. - Verifies: `REQ-CFG-081`, `REQ-CFG-082`, `REQ-CFG-083`, `REQ-CFG-084`, `REQ-CMP-018`, `REQ-CMP-019`, `REQ-CMP-020`, `REQ-CMP-021`, `REQ-CMP-022` - **AT-VAL-010:** All assessable differences are collected in one run. - Verifies: `REQ-CMP-026`, `REQ-CMP-027` -- **AT-VAL-011:** Every defined field is included in comparison; under `fields.undeclared: "exclude"`, undeclared fields are `not_assessed` and represented by the required exclusion summary. +- **AT-VAL-011:** Every field listed in `fields.in_scope` is assessed; defined fields omitted from that list and observed source fields not represented by a definition are `not_assessed` and represented by the required exclusion summaries. - Verifies: `REQ-CFG-039`, `REQ-CMP-004`, `REQ-CMP-013`, `REQ-CMP-031`, `REQ-OUT-041` ### 14.4 Phase and run-status scenarios @@ -1603,7 +1694,7 @@ This catalogue summarises minimum executable scenarios for the first release. It - **AT-OUT-008:** An end-to-end mixed-defect comparison exercises every conceptual processing step, including invalid contract values, deliberate exclusions, failed prerequisites, mandatory key processing, and continued comparison of unaffected values. This consolidated parameterized scenario also covers: A large mixed-content fixture verifies that every configured value opportunity is either assessed or explicitly classified and that no valid accepted content is silently skipped. - Verifies: `REQ-CMP-005`, `REQ-CMP-008`, `REQ-CMP-014`, `REQ-CMP-030`, `REQ-CMP-037`, `REQ-CMP-038`, `REQ-CMP-039`, `REQ-RUN-006`, `REQ-DEP-010` - **AT-OUT-010:** A table-driven finding suite generates every finding code and verifies its fixed classification, structured fields, required contextual evidence, deterministic local ID, human-readable message, summary-only successful matches, and absence of message-only semantics. `missing_record`, `extra_record`, and `value_mismatch` are always `comparison_difference`; every other finding code is always `assessment_issue`; and phase remains independently correct. This consolidated parameterized scenario also covers: For every finding code, result fixtures enforce the code-specific inclusion and omission of source and canonical values, deterministic messages derived from structured data, classification-first total finding order, duplicate coalescence, and sequential post-sort report-local IDs. Category incompatibility, invalid content within an accepted category, and comparison of two valid unequal values produce mutually exclusive `type_mismatch`, `invalid_field_value`, and `value_mismatch` findings. Independent reference and query contract violations produce separate side-specific findings, while comparison of two valid unequal values produces one `both` finding. Each not-assessable value opportunity has one applicable causal finding and does not generate a separate `not_assessable` finding. - - Verifies: `REQ-PRN-006`, `REQ-FND-001`, `REQ-FND-002`, `REQ-FND-003`, `REQ-FND-004`, `REQ-FND-005`, `REQ-FND-006`, `REQ-FND-012`, `REQ-FND-013`, `REQ-FND-014`, `REQ-FND-015`, `REQ-FND-016`, `REQ-FND-017`, `REQ-FND-018`, `REQ-FND-019`, `REQ-FND-020`, `REQ-FND-023`, `REQ-FND-024`, `REQ-FND-025`, `REQ-OUT-032`, `REQ-OUT-033`, `REQ-OUT-034`, `REQ-OUT-035`, `REQ-OUT-099`, `REQ-OUT-061`, `REQ-OUT-062`, `REQ-OUT-063`, `REQ-OUT-064`, `REQ-OUT-065`, `REQ-OUT-066`, `REQ-OUT-067`, `REQ-OUT-068` + - Verifies: `REQ-PRN-006`, `REQ-FND-001`, `REQ-FND-002`, `REQ-FND-003`, `REQ-FND-004`, `REQ-FND-005`, `REQ-FND-006`, `REQ-FND-012`, `REQ-FND-013`, `REQ-FND-014`, `REQ-FND-015`, `REQ-FND-016`, `REQ-FND-017`, `REQ-FND-018`, `REQ-FND-019`, `REQ-FND-020`, `REQ-FND-023`, `REQ-FND-024`, `REQ-FND-025`, `REQ-OUT-032`, `REQ-OUT-033`, `REQ-OUT-034`, `REQ-OUT-035`, `REQ-OUT-099`, `REQ-OUT-061`, `REQ-OUT-062`, `REQ-OUT-064`, `REQ-OUT-065`, `REQ-OUT-066`, `REQ-OUT-067`, `REQ-OUT-068` ### 14.5 Evidence, CLI, and packaging scenarios @@ -1633,13 +1724,13 @@ This catalogue summarises minimum executable scenarios for the first release. It - Verifies: `REQ-OUT-009`, `REQ-OUT-010`, `REQ-OUT-011`, `REQ-OUT-014`, `REQ-OUT-015`, `REQ-OUT-016`, `REQ-OUT-020`, `REQ-OUT-121`, `REQ-OUT-122`, `REQ-OUT-123`, `REQ-OUT-124` - **AT-EVD-016:** Parameterized single and composite record identities verify ordered source and canonical key arrays, null identities for an absent side, presence-state distinctions, recursive parser-native typed values, unique canonical rendering of every scalar type, UTC datetime rendering, canonical key-component rendering, deterministic set and multiset element order, and deterministic object-member ordering. Source-location fixtures verify tabular data-record and starting-line numbers across multiline records, field-name inclusion only for present fields, record-level locations for absent fields, JSON record and field pointers, RFC 6901 escaping, and null locations for absent records. - Verifies: `REQ-OUT-021`, `REQ-OUT-026`, `REQ-OUT-027`, `REQ-OUT-028`, `REQ-OUT-029`, `REQ-OUT-089`, `REQ-OUT-090`, `REQ-OUT-091`, `REQ-OUT-092`, `REQ-OUT-093`, `REQ-OUT-094`, `REQ-OUT-117`, `REQ-OUT-118`, `REQ-OUT-119` -- **AT-EVD-018:** Parameterized result fixtures verify fixed-shape zero-inclusive summaries, record-count reconciliation, finding totals by classification, code, and priority, and exactly one matched, mismatched, or not-assessable classification for every in-scope non-key value opportunity, including optional, required, invalid, and prohibited values. +- **AT-EVD-018:** Parameterized result fixtures verify fixed-shape zero-inclusive summaries, record-count reconciliation, finding totals by classification, code, and priority, and exactly one matched, mismatched, or not-assessable classification for every in-scope non-key value opportunity, including optional, required, invalid, and prohibited values. Semantic checks also verify that configured-field exclusions agree with `fields.in_scope` and that undeclared-field exclusions agree with observed source fields and record counts. - Verifies: `REQ-OUT-037`, `REQ-OUT-038`, `REQ-OUT-039`, `REQ-OUT-040`, `REQ-OUT-069`, `REQ-OUT-070`, `REQ-OUT-071`, `REQ-OUT-072`, `REQ-OUT-073`, `REQ-OUT-101`, `REQ-OUT-102`, `REQ-OUT-103`, `REQ-OUT-104`, `REQ-OUT-105`, `REQ-OUT-106`, `REQ-OUT-107`, `REQ-OUT-108` - **AT-EVD-022:** Before publication, representative generated results pass both structural and semantic self-validation with Draft 2020-12 format assertion enabled. Injected schema and semantic inconsistencies, including malformed UUIDs, dates, and date-times, produce an unexpected internal error and publish no canonical result, HTML report, or evidence-package directory. - Verifies: `REQ-OUT-109`, `REQ-OUT-110`, `REQ-OUT-114` - **AT-EVD-019:** HTML generated solely from canonical JSON uses an open non-proprietary format and presents the run purpose, phase and run statuses, input and configuration identities, counts, findings, discrepancies, limitations, exclusions, unassessed content, provenance, and no acceptance verdict; HTML is never the sole canonical evidence format. Its headline distinguishes run completeness, comparison-difference count, and assessment-issue count. Detailed findings are organized primarily by classification with comparison differences first and phase retained as secondary traceability information. This consolidated parameterized scenario also covers: Complete and incomplete reports contain raw findings and metrics but no acceptance threshold, pass/fail verdict, or equivalent inferred laboratory decision. - Verifies: `REQ-PRN-009`, `REQ-OUT-042`, `REQ-OUT-074`, `REQ-OUT-075`, `REQ-OUT-076`, `REQ-OUT-077`, `REQ-OUT-078`, `REQ-OUT-079`, `REQ-OUT-080`, `REQ-OUT-081`, `REQ-OUT-120`, `REQ-OUT-126`, `REQ-OUT-127`, `REQ-OUT-128`, `REQ-DEP-004`, `REQ-DEP-005`, `REQ-RUN-013` -- **AT-EVD-021:** Installed-CLI tests cover the required `adviso-validation compare` form, required reference, query, and output arguments, optional configuration, enumerated choices, invalid combinations before input access, normal summaries on standard output, and optional progress that never contaminates machine-readable output. This consolidated parameterized scenario also covers: Omitting `--configuration` selects and records the bundled strict-exact configuration; a supplied value is treated only as a custom comparison-configuration path. The reference and query formats are independently determined and recorded from recognised, case-insensitive filename extensions; a missing, unknown, or ambiguous extension causes an input error, and content that does not conform to the indicated format is rejected. Command-line `--help` describes every version 1 command and option and directs users to the full guide. +- **AT-EVD-021:** Installed-CLI tests cover the required `adviso-validation compare` form, required reference, query, and output arguments, optional configuration, enumerated choices, invalid combinations before input access, normal summaries on standard output, and optional progress that never contaminates machine-readable output. This consolidated parameterized scenario also covers: Omitting `--configuration` selects automatic strict-exact mode and records its generated effective configuration; a supplied value is treated only as a custom comparison-configuration path. The reference and query formats are independently determined and recorded from recognised, case-insensitive filename extensions; a missing, unknown, or ambiguous extension causes an input error, and content that does not conform to the indicated format is rejected. Command-line `--help` describes every version 1 command and option and directs users to the full guide. - Verifies: `REQ-PRD-002`, `REQ-INP-001`, `REQ-INP-002`, `REQ-INP-004`, `REQ-INP-060`, `REQ-INP-061`, `REQ-INP-062`, `REQ-CLI-001`, `REQ-CLI-002`, `REQ-CLI-003`, `REQ-CLI-006`, `REQ-CLI-007`, `REQ-CLI-008`, `REQ-INP-003`, `REQ-CFG-092`, `REQ-CLI-009`, `REQ-CLI-010`, `REQ-INP-005`, `REQ-INP-006`, `REQ-INP-007`, `REQ-INP-008`, `REQ-INP-009`, `REQ-CLI-013`, `REQ-CLI-004`, `REQ-DOC-005` ### 14.6 Documentation scenarios diff --git a/src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json b/src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json index 5133346..ec74656 100644 --- a/src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json +++ b/src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json @@ -61,19 +61,15 @@ "additionalProperties": false, "required": [ "record_key", - "undeclared", + "in_scope", "definitions" ], "properties": { "record_key": { "$ref": "#/$defs/uniqueNonEmptyStrings" }, - "undeclared": { - "enum": [ - "compare", - "exclude", - "fail" - ] + "in_scope": { + "$ref": "#/$defs/uniqueNonEmptyStrings" }, "definitions": { "type": "object", @@ -91,13 +87,13 @@ "type": "object", "additionalProperties": false, "required": [ - "source", + "source_fields", "required", - "nulls", + "null_policy", "comparator" ], "properties": { - "source": { + "source_fields": { "type": "object", "additionalProperties": false, "required": [ @@ -132,7 +128,7 @@ "logical_type": { "$ref": "#/$defs/logicalType" }, - "nulls": { + "null_policy": { "$ref": "#/$defs/nullPolicy" }, "source_normalisation": { @@ -156,6 +152,9 @@ } } }, + "collection_representation": { + "$ref": "#/$defs/collectionRepresentation" + }, "value_mapping": { "$ref": "#/$defs/valueMapping" }, @@ -208,6 +207,11 @@ "required": [ "typed_normalisation" ] + }, + { + "required": [ + "collection_representation" + ] } ] } @@ -254,16 +258,71 @@ "typed_normalisation" ] }, + { + "required": [ + "collection_representation" + ] + }, { "properties": { - "nulls": { + "null_policy": { "required": [ "missing_tokens" ] } }, "required": [ - "nulls" + "null_policy" + ] + } + ] + } + } + }, + { + "if": { + "required": [ + "collection_representation" + ] + }, + "then": { + "properties": { + "logical_type": { + "type": "object" + } + }, + "required": [ + "logical_type" + ] + } + }, + { + "if": { + "properties": { + "logical_type": { + "type": "object" + } + }, + "required": [ + "logical_type" + ] + }, + "then": { + "not": { + "anyOf": [ + { + "required": [ + "source_normalisation" + ] + }, + { + "required": [ + "value_mapping" + ] + }, + { + "required": [ + "typed_normalisation" ] } ] @@ -291,12 +350,12 @@ "type": "object", "additionalProperties": false, "required": [ - "name", + "collection_type", "element_type", "element_comparator" ], "properties": { - "name": { + "collection_type": { "enum": [ "list", "set", @@ -328,7 +387,7 @@ { "if": { "properties": { - "name": { + "collection_type": { "enum": [ "set", "multiset" @@ -336,7 +395,7 @@ } }, "required": [ - "name" + "collection_type" ] }, "then": { @@ -371,7 +430,7 @@ }, "then": { "properties": { - "name": { + "collection_type": { "const": "list" }, "element_type": { @@ -625,6 +684,78 @@ } ] }, + "collectionRepresentation": { + "type": "object", + "additionalProperties": false, + "minProperties": 1, + "properties": { + "reference": { + "$ref": "#/$defs/collectionRepresentationSide" + }, + "query": { + "$ref": "#/$defs/collectionRepresentationSide" + } + } + }, + "collectionRepresentationSide": { + "type": "object", + "additionalProperties": false, + "required": [ + "format" + ], + "properties": { + "format": { + "enum": [ + "json_array", + "delimited_text" + ] + }, + "delimiter": { + "type": "string", + "minLength": 1 + }, + "element_normalisation": { + "type": "array", + "minItems": 1, + "items": { + "$ref": "#/$defs/sourceNormalisation" + } + }, + "element_mapping": { + "type": "array", + "minItems": 1, + "items": { + "$ref": "#/$defs/nonNullValueMappingEntry" + } + } + }, + "allOf": [ + { + "if": { + "properties": { + "format": { + "const": "delimited_text" + } + }, + "required": [ + "format" + ] + }, + "then": { + "required": [ + "delimiter" + ] + }, + "else": { + "not": { + "required": [ + "delimiter" + ] + } + } + } + ] + }, "typedNormalisation": { "oneOf": [ { @@ -729,6 +860,22 @@ } } }, + "nonNullValueMappingEntry": { + "type": "object", + "additionalProperties": false, + "required": [ + "from", + "to" + ], + "properties": { + "from": { + "$ref": "#/$defs/nonNullSourceValueDescriptor" + }, + "to": { + "$ref": "#/$defs/nonNullSourceValueDescriptor" + } + } + }, "comparator": { "oneOf": [ { diff --git a/src/adviso_validation/schemas/comparison-results-v1.0.schema.json b/src/adviso_validation/schemas/comparison-results-v1.0.schema.json index 2862624..2f95b8e 100644 --- a/src/adviso_validation/schemas/comparison-results-v1.0.schema.json +++ b/src/adviso_validation/schemas/comparison-results-v1.0.schema.json @@ -320,7 +320,6 @@ "missing_record", "extra_record", "missing_required_field", - "undeclared_field", "type_mismatch", "invalid_field_value", "null_policy_violation", @@ -337,9 +336,6 @@ "missing_required_field": { "$ref": "#/$defs/nonNegativeInteger" }, - "undeclared_field": { - "$ref": "#/$defs/nonNegativeInteger" - }, "type_mismatch": { "$ref": "#/$defs/nonNegativeInteger" }, @@ -442,31 +438,51 @@ } }, "exclusion": { - "type": "object", - "additionalProperties": false, - "required": [ - "code", - "side", - "source_field", - "record_count" - ], - "properties": { - "code": { - "const": "undeclared_field_excluded" - }, - "side": { - "enum": [ - "reference", - "query" - ] - }, - "source_field": { - "$ref": "#/$defs/nonEmptyString" + "oneOf": [ + { + "type": "object", + "additionalProperties": false, + "required": [ + "code", + "canonical_field" + ], + "properties": { + "code": { + "const": "configured_field_excluded" + }, + "canonical_field": { + "$ref": "#/$defs/nonEmptyString" + } + } }, - "record_count": { - "$ref": "#/$defs/nonNegativeInteger" + { + "type": "object", + "additionalProperties": false, + "required": [ + "code", + "side", + "source_field", + "record_count" + ], + "properties": { + "code": { + "const": "undeclared_field_excluded" + }, + "side": { + "enum": [ + "reference", + "query" + ] + }, + "source_field": { + "$ref": "#/$defs/nonEmptyString" + }, + "record_count": { + "$ref": "#/$defs/nonNegativeInteger" + } + } } - } + ] }, "recordIdentityValues": { "type": "array", @@ -1088,8 +1104,6 @@ "not": { "anyOf": [ { "required": ["field"] }, - { "required": ["source_field"] }, - { "required": ["record_count"] }, { "required": ["review_priority"] }, { "required": ["values"] } ] @@ -1114,8 +1128,6 @@ "not": { "anyOf": [ { "required": ["field"] }, - { "required": ["source_field"] }, - { "required": ["record_count"] }, { "required": ["review_priority"] }, { "required": ["values"] } ] @@ -1141,13 +1153,7 @@ "field", "review_priority", "values" - ], - "not": { - "anyOf": [ - { "required": ["source_field"] }, - { "required": ["record_count"] } - ] - } + ] } ] }, @@ -1171,8 +1177,6 @@ "anyOf": [ { "required": ["record_identity"] }, { "required": ["source_locations"] }, - { "required": ["source_field"] }, - { "required": ["record_count"] }, { "required": ["values"] } ] } @@ -1202,41 +1206,7 @@ "field", "review_priority", "values" - ], - "not": { - "anyOf": [ - { "required": ["source_field"] }, - { "required": ["record_count"] } - ] - } - } - ] - }, - { - "allOf": [ - { - "$ref": "#/$defs/findingShape" - }, - { - "properties": { - "code": { - "const": "undeclared_field" - } - }, - "required": [ - "code", - "source_field", - "record_count" - ], - "not": { - "anyOf": [ - { "required": ["record_identity"] }, - { "required": ["source_locations"] }, - { "required": ["field"] }, - { "required": ["review_priority"] }, - { "required": ["values"] } - ] - } + ] } ] }, @@ -1258,41 +1228,7 @@ "field", "review_priority", "values" - ], - "not": { - "anyOf": [ - { "required": ["source_field"] }, - { "required": ["record_count"] } - ] - } - } - ] - }, - { - "allOf": [ - { - "$ref": "#/$defs/findingShape" - }, - { - "properties": { - "code": { - "const": "value_mismatch" - } - }, - "required": [ - "code", - "record_identity", - "source_locations", - "source_field", - "values" - ], - "not": { - "anyOf": [ - { "required": ["field"] }, - { "required": ["record_count"] }, - { "required": ["review_priority"] } - ] - } + ] } ] } @@ -1325,7 +1261,6 @@ "missing_record", "extra_record", "missing_required_field", - "undeclared_field", "type_mismatch", "invalid_field_value", "null_policy_violation", @@ -1359,12 +1294,6 @@ "field": { "$ref": "#/$defs/nonEmptyString" }, - "source_field": { - "$ref": "#/$defs/nonEmptyString" - }, - "record_count": { - "$ref": "#/$defs/nonNegativeInteger" - }, "review_priority": { "enum": [ "low", @@ -1557,38 +1486,6 @@ ] } }, - { - "if": { - "properties": { - "code": { - "const": "undeclared_field" - } - }, - "required": [ - "code" - ] - }, - "then": { - "required": [ - "source_field", - "record_count" - ], - "properties": { - "classification": { - "const": "assessment_issue" - }, - "phase": { - "const": "contract" - }, - "affected_side": { - "enum": [ - "reference", - "query" - ] - } - } - } - }, { "if": { "properties": { @@ -1604,40 +1501,10 @@ "required": [ "record_identity", "source_locations", + "field", + "review_priority", "values" ], - "oneOf": [ - { - "required": [ - "field", - "review_priority" - ], - "not": { - "required": [ - "source_field" - ] - } - }, - { - "required": [ - "source_field" - ], - "not": { - "anyOf": [ - { - "required": [ - "field" - ] - }, - { - "required": [ - "review_priority" - ] - } - ] - } - } - ], "properties": { "classification": { "const": "comparison_difference" From 883e9f0ad2b965bfa2e1df1943c877c282eff31c Mon Sep 17 00:00:00 2001 From: Kevin Howe Date: Fri, 30 Oct 2026 14:30:38 +0000 Subject: [PATCH 12/45] Renamed adviso-validation compare to adviso-compare --- AGENTS.md | 10 +-- README.md | 10 +-- docs/SPECIFICATION.md | 83 +++++++++---------- pyproject.toml | 10 +-- src/adviso_compare/__init__.py | 1 + src/adviso_compare/schemas/__init__.py | 1 + .../comparison-configuration-v1.0.schema.json | 0 .../comparison-results-v1.0.schema.json | 2 +- src/adviso_validation/__init__.py | 1 - src/adviso_validation/schemas/__init__.py | 1 - 10 files changed, 59 insertions(+), 60 deletions(-) create mode 100644 src/adviso_compare/__init__.py create mode 100644 src/adviso_compare/schemas/__init__.py rename src/{adviso_validation => adviso_compare}/schemas/comparison-configuration-v1.0.schema.json (100%) rename src/{adviso_validation => adviso_compare}/schemas/comparison-results-v1.0.schema.json (99%) delete mode 100644 src/adviso_validation/__init__.py delete mode 100644 src/adviso_validation/schemas/__init__.py diff --git a/AGENTS.md b/AGENTS.md index 6452f18..2274341 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,4 +1,4 @@ -# Agent guidance for `adviso-validation` +# Agent guidance for `adviso-compare` ## Purpose of this file @@ -9,8 +9,8 @@ This file governs how agents work in this repository. It does not define product Use this order of authority: 1. `docs/SPECIFICATION.md` — normative behaviour and requirement identifiers. -2. `src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json` — normative comparison-configuration structure. -3. `src/adviso_validation/schemas/comparison-results-v1.0.schema.json` — normative canonical-results structure. +2. `src/adviso_compare/schemas/comparison-configuration-v1.0.schema.json` — normative comparison-configuration structure. +3. `src/adviso_compare/schemas/comparison-results-v1.0.schema.json` — normative canonical-results structure. 4. Version 1 tests and controlled fixtures — executable verification of the normative sources. 5. User documentation — explanatory material that must agree with the sources above. @@ -18,10 +18,10 @@ If these sources appear inconsistent, stop and report the exact conflict and its ## Repository layout -- Put production code under `src/adviso_validation/`. +- Put production code under `src/adviso_compare/`. - Put automated tests under `tests/`. - Put maintained user and developer documentation under `docs/`. -- Keep runtime package resources inside `src/adviso_validation/`. +- Keep runtime package resources inside `src/adviso_compare/`. - Treat generated version files, build outputs, caches, coverage output, and virtual environments as generated artefacts rather than source. Do not create new top-level directories or reorganize established modules without a concrete need in the approved implementation plan. diff --git a/README.md b/README.md index 398cbdf..9dab143 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ -# AdvISO Validation +# AdvISO Compare -`adviso-validation` is a command-line tool for comparing a reference dataset with a query dataset. It uses explicit configuration rules to translate different source representations into a common canonical representation, then reports post-translation differences separately from issues that prevented assessment. +`adviso-compare` is a command-line tool for comparing a reference dataset with a query dataset. It uses explicit configuration rules to translate different source representations into a common canonical representation, then reports post-translation differences separately from issues that prevented assessment. The project is intended to support bioinformatics validation and verification activities associated with ISO 15189 accreditation. It produces comparison evidence; it does not determine whether a pipeline is valid, acceptable, or compliant. @@ -11,7 +11,7 @@ Version 1 is currently specified but not yet implemented. The repository is bein The current normative artefacts are: - [Behaviour specification](docs/SPECIFICATION.md) -- [Comparison-configuration schema](src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json) -- [Canonical-results schema](src/adviso_validation/schemas/comparison-results-v1.0.schema.json) +- [Comparison-configuration schema](src/adviso_compare/schemas/comparison-configuration-v1.0.schema.json) +- [Canonical-results schema](src/adviso_compare/schemas/comparison-results-v1.0.schema.json) -The planned distribution and executable are named `adviso-validation`; the Python import package is `adviso_validation`. +The planned distribution and executable are named `adviso-compare`; the Python import package is `adviso_compare`. diff --git a/docs/SPECIFICATION.md b/docs/SPECIFICATION.md index 2c78022..cfbd356 100644 --- a/docs/SPECIFICATION.md +++ b/docs/SPECIFICATION.md @@ -1,21 +1,21 @@ -# AdvISO Validation Toolkit: `compare` Behaviour Specification +# AdvISO Compare Behaviour Specification **Document status:** Draft for review **Specification version:** 0.1.0-draft -**Applies to:** The `compare` command and its supporting comparison engine +**Applies to:** The `adviso-compare` command and its supporting comparison engine **Normative language:** The terms **MUST**, **MUST NOT**, **SHOULD**, **SHOULD NOT**, and **MAY** indicate requirement strength. ## 1. Purpose -This document defines a specification for version 1.0 of the AdvISO Validation Toolkit: a command-line namespace for cohesive capabilities that support bioinformatics validation and verification activities associated with ISO 15189 accreditation. +This document defines a specification for version 1.0 of AdvISO Compare, a command-line tool that supports bioinformatics validation and verification activities associated with ISO 15189 accreditation. -The initial sole user-facing subcommand within the toolkit is `compare`. It is designed for two datasets that describe the same kinds of samples or observations but may express their fields and values using different names, types, vocabularies, or other source dialects. The declared comparison configuration translates those representations into a common canonical representation, after which the tool compares the canonical records and values. It produces deterministic, machine-readable findings and a human-readable summary suitable for review and retention as one component of a bioinformatics pipeline validation or verification record. +AdvISO Compare is designed for two datasets that describe the same kinds of samples or observations but may express their fields and values using different names, types, vocabularies, or other source dialects. The declared comparison configuration translates those representations into a common canonical representation, after which the tool compares the canonical records and values. It produces deterministic, machine-readable findings and a human-readable summary suitable for review and retention as one component of a bioinformatics pipeline validation or verification record. The tool distinguishes differences that remain after successful translation from issues that prevented some intended translation or assessment. It is intended to answer these questions: > Given these identified inputs and declared translation and comparison rules, which expected and observed records and values are equivalent, different, missing, or extra after translation? Which input or configuration-related issues prevented an intended assessment? -Given this narrow and precisely defined scope, the toolkit cannot be used _alone_ to assert that: +Given this narrow and precisely defined scope, the tool cannot be used _alone_ to assert that: - a pipeline is clinically or analytically valid - a pipeline conforms with ISO 15189 or any other standard @@ -28,44 +28,43 @@ Responsibility for the provenance and suitability of reference material, the com ### 2.1 Product structure -The organisation of the toolkit within the AdvISO GitHub organisation is: +The organisation of the tool within the AdvISO GitHub organisation is: ```text adviso-project └── validation_toolkit repository - └── adviso-validation distribution and executable - └── adviso_validation Python import package - ├── internal comparison, configuration, provenance, findings, and reporting components - └── compare subcommand + └── adviso-compare distribution and executable + └── adviso_compare Python import package + └── internal comparison, configuration, provenance, findings, and reporting components ``` -**REQ-PRD-001.** The **AdvISO Validation Toolkit** is the user-facing product and a minimal command namespace. The installable distribution package MUST be named **`adviso-validation`**. The command-line executable MUST be named **`adviso-validation`**. **`compare`** is currently the sole specified subcommand. The other components, including the **comparison engine**, are internal reusable software components and are not separate products. +**REQ-PRD-001.** **AdvISO Compare** is the user-facing product. The installable distribution package MUST be named **`adviso-compare`**. The command-line executable MUST be named **`adviso-compare`**. The other components, including the **comparison engine**, are internal reusable software components and are not separate products. **REQ-PRD-002.** The command-line form MUST be: ```text -adviso-validation compare ... +adviso-compare ... ``` -**REQ-PRD-003.** The Python import package MUST be named `adviso_validation`. +**REQ-PRD-003.** The Python import package MUST be named `adviso_compare`. **REQ-PRD-004.** The repository name MAY differ from the distribution and import-package names. Any difference between these names MUST be documented clearly. **REQ-PRD-005.** Once released, changes to the distribution name MUST follow the documented compatibility policy. Changes to the executable name MUST follow that policy. Changes to the import-package name MUST follow that policy. -**REQ-PRD-006.** Additional commands are not a development objective for the first release. They MUST NOT be added speculatively. +**REQ-PRD-006.** The executable MUST perform its comparison operation directly and MUST NOT expose a subcommand hierarchy. -**REQ-PRD-007.** A future command MAY be introduced only for a demonstrated, materially distinct operation in the same validation-evidence workflow. +**REQ-PRD-007.** Any future proposal to expand this executable to multiple comparison operations MUST define and justify a cohesive revised command-line interface rather than reserve speculative subcommands in version 1. -**REQ-PRD-008.** The top-level dispatcher MUST remain thin. It MUST NOT create speculative shared abstractions. +**REQ-PRD-008.** The command-line layer MUST remain thin. It MUST NOT create speculative shared abstractions. **REQ-PRD-009.** The initial implementation SHOULD use ordinary modules and a small comparator registry rather than a general plugin framework. **REQ-PRD-010.** Capabilities with materially different purposes, dependencies, governance, or release lifecycles SHOULD be separate AdvISO packages. -### 2.2 Scope of the `compare` subcommand +### 2.2 Scope of `adviso-compare` -The first-release scope of `compare` is: +The first-release scope of `adviso-compare` is: - comparison of structured records represented in CSV, TSV or JSON; - configuration-defined record keys and conservative single-field key selection in automatic mode; @@ -80,7 +79,7 @@ The first-release scope of `compare` is: ### 2.3 Out of scope -The `compare` command does not: +`adviso-compare` does not: - establish analytical sensitivity, analytical specificity, precision, limit of detection, reportable range, clinical validity, or clinical utility; - generate or adjudicate truth data; @@ -100,7 +99,7 @@ The following principles govern all requirements: 3. **REQ-PRN-003.** **Symmetry:** Missing and extra content MUST be assessed in both directions. 4. **REQ-PRN-009.** **Separation of concerns:** Run status, comparison evidence, and the laboratory's acceptance decision MUST remain distinct concepts. 5. **REQ-PRN-004.** **Complete evidence:** A normal comparison mismatch MUST still produce a complete report. -6. **REQ-PRN-005.** **Determinism:** The same exact input bytes, input basenames, configuration selection, toolkit version, and runtime identity MUST produce identical result content except for the explicitly variable run identifier and completion time. Configuration selection includes omission of `--configuration`, in which case automatic construction MUST itself be deterministic. Comparison outcomes MUST additionally satisfy the invariance rules in Section 9.1. +6. **REQ-PRN-005.** **Determinism:** The same exact input bytes, input basenames, configuration selection, software version, and runtime identity MUST produce identical result content except for the explicitly variable run identifier and completion time. Configuration selection includes omission of `--configuration`, in which case automatic construction MUST itself be deterministic. Comparison outcomes MUST additionally satisfy the invariance rules in Section 9.1. 7. **REQ-PRN-006.** **Traceability:** Findings MUST identify the rule, record, and field that produced them. 8. **REQ-PRN-007.** **Data minimisation:** Reports MUST include only the source identifiers and values necessary to explain and review the comparison. Routine logs, progress output, terminal diagnostics, and execution-error diagnostics MUST NOT reproduce them. 9. **REQ-PRN-008.** **Fail clearly:** Invalid or ambiguous input MUST produce an explicit error rather than an assumed interpretation. @@ -122,7 +121,7 @@ The following principles govern all requirements: - **Field:** A named value within a record. - **Finding:** A structured observation generated by applying one comparison rule to a record, field, dataset, or run. - **Run status:** Whether the requested comparison is `complete`, `incomplete`, or ended in `error`. -- **Laboratory decision:** The externally reviewed conclusion about validation or verification. The toolkit does not make this decision. +- **Laboratory decision:** The externally reviewed conclusion about validation or verification. The tool does not make this decision. - **Missing:** Expected in the reference but absent from the query. - **Extra:** Present in the query but absent from the reference. - **Null:** A present field with an explicitly empty/no-value representation defined by the input format or comparison configuration. @@ -251,7 +250,7 @@ For CSV and TSV: **REQ-CFG-001.** Every comparison uses an effective comparison configuration. A custom configuration MUST: -- **REQ-CFG-095.** use a published, independently versioned comparison-configuration schema supported by the installed toolkit release; +- **REQ-CFG-095.** use a published, independently versioned comparison-configuration schema supported by the installed software release; - **REQ-CFG-096.** declare its schema version and a stable configuration identifier and revision; - **REQ-CFG-097.** be validated fully before either dataset is compared; - **REQ-CFG-098.** reject unknown configuration keys by default; @@ -656,7 +655,7 @@ The first-release typed-value operation identifiers are: #### 6.2.9 Normative machine-readable schema and semantic validation -The normative JSON Schema Draft 2020-12 definition of comparison-configuration schema version `1.0` is [`src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json`](../src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json). +The normative JSON Schema Draft 2020-12 definition of comparison-configuration schema version `1.0` is [`src/adviso_compare/schemas/comparison-configuration-v1.0.schema.json`](../src/adviso_compare/schemas/comparison-configuration-v1.0.schema.json). **REQ-CFG-128.** The schema MUST use `https://adviso-project.github.io/validation_toolkit/schemas/comparison-configuration-v1.0.schema.json` as its canonical `$id`. @@ -664,7 +663,7 @@ The normative JSON Schema Draft 2020-12 definition of comparison-configuration s The standalone schema file is the authoritative machine-readable definition. -**REQ-CFG-087.** It MUST be version controlled in the source repository. It MUST be included as package data in the installed `adviso_validation.schemas` package. It MUST be loaded at runtime through the package-resource API rather than through a path relative to the working directory. +**REQ-CFG-087.** It MUST be version controlled in the source repository. It MUST be included as package data in the installed `adviso_compare.schemas` package. It MUST be loaded at runtime through the package-resource API rather than through a path relative to the working directory. **REQ-CFG-088.** CI MUST verify the schema's presence in built distributions. CI MUST validate every documented example against it. CI MUST validate every controlled configuration fixture against it. @@ -1159,13 +1158,13 @@ No separate package manifest is required in version 1; `comparison-results.json` ### 9.1 Machine-readable output -**REQ-OUT-004.** JSON MUST be the canonical output format. Its normative JSON Schema Draft 2020-12 definition is [`src/adviso_validation/schemas/comparison-results-v1.0.schema.json`](../src/adviso_validation/schemas/comparison-results-v1.0.schema.json). +**REQ-OUT-004.** JSON MUST be the canonical output format. Its normative JSON Schema Draft 2020-12 definition is [`src/adviso_compare/schemas/comparison-results-v1.0.schema.json`](../src/adviso_compare/schemas/comparison-results-v1.0.schema.json). **REQ-OUT-125.** `comparison-results.json` MUST be serialized as RFC 8259 JSON encoded as UTF-8 without a byte-order mark, using two-space indentation, LF line endings, and exactly one final newline. Non-ASCII characters MUST be emitted directly rather than escaped unless escaping is required by JSON syntax. Members of fixed-schema objects MUST be emitted in their documented schema order; members of dynamically named objects MUST be emitted in Unicode code-point order. **REQ-OUT-112.** The results schema MUST use `https://adviso-project.github.io/validation_toolkit/schemas/comparison-results-v1.0.schema.json` as its canonical `$id`. -**REQ-OUT-005.** The standalone results-schema file MUST be version controlled. It MUST be included as package data in `adviso_validation.schemas`. It MUST be resolvable together with the comparison-configuration schema that it references. +**REQ-OUT-005.** The standalone results-schema file MUST be version controlled. It MUST be included as package data in `adviso_compare.schemas`. It MUST be resolvable together with the comparison-configuration schema that it references. The result-schema version and comparison-configuration-schema version are independent. @@ -1199,7 +1198,7 @@ Every property is required, including empty `exclusions` and `findings` arrays. **REQ-OUT-095.** For determinism requirements, the comparison outcome consists of run and phase statuses; summary counts; exclusions considered without input occurrence order; and each finding's classification, code, phase, affected side, record identity, canonical or source field identity, review priority, and value evidence. It excludes run ID, completion time, software-runtime provenance, input byte provenance, source locations, human-readable messages that reproduce variable locations, and report-local finding IDs. -**REQ-OUT-096.** Given the same exact input bytes, input basenames, configuration selection, toolkit version, and runtime identity, all canonical result content other than `run.id` and `run.completed_at` MUST be identical. Repeated automatic-mode runs therefore MUST generate byte-identical effective configuration documents. +**REQ-OUT-096.** Given the same exact input bytes, input basenames, configuration selection, software version, and runtime identity, all canonical result content other than `run.id` and `run.completed_at` MUST be identical. Repeated automatic-mode runs therefore MUST generate byte-identical effective configuration documents. **REQ-OUT-097.** Changing only input record order or accepted LF/CRLF line endings MUST leave the comparison outcome unchanged. Exact-byte input size and digest, source locations, location-dependent messages, and report-local finding IDs MAY change. The result for each particular ordered byte input MUST still satisfy all deterministic ordering rules. @@ -1209,7 +1208,7 @@ Every property is required, including empty `exclusions` and `findings` arrays. **REQ-OUT-014.** Overall run status MUST be `complete` if and only if all three phases are complete. -**REQ-OUT-015.** The `software` object MUST identify `adviso-validation` and its version. A build identifier MAY be included when available. +**REQ-OUT-015.** The `software` object MUST identify `adviso-compare` and its version. A build identifier MAY be included when available. **REQ-OUT-016.** The `runtime` object MUST record Python version, a normalized non-identifying platform identifier, and the Unicode-data version. @@ -1407,7 +1406,7 @@ Configured-field exclusions MUST be ordered first by `canonical_field` using Uni ### 10.1 General behaviour -- **REQ-CLI-001.** A subcommand MUST be required. +- **REQ-CLI-001.** Comparison arguments MUST be accepted directly by the `adviso-compare` executable without a subcommand. - **REQ-CLI-002.** Invalid argument combinations MUST be rejected before input processing. - **REQ-CLI-003.** Modes and formats MUST use enumerated choices. - **REQ-CLI-004.** `--help` and `--version` MUST work without other arguments. @@ -1416,12 +1415,12 @@ Configured-field exclusions MUST be ordered first by `canonical_field` using Uni - **REQ-CLI-007.** Comparison progress SHOULD be optional. - **REQ-CLI-008.** Comparison progress MUST NOT contaminate machine-readable output. -### 10.2 Proposed comparison interface +### 10.2 Comparison interface The conceptual interface is: ```text -adviso-validation compare \ +adviso-compare \ --reference REFERENCE \ --query QUERY \ --output OUTPUT \ @@ -1456,9 +1455,9 @@ adviso-validation compare \ Error classes represented by exit codes 2 and 3 have run status `error`. The distinct codes allow automation to distinguish expected operational errors from unexpected implementation failures. -### 10.4 Additional commands +### 10.4 Command hierarchy -**REQ-CLI-017.** The version 1 executable MUST expose no user-facing subcommand other than `compare`. +**REQ-CLI-017.** The version 1 executable MUST NOT expose user-facing subcommands. ## 11. Security and data governance @@ -1480,7 +1479,7 @@ Error classes represented by exit codes 2 and 3 have run status `error`. The dis ## 12. Deployment, accessibility, and reliability - **REQ-DEP-001.** all core comparison functionality MUST operate fully offline; -- **REQ-DEP-002.** the toolkit MUST NOT require a hosted service, account, licence server, or telemetry connection; +- **REQ-DEP-002.** the tool MUST NOT require a hosted service, account, licence server, or telemetry connection; - **REQ-DEP-003.** an offline-installable artefact with integrity information SHOULD be provided; - **REQ-DEP-004.** reports MUST use open formats; - **REQ-DEP-005.** reports MUST NOT require proprietary software; @@ -1569,13 +1568,13 @@ In operational use, each laboratory is responsible for creating, approving, main - **REQ-DOC-013.** every property in the canonical results JSON schema, including finding classifications and codes, typed source and canonical values, counts, exclusions, run and phase statuses, and provenance fields; the guide MUST explain that software consuming a result must first validate `results_schema_version` and must use structured properties, classifications, and finding codes rather than parsing presentation messages; - **REQ-DOC-014.** execution-error diagnostics and common corrective actions; - **REQ-DOC-015.** the tool's intended use, non-claims, security and data-governance boundaries, and known limitations; and -- **REQ-DOC-016.** the toolkit, comparison-configuration-schema, and results-schema versions to which the documentation applies. +- **REQ-DOC-016.** the software, comparison-configuration-schema, and results-schema versions to which the documentation applies. **REQ-DOC-002.** The guide MUST include complete, runnable examples covering automatic strict-exact mode, including automatic key discovery, and representative custom configuration features. **REQ-DOC-003.** Configuration examples MUST validate against the published comparison-configuration schema. Results examples MUST validate against the published results schema. -**REQ-DOC-004.** Documentation MUST distinguish machine-enforced schema constraints from explanatory guidance. It MUST accurately describe the behaviour of the documented toolkit version and MUST NOT contradict the machine-readable schemas it supports. Any material inconsistency is a release-blocking documentation defect. +**REQ-DOC-004.** Documentation MUST distinguish machine-enforced schema constraints from explanatory guidance. It MUST accurately describe the behaviour of the documented software version and MUST NOT contradict the machine-readable schemas it supports. Any material inconsistency is a release-blocking documentation defect. **REQ-DOC-005.** Command-line `--help` output MUST provide a concise description of every command and option. It MUST direct users to the full guide. @@ -1591,9 +1590,9 @@ In operational use, each laboratory is responsible for creating, approving, main - **REQ-REL-011.** a reproducible lockfile or equivalent dependency record; - **REQ-REL-012.** evidence that required verification tests passed. -**REQ-REL-002.** Comparison-configuration schemas, results schemas, and toolkit releases MUST each have their own version identifier and MUST NOT be assumed to advance in lockstep or to use matching version numbers. Each results-schema version nevertheless references the exact comparison-configuration schema version or versions permitted in that result, as required by `REQ-OUT-113`. +**REQ-REL-002.** Comparison-configuration schemas, results schemas, and software releases MUST each have their own version identifier and MUST NOT be assumed to advance in lockstep or to use matching version numbers. Each results-schema version nevertheless references the exact comparison-configuration schema version or versions permitted in that result, as required by `REQ-OUT-113`. -**REQ-REL-003.** Each toolkit release MUST document the comparison-configuration schema versions it supports. It MUST document the results schema versions it supports. +**REQ-REL-003.** Each software release MUST document the comparison-configuration schema versions it supports. It MUST document the results schema versions it supports. **REQ-REL-004.** An unsupported schema version MUST be rejected clearly rather than interpreted approximately. @@ -1698,7 +1697,7 @@ This catalogue summarises minimum executable scenarios for the first release. It ### 14.5 Evidence, CLI, and packaging scenarios -- **AT-EVD-001:** Runs with identical exact bytes, basenames, configuration, toolkit version, and runtime identity produce byte-identical canonical JSON after normalizing only run ID and completion time. Serialization fixtures verify UTF-8 without a byte-order mark, two-space indentation, LF line endings, exactly one final newline, direct non-ASCII output, fixed-schema member order, and Unicode code-point order for dynamic members. Parameterized record-order and LF/CRLF variants preserve the defined comparison outcome while permitting only the documented byte-provenance, source-location, location-dependent message, and report-local ID differences. The comparison outcome also remains invariant across every supported Python and operating-system environment, including Unicode-sensitive fixtures. +- **AT-EVD-001:** Runs with identical exact bytes, basenames, configuration, software version, and runtime identity produce byte-identical canonical JSON after normalizing only run ID and completion time. Serialization fixtures verify UTF-8 without a byte-order mark, two-space indentation, LF line endings, exactly one final newline, direct non-ASCII output, fixed-schema member order, and Unicode code-point order for dynamic members. Parameterized record-order and LF/CRLF variants preserve the defined comparison outcome while permitting only the documented byte-provenance, source-location, location-dependent message, and report-local ID differences. The comparison outcome also remains invariant across every supported Python and operating-system environment, including Unicode-sensitive fixtures. - Verifies: `REQ-PRN-005`, `REQ-INP-014`, `REQ-CMP-033`, `REQ-OUT-012`, `REQ-OUT-095`, `REQ-OUT-096`, `REQ-OUT-097`, `REQ-OUT-098`, `REQ-OUT-125`, `REQ-DEP-013`, `REQ-CMP-032` - **AT-EVD-002:** Results contain exact-byte SHA-256 input digests and input basenames without exposing absolute paths, raw command lines, environment variables, or secrets; the complete validated comparison configuration is embedded without a separate configuration digest. - Verifies: `REQ-CFG-023`, `REQ-OUT-017`, `REQ-OUT-018`, `REQ-OUT-019`, `REQ-OUT-022`, `REQ-OUT-023`, `REQ-SEC-007`, `REQ-SEC-008` @@ -1718,7 +1717,7 @@ This catalogue summarises minimum executable scenarios for the first release. It - Verifies: `REQ-SEC-002`, `REQ-SEC-003`, `REQ-DEP-001`, `REQ-DEP-002` - **AT-EVD-013:** Findings attributable to declared fields inherit their resolved review priority; other findings do not. HTML presentation may distinguish priorities but does not rely on colour alone, and priority never changes findings, counts, status, or exit code. - Verifies: `REQ-CFG-041`, `REQ-CFG-042`, `REQ-CFG-043`, `REQ-CFG-044`, `REQ-CFG-045`, `REQ-OUT-034` -- **AT-PKG-001:** The built distribution is named `adviso-validation`, contains an importable `adviso_validation` package and every required subpackage, installs the `adviso-validation` executable, and completes an end-to-end comparison through that executable. This consolidated parameterized scenario also covers: The release artefact can be installed and used without network access, contains offline documentation and integrity information, and supports an end-to-end comparison in an isolated environment. +- **AT-PKG-001:** The built distribution is named `adviso-compare`, contains an importable `adviso_compare` package and every required subpackage, installs the `adviso-compare` executable, and completes an end-to-end comparison through that executable. This consolidated parameterized scenario also covers: The release artefact can be installed and used without network access, contains offline documentation and integrity information, and supports an end-to-end comparison in an isolated environment. - Verifies: `REQ-PRD-001`, `REQ-PRD-003`, `REQ-CFG-087`, `REQ-TST-013`, `REQ-TST-030`, `REQ-DEP-003`, `REQ-DEP-007` - **AT-EVD-015:** Canonical results contain valid run, phase, software, runtime, and fixed-dialect metadata; run IDs are canonical lowercase UUIDv4 values; completion timestamps are UTC RFC 3339 values using `Z` and whole-second precision; Python implementation/version, normalized platform, and Unicode database version use their prescribed forms without identifying host details; start time and duration are absent; and overall status reconciles with phase statuses. - Verifies: `REQ-OUT-009`, `REQ-OUT-010`, `REQ-OUT-011`, `REQ-OUT-014`, `REQ-OUT-015`, `REQ-OUT-016`, `REQ-OUT-020`, `REQ-OUT-121`, `REQ-OUT-122`, `REQ-OUT-123`, `REQ-OUT-124` @@ -1730,12 +1729,12 @@ This catalogue summarises minimum executable scenarios for the first release. It - Verifies: `REQ-OUT-109`, `REQ-OUT-110`, `REQ-OUT-114` - **AT-EVD-019:** HTML generated solely from canonical JSON uses an open non-proprietary format and presents the run purpose, phase and run statuses, input and configuration identities, counts, findings, discrepancies, limitations, exclusions, unassessed content, provenance, and no acceptance verdict; HTML is never the sole canonical evidence format. Its headline distinguishes run completeness, comparison-difference count, and assessment-issue count. Detailed findings are organized primarily by classification with comparison differences first and phase retained as secondary traceability information. This consolidated parameterized scenario also covers: Complete and incomplete reports contain raw findings and metrics but no acceptance threshold, pass/fail verdict, or equivalent inferred laboratory decision. - Verifies: `REQ-PRN-009`, `REQ-OUT-042`, `REQ-OUT-074`, `REQ-OUT-075`, `REQ-OUT-076`, `REQ-OUT-077`, `REQ-OUT-078`, `REQ-OUT-079`, `REQ-OUT-080`, `REQ-OUT-081`, `REQ-OUT-120`, `REQ-OUT-126`, `REQ-OUT-127`, `REQ-OUT-128`, `REQ-DEP-004`, `REQ-DEP-005`, `REQ-RUN-013` -- **AT-EVD-021:** Installed-CLI tests cover the required `adviso-validation compare` form, required reference, query, and output arguments, optional configuration, enumerated choices, invalid combinations before input access, normal summaries on standard output, and optional progress that never contaminates machine-readable output. This consolidated parameterized scenario also covers: Omitting `--configuration` selects automatic strict-exact mode and records its generated effective configuration; a supplied value is treated only as a custom comparison-configuration path. The reference and query formats are independently determined and recorded from recognised, case-insensitive filename extensions; a missing, unknown, or ambiguous extension causes an input error, and content that does not conform to the indicated format is rejected. Command-line `--help` describes every version 1 command and option and directs users to the full guide. +- **AT-EVD-021:** Installed-CLI tests cover the required direct `adviso-compare` form, required reference, query, and output arguments, optional configuration, enumerated choices, invalid combinations before input access, rejection of subcommands, normal summaries on standard output, and optional progress that never contaminates machine-readable output. This consolidated parameterized scenario also covers: Omitting `--configuration` selects automatic strict-exact mode and records its generated effective configuration; a supplied value is treated only as a custom comparison-configuration path. The reference and query formats are independently determined and recorded from recognised, case-insensitive filename extensions; a missing, unknown, or ambiguous extension causes an input error, and content that does not conform to the indicated format is rejected. Command-line `--help` describes every version 1 option and directs users to the full guide. - Verifies: `REQ-PRD-002`, `REQ-INP-001`, `REQ-INP-002`, `REQ-INP-004`, `REQ-INP-060`, `REQ-INP-061`, `REQ-INP-062`, `REQ-CLI-001`, `REQ-CLI-002`, `REQ-CLI-003`, `REQ-CLI-006`, `REQ-CLI-007`, `REQ-CLI-008`, `REQ-INP-003`, `REQ-CFG-092`, `REQ-CLI-009`, `REQ-CLI-010`, `REQ-INP-005`, `REQ-INP-006`, `REQ-INP-007`, `REQ-INP-008`, `REQ-INP-009`, `REQ-CLI-013`, `REQ-CLI-004`, `REQ-DOC-005` ### 14.6 Documentation scenarios -- **AT-DOC-001:** The built distribution contains an offline user guide covering every topic required by Section 13.4, and the guide identifies the toolkit and schema versions to which it applies. This consolidated parameterized scenario also covers: Every complete comparison-configuration and canonical-results example in the guide validates against its corresponding packaged schema, and every documented command-line invocation uses the version 1 interface. +- **AT-DOC-001:** The built distribution contains an offline user guide covering every topic required by Section 13.4, and the guide identifies the software and schema versions to which it applies. This consolidated parameterized scenario also covers: Every complete comparison-configuration and canonical-results example in the guide validates against its corresponding packaged schema, and every documented command-line invocation uses the version 1 interface. - Verifies: `REQ-DOC-001`, `REQ-DOC-006`, `REQ-DOC-007`, `REQ-DOC-008`, `REQ-DOC-009`, `REQ-DOC-010`, `REQ-DOC-011`, `REQ-DOC-012`, `REQ-DOC-013`, `REQ-DOC-014`, `REQ-DOC-015`, `REQ-DOC-016`, `REQ-DOC-002`, `REQ-DOC-003`, `REQ-DOC-004`, `REQ-OUT-111` ### 14.7 Fixture and compatibility scenarios diff --git a/pyproject.toml b/pyproject.toml index 86a5fcd..71ce9f5 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -7,7 +7,7 @@ requires = [ build-backend = "setuptools.build_meta" [project] -name = "adviso-validation" +name = "adviso-compare" dynamic = ["version"] description = "Translate, compare, and report differences between structured bioinformatics datasets." readme = "README.md" @@ -40,7 +40,7 @@ dev = [ ] [project.scripts] -adviso-validation = "adviso_validation.cli:main" +adviso-compare = "adviso_compare.cli:main" [tool.setuptools] package-dir = { "" = "src" } @@ -48,14 +48,14 @@ include-package-data = true [tool.setuptools.packages.find] where = ["src"] -include = ["adviso_validation*"] +include = ["adviso_compare*"] namespaces = false [tool.setuptools.package-data] -"adviso_validation.schemas" = ["*.json"] +"adviso_compare.schemas" = ["*.json"] [tool.setuptools_scm] -write_to = "src/adviso_validation/_version.py" +write_to = "src/adviso_compare/_version.py" fallback_version = "0.0.0" [tool.pytest.ini_options] diff --git a/src/adviso_compare/__init__.py b/src/adviso_compare/__init__.py new file mode 100644 index 0000000..f805424 --- /dev/null +++ b/src/adviso_compare/__init__.py @@ -0,0 +1 @@ +"""AdvISO Compare.""" diff --git a/src/adviso_compare/schemas/__init__.py b/src/adviso_compare/schemas/__init__.py new file mode 100644 index 0000000..74519cb --- /dev/null +++ b/src/adviso_compare/schemas/__init__.py @@ -0,0 +1 @@ +"""Versioned machine-readable schemas distributed with adviso-compare.""" diff --git a/src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json b/src/adviso_compare/schemas/comparison-configuration-v1.0.schema.json similarity index 100% rename from src/adviso_validation/schemas/comparison-configuration-v1.0.schema.json rename to src/adviso_compare/schemas/comparison-configuration-v1.0.schema.json diff --git a/src/adviso_validation/schemas/comparison-results-v1.0.schema.json b/src/adviso_compare/schemas/comparison-results-v1.0.schema.json similarity index 99% rename from src/adviso_validation/schemas/comparison-results-v1.0.schema.json rename to src/adviso_compare/schemas/comparison-results-v1.0.schema.json index 2f95b8e..a520a14 100644 --- a/src/adviso_validation/schemas/comparison-results-v1.0.schema.json +++ b/src/adviso_compare/schemas/comparison-results-v1.0.schema.json @@ -127,7 +127,7 @@ ], "properties": { "name": { - "const": "adviso-validation" + "const": "adviso-compare" }, "version": { "$ref": "#/$defs/nonEmptyString" diff --git a/src/adviso_validation/__init__.py b/src/adviso_validation/__init__.py deleted file mode 100644 index 76eb0da..0000000 --- a/src/adviso_validation/__init__.py +++ /dev/null @@ -1 +0,0 @@ -"""AdvISO validation toolkit.""" diff --git a/src/adviso_validation/schemas/__init__.py b/src/adviso_validation/schemas/__init__.py deleted file mode 100644 index 3011ae2..0000000 --- a/src/adviso_validation/schemas/__init__.py +++ /dev/null @@ -1 +0,0 @@ -"""Versioned machine-readable schemas distributed with adviso-validation.""" From d956ba580448b78f9da03e97c580a6e8af552207 Mon Sep 17 00:00:00 2001 From: Kevin Howe Date: Fri, 30 Oct 2026 14:49:25 +0000 Subject: [PATCH 13/45] Restructured to accommodate future additional command-line tools --- .gitignore | 9 ++---- AGENTS.md | 28 +++++++++++-------- README.md | 28 +++++++++++-------- packages/adviso-compare/README.md | 17 +++++++++++ .../adviso-compare/docs}/SPECIFICATION.md | 24 ++++++++++------ .../adviso-compare/pyproject.toml | 4 ++- .../src}/adviso_compare/__init__.py | 0 .../src}/adviso_compare/schemas/__init__.py | 0 .../comparison-configuration-v1.0.schema.json | 0 .../comparison-results-v1.0.schema.json | 0 .../adviso-compare/tests}/__init__.py | 0 11 files changed, 70 insertions(+), 40 deletions(-) create mode 100644 packages/adviso-compare/README.md rename {docs => packages/adviso-compare/docs}/SPECIFICATION.md (99%) rename pyproject.toml => packages/adviso-compare/pyproject.toml (93%) rename {src => packages/adviso-compare/src}/adviso_compare/__init__.py (100%) rename {src => packages/adviso-compare/src}/adviso_compare/schemas/__init__.py (100%) rename {src => packages/adviso-compare/src}/adviso_compare/schemas/comparison-configuration-v1.0.schema.json (100%) rename {src => packages/adviso-compare/src}/adviso_compare/schemas/comparison-results-v1.0.schema.json (100%) rename {tests => packages/adviso-compare/tests}/__init__.py (100%) diff --git a/.gitignore b/.gitignore index 9c14cd4..edc8508 100644 --- a/.gitignore +++ b/.gitignore @@ -1,8 +1,8 @@ # artefacts .DS_Store -# dynamic version -version.py +# dynamic versions +**/_version.py ## dev/env .venv @@ -79,7 +79,7 @@ instance/ .scrapy # Sphinx documentation -docs/_build/ +**/docs/_build/ # PyBuilder .pybuilder/ @@ -146,6 +146,3 @@ dmypy.json # Cython debug symbols cython_debug/ - -# file generated by setuptools_scm -version.py \ No newline at end of file diff --git a/AGENTS.md b/AGENTS.md index 2274341..bd5f2e0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,16 +1,16 @@ -# Agent guidance for `adviso-compare` +# Agent guidance for the AdvISO validation toolkit ## Purpose of this file -This file governs how agents work in this repository. It does not define product behaviour. Product requirements belong in `docs/SPECIFICATION.md` and the normative schemas; do not duplicate them here. +This file governs how agents work in this monorepo. It does not define product behaviour. Product requirements belong in the applicable package specification and normative schemas; do not duplicate them here. ## Sources of truth -Use this order of authority: +For `adviso-compare`, use this order of authority: -1. `docs/SPECIFICATION.md` — normative behaviour and requirement identifiers. -2. `src/adviso_compare/schemas/comparison-configuration-v1.0.schema.json` — normative comparison-configuration structure. -3. `src/adviso_compare/schemas/comparison-results-v1.0.schema.json` — normative canonical-results structure. +1. `packages/adviso-compare/docs/SPECIFICATION.md` — normative behaviour and requirement identifiers. +2. `packages/adviso-compare/src/adviso_compare/schemas/comparison-configuration-v1.0.schema.json` — normative comparison-configuration structure. +3. `packages/adviso-compare/src/adviso_compare/schemas/comparison-results-v1.0.schema.json` — normative canonical-results structure. 4. Version 1 tests and controlled fixtures — executable verification of the normative sources. 5. User documentation — explanatory material that must agree with the sources above. @@ -18,13 +18,17 @@ If these sources appear inconsistent, stop and report the exact conflict and its ## Repository layout -- Put production code under `src/adviso_compare/`. -- Put automated tests under `tests/`. -- Put maintained user and developer documentation under `docs/`. -- Keep runtime package resources inside `src/adviso_compare/`. +- Keep each independently installable tool under `packages//`. +- Give each tool its own `pyproject.toml`, README, `docs/`, `src/`, and `tests/`. +- For `adviso-compare`, put production code and runtime resources under + `packages/adviso-compare/src/adviso_compare/`, tests under + `packages/adviso-compare/tests/`, and maintained product documentation under + `packages/adviso-compare/docs/`. +- Keep repository-wide guidance at the repository root. Do not place + product-specific requirements there. - Treat generated version files, build outputs, caches, coverage output, and virtual environments as generated artefacts rather than source. -Do not create new top-level directories or reorganize established modules without a concrete need in the approved implementation plan. +Do not create a package for a proposed tool until its scope and implementation work have been approved. Do not create new top-level directories or reorganize established modules without a concrete need in an approved plan. ## Working method @@ -42,7 +46,7 @@ Use test-driven development for the version 1 implementation. A failing test is ## Change control -- Do not change `docs/SPECIFICATION.md`, either normative schema, or established requirement semantics unless the task explicitly authorizes a specification change. +- Do not change a package specification, its normative schemas, or established requirement semantics unless the task explicitly authorizes a specification change. - Preserve existing requirement IDs. Add new IDs rather than renumbering requirements. - Preserve schema version identifiers and canonical `$id` values unless a deliberate version change has been approved. - When behaviour changes are authorized, update all affected normative text, schemas, tests, fixtures, and user documentation together. diff --git a/README.md b/README.md index 9dab143..0bfbb5a 100644 --- a/README.md +++ b/README.md @@ -1,17 +1,21 @@ -# AdvISO Compare +# AdvISO validation toolkit -`adviso-compare` is a command-line tool for comparing a reference dataset with a query dataset. It uses explicit configuration rules to translate different source representations into a common canonical representation, then reports post-translation differences separately from issues that prevented assessment. +This repository is a monorepo for independently installable AdvISO tools that +support bioinformatics validation and verification work. Repository membership +groups related software; it does not imply a shared executable, Python package, +dependency set, version, or release lifecycle. -The project is intended to support bioinformatics validation and verification activities associated with ISO 15189 accreditation. It produces comparison evidence; it does not determine whether a pipeline is valid, acceptable, or compliant. +## Tools -## Project status +| Tool | Purpose | Status | +| --- | --- | --- | +| [`adviso-compare`](packages/adviso-compare/README.md) | Translate and compare structured reference and query datasets | Version 1 specified; implementation not yet started | -Version 1 is currently specified but not yet implemented. The repository is being prepared for a test-driven implementation. +Each tool lives under `packages/` with its own packaging metadata, source, +tests, documentation, normative artefacts, and release version. A future +dataset-oriented tool can be added as a peer package without adding unrelated +commands or dependencies to `adviso-compare`. -The current normative artefacts are: - -- [Behaviour specification](docs/SPECIFICATION.md) -- [Comparison-configuration schema](src/adviso_compare/schemas/comparison-configuration-v1.0.schema.json) -- [Canonical-results schema](src/adviso_compare/schemas/comparison-results-v1.0.schema.json) - -The planned distribution and executable are named `adviso-compare`; the Python import package is `adviso_compare`. +Repository-wide contribution and code-quality guidance is in +[`AGENTS.md`](AGENTS.md). Product behaviour is defined within each tool's own +specification and schemas. diff --git a/packages/adviso-compare/README.md b/packages/adviso-compare/README.md new file mode 100644 index 0000000..2138925 --- /dev/null +++ b/packages/adviso-compare/README.md @@ -0,0 +1,17 @@ +# AdvISO Compare + +`adviso-compare` is a command-line tool for comparing a reference dataset with a query dataset. It uses explicit configuration rules to translate different source representations into a common canonical representation, then reports post-translation differences separately from issues that prevented assessment. + +The project is intended to support bioinformatics validation and verification activities associated with ISO 15189 accreditation. It produces comparison evidence; it does not determine whether a pipeline is valid, acceptable, or compliant. + +## Project status + +Version 1 is currently specified but not yet implemented. The package is being prepared for a test-driven implementation. + +The current normative artefacts are: + +- [Behaviour specification](docs/SPECIFICATION.md) +- [Comparison-configuration schema](src/adviso_compare/schemas/comparison-configuration-v1.0.schema.json) +- [Canonical-results schema](src/adviso_compare/schemas/comparison-results-v1.0.schema.json) + +The planned distribution and executable are named `adviso-compare`; the Python import package is `adviso_compare`. diff --git a/docs/SPECIFICATION.md b/packages/adviso-compare/docs/SPECIFICATION.md similarity index 99% rename from docs/SPECIFICATION.md rename to packages/adviso-compare/docs/SPECIFICATION.md index cfbd356..72f9d24 100644 --- a/docs/SPECIFICATION.md +++ b/packages/adviso-compare/docs/SPECIFICATION.md @@ -2,7 +2,7 @@ **Document status:** Draft for review **Specification version:** 0.1.0-draft -**Applies to:** The `adviso-compare` command and its supporting comparison engine +**Applies to:** The `adviso-compare` command and its supporting comparison engine
**Normative language:** The terms **MUST**, **MUST NOT**, **SHOULD**, **SHOULD NOT**, and **MAY** indicate requirement strength. ## 1. Purpose @@ -28,15 +28,21 @@ Responsibility for the provenance and suitability of reference material, the com ### 2.1 Product structure -The organisation of the tool within the AdvISO GitHub organisation is: +The product identities and their roles are: -```text -adviso-project -└── validation_toolkit repository - └── adviso-compare distribution and executable - └── adviso_compare Python import package - └── internal comparison, configuration, provenance, findings, and reporting components -``` +| Concept | Name | Role | +| --- | --- | --- | +| Product | **AdvISO Compare** | User-facing comparison tool | +| Distribution | `adviso-compare` | Installable Python distribution | +| Executable | `adviso-compare` | Direct command-line interface | +| Import package | `adviso_compare` | Python implementation package | +| Comparison engine | Internal component | Implements the comparison behaviour | + +The distribution contains the `adviso_compare` import package and installs the +`adviso-compare` executable. The executable delegates to a thin command-line +layer in that package. The comparison engine and supporting configuration, +provenance, findings, and reporting components are internal modules rather than +separate products. **REQ-PRD-001.** **AdvISO Compare** is the user-facing product. The installable distribution package MUST be named **`adviso-compare`**. The command-line executable MUST be named **`adviso-compare`**. The other components, including the **comparison engine**, are internal reusable software components and are not separate products. diff --git a/pyproject.toml b/packages/adviso-compare/pyproject.toml similarity index 93% rename from pyproject.toml rename to packages/adviso-compare/pyproject.toml index 71ce9f5..b6d9c51 100644 --- a/pyproject.toml +++ b/packages/adviso-compare/pyproject.toml @@ -55,7 +55,9 @@ namespaces = false "adviso_compare.schemas" = ["*.json"] [tool.setuptools_scm] -write_to = "src/adviso_compare/_version.py" +root = "../.." +relative_to = "pyproject.toml" +version_file = "src/adviso_compare/_version.py" fallback_version = "0.0.0" [tool.pytest.ini_options] diff --git a/src/adviso_compare/__init__.py b/packages/adviso-compare/src/adviso_compare/__init__.py similarity index 100% rename from src/adviso_compare/__init__.py rename to packages/adviso-compare/src/adviso_compare/__init__.py diff --git a/src/adviso_compare/schemas/__init__.py b/packages/adviso-compare/src/adviso_compare/schemas/__init__.py similarity index 100% rename from src/adviso_compare/schemas/__init__.py rename to packages/adviso-compare/src/adviso_compare/schemas/__init__.py diff --git a/src/adviso_compare/schemas/comparison-configuration-v1.0.schema.json b/packages/adviso-compare/src/adviso_compare/schemas/comparison-configuration-v1.0.schema.json similarity index 100% rename from src/adviso_compare/schemas/comparison-configuration-v1.0.schema.json rename to packages/adviso-compare/src/adviso_compare/schemas/comparison-configuration-v1.0.schema.json diff --git a/src/adviso_compare/schemas/comparison-results-v1.0.schema.json b/packages/adviso-compare/src/adviso_compare/schemas/comparison-results-v1.0.schema.json similarity index 100% rename from src/adviso_compare/schemas/comparison-results-v1.0.schema.json rename to packages/adviso-compare/src/adviso_compare/schemas/comparison-results-v1.0.schema.json diff --git a/tests/__init__.py b/packages/adviso-compare/tests/__init__.py similarity index 100% rename from tests/__init__.py rename to packages/adviso-compare/tests/__init__.py From 442f0a0e8afaf9305c954121055115840d440631 Mon Sep 17 00:00:00 2001 From: Kevin Howe Date: Fri, 6 Nov 2026 11:03:52 +0000 Subject: [PATCH 14/45] Phase one of implementation completed and reviewed: establish the baseline package structure, schemas, and test architecture --- AGENTS.md | 12 +- packages/adviso-compare/README.md | 3 + .../docs/IMPLEMENTATION_PLAN.md | 413 ++++++++++++++++++ packages/adviso-compare/docs/SPECIFICATION.md | 8 +- packages/adviso-compare/pyproject.toml | 3 + packages/adviso-compare/tests/README.md | 17 + .../adviso-compare/tests/support/__init__.py | 2 + .../adviso-compare/tests/support/documents.py | 172 ++++++++ .../adviso-compare/tests/support/schemas.py | 52 +++ .../adviso-compare/tests/test_distribution.py | 112 +++++ packages/adviso-compare/tests/test_schemas.py | 89 ++++ 11 files changed, 874 insertions(+), 9 deletions(-) create mode 100644 packages/adviso-compare/docs/IMPLEMENTATION_PLAN.md create mode 100644 packages/adviso-compare/tests/README.md create mode 100644 packages/adviso-compare/tests/support/__init__.py create mode 100644 packages/adviso-compare/tests/support/documents.py create mode 100644 packages/adviso-compare/tests/support/schemas.py create mode 100644 packages/adviso-compare/tests/test_distribution.py create mode 100644 packages/adviso-compare/tests/test_schemas.py diff --git a/AGENTS.md b/AGENTS.md index bd5f2e0..a207efd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -8,7 +8,7 @@ This file governs how agents work in this monorepo. It does not define product b For `adviso-compare`, use this order of authority: -1. `packages/adviso-compare/docs/SPECIFICATION.md` — normative behaviour and requirement identifiers. +1. `packages/adviso-compare/docs/SPECIFICATION.md` — normative behaviour. 2. `packages/adviso-compare/src/adviso_compare/schemas/comparison-configuration-v1.0.schema.json` — normative comparison-configuration structure. 3. `packages/adviso-compare/src/adviso_compare/schemas/comparison-results-v1.0.schema.json` — normative canonical-results structure. 4. Version 1 tests and controlled fixtures — executable verification of the normative sources. @@ -36,8 +36,8 @@ Before changing behaviour: 1. Read the relevant specification sections in full, including referenced requirements elsewhere in the document. 2. Read the relevant schema definitions. -3. Identify the requirement IDs affected by the work. -4. Add or update the smallest coherent set of tests that verifies those requirements. +3. Identify the specified behaviours affected by the work. +4. Add or update the smallest coherent set of tests that verifies those behaviours. 5. Implement until those tests pass. 6. Run proportionate wider tests, static checks, schema checks, and packaging checks. 7. Update affected documentation and controlled fixtures. @@ -47,7 +47,6 @@ Use test-driven development for the version 1 implementation. A failing test is ## Change control - Do not change a package specification, its normative schemas, or established requirement semantics unless the task explicitly authorizes a specification change. -- Preserve existing requirement IDs. Add new IDs rather than renumbering requirements. - Preserve schema version identifiers and canonical `$id` values unless a deliberate version change has been approved. - When behaviour changes are authorized, update all affected normative text, schemas, tests, fixtures, and user documentation together. - Do not add speculative features, extension points, compatibility layers, dependencies, or abstractions. @@ -78,7 +77,8 @@ Do not introduce or replace a formatter, linter, type checker, build backend, te ## Testing practice -- Cite the requirement IDs verified by each behavioural test or coherent parameterized scenario. +- Give tests descriptive behavioural names that remain meaningful without the specification. +- Do not embed specification requirement identifiers or acceptance-scenario identifiers in code, tests, fixtures, snapshots, or package metadata. Development traceability belongs in the specification and review records, and must remain a one-way reference from those records to the software artefacts. - Prefer parameterization when it expresses one rule table clearly; do not merge unrelated behaviours merely to reduce test count. - Test public behaviour and stable internal contracts rather than private implementation details. - Use controlled, synthetic, non-sensitive fixtures. @@ -111,7 +111,7 @@ Refer to the specification for the software's normative security and data-govern Report: -- the requirement IDs affected; +- the specified behaviours affected; - tests added or updated and why they are sufficient; - commands run and their results; - schema, static, packaging, and documentation checks performed; diff --git a/packages/adviso-compare/README.md b/packages/adviso-compare/README.md index 2138925..0e3e745 100644 --- a/packages/adviso-compare/README.md +++ b/packages/adviso-compare/README.md @@ -14,4 +14,7 @@ The current normative artefacts are: - [Comparison-configuration schema](src/adviso_compare/schemas/comparison-configuration-v1.0.schema.json) - [Canonical-results schema](src/adviso_compare/schemas/comparison-results-v1.0.schema.json) +The proposed staged delivery sequence is documented in the +[version 1 implementation plan](docs/IMPLEMENTATION_PLAN.md). + The planned distribution and executable are named `adviso-compare`; the Python import package is `adviso_compare`. diff --git a/packages/adviso-compare/docs/IMPLEMENTATION_PLAN.md b/packages/adviso-compare/docs/IMPLEMENTATION_PLAN.md new file mode 100644 index 0000000..3e76808 --- /dev/null +++ b/packages/adviso-compare/docs/IMPLEMENTATION_PLAN.md @@ -0,0 +1,413 @@ +# AdvISO Compare version 1 implementation plan + +## 1. Purpose + +This document divides the version 1 implementation of `adviso-compare` into +reviewable, test-driven increments. Each increment ends in a coherent and +demonstrable result. Work on the next increment starts only after the completed +increment has been reviewed and approved. + +The behaviour specification and the two normative schemas remain the sources of +truth. This plan sequences their implementation; it does not add, remove, or +reinterpret product requirements. + +The plan covers only `adviso-compare`. Dataset acquisition, dataset subsetting, +pipeline execution, validation-dataset curation, and the proposed +`adviso-datasets` product remain outside its scope. The implementation also does +not add a laboratory acceptance verdict: its output is comparison evidence for +the validation process. + +## 2. Delivery method + +Each implementation step follows the same cycle: + +1. Re-read the relevant specification sections and schemas. +2. Identify the specified behaviours and validation scenarios in scope. +3. Add the smallest coherent set of tests and controlled fixtures for that + behaviour. Confirm that new behavioural tests fail for the expected reason. +4. Implement only enough production code to satisfy the step. +5. Run the new tests and all previously passing tests. +6. Run the proportionate schema, packaging, static, and documentation checks + that are already configured for the project. +7. Present the increment for review and approval. + +Tests will use descriptive behavioural names and remain understandable without +the development specification. Parameterization will be used for genuine rule +tables, without merging unrelated behaviours merely to reduce the test count. +Fixtures will be synthetic, small, non-sensitive, and deterministically +reproducible. + +No implementation difficulty will be resolved by silently changing the +specification or a schema. Any conflict, ambiguity, or impracticable requirement +will stop the affected step and be presented as a decision before implementation +continues. + +### Approval packet for every step + +The review at the end of each step will include: + +- the implemented capability and its architectural boundaries; +- the specified behaviours and validation scenarios addressed; +- tests and fixtures added or changed; +- commands run and their results; +- any schema, packaging, static, or documentation checks performed; +- a focused demonstration using representative inputs; +- known limitations and behaviour intentionally deferred to later steps; and +- any questions or proposed deviations requiring a decision. + +Approval means that the increment becomes the baseline for subsequent work. A +review may instead request corrections within the same step; those corrections +are completed and re-presented before the next step begins. + +## 3. Implementation sequence + +### Step 0 — Establish the verified baseline and test architecture + +**Goal:** create a trustworthy starting point without implementing comparison +behaviour. + +Work in this step: + +- verify that the existing specification and both JSON Schemas can be loaded; +- verify their canonical `$id` values and internal schema references; +- define the test layout, behavioural naming convention, shared fixture builders, + and deterministic fixture-generation approach; +- add schema-validation helpers that use Draft 2020-12 with format assertion; +- establish tests for loading packaged schemas through the package-resource API; +- build the source distribution and wheel and inspect their contents; and +- record any source/schema conflict as a blocker rather than choosing an + interpretation. + +This step does not create a comparison engine or claim that the executable is +usable. + +**Approval demonstration:** the schemas validate themselves and representative +schema fixtures, packaged schema resources can be loaded, and both distribution +formats build reproducibly enough for inspection. + +### Step 1 — Define package boundaries, domain vocabulary, and CLI invocation + +**Goal:** establish stable outer interfaces before domain behaviour is added. + +Work in this step: + +- create the intended package/module boundaries for input, configuration, + translation, comparison, results, reporting, and CLI orchestration; +- define narrowly scoped domain value types and exception categories at those + boundaries, without speculative framework abstractions; +- implement parsing of the direct `adviso-compare` command form into an + invocation value object; +- implement required arguments, `--configuration`, `--overwrite`, help, and + invalid-invocation handling that does not access inputs; and +- keep orchestration injectable so CLI tests need not depend on private kernel + details. + +The executable will not be presented as end-to-end complete in this increment. +The installed entry point and complete diagnostic behaviour are closed in Step +11. + +**Approval demonstration:** accepted arguments produce the expected invocation +object, invalid combinations fail before file access, subcommands are rejected, +and help accurately describes the version 1 interface. + +### Step 2 — Implement deterministic input discovery and parsing + +**Goal:** turn reference and query files into lossless parser-native datasets +with provenance, without yet interpreting configured field types. + +Work in this step: + +- determine CSV, TSV, and JSON formats independently from filename extensions; +- read exact input bytes, calculate SHA-256 digests, and retain basenames; +- enforce UTF-8 and the fixed tabular dialects; +- parse tabular headers and records while retaining record and physical-line + locations; +- parse JSON arrays of objects while rejecting duplicate member names, + non-object records, non-finite numbers, and invalid root structures; +- preserve parser-native categories, exact decimal values, numerical lexical + evidence, presence state, and JSON Pointer locations; +- distinguish valid empty datasets from invalid zero-byte inputs; and +- implement the applicable input-error taxonomy without logging sensitive + record content. + +**Approval demonstration:** a table-driven parser suite shows accepted edge +cases and precise rejection of malformed inputs across all three formats, with +exact categories and source locations visible through the domain API. + +### Step 3 — Load, validate, and resolve custom comparison configurations + +**Goal:** convert a custom YAML document into one validated, immutable effective +configuration before either dataset is read. + +Work in this step: + +- load YAML safely using the required scalar resolution and duplicate-key + handling; +- validate structure with the packaged configuration schema; +- implement every cross-property semantic validation rule that JSON Schema + cannot express; +- report precise configuration paths and stable configuration error categories; +- expand only the documented defaults and omitted empty structures; +- validate field definitions, source-field correspondence, in-scope fields, + record keys, logical types, collection representations, mappings, + normalisations, comparators, and review priorities; and +- serialize the resolved configuration into its schema-valid embedded form. + +**Approval demonstration:** representative valid configurations resolve to the +expected effective model, while a parameterized semantic-error suite produces +precise paths and reasons. A test proves that custom configuration failure +precedes dataset access. + +### Step 4 — Implement the source-to-canonical translation pipeline + +**Goal:** translate an individual declared field value from either dataset into +canonical typed evidence using the exact configured sequence of operations. + +Work in this step: + +- implement presence and null-policy handling; +- implement closed value mappings with typed source descriptors; +- implement source-text and typed-value normalisation registries; +- implement declared scalar parsing for string, integer, decimal, boolean, + date, and datetime values using exact arithmetic where required; +- implement list, set, and multiset source representations, element processing, + duplicate semantics, and deterministic canonical ordering; +- retain source and canonical evidence needed by later findings; and +- distinguish valid values, invalid attributable values, prohibited nulls, and + deliberately out-of-scope content without yet deciding whole-run status. + +**Approval demonstration:** table-driven unit tests trace representative scalar +and collection values through every configured operation, including failure +states, with no implicit coercion or inference. + +### Step 5 — Resolve record identity and automatic strict-exact mode + +**Goal:** obtain an unambiguous ordered record key for each dataset in custom and +configuration-free operation. + +Work in this step: + +- process configured single and composite record keys through the same explicit + translation rules that apply to their declared fields; +- detect missing, null, invalid, and duplicate keys as execution errors without + overwriting records; +- generate the automatic strict-exact effective configuration after parsing and + dataset-wide field discovery; +- apply the specified `record_id` preference and conservative single-field key + discovery algorithm, including its no-fallback rule; +- generate same-name non-key field definitions and one-sided exclusions without + type inference, mapping, normalisation, or coercion; and +- validate the generated configuration before comparison. + +**Approval demonstration:** custom aliases and composite keys work; duplicate or +invalid keys fail with attributable locations; and automatic-mode fixtures show +deterministic success, zero-candidate failure, ambiguous-candidate failure, and +the special handling of shared `record_id`. + +### Step 6 — Implement the contract phase + +**Goal:** assess whether the declared correspondence can be applied to every +record while allowing unaffected content to proceed. + +Work in this step: + +- discover dataset-wide structure from tabular headers and the union of JSON + members; +- apply required/optional field rules independently on each side; +- classify invalid, missing, null, prohibited, and type-incompatible configured + values; +- create side-specific contract findings and causal not-assessable states; +- consolidate defined-but-out-of-scope and undeclared-source-field exclusions; +- attach resolved review priority where applicable; and +- mark the contract phase complete when it was fully performed, even when it + contains findings. + +**Approval demonstration:** a mixed dataset produces all expected contract +findings, exclusions, and blocked opportunities while still exposing unaffected +records and values to later phases. + +### Step 7 — Implement the records phase + +**Goal:** align records by canonical identity and report the complete set of +reference-only and query-only records. + +Work in this step: + +- construct collision-safe indexes for ordered single and composite keys; +- match record identities independently of input order; +- emit every missing-record and extra-record difference; +- retain the reference and query source identities required for readable + evidence; and +- reconcile zero-record, one-sided-empty, and mixed missing/extra cases. + +**Approval demonstration:** reordered datasets match identically, composite keys +preserve multiple records for a sample, and simultaneous missing and extra +records are all reported deterministically. + +### Step 8 — Implement comparators and the values phase + +**Goal:** compare every assessable in-scope non-key value and distinguish true +post-translation differences from assessment issues. + +Work in this step: + +- implement the fixed comparator registry and legal type/comparator + combinations; +- implement preserved, typed, normalized-string, numeric, date, datetime, list, + set, and multiset equality semantics; +- implement absolute or reference-directed relative numeric tolerance with exact + decimal arithmetic and inclusive boundaries; +- classify every value opportunity as matched, mismatched, not assessable, or + deliberately not assessed as specified; +- collect all assessable differences rather than stopping at the first; and +- preserve the independence of finding classification, phase, and review + priority. + +**Approval demonstration:** a matrix covering every comparator and its boundary +conditions produces the expected canonical outcomes, and one mixed comparison +accounts for every configured value opportunity. + +### Step 9 — Assemble and validate canonical results + +**Goal:** construct the complete canonical result document deterministically and +prove that it satisfies both structural and semantic invariants. + +Work in this step: + +- finalize the finding registry, fixed classifications, structured fields, + deterministic messages, coalescence, total order, and report-local IDs; +- derive phase completion and overall `complete` or `incomplete` status; +- build fixed-shape, zero-inclusive summaries and exclusion summaries; +- render record identities, values, locations, software/runtime identity, + dialect metadata, UUID, and completion timestamp in their prescribed forms; +- embed the effective configuration and exact input provenance; +- serialize canonical JSON using the required member order, Unicode handling, + indentation, line endings, and final newline; +- validate the generated result against the packaged results schema with format + assertion; and +- implement additional semantic self-validation and treat self-validation + failure as an internal error. + +**Approval demonstration:** representative complete and incomplete documents +validate structurally and semantically, summaries reconcile with findings and +opportunities, and repeated/order-varied runs differ only in the explicitly +permitted runtime and location evidence. + +### Step 10 — Produce the evidence package and HTML report + +**Goal:** publish a safe, atomic, human-readable two-file evidence package whose +canonical source is the validated JSON result. + +Work in this step: + +- generate `comparison-report.html` solely from canonical JSON; +- present completeness, comparison differences, assessment issues, phase + traceability, exclusions, provenance, and sensitive-content warnings without + an acceptance verdict; +- provide accessible priority presentation that does not rely on colour alone; +- implement exact destination and overwrite rules, including rejection of + symlinks, special objects, partial packages, and directories with extra + entries; +- stage and replace output atomically while preserving an existing package after + failed replacement; +- apply restrictive temporary-file handling and cleanup; and +- keep operation wholly offline and avoid exposing identifiers or values in + routine terminal output. + +**Approval demonstration:** complete and incomplete sample packages contain +exactly the two required files; the HTML can be regenerated from JSON alone; and +fault-injection tests demonstrate safe creation, refusal, overwrite, rollback, +permissions, and cleanup. + +### Step 11 — Integrate the installed CLI and execution-error boundary + +**Goal:** deliver the complete product path from installed executable to +evidence package, with stable exit and diagnostic behaviour. + +Work in this step: + +- connect the CLI invocation, parsers, configuration resolution, comparison + kernel, result construction, and evidence writer; +- implement normal standard-output summaries and optional progress without + contaminating machine-readable evidence; +- map every expected invocation, configuration, input, resource, and output + failure to its specified diagnostic and exit code; +- isolate unexpected internal failures, omit ordinary-use tracebacks, and + publish no partial report for execution errors; +- confirm that comparison findings still produce exit code 0 when assessment is + complete, and that not-assessable content produces the documented incomplete + result and exit code; +- build and install the wheel in an isolated environment; and +- run representative custom and automatic comparisons through the installed + `adviso-compare` executable without network access. + +**Approval demonstration:** an installed wheel completes end-to-end successful, +difference-containing, incomplete, and expected-error runs with the precise +files, diagnostics, and exit codes required by the specification. + +### Step 12 — Complete user documentation and release-candidate verification + +**Goal:** make the implemented version understandable, reproducible, and ready +for a release decision. + +Work in this step: + +- write the offline user guide covering installation, supported inputs, custom + and automatic operation, the complete configuration model, canonical result + fields, HTML interpretation, exit behaviour, security cautions, and worked + examples; +- validate every complete configuration and result example against its packaged + schema; +- verify every documented command against the installed interface; +- document deterministic fixture regeneration; +- run the complete behavioural acceptance suite; +- run all configured schema, packaging, static, and documentation checks across + the supported environments available to the project; and +- produce a final verification report that distinguishes automated, + inspection-based, process, and release-time evidence. + +This step does not silently introduce a formatter, linter, type checker, +dependency manager, CI system, or release policy. Any new project-wide tooling +or policy identified as desirable is proposed separately for approval. + +**Approval demonstration:** a fresh user can install the built artefact offline, +follow the packaged guide, run both documented modes, validate the resulting +canonical JSON, interpret the HTML, and reproduce the controlled fixtures. The +final review identifies any requirements that remain dependent on manual or +release-process evidence. + +## 4. Recommended review boundaries + +The steps are intentionally ordered by dependency. In particular: + +- result and report design is not allowed to drive undocumented changes to the + comparison model; +- automatic mode is built on the same validated effective-configuration model as + custom mode; +- the HTML report is built from canonical JSON rather than from private engine + state; and +- installed-product testing follows component integration, but packaging and + schema-resource checks begin at the baseline so problems are not deferred. + +If a step proves too large for effective review, it may be divided into smaller +approval checkpoints without changing its scope or starting work assigned to a +later step. Conversely, steps should not be merged merely for speed: the purpose +of the sequence is to make behavioural and architectural corrections inexpensive +before later components depend on them. + +## 5. Definition of version 1 implementation complete + +Implementation is complete only when: + +- the baseline step and all twelve implementation steps have been approved; +- the installed executable satisfies the normative specification and schemas; +- every acceptance-catalogue scenario passes in its intended test environment; +- tests and fixtures remain understandable without the development specification; +- generated configuration and result documents pass structural and semantic + validation; +- the built distributions contain all required code, schemas, and offline + documentation; +- no known inconsistency remains between implementation, tests, schemas, and + documentation; and +- specified behaviours not directly automated are explicitly accounted for by + inspection, packaging, environment, or release-process evidence. diff --git a/packages/adviso-compare/docs/SPECIFICATION.md b/packages/adviso-compare/docs/SPECIFICATION.md index 72f9d24..b8a56b0 100644 --- a/packages/adviso-compare/docs/SPECIFICATION.md +++ b/packages/adviso-compare/docs/SPECIFICATION.md @@ -1531,8 +1531,8 @@ Error classes represented by exit codes 2 and 3 have run status `error`. The dis **REQ-TST-004.** Controlled test cases SHOULD record: -- **REQ-TST-020.** stable scenario identifier; -- **REQ-TST-021.** requirement(s) tested; +- **REQ-TST-020.** stable, descriptive test or fixture name independent of the identifiers used in this specification; +- **REQ-TST-021.** behaviour under test, expressed in domain terms that remain meaningful without this specification; - **REQ-TST-022.** purpose and rationale; - **REQ-TST-023.** input data and comparison configuration; - **REQ-TST-024.** expected findings, counts, run status, per-phase statuses, and exit code; @@ -1546,6 +1546,8 @@ Error classes represented by exit codes 2 and 3 have run status `error`. The dis **REQ-TST-007.** Controlled test fixtures, including test comparison configurations, MUST be maintained by the development team under version control and its documented review process. +**REQ-TST-035.** Traceability to this specification MUST be one-way. The acceptance-test catalogue and development review records MAY map specified requirements to tests, but production code, test code, test names, markers, fixtures, snapshots, package metadata, and runtime artefacts MUST NOT embed specification requirement identifiers or acceptance-scenario identifiers. Archiving or removing this development specification MUST NOT require changes to those software artefacts. + In operational use, each laboratory is responsible for creating, approving, maintaining, and retaining its own comparison configurations under its local quality-management arrangements. The AdvISO project does not centrally approve or manage laboratory configurations. ### 13.3 Continuous integration @@ -1606,7 +1608,7 @@ In operational use, each laboratory is responsible for creating, approving, main ## 14. Acceptance-test catalogue -This catalogue summarises minimum executable scenarios for the first release. It verifies, but does not independently define, the normative behaviour above. Every scenario lists the requirements it verifies. +This catalogue summarises minimum executable scenarios for the first release. It verifies, but does not independently define, the normative behaviour above. Every scenario lists the requirements it verifies. The scenario and requirement identifiers are development traceability within this specification; they are not identifiers for use in test or runtime artefacts. ### 14.1 Input and contract scenarios diff --git a/packages/adviso-compare/pyproject.toml b/packages/adviso-compare/pyproject.toml index b6d9c51..dd32a12 100644 --- a/packages/adviso-compare/pyproject.toml +++ b/packages/adviso-compare/pyproject.toml @@ -37,6 +37,9 @@ dependencies = [ dev = [ "build>=1.2,<2", "pytest>=8,<9", + "setuptools>=69", + "setuptools-scm>=8", + "wheel", ] [project.scripts] diff --git a/packages/adviso-compare/tests/README.md b/packages/adviso-compare/tests/README.md new file mode 100644 index 0000000..1b156a7 --- /dev/null +++ b/packages/adviso-compare/tests/README.md @@ -0,0 +1,17 @@ +# Version 1 test conventions + +Tests use descriptive behavioural names and assertions that remain meaningful +without the development specification. Test modules, names, markers, fixtures, +and snapshots do not embed specification requirement identifiers or acceptance +scenario identifiers. During development, any formal traceability is maintained +in the specification and review records as a one-way reference to the tests. + +Reusable synthetic documents are built in `tests/support/documents.py`. Builders +return new objects on every call so tests can modify a document without leaking +state into another test. Fixed values are intentional: generated fixtures must +be deterministic and reviewable. A future generated on-disk fixture must record +its generator and regeneration command alongside the fixture. + +Schema loading and validator construction live in `tests/support/schemas.py`. +Canonical-results validators register the packaged comparison-configuration +schema explicitly and enable JSON Schema format assertion. diff --git a/packages/adviso-compare/tests/support/__init__.py b/packages/adviso-compare/tests/support/__init__.py new file mode 100644 index 0000000..df1f10d --- /dev/null +++ b/packages/adviso-compare/tests/support/__init__.py @@ -0,0 +1,2 @@ +"""Reusable support for the AdvISO Compare test suite.""" + diff --git a/packages/adviso-compare/tests/support/documents.py b/packages/adviso-compare/tests/support/documents.py new file mode 100644 index 0000000..46ca759 --- /dev/null +++ b/packages/adviso-compare/tests/support/documents.py @@ -0,0 +1,172 @@ +"""Deterministic synthetic documents used by schema-level tests.""" + +from __future__ import annotations + +from typing import Any, Literal + + +JsonObject = dict[str, Any] + + +def minimal_configuration() -> JsonObject: + """Return a fresh, structurally valid comparison configuration.""" + return { + "schema_version": "1.0", + "metadata": { + "id": "org.example.synthetic-exact-comparison", + "revision": "1.0.0", + }, + "fields": { + "record_key": ["record_id"], + "in_scope": ["record_id", "measured_value"], + "definitions": { + "record_id": { + "source_fields": { + "reference": "reference_id", + "query": "query_id", + }, + "required": {"reference": True, "query": True}, + "logical_type": "string", + "null_policy": {"allowed": False, "nulls_equal": False}, + "comparator": {"name": "typed_exact"}, + }, + "measured_value": { + "source_fields": { + "reference": "expected_value", + "query": "observed_value", + }, + "required": {"reference": True, "query": True}, + "logical_type": "decimal", + "null_policy": {"allowed": False, "nulls_equal": False}, + "comparator": {"name": "numeric_exact"}, + }, + }, + }, + } + + +def minimal_result( + *, status: Literal["complete", "incomplete"] = "complete" +) -> JsonObject: + """Return a fresh, structurally valid canonical result.""" + zero_by_code = { + "missing_record": 0, + "extra_record": 0, + "missing_required_field": 0, + "type_mismatch": 0, + "invalid_field_value": 0, + "null_policy_violation": 0, + "mapping_failure": 0, + "value_mismatch": 0, + } + records = { + "reference": 0, + "query": 0, + "matched": 0, + "missing": 0, + "extra": 0, + } + values = {"matched": 0, "mismatched": 0, "not_assessable": 0} + findings: list[JsonObject] = [] + phases = { + "contract": "complete", + "records": "complete", + "values": "complete", + } + + if status == "incomplete": + records = { + "reference": 1, + "query": 1, + "matched": 1, + "missing": 0, + "extra": 0, + } + values = {"matched": 0, "mismatched": 0, "not_assessable": 1} + zero_by_code["missing_required_field"] = 1 + phases["values"] = "incomplete" + findings = [ + { + "id": "F000001", + "classification": "assessment_issue", + "code": "missing_required_field", + "phase": "contract", + "message": "Required query field observed_value is absent.", + "affected_side": "query", + "record_identity": { + "reference": ["sample-1"], + "query": ["sample-1"], + "canonical": ["sample-1"], + }, + "source_locations": { + "reference": {"json_pointer": "/0/expected_value"}, + "query": {"json_pointer": "/0"}, + }, + "field": "measured_value", + "review_priority": "normal", + "values": { + "reference": {"state": "present"}, + "query": {"state": "absent"}, + }, + } + ] + + return { + "results_schema_version": "1.0", + "run": { + "id": "123e4567-e89b-42d3-a456-426614174000", + "status": status, + "completed_at": "2026-01-02T03:04:05Z", + }, + "phases": { + **phases, + }, + "software": {"name": "adviso-compare", "version": "1.0.0"}, + "runtime": { + "python": "cpython-3.10.12", + "platform": "linux-x86_64", + "unicode_version": "14.0.0", + }, + "inputs": { + "reference": { + "basename": "reference.json", + "format": "json", + "encoding": "utf-8", + "size_bytes": 2, + "digest": {"algorithm": "sha256", "value": "0" * 64}, + }, + "query": { + "basename": "query.json", + "format": "json", + "encoding": "utf-8", + "size_bytes": 2, + "digest": {"algorithm": "sha256", "value": "1" * 64}, + }, + }, + "record_key_fields": { + "reference": ["reference_id"], + "query": ["query_id"], + "canonical": ["record_id"], + }, + "comparison_configuration": {"document": minimal_configuration()}, + "summary": { + "records": records, + "values": values, + "findings": { + "total": len(findings), + "by_classification": { + "comparison_difference": 0, + "assessment_issue": len(findings), + }, + "by_code": zero_by_code, + "by_review_priority": { + "low": 0, + "normal": len(findings), + "high": 0, + "unassigned": 0, + }, + }, + }, + "exclusions": [], + "findings": findings, + } diff --git a/packages/adviso-compare/tests/support/schemas.py b/packages/adviso-compare/tests/support/schemas.py new file mode 100644 index 0000000..b703517 --- /dev/null +++ b/packages/adviso-compare/tests/support/schemas.py @@ -0,0 +1,52 @@ +"""Helpers for loading and validating packaged version 1 schemas.""" + +from __future__ import annotations + +import json +from importlib.resources import files +from typing import Any + +from jsonschema import Draft202012Validator, FormatChecker +from referencing import Registry, Resource + + +CONFIGURATION_SCHEMA_FILE = "comparison-configuration-v1.0.schema.json" +RESULTS_SCHEMA_FILE = "comparison-results-v1.0.schema.json" +SCHEMA_PACKAGE = "adviso_compare.schemas" + +JsonObject = dict[str, Any] + + +def load_packaged_schema(filename: str) -> JsonObject: + """Load a schema through the installed-package resource interface.""" + resource = files(SCHEMA_PACKAGE).joinpath(filename) + return json.loads(resource.read_text(encoding="utf-8")) + + +def schema_registry(*schemas: JsonObject) -> Registry: + """Return a registry containing each schema under its canonical identifier.""" + registry = Registry() + for schema in schemas: + registry = registry.with_resource( + schema["$id"], Resource.from_contents(schema) + ) + return registry + + +def configuration_validator() -> Draft202012Validator: + """Return the version 1 comparison-configuration validator.""" + schema = load_packaged_schema(CONFIGURATION_SCHEMA_FILE) + return Draft202012Validator(schema, format_checker=FormatChecker()) + + +def results_validator() -> Draft202012Validator: + """Return the version 1 canonical-results validator with references resolved.""" + configuration = load_packaged_schema(CONFIGURATION_SCHEMA_FILE) + results = load_packaged_schema(RESULTS_SCHEMA_FILE) + registry = schema_registry(configuration, results) + return Draft202012Validator( + results, + registry=registry, + format_checker=FormatChecker(), + ) + diff --git a/packages/adviso-compare/tests/test_distribution.py b/packages/adviso-compare/tests/test_distribution.py new file mode 100644 index 0000000..b3fbf47 --- /dev/null +++ b/packages/adviso-compare/tests/test_distribution.py @@ -0,0 +1,112 @@ +"""Baseline checks for schema resources in built distributions.""" + +from __future__ import annotations + +import subprocess +import sys +import tarfile +import venv +import zipfile +from email.parser import BytesParser +from pathlib import Path + +REQUIRED_PACKAGE_MEMBERS = { + "adviso_compare/__init__.py", + "adviso_compare/schemas/__init__.py", + "adviso_compare/schemas/comparison-configuration-v1.0.schema.json", + "adviso_compare/schemas/comparison-results-v1.0.schema.json", +} + + +def _relative_sdist_members(archive: Path) -> set[str]: + """Return source-distribution members without its generated root directory.""" + with tarfile.open(archive, mode="r:gz") as distribution: + members = {member.name for member in distribution.getmembers()} + + roots = {member.partition("/")[0] for member in members} + assert len(roots) == 1 + root = roots.pop() + return { + member.removeprefix(f"{root}/") + for member in members + if member != root + } + + +def _installed_python(environment: Path) -> Path: + """Return the Python executable created by ``venv`` on this platform.""" + executable = "python.exe" if sys.platform == "win32" else "python" + return environment / ("Scripts" if sys.platform == "win32" else "bin") / executable + + +def test_built_distributions_contain_import_package_and_schemas( + tmp_path: Path, +) -> None: + """Both distribution formats carry the import package and normative schemas.""" + project_root = Path(__file__).resolve().parents[1] + output = tmp_path / "dist" + subprocess.run( + [ + sys.executable, + "-m", + "build", + "--no-isolation", + "--outdir", + str(output), + str(project_root), + ], + check=True, + capture_output=True, + text=True, + ) + + wheel = next(output.glob("*.whl")) + source_distribution = next(output.glob("*.tar.gz")) + + with zipfile.ZipFile(wheel) as distribution: + wheel_members = set(distribution.namelist()) + metadata_member = next( + member + for member in wheel_members + if member.endswith(".dist-info/METADATA") + ) + metadata = BytesParser().parsebytes(distribution.read(metadata_member)) + sdist_members = _relative_sdist_members(source_distribution) + + assert metadata["Name"] == "adviso-compare" + assert REQUIRED_PACKAGE_MEMBERS <= wheel_members + assert { + f"src/{member}" for member in REQUIRED_PACKAGE_MEMBERS + } <= sdist_members + + environment = tmp_path / "installed" + venv.EnvBuilder(with_pip=True).create(environment) + installed_python = _installed_python(environment) + subprocess.run( + [ + str(installed_python), + "-m", + "pip", + "install", + "--no-deps", + str(wheel), + ], + check=True, + capture_output=True, + text=True, + ) + resource_check = ( + "import json; " + "from importlib.resources import files; " + "root = files('adviso_compare.schemas'); " + "names = ['comparison-configuration-v1.0.schema.json', " + "'comparison-results-v1.0.schema.json']; " + "assert all(json.loads(root.joinpath(name).read_text(encoding='utf-8')) " + "for name in names)" + ) + subprocess.run( + [str(installed_python), "-c", resource_check], + check=True, + capture_output=True, + text=True, + ) diff --git a/packages/adviso-compare/tests/test_schemas.py b/packages/adviso-compare/tests/test_schemas.py new file mode 100644 index 0000000..47609aa --- /dev/null +++ b/packages/adviso-compare/tests/test_schemas.py @@ -0,0 +1,89 @@ +"""Baseline verification of the authoritative version 1 schemas.""" + +from __future__ import annotations + +from copy import deepcopy + +import pytest +from jsonschema import Draft202012Validator + +from tests.support.documents import minimal_configuration, minimal_result +from tests.support.schemas import ( + CONFIGURATION_SCHEMA_FILE, + RESULTS_SCHEMA_FILE, + configuration_validator, + load_packaged_schema, + results_validator, +) + + +CONFIGURATION_SCHEMA_ID = ( + "https://adviso-project.github.io/validation_toolkit/schemas/" + "comparison-configuration-v1.0.schema.json" +) +RESULTS_SCHEMA_ID = ( + "https://adviso-project.github.io/validation_toolkit/schemas/" + "comparison-results-v1.0.schema.json" +) + + +@pytest.mark.parametrize( + ("filename", "canonical_id"), + [ + (CONFIGURATION_SCHEMA_FILE, CONFIGURATION_SCHEMA_ID), + (RESULTS_SCHEMA_FILE, RESULTS_SCHEMA_ID), + ], +) +def test_authoritative_schemas_are_packaged_valid_draft_2020_12_documents( + filename: str, + canonical_id: str, +) -> None: + """The source-of-truth schemas are loadable resources with stable identities.""" + schema = load_packaged_schema(filename) + + assert schema["$schema"] == "https://json-schema.org/draft/2020-12/schema" + assert schema["$id"] == canonical_id + Draft202012Validator.check_schema(schema) + + +@pytest.mark.parametrize("status", ["complete", "incomplete"]) +def test_representative_documents_validate_with_reference_and_format_assertion( + status: str, +) -> None: + """Representative documents exercise both schemas and their external reference.""" + configuration = minimal_configuration() + result = minimal_result(status=status) + + configuration_validator().validate(configuration) + results_validator().validate(result) + + +def test_results_validation_asserts_recognised_formats() -> None: + """A lexically plausible but impossible RFC 3339 date-time is invalid.""" + result = minimal_result() + invalid_result = deepcopy(result) + invalid_result["run"]["completed_at"] = "2026-13-40T03:04:05Z" + + errors = list(results_validator().iter_errors(invalid_result)) + + assert errors + assert any(error.validator == "format" for error in errors) + + +def test_synthetic_document_builders_are_deterministic_and_independent() -> None: + """Generated documents are reproducible without sharing mutable state.""" + first_configuration = minimal_configuration() + second_configuration = minimal_configuration() + first_result = minimal_result(status="incomplete") + second_result = minimal_result(status="incomplete") + + assert first_configuration == second_configuration + assert first_result == second_result + + first_configuration["metadata"]["id"] = "changed" + first_result["run"]["status"] = "changed" + + assert second_configuration["metadata"]["id"] == ( + "org.example.synthetic-exact-comparison" + ) + assert second_result["run"]["status"] == "incomplete" From 054bbd01a909ed15fef575f351bb189f089fc9fd Mon Sep 17 00:00:00 2001 From: Kevin Howe Date: Fri, 6 Nov 2026 11:53:32 +0000 Subject: [PATCH 15/45] Added command-line invocation layer --- packages/adviso-compare/README.md | 4 +- .../src/adviso_compare/application.py | 13 ++ .../adviso-compare/src/adviso_compare/cli.py | 103 ++++++++++ .../src/adviso_compare/invocation.py | 26 +++ packages/adviso-compare/tests/test_cli.py | 184 ++++++++++++++++++ 5 files changed, 329 insertions(+), 1 deletion(-) create mode 100644 packages/adviso-compare/src/adviso_compare/application.py create mode 100644 packages/adviso-compare/src/adviso_compare/cli.py create mode 100644 packages/adviso-compare/src/adviso_compare/invocation.py create mode 100644 packages/adviso-compare/tests/test_cli.py diff --git a/packages/adviso-compare/README.md b/packages/adviso-compare/README.md index 0e3e745..35bc3b1 100644 --- a/packages/adviso-compare/README.md +++ b/packages/adviso-compare/README.md @@ -6,7 +6,9 @@ The project is intended to support bioinformatics validation and verification ac ## Project status -Version 1 is currently specified but not yet implemented. The package is being prepared for a test-driven implementation. +Version 1 implementation is in progress. The schema-validation baseline and +command-line invocation layer are implemented; input processing, the comparison +kernel, and evidence generation are not yet available. The current normative artefacts are: diff --git a/packages/adviso-compare/src/adviso_compare/application.py b/packages/adviso-compare/src/adviso_compare/application.py new file mode 100644 index 0000000..197bad1 --- /dev/null +++ b/packages/adviso-compare/src/adviso_compare/application.py @@ -0,0 +1,13 @@ +"""Application orchestration boundary for AdvISO Compare.""" + +from __future__ import annotations + +from adviso_compare.invocation import ComparisonInvocation + + +def execute_comparison(invocation: ComparisonInvocation) -> int: + """Execute a comparison once later implementation increments supply the core.""" + raise NotImplementedError( + "comparison execution is not available at this implementation stage" + ) + diff --git a/packages/adviso-compare/src/adviso_compare/cli.py b/packages/adviso-compare/src/adviso_compare/cli.py new file mode 100644 index 0000000..a151ea5 --- /dev/null +++ b/packages/adviso-compare/src/adviso_compare/cli.py @@ -0,0 +1,103 @@ +"""Thin command-line adapter for AdvISO Compare.""" + +from __future__ import annotations + +import argparse +from importlib.metadata import version +from pathlib import Path +from typing import NoReturn, Sequence + +from adviso_compare.invocation import ComparisonExecutor, ComparisonInvocation + + +class _ArgumentParser(argparse.ArgumentParser): + """Argument parser that emits the stable invocation-error diagnostic.""" + + def error(self, message: str) -> NoReturn: + """Report invalid command-line syntax without touching comparison inputs.""" + self.exit(2, f"ERROR [invocation.invalid_arguments]: {message}\n") + + +def _build_parser() -> argparse.ArgumentParser: + """Construct the version 1 direct comparison interface.""" + parser = _ArgumentParser( + prog="adviso-compare", + description=( + "Compare a reference dataset with a query dataset and write a " + "two-file evidence package." + ), + epilog=( + "See the AdvISO Compare user guide for complete usage, input-format, " + "and comparison-configuration documentation." + ), + allow_abbrev=False, + ) + parser.add_argument( + "--version", + action="version", + version=f"%(prog)s {version('adviso-compare')}", + help="show the installed software version and exit", + ) + parser.add_argument( + "--reference", + required=True, + type=Path, + metavar="REFERENCE", + help="reference dataset in CSV, TSV, or JSON format", + ) + parser.add_argument( + "--query", + required=True, + type=Path, + metavar="QUERY", + help="query dataset in CSV, TSV, or JSON format", + ) + parser.add_argument( + "--output", + required=True, + type=Path, + metavar="OUTPUT", + help="destination directory for the evidence package", + ) + parser.add_argument( + "--configuration", + type=Path, + metavar="CONFIGURATION", + help=( + "custom YAML comparison configuration; omit to use automatic " + "strict-exact mode" + ), + ) + parser.add_argument( + "--overwrite", + action="store_true", + help="replace an existing complete version 1 evidence package", + ) + return parser + + +def parse_invocation(arguments: Sequence[str] | None = None) -> ComparisonInvocation: + """Parse command-line text into an immutable comparison invocation.""" + namespace = _build_parser().parse_args(arguments) + return ComparisonInvocation( + reference=namespace.reference, + query=namespace.query, + output=namespace.output, + configuration=namespace.configuration, + overwrite=namespace.overwrite, + ) + + +def main( + arguments: Sequence[str] | None = None, + *, + executor: ComparisonExecutor | None = None, +) -> int: + """Parse one invocation and dispatch it to the comparison application.""" + invocation = parse_invocation(arguments) + if executor is None: + from adviso_compare.application import execute_comparison + + executor = execute_comparison + return executor(invocation) + diff --git a/packages/adviso-compare/src/adviso_compare/invocation.py b/packages/adviso-compare/src/adviso_compare/invocation.py new file mode 100644 index 0000000..b43c3f0 --- /dev/null +++ b/packages/adviso-compare/src/adviso_compare/invocation.py @@ -0,0 +1,26 @@ +"""Domain values at the command-line application boundary.""" + +from __future__ import annotations + +from dataclasses import dataclass +from pathlib import Path +from typing import Protocol + + +@dataclass(frozen=True, slots=True) +class ComparisonInvocation: + """User-selected paths and overwrite authorization for one comparison.""" + + reference: Path + query: Path + output: Path + configuration: Path | None = None + overwrite: bool = False + + +class ComparisonExecutor(Protocol): + """Callable application boundary used by the command-line adapter.""" + + def __call__(self, invocation: ComparisonInvocation) -> int: + """Execute one comparison and return its process exit code.""" + ... diff --git a/packages/adviso-compare/tests/test_cli.py b/packages/adviso-compare/tests/test_cli.py new file mode 100644 index 0000000..f8b2ccd --- /dev/null +++ b/packages/adviso-compare/tests/test_cli.py @@ -0,0 +1,184 @@ +"""Behavioural tests for the direct AdvISO Compare command line.""" + +from __future__ import annotations + +from dataclasses import FrozenInstanceError +from importlib.metadata import version +from pathlib import Path + +import pytest + +from adviso_compare.cli import main, parse_invocation +from adviso_compare.invocation import ComparisonInvocation + + +@pytest.mark.parametrize( + ("optional_arguments", "expected_configuration", "expected_overwrite"), + [ + ([], None, False), + (["--configuration", "rules.yaml", "--overwrite"], Path("rules.yaml"), True), + ], +) +def test_direct_arguments_create_an_immutable_invocation_without_reading_inputs( + optional_arguments: list[str], + expected_configuration: Path | None, + expected_overwrite: bool, +) -> None: + """Parsing records paths and flags without requiring any path to exist.""" + arguments = [ + "--reference", + "does-not-exist-reference.csv", + "--query", + "does-not-exist-query.json", + "--output", + "does-not-exist-output", + *optional_arguments, + ] + + invocation = parse_invocation(arguments) + + assert invocation == ComparisonInvocation( + reference=Path("does-not-exist-reference.csv"), + query=Path("does-not-exist-query.json"), + output=Path("does-not-exist-output"), + configuration=expected_configuration, + overwrite=expected_overwrite, + ) + with pytest.raises(FrozenInstanceError): + invocation.overwrite = not expected_overwrite # type: ignore[misc] + + +def test_main_dispatches_the_invocation_through_an_injected_executor() -> None: + """CLI orchestration passes one domain value to an independently testable core.""" + received: list[ComparisonInvocation] = [] + + def execute(invocation: ComparisonInvocation) -> int: + received.append(invocation) + return 17 + + exit_code = main( + [ + "--reference", + "reference.csv", + "--query", + "query.tsv", + "--output", + "evidence", + ], + executor=execute, + ) + + assert exit_code == 17 + assert received == [ + ComparisonInvocation( + reference=Path("reference.csv"), + query=Path("query.tsv"), + output=Path("evidence"), + ) + ] + + +@pytest.mark.parametrize("missing_option", ["--reference", "--query", "--output"]) +def test_each_required_argument_is_rejected_before_execution( + missing_option: str, + capsys: pytest.CaptureFixture[str], +) -> None: + """Missing required paths are concise invocation errors and never reach the core.""" + arguments = [ + "--reference", + "reference.csv", + "--query", + "query.csv", + "--output", + "evidence", + ] + option_index = arguments.index(missing_option) + del arguments[option_index : option_index + 2] + + def unexpected_executor(invocation: ComparisonInvocation) -> int: + raise AssertionError(f"executor received invalid invocation: {invocation!r}") + + with pytest.raises(SystemExit) as error: + main(arguments, executor=unexpected_executor) + + captured = capsys.readouterr() + assert error.value.code == 2 + assert captured.out == "" + assert captured.err.startswith("ERROR [invocation.invalid_arguments]: ") + assert missing_option in captured.err + assert captured.err.count("\n") == 1 + + +def test_subcommand_syntax_is_rejected_before_execution( + capsys: pytest.CaptureFixture[str], +) -> None: + """The executable accepts comparison options directly and has no subcommands.""" + arguments = [ + "compare", + "--reference", + "reference.csv", + "--query", + "query.csv", + "--output", + "evidence", + ] + + def unexpected_executor(invocation: ComparisonInvocation) -> int: + raise AssertionError(f"executor received invalid invocation: {invocation!r}") + + with pytest.raises(SystemExit) as error: + main(arguments, executor=unexpected_executor) + + captured = capsys.readouterr() + assert error.value.code == 2 + assert captured.out == "" + assert captured.err == ( + "ERROR [invocation.invalid_arguments]: unrecognized arguments: compare\n" + ) + + +def test_help_documents_the_complete_direct_interface_and_user_guide( + capsys: pytest.CaptureFixture[str], +) -> None: + """Help works alone and describes every version 1 command-line option.""" + + def unexpected_executor(invocation: ComparisonInvocation) -> int: + raise AssertionError(f"executor was called for help: {invocation!r}") + + with pytest.raises(SystemExit) as error: + main(["--help"], executor=unexpected_executor) + + captured = capsys.readouterr() + assert error.value.code == 0 + assert captured.err == "" + assert "usage: adviso-compare" in captured.out + for option in ( + "--reference", + "--query", + "--output", + "--configuration", + "--overwrite", + "--help", + "--version", + ): + assert option in captured.out + assert "user guide" in captured.out.lower() + assert "subcommand" not in captured.out.lower() + + +def test_version_works_without_comparison_arguments( + capsys: pytest.CaptureFixture[str], +) -> None: + """Version reporting exits successfully without dispatching a comparison.""" + + def unexpected_executor(invocation: ComparisonInvocation) -> int: + raise AssertionError(f"executor was called for version: {invocation!r}") + + with pytest.raises(SystemExit) as error: + main(["--version"], executor=unexpected_executor) + + captured = capsys.readouterr() + assert error.value.code == 0 + assert captured.err == "" + assert captured.out == f"adviso-compare {version('adviso-compare')}\n" + From 453db3775807ec7f7bbe9cf623abfa07b2a4fc8e Mon Sep 17 00:00:00 2001 From: Kevin Howe Date: Fri, 6 Nov 2026 12:22:13 +0000 Subject: [PATCH 16/45] Added input parsing (supporting JSON, and CSV/TSV as input formats) --- packages/adviso-compare/README.md | 7 +- .../src/adviso_compare/inputs.py | 442 ++++++++++++++++++ packages/adviso-compare/tests/test_inputs.py | 272 +++++++++++ 3 files changed, 718 insertions(+), 3 deletions(-) create mode 100644 packages/adviso-compare/src/adviso_compare/inputs.py create mode 100644 packages/adviso-compare/tests/test_inputs.py diff --git a/packages/adviso-compare/README.md b/packages/adviso-compare/README.md index 35bc3b1..ef8432e 100644 --- a/packages/adviso-compare/README.md +++ b/packages/adviso-compare/README.md @@ -6,9 +6,10 @@ The project is intended to support bioinformatics validation and verification ac ## Project status -Version 1 implementation is in progress. The schema-validation baseline and -command-line invocation layer are implemented; input processing, the comparison -kernel, and evidence generation are not yet available. +Version 1 implementation is in progress. The schema-validation baseline, +command-line invocation layer, and deterministic CSV, TSV, and JSON input parser +are implemented. Comparison-configuration processing, the comparison kernel, +and evidence generation are not yet available. The current normative artefacts are: diff --git a/packages/adviso-compare/src/adviso_compare/inputs.py b/packages/adviso-compare/src/adviso_compare/inputs.py new file mode 100644 index 0000000..a94906e --- /dev/null +++ b/packages/adviso-compare/src/adviso_compare/inputs.py @@ -0,0 +1,442 @@ +"""Deterministic, lossless parsing of supported comparison inputs.""" + +from __future__ import annotations + +import csv +import io +import json +from dataclasses import dataclass, field +from decimal import Decimal +from enum import Enum +from hashlib import sha256 +from pathlib import Path +from typing import TypeAlias + + +class InputFormat(str, Enum): + """A supported dataset serialization format.""" + + CSV = "csv" + TSV = "tsv" + JSON = "json" + + +class InputSide(str, Enum): + """The role an input dataset has in a comparison.""" + + REFERENCE = "reference" + QUERY = "query" + + +class NativeCategory(str, Enum): + """A value category assigned by the source-format parser.""" + + STRING = "string" + INTEGER = "integer" + DECIMAL = "decimal" + BOOLEAN = "boolean" + NULL = "null" + ARRAY = "array" + OBJECT = "object" + + +@dataclass(frozen=True, slots=True) +class NativeValue: + """A parser-native value with optional numerical lexical evidence. + + Lexical evidence is deliberately excluded from equality: numerical values + in the same category compare by their exact numerical content. + """ + + category: NativeCategory + value: object = None + lexical: str | None = field(default=None, compare=False) + + +@dataclass(frozen=True, slots=True) +class TabularLocation: + """A one-based tabular record and starting physical-line location.""" + + record_number: int + line_number: int + field_name: str | None = None + + +@dataclass(frozen=True, slots=True) +class JsonLocation: + """An RFC 6901 JSON Pointer location.""" + + pointer: str + + +SourceLocation: TypeAlias = TabularLocation | JsonLocation + + +@dataclass(frozen=True, slots=True) +class ParsedRecord: + """One immutable source record and its attributable location.""" + + members: tuple[tuple[str, NativeValue], ...] + location: SourceLocation + + @property + def field_names(self) -> tuple[str, ...]: + """Return source field names in deterministic parser order.""" + return tuple(name for name, _ in self.members) + + def value(self, field_name: str) -> NativeValue: + """Return the present source value for ``field_name``.""" + for name, value in self.members: + if name == field_name: + return value + raise KeyError(field_name) + + def has_field(self, field_name: str) -> bool: + """Return whether the source record contains ``field_name``.""" + return any(name == field_name for name, _ in self.members) + + def location_for(self, field_name: str) -> SourceLocation: + """Return a present field's location, or the record location if absent.""" + if not self.has_field(field_name): + return self.location + if isinstance(self.location, TabularLocation): + return TabularLocation( + record_number=self.location.record_number, + line_number=self.location.line_number, + field_name=field_name, + ) + return JsonLocation( + pointer=f"{self.location.pointer}/{_escape_json_pointer(field_name)}" + ) + + +@dataclass(frozen=True, slots=True) +class InputProvenance: + """Automatically captured evidence about exact input bytes.""" + + basename: str + format: InputFormat + encoding: str + size_bytes: int + sha256: str + + +@dataclass(frozen=True, slots=True) +class ParsedDataset: + """A losslessly parsed dataset before configured interpretation.""" + + provenance: InputProvenance + field_names: tuple[str, ...] + records: tuple[ParsedRecord, ...] + + +@dataclass(frozen=True, slots=True) +class InputError(Exception): + """A classified expected failure while discovering or parsing an input.""" + + code: str + side: InputSide + message: str + + @property + def category(self) -> str: + """Return the stable execution-error category.""" + return "input" + + @property + def diagnostic(self) -> str: + """Render the stable, single-line non-sensitive diagnostic.""" + return f"ERROR [input.{self.code}]: {self.message}" + + def __str__(self) -> str: + """Return the actionable message without exposing input content.""" + return self.message + + +class _DuplicateMember(ValueError): + """Signal a duplicate JSON object member before mapping construction.""" + + +class _NonFiniteNumber(ValueError): + """Signal a non-standard non-finite JSON numerical token.""" + + +def load_dataset(path: Path, side: InputSide) -> ParsedDataset: + """Discover and parse one supported dataset from its exact bytes.""" + input_format = _discover_format(path, side) + try: + content = path.read_bytes() + except OSError as error: + raise InputError( + code="unreadable", + side=side, + message=f"{side.value} input could not be read", + ) from error + + provenance = InputProvenance( + basename=path.name, + format=input_format, + encoding="utf-8", + size_bytes=len(content), + sha256=sha256(content).hexdigest(), + ) + try: + text = content.decode("utf-8", errors="strict") + except UnicodeDecodeError as error: + raise InputError( + code="invalid_encoding", + side=side, + message=f"{side.value} input is not valid UTF-8", + ) from error + + if input_format is InputFormat.JSON: + field_names, records = _parse_json(text, side) + else: + delimiter = "," if input_format is InputFormat.CSV else "\t" + field_names, records = _parse_table(text, delimiter, side) + return ParsedDataset( + provenance=provenance, + field_names=field_names, + records=records, + ) + + +def _discover_format(path: Path, side: InputSide) -> InputFormat: + """Select a format solely from the final case-insensitive suffix.""" + suffix = path.suffix.casefold() + formats = { + ".csv": InputFormat.CSV, + ".tsv": InputFormat.TSV, + ".json": InputFormat.JSON, + } + try: + return formats[suffix] + except KeyError as error: + raise InputError( + code="unsupported_format", + side=side, + message=( + f"{side.value} input must have a .csv, .tsv, or .json " + "filename extension" + ), + ) from error + + +def _parse_table( + text: str, + delimiter: str, + side: InputSide, +) -> tuple[tuple[str, ...], tuple[ParsedRecord, ...]]: + """Parse one CSV or TSV document using the fixed version 1 dialect.""" + reader = csv.reader( + io.StringIO(text, newline=""), + delimiter=delimiter, + quotechar='"', + doublequote=True, + escapechar=None, + strict=True, + ) + try: + header = next(reader) + except StopIteration as error: + raise InputError( + code="invalid_structure", + side=side, + message=f"{side.value} tabular input must contain a header row", + ) from error + except csv.Error as error: + raise _tabular_syntax_error(side, reader.line_num) from error + + if not header or any(name == "" for name in header): + raise InputError( + code="invalid_structure", + side=side, + message=f"{side.value} tabular header contains an empty column name", + ) + if len(set(header)) != len(header): + raise InputError( + code="invalid_structure", + side=side, + message=f"{side.value} tabular header contains duplicate column names", + ) + + records: list[ParsedRecord] = [] + previous_ending_line = reader.line_num + while True: + starting_line = previous_ending_line + 1 + try: + row = next(reader) + except StopIteration: + break + except csv.Error as error: + raise _tabular_syntax_error(side, reader.line_num) from error + ending_line = reader.line_num + record_number = len(records) + 1 + if len(row) != len(header): + raise InputError( + code="invalid_structure", + side=side, + message=( + f"{side.value} tabular data record {record_number} starting " + f"on physical line {starting_line} has {len(row)} fields; " + f"expected {len(header)}" + ), + ) + members = tuple( + (name, NativeValue(NativeCategory.STRING, value)) + for name, value in zip(header, row) + ) + records.append( + ParsedRecord( + members=members, + location=TabularLocation( + record_number=record_number, + line_number=starting_line, + ), + ) + ) + previous_ending_line = ending_line + return tuple(header), tuple(records) + + +def _tabular_syntax_error(side: InputSide, line_number: int) -> InputError: + """Construct a stable syntax error without reproducing parser input.""" + line_context = f" near physical line {line_number}" if line_number else "" + return InputError( + code="invalid_syntax", + side=side, + message=f"{side.value} tabular input has invalid quoted syntax{line_context}", + ) + + +def _parse_json( + text: str, + side: InputSide, +) -> tuple[tuple[str, ...], tuple[ParsedRecord, ...]]: + """Parse a record-oriented JSON array without lossy numerical conversion.""" + + def parse_integer(lexical: str) -> NativeValue: + return NativeValue(NativeCategory.INTEGER, int(lexical), lexical) + + def parse_decimal(lexical: str) -> NativeValue: + return NativeValue(NativeCategory.DECIMAL, Decimal(lexical), lexical) + + def reject_constant(_lexical: str) -> object: + raise _NonFiniteNumber + + def parse_object(pairs: list[tuple[str, object]]) -> NativeValue: + names = [name for name, _ in pairs] + if len(set(names)) != len(names): + raise _DuplicateMember + members = tuple( + sorted( + ((name, _to_native(value)) for name, value in pairs), + key=lambda item: item[0], + ) + ) + return NativeValue(NativeCategory.OBJECT, members) + + try: + root = json.loads( + text, + parse_int=parse_integer, + parse_float=parse_decimal, + parse_constant=reject_constant, + object_pairs_hook=parse_object, + ) + except _NonFiniteNumber as error: + raise InputError( + code="invalid_syntax", + side=side, + message=f"{side.value} JSON input contains a non-finite number", + ) from error + except json.JSONDecodeError as error: + raise InputError( + code="invalid_syntax", + side=side, + message=( + f"{side.value} JSON input has invalid syntax at line " + f"{error.lineno}, column {error.colno}" + ), + ) from error + except _DuplicateMember as error: + raise InputError( + code="invalid_structure", + side=side, + message=f"{side.value} JSON input contains a duplicate object member", + ) from error + + if not isinstance(root, list): + raise InputError( + code="invalid_structure", + side=side, + message=( + f"{side.value} JSON root is {_json_category(root)}; it must be " + "one array of record objects" + ), + ) + + records: list[ParsedRecord] = [] + all_field_names: set[str] = set() + for index, item in enumerate(root): + if ( + not isinstance(item, NativeValue) + or item.category is not NativeCategory.OBJECT + ): + raise InputError( + code="invalid_structure", + side=side, + message=( + f"{side.value} JSON array item {index} is " + f"{_json_category(item)}; every item must be a record object" + ), + ) + members = item.value + if not isinstance(members, tuple): + raise AssertionError("JSON object members were not normalized") + all_field_names.update(name for name, _ in members) + records.append( + ParsedRecord( + members=members, + location=JsonLocation(pointer=f"/{index}"), + ) + ) + return tuple(sorted(all_field_names)), tuple(records) + + +def _to_native(value: object) -> NativeValue: + """Recursively assign parser-native categories to JSON library values.""" + if isinstance(value, NativeValue): + return value + if value is None: + return NativeValue(NativeCategory.NULL) + if isinstance(value, bool): + return NativeValue(NativeCategory.BOOLEAN, value) + if isinstance(value, str): + return NativeValue(NativeCategory.STRING, value) + if isinstance(value, list): + return NativeValue( + NativeCategory.ARRAY, + tuple(_to_native(element) for element in value), + ) + raise TypeError(f"unsupported JSON parser value: {type(value).__name__}") + + +def _escape_json_pointer(member_name: str) -> str: + """Escape one JSON object member for use in an RFC 6901 pointer.""" + return member_name.replace("~", "~0").replace("/", "~1") + + +def _json_category(value: object) -> str: + """Describe a parsed JSON value without reproducing its content.""" + if isinstance(value, NativeValue): + return value.category.value + if isinstance(value, list): + return "array" + if value is None: + return "null" + if isinstance(value, bool): + return "boolean" + if isinstance(value, str): + return "string" + return type(value).__name__ diff --git a/packages/adviso-compare/tests/test_inputs.py b/packages/adviso-compare/tests/test_inputs.py new file mode 100644 index 0000000..87a3ae9 --- /dev/null +++ b/packages/adviso-compare/tests/test_inputs.py @@ -0,0 +1,272 @@ +"""Behavioural tests for deterministic dataset input parsing.""" + +from __future__ import annotations + +from decimal import Decimal +from hashlib import sha256 +from pathlib import Path + +import pytest + +from adviso_compare.inputs import ( + InputError, + InputFormat, + InputSide, + JsonLocation, + NativeCategory, + NativeValue, + TabularLocation, + load_dataset, +) + + +def _write_bytes(path: Path, content: bytes) -> Path: + """Write one controlled input and return its path.""" + path.write_bytes(content) + return path + + +@pytest.mark.parametrize( + ("filename", "content", "expected_format"), + [ + ("REFERENCE.CSV", b"record_id,value\nS1,alpha\n", InputFormat.CSV), + ("reference.TsV", b"record_id\tvalue\nS1\talpha\n", InputFormat.TSV), + ("reference.JsOn", b'[{"record_id":"S1","value":"alpha"}]', InputFormat.JSON), + ], +) +def test_format_and_exact_byte_provenance_are_discovered_independently( + tmp_path: Path, + filename: str, + content: bytes, + expected_format: InputFormat, +) -> None: + """The final case-insensitive suffix selects format and bytes supply provenance.""" + path = _write_bytes(tmp_path / filename, content) + + dataset = load_dataset(path, InputSide.REFERENCE) + + assert dataset.provenance.basename == filename + assert dataset.provenance.format is expected_format + assert dataset.provenance.encoding == "utf-8" + assert dataset.provenance.size_bytes == len(content) + assert dataset.provenance.sha256 == sha256(content).hexdigest() + assert len(dataset.records) == 1 + + +@pytest.mark.parametrize( + ("suffix", "delimiter"), + [("csv", ","), ("tsv", "\t")], +) +def test_fixed_tabular_dialects_preserve_strings_and_record_locations( + tmp_path: Path, + suffix: str, + delimiter: str, +) -> None: + """Quoted syntax, multiline cells, and physical locations follow one dialect.""" + content = ( + f'record_id{delimiter}result{delimiter}note\r\n' + f'S1{delimiter}"a{delimiter}b"{delimiter}"first\r\nsecond"\r\n' + f'S2{delimiter}"""quoted"""{delimiter}\r\n' + ).encode() + path = _write_bytes(tmp_path / f"records.{suffix}", content) + + dataset = load_dataset(path, InputSide.QUERY) + + assert dataset.field_names == ("record_id", "result", "note") + assert len(dataset.records) == 2 + first, second = dataset.records + assert first.value("result").category is NativeCategory.STRING + assert first.value("result").value == f"a{delimiter}b" + assert first.value("note").value == "first\r\nsecond" + assert second.value("result").value == '"quoted"' + assert second.value("note").value == "" + assert first.location == TabularLocation(record_number=1, line_number=2) + assert first.location_for("note") == TabularLocation( + record_number=1, + line_number=2, + field_name="note", + ) + assert second.location == TabularLocation(record_number=2, line_number=4) + + +def test_lf_and_crlf_inputs_have_equal_records_but_distinct_byte_provenance( + tmp_path: Path, +) -> None: + """Accepted line-ending variants preserve parsed content, not byte evidence.""" + lf_path = _write_bytes(tmp_path / "lf.csv", b"record_id,value\nS1,alpha\n") + crlf_path = _write_bytes( + tmp_path / "crlf.csv", b"record_id,value\r\nS1,alpha\r\n" + ) + + lf_dataset = load_dataset(lf_path, InputSide.REFERENCE) + crlf_dataset = load_dataset(crlf_path, InputSide.REFERENCE) + + assert lf_dataset.records == crlf_dataset.records + assert lf_dataset.provenance.sha256 != crlf_dataset.provenance.sha256 + assert lf_dataset.provenance.size_bytes != crlf_dataset.provenance.size_bytes + + +def test_json_preserves_native_categories_lexical_numbers_and_pointers( + tmp_path: Path, +) -> None: + """JSON values remain lossless and attributable before configured processing.""" + content = ( + b'[{"text":"01","integer":1,"decimal":1.00,"exponent":1e-1,' + b'"boolean":true,"nothing":null,"array":[1,"x"],' + b'"object":{"z":2,"a":1},"quoted_nonfinite":"NaN",' + b'"a/b~c":"located"}]' + ) + path = _write_bytes(tmp_path / "records.json", content) + + dataset = load_dataset(path, InputSide.REFERENCE) + + assert dataset.field_names == ( + "a/b~c", + "array", + "boolean", + "decimal", + "exponent", + "integer", + "nothing", + "object", + "quoted_nonfinite", + "text", + ) + record = dataset.records[0] + assert record.location == JsonLocation(pointer="/0") + assert record.location_for("a/b~c") == JsonLocation(pointer="/0/a~1b~0c") + assert record.value("text").category is NativeCategory.STRING + assert record.value("text").value == "01" + assert record.value("integer").category is NativeCategory.INTEGER + assert record.value("integer").value == 1 + assert record.value("integer").lexical == "1" + assert record.value("decimal").category is NativeCategory.DECIMAL + assert record.value("decimal").value == Decimal("1.00") + assert record.value("decimal").lexical == "1.00" + assert record.value("exponent").value == Decimal("1e-1") + assert record.value("exponent").lexical == "1e-1" + assert record.value("boolean").category is NativeCategory.BOOLEAN + assert record.value("nothing").category is NativeCategory.NULL + assert record.has_field("nothing") + assert not record.has_field("missing") + assert record.location_for("missing") == record.location + assert record.value("quoted_nonfinite") == NativeValue( + NativeCategory.STRING, "NaN" + ) + assert record.value("array").category is NativeCategory.ARRAY + assert record.value("array").value[0].category is NativeCategory.INTEGER + assert record.value("object").category is NativeCategory.OBJECT + assert tuple(name for name, _ in record.value("object").value) == ("a", "z") + + +def test_parser_native_equality_is_recursive_and_category_preserving( + tmp_path: Path, +) -> None: + """Objects ignore member order while arrays and native categories remain exact.""" + first = load_dataset( + _write_bytes( + tmp_path / "first.json", + b'[{"value":{"b":1.00,"a":[1,2]},"other":1}]', + ), + InputSide.REFERENCE, + ).records[0] + second = load_dataset( + _write_bytes( + tmp_path / "second.json", + b'[{"other":1.0,"value":{"a":[1,2],"b":1.0}}]', + ), + InputSide.QUERY, + ).records[0] + reversed_array = load_dataset( + _write_bytes( + tmp_path / "third.json", + b'[{"value":{"a":[2,1],"b":1.0}}]', + ), + InputSide.QUERY, + ).records[0] + + assert first.value("value") == second.value("value") + assert first.value("other") != second.value("other") + assert first.value("value") != reversed_array.value("value") + + +@pytest.mark.parametrize( + ("filename", "content", "expected_fields"), + [ + ("empty.json", b"[]", ()), + ("empty.csv", b"record_id,value\n", ("record_id", "value")), + ("empty.tsv", b"record_id\tvalue\n", ("record_id", "value")), + ], +) +def test_structurally_valid_empty_datasets_remain_valid( + tmp_path: Path, + filename: str, + content: bytes, + expected_fields: tuple[str, ...], +) -> None: + """An empty record collection is distinct from an empty input file.""" + dataset = load_dataset( + _write_bytes(tmp_path / filename, content), InputSide.REFERENCE + ) + + assert dataset.records == () + assert dataset.field_names == expected_fields + + +@pytest.mark.parametrize( + ("filename", "content", "expected_code"), + [ + ("records", b"record_id\n", "unsupported_format"), + ("records.txt", b"record_id\n", "unsupported_format"), + ("records.csv", b"record_id\n\xff", "invalid_encoding"), + ("records.json", b"[", "invalid_syntax"), + ("records.json", b"[NaN]", "invalid_syntax"), + ("records.json", b"", "invalid_syntax"), + ("records.csv", b"", "invalid_structure"), + ("records.tsv", b"", "invalid_structure"), + ("records.csv", b'a,b\n"unterminated,b\n', "invalid_syntax"), + ("records.csv", b"a,a\n1,2\n", "invalid_structure"), + ("records.csv", b"a,\n1,2\n", "invalid_structure"), + ("records.csv", b"a,b\n1\n", "invalid_structure"), + ("records.tsv", b"a\tb\n1\t2\t3\n", "invalid_structure"), + ("records.json", b"{}", "invalid_structure"), + ("records.json", b"1", "invalid_structure"), + ("records.json", b"[1]", "invalid_structure"), + ("records.json", b'[{"a":1,"a":2}]', "invalid_structure"), + ("records.json", b'[{"a":{"b":1,"b":2}}]', "invalid_structure"), + ], +) +def test_invalid_inputs_are_rejected_with_the_specific_input_error( + tmp_path: Path, + filename: str, + content: bytes, + expected_code: str, +) -> None: + """Malformed bytes, syntax, and structures are not repaired or inferred.""" + path = _write_bytes(tmp_path / filename, content) + + with pytest.raises(InputError) as raised: + load_dataset(path, InputSide.QUERY) + + assert raised.value.category == "input" + assert raised.value.code == expected_code + assert raised.value.side is InputSide.QUERY + assert raised.value.diagnostic.startswith(f"ERROR [input.{expected_code}]: ") + assert str(tmp_path) not in raised.value.diagnostic + assert "\n" not in raised.value.diagnostic + + +def test_unreadable_input_is_classified_without_exposing_its_path( + tmp_path: Path, +) -> None: + """Filesystem failures use a stable error without leaking an absolute path.""" + path = tmp_path / "missing.json" + + with pytest.raises(InputError) as raised: + load_dataset(path, InputSide.REFERENCE) + + assert raised.value.code == "unreadable" + assert raised.value.diagnostic == ( + "ERROR [input.unreadable]: reference input could not be read" + ) + assert str(path) not in raised.value.diagnostic From cd468e79921282cedcf99eb7e4cf296949746761 Mon Sep 17 00:00:00 2001 From: Kevin Howe Date: Fri, 6 Nov 2026 12:29:31 +0000 Subject: [PATCH 17/45] Additions to input parsing logic and tests picked up in review --- .../src/adviso_compare/inputs.py | 59 +++++++ packages/adviso-compare/tests/test_inputs.py | 147 ++++++++++++++++-- 2 files changed, 196 insertions(+), 10 deletions(-) diff --git a/packages/adviso-compare/src/adviso_compare/inputs.py b/packages/adviso-compare/src/adviso_compare/inputs.py index a94906e..a81bc65 100644 --- a/packages/adviso-compare/src/adviso_compare/inputs.py +++ b/packages/adviso-compare/src/adviso_compare/inputs.py @@ -228,6 +228,7 @@ def _parse_table( side: InputSide, ) -> tuple[tuple[str, ...], tuple[ParsedRecord, ...]]: """Parse one CSV or TSV document using the fixed version 1 dialect.""" + _validate_tabular_lexical_syntax(text, delimiter, side) reader = csv.reader( io.StringIO(text, newline=""), delimiter=delimiter, @@ -299,6 +300,64 @@ def _parse_table( return tuple(header), tuple(records) +def _validate_tabular_lexical_syntax( + text: str, + delimiter: str, + side: InputSide, +) -> None: + """Reject quote and line-ending forms accepted by permissive CSV readers.""" + start_field = "start_field" + unquoted = "unquoted" + quoted = "quoted" + after_quote = "after_quote" + state = start_field + line_number = 1 + quoted_start_line: int | None = None + index = 0 + + while index < len(text): + character = text[index] + if character == "\r": + if index + 1 >= len(text) or text[index + 1] != "\n": + raise _tabular_syntax_error(side, line_number) + if state != quoted: + state = start_field + line_number += 1 + index += 2 + continue + if character == "\n": + if state != quoted: + state = start_field + line_number += 1 + index += 1 + continue + if character == delimiter: + if state != quoted: + state = start_field + index += 1 + continue + if character == '"': + if state == start_field: + state = quoted + quoted_start_line = line_number + elif state == unquoted: + raise _tabular_syntax_error(side, line_number) + elif state == quoted: + state = after_quote + else: + state = quoted + index += 1 + continue + if state == start_field: + state = unquoted + elif state == after_quote: + raise _tabular_syntax_error(side, line_number) + index += 1 + + if state == quoted: + raise _tabular_syntax_error(side, quoted_start_line or line_number) + + def _tabular_syntax_error(side: InputSide, line_number: int) -> InputError: """Construct a stable syntax error without reproducing parser input.""" line_context = f" near physical line {line_number}" if line_number else "" diff --git a/packages/adviso-compare/tests/test_inputs.py b/packages/adviso-compare/tests/test_inputs.py index 87a3ae9..54f525e 100644 --- a/packages/adviso-compare/tests/test_inputs.py +++ b/packages/adviso-compare/tests/test_inputs.py @@ -213,25 +213,152 @@ def test_structurally_valid_empty_datasets_remain_valid( assert dataset.field_names == expected_fields +@pytest.mark.parametrize(("suffix", "delimiter"), [("csv", ","), ("tsv", "\t")]) +@pytest.mark.parametrize( + ("body", "line_number"), + [ + ('"unterminated', 2), + ('unquoted"quote', 2), + ('"closed"trailing', 2), + ('"backslash\\"escape"', 2), + ("bare\rcarriage-return", 2), + ], +) +def test_malformed_tabular_quote_and_line_syntax_is_rejected_for_each_dialect( + tmp_path: Path, + suffix: str, + delimiter: str, + body: str, + line_number: int, +) -> None: + """Both fixed dialects reject non-standard quoting and record endings.""" + content = f"record_id{delimiter}value\nS1{delimiter}{body}\n".encode() + path = _write_bytes(tmp_path / f"records.{suffix}", content) + + with pytest.raises(InputError) as raised: + load_dataset(path, InputSide.QUERY) + + assert raised.value.code == "invalid_syntax" + assert f"physical line {line_number}" in raised.value.message + + +@pytest.mark.parametrize(("suffix", "delimiter"), [("csv", ","), ("tsv", "\t")]) +@pytest.mark.parametrize( + ("document_template", "message_fragment"), + [ + ("field{d}field\n", "duplicate column names"), + ("field{d}\n", "empty column name"), + ("field{d}other\none\n", "has 1 fields; expected 2"), + ("field{d}other\none{d}two{d}three\n", "has 3 fields; expected 2"), + ("field{d}other\n\n", "has 0 fields; expected 2"), + ], +) +def test_invalid_tabular_structure_is_rejected_for_each_dialect( + tmp_path: Path, + suffix: str, + delimiter: str, + document_template: str, + message_fragment: str, +) -> None: + """Headers, row widths, and blank physical records are checked symmetrically.""" + content = document_template.format(d=delimiter) + path = _write_bytes(tmp_path / f"records.{suffix}", content.encode()) + + with pytest.raises(InputError) as raised: + load_dataset(path, InputSide.REFERENCE) + + assert raised.value.code == "invalid_structure" + assert message_fragment in raised.value.message + + +@pytest.mark.parametrize("suffix", ["csv", "tsv", "json"]) +def test_invalid_utf8_is_rejected_before_format_parsing( + tmp_path: Path, + suffix: str, +) -> None: + """The UTF-8 requirement is identical for every supported format.""" + path = _write_bytes(tmp_path / f"records.{suffix}", b"\xff") + + with pytest.raises(InputError) as raised: + load_dataset(path, InputSide.REFERENCE) + + assert raised.value.code == "invalid_encoding" + + +@pytest.mark.parametrize("token", ["NaN", "Infinity", "-Infinity"]) +@pytest.mark.parametrize( + "document_template", + [ + "{}", + '[{{"value":{}}}]', + '[{{"value":{{"nested":{}}}}}]', + '[{{"value":[{}]}}]', + ], +) +def test_non_finite_json_tokens_are_rejected_in_every_structural_context( + tmp_path: Path, + token: str, + document_template: str, +) -> None: + """No parser location enables non-standard JSON numerical constants.""" + document = document_template.format(token).encode() + path = _write_bytes(tmp_path / "records.json", document) + + with pytest.raises(InputError) as raised: + load_dataset(path, InputSide.QUERY) + + assert raised.value.code == "invalid_syntax" + assert raised.value.message == "query JSON input contains a non-finite number" + + +@pytest.mark.parametrize( + ("content", "message_fragment"), + [ + (b"{}", "JSON root is object"), + (b"1", "JSON root is integer"), + (b'"record"', "JSON root is string"), + (b"[1]", "array item 0 is integer"), + (b'[{"record_id":"S1"},[]]', "array item 1 is array"), + ], +) +def test_invalid_json_roots_identify_the_value_and_required_structure( + tmp_path: Path, + content: bytes, + message_fragment: str, +) -> None: + """Root errors identify the safe structural category and offending index.""" + path = _write_bytes(tmp_path / "records.json", content) + + with pytest.raises(InputError) as raised: + load_dataset(path, InputSide.REFERENCE) + + assert raised.value.code == "invalid_structure" + assert message_fragment in raised.value.message + assert "record object" in raised.value.message + + +def test_malformed_json_reports_its_parser_location(tmp_path: Path) -> None: + """JSON syntax errors retain safe one-based line and column context.""" + path = _write_bytes(tmp_path / "records.json", b'[\n {"a": 1},\n]') + + with pytest.raises(InputError) as raised: + load_dataset(path, InputSide.QUERY) + + assert raised.value.code == "invalid_syntax" + assert raised.value.message == ( + "query JSON input has invalid syntax at line 3, column 1" + ) + + @pytest.mark.parametrize( ("filename", "content", "expected_code"), [ ("records", b"record_id\n", "unsupported_format"), ("records.txt", b"record_id\n", "unsupported_format"), - ("records.csv", b"record_id\n\xff", "invalid_encoding"), ("records.json", b"[", "invalid_syntax"), - ("records.json", b"[NaN]", "invalid_syntax"), ("records.json", b"", "invalid_syntax"), ("records.csv", b"", "invalid_structure"), ("records.tsv", b"", "invalid_structure"), - ("records.csv", b'a,b\n"unterminated,b\n', "invalid_syntax"), - ("records.csv", b"a,a\n1,2\n", "invalid_structure"), - ("records.csv", b"a,\n1,2\n", "invalid_structure"), - ("records.csv", b"a,b\n1\n", "invalid_structure"), - ("records.tsv", b"a\tb\n1\t2\t3\n", "invalid_structure"), - ("records.json", b"{}", "invalid_structure"), - ("records.json", b"1", "invalid_structure"), - ("records.json", b"[1]", "invalid_structure"), ("records.json", b'[{"a":1,"a":2}]', "invalid_structure"), ("records.json", b'[{"a":{"b":1,"b":2}}]', "invalid_structure"), ], From 62414c5596122167dcd714cf231aba77e2a0ae9c Mon Sep 17 00:00:00 2001 From: Kevin Howe Date: Fri, 6 Nov 2026 12:52:35 +0000 Subject: [PATCH 18/45] Implemented first attempt at comparison configuration parsing and representation (with tests) --- packages/adviso-compare/README.md | 7 +- .../src/adviso_compare/application.py | 42 +- .../src/adviso_compare/configuration.py | 836 ++++++++++++++++++ .../tests/test_configuration.py | 661 ++++++++++++++ 4 files changed, 1542 insertions(+), 4 deletions(-) create mode 100644 packages/adviso-compare/src/adviso_compare/configuration.py create mode 100644 packages/adviso-compare/tests/test_configuration.py diff --git a/packages/adviso-compare/README.md b/packages/adviso-compare/README.md index ef8432e..6d4fe27 100644 --- a/packages/adviso-compare/README.md +++ b/packages/adviso-compare/README.md @@ -7,9 +7,10 @@ The project is intended to support bioinformatics validation and verification ac ## Project status Version 1 implementation is in progress. The schema-validation baseline, -command-line invocation layer, and deterministic CSV, TSV, and JSON input parser -are implemented. Comparison-configuration processing, the comparison kernel, -and evidence generation are not yet available. +command-line invocation layer, deterministic CSV, TSV, and JSON input parser, +and custom comparison-configuration loader are implemented. Automatic +configuration generation, the comparison kernel, and evidence generation are +not yet available. The current normative artefacts are: diff --git a/packages/adviso-compare/src/adviso_compare/application.py b/packages/adviso-compare/src/adviso_compare/application.py index 197bad1..3aea15a 100644 --- a/packages/adviso-compare/src/adviso_compare/application.py +++ b/packages/adviso-compare/src/adviso_compare/application.py @@ -2,12 +2,52 @@ from __future__ import annotations +from dataclasses import dataclass +from pathlib import Path +from typing import Callable + +from adviso_compare.configuration import ( + EffectiveConfiguration, + load_comparison_configuration, +) +from adviso_compare.inputs import InputSide, ParsedDataset, load_dataset from adviso_compare.invocation import ComparisonInvocation +@dataclass(frozen=True, slots=True) +class PreparedComparison: + """Validated configuration selection and losslessly parsed input datasets.""" + + invocation: ComparisonInvocation + configuration: EffectiveConfiguration | None + reference: ParsedDataset + query: ParsedDataset + + +def prepare_comparison( + invocation: ComparisonInvocation, + *, + configuration_loader: Callable[ + [Path], EffectiveConfiguration + ] = load_comparison_configuration, + dataset_loader: Callable[[Path, InputSide], ParsedDataset] = load_dataset, +) -> PreparedComparison: + """Validate any custom configuration before accessing either dataset.""" + configuration = None + if invocation.configuration is not None: + configuration = configuration_loader(invocation.configuration) + reference = dataset_loader(invocation.reference, InputSide.REFERENCE) + query = dataset_loader(invocation.query, InputSide.QUERY) + return PreparedComparison( + invocation=invocation, + configuration=configuration, + reference=reference, + query=query, + ) + + def execute_comparison(invocation: ComparisonInvocation) -> int: """Execute a comparison once later implementation increments supply the core.""" raise NotImplementedError( "comparison execution is not available at this implementation stage" ) - diff --git a/packages/adviso-compare/src/adviso_compare/configuration.py b/packages/adviso-compare/src/adviso_compare/configuration.py new file mode 100644 index 0000000..1252462 --- /dev/null +++ b/packages/adviso-compare/src/adviso_compare/configuration.py @@ -0,0 +1,836 @@ +"""Safe loading and validation of custom comparison configurations.""" + +from __future__ import annotations + +import copy +import json +import re +import unicodedata +from dataclasses import dataclass +from decimal import Decimal +from importlib.resources import files +from pathlib import Path +from typing import Any, Generic, Iterator, Mapping, TypeVar + +import yaml +from jsonschema import Draft202012Validator, FormatChecker, ValidationError +from yaml.constructor import ConstructorError +from yaml.nodes import MappingNode, ScalarNode +from yaml.tokens import AliasToken, AnchorToken, TagToken + + +CONFIGURATION_SCHEMA_FILE = "comparison-configuration-v1.0.schema.json" +CONFIGURATION_SCHEMA_PACKAGE = "adviso_compare.schemas" +SUPPORTED_SCHEMA_VERSION = "1.0" +AUTOMATIC_CONFIGURATION_ID = "org.adviso.automatic-strict-exact" + +JsonObject = dict[str, Any] +T = TypeVar("T") + + +@dataclass(frozen=True, slots=True) +class DatasetSides(Generic[T]): + """Resolved reference-side and query-side values of one declaration.""" + + reference: T + query: T + + +@dataclass(frozen=True, slots=True) +class Operation: + """One registered processing operation and its immutable parameters.""" + + name: str + parameters: tuple[tuple[str, object], ...] = () + + +@dataclass(frozen=True, slots=True) +class SourceValueDescriptor: + """One explicitly typed value used by null or mapping rules.""" + + category: str + value: object = None + + +@dataclass(frozen=True, slots=True) +class ValueMappingEntry: + """One closed source-to-destination value mapping.""" + + source: SourceValueDescriptor + destination: SourceValueDescriptor + + +@dataclass(frozen=True, slots=True) +class Comparator: + """A comparator declaration and its immutable parameters.""" + + name: str + parameters: tuple[tuple[str, object], ...] = () + + +@dataclass(frozen=True, slots=True) +class CollectionLogicalType: + """The resolved logical type of a collection-valued field.""" + + collection_type: str + element_type: str + element_comparator: Comparator + + +@dataclass(frozen=True, slots=True) +class NullPolicy: + """Resolved null acceptance and side-specific missing tokens.""" + + allowed: bool + nulls_equal: bool + missing_tokens: DatasetSides[tuple[SourceValueDescriptor, ...]] + + +@dataclass(frozen=True, slots=True) +class CollectionRepresentation: + """Resolved representation and element processing for one dataset side.""" + + format: str + delimiter: str | None + element_normalisation: tuple[Operation, ...] + element_mapping: tuple[ValueMappingEntry, ...] + + +@dataclass(frozen=True, slots=True) +class FieldDefinition: + """An immutable effective declaration for one canonical field.""" + + name: str + source_fields: DatasetSides[str] + required: DatasetSides[bool] + logical_type: str | CollectionLogicalType | None + null_policy: NullPolicy + source_normalisation: DatasetSides[tuple[Operation, ...]] + collection_representation: DatasetSides[CollectionRepresentation | None] + value_mapping: DatasetSides[tuple[ValueMappingEntry, ...]] + typed_normalisation: tuple[Operation, ...] + comparator: Comparator + review_priority: str + + +@dataclass(frozen=True, slots=True) +class ConfigurationMetadata: + """Identity and optional human-readable metadata for one configuration.""" + + id: str + revision: str + title: str | None + description: str | None + + +@dataclass(frozen=True, slots=True) +class EffectiveConfiguration: + """A validated custom configuration with immutable resolved application state.""" + + schema_version: str + metadata: ConfigurationMetadata + record_key: tuple[str, ...] + in_scope: tuple[str, ...] + _fields: tuple[tuple[str, FieldDefinition], ...] + _document_json: str + + @property + def document(self) -> JsonObject: + """Return a fresh JSON-compatible copy of the validated source document.""" + document = json.loads(self._document_json) + if not isinstance(document, dict): + raise AssertionError("validated configuration document is not an object") + return document + + @property + def fields(self) -> Mapping[str, FieldDefinition]: + """Return the resolved field definitions as a read-only mapping.""" + return _TupleMapping(self._fields) + + def field(self, name: str) -> FieldDefinition: + """Return one resolved canonical field definition.""" + try: + return self.fields[name] + except KeyError as error: + raise KeyError(f"unknown canonical field: {name}") from error + + +class _TupleMapping(Mapping[str, T]): + """A minimal immutable mapping backed by ordered key-value tuples.""" + + def __init__(self, items: tuple[tuple[str, T], ...]) -> None: + self._items = items + + def __getitem__(self, key: str) -> T: + for candidate, value in self._items: + if candidate == key: + return value + raise KeyError(key) + + def __iter__(self) -> Iterator[str]: + return (key for key, _ in self._items) + + def __len__(self) -> int: + return len(self._items) + + +@dataclass(frozen=True, slots=True) +class ConfigurationError(Exception): + """A classified expected failure while loading a custom configuration.""" + + code: str + path: str + message: str + + @property + def category(self) -> str: + """Return the stable execution-error category.""" + return "configuration" + + @property + def diagnostic(self) -> str: + """Render the stable single-line diagnostic.""" + return f"ERROR [configuration.{self.code}]: {self.path}: {self.message}" + + def __str__(self) -> str: + """Return the actionable path and message.""" + return f"{self.path}: {self.message}" + + +class _Yaml12SafeLoader(yaml.SafeLoader): + """Safe loader with YAML 1.2 core scalar resolution and strict mappings.""" + + yaml_implicit_resolvers = copy.deepcopy(yaml.SafeLoader.yaml_implicit_resolvers) + + def construct_mapping( + self, + node: MappingNode, + deep: bool = False, + ) -> dict[str, object]: + """Construct a string-keyed mapping while rejecting merges and duplicates.""" + if not isinstance(node, MappingNode): + raise ConstructorError(None, None, "expected a mapping", node.start_mark) + mapping: dict[str, object] = {} + for key_node, value_node in node.value: + if key_node.tag == "tag:yaml.org,2002:merge": + raise ConstructorError( + "while constructing a mapping", + node.start_mark, + "merge keys are not permitted", + key_node.start_mark, + ) + key = self.construct_object(key_node, deep=deep) + if not isinstance(key, str): + raise ConstructorError( + "while constructing a mapping", + node.start_mark, + "mapping keys must be strings", + key_node.start_mark, + ) + if key in mapping: + raise ConstructorError( + "while constructing a mapping", + node.start_mark, + "duplicate mapping key", + key_node.start_mark, + ) + mapping[key] = self.construct_object(value_node, deep=deep) + return mapping + + +for first_character, resolvers in tuple( + _Yaml12SafeLoader.yaml_implicit_resolvers.items() +): + _Yaml12SafeLoader.yaml_implicit_resolvers[first_character] = [ + (tag, expression) + for tag, expression in resolvers + if tag + not in { + "tag:yaml.org,2002:bool", + "tag:yaml.org,2002:float", + "tag:yaml.org,2002:int", + "tag:yaml.org,2002:null", + "tag:yaml.org,2002:timestamp", + } + ] + +_Yaml12SafeLoader.add_implicit_resolver( + "tag:yaml.org,2002:null", + re.compile(r"^(?:~|null|Null|NULL)$"), + ["~", "n", "N"], +) +_Yaml12SafeLoader.add_implicit_resolver( + "tag:yaml.org,2002:bool", + re.compile(r"^(?:true|True|TRUE|false|False|FALSE)$"), + ["t", "T", "f", "F"], +) +_Yaml12SafeLoader.add_implicit_resolver( + "tag:yaml.org,2002:int", + re.compile(r"^[-+]?(?:[0-9][0-9_]*|0o[0-7_]+|0x[0-9a-fA-F_]+)$"), + list("-+0123456789"), +) +_Yaml12SafeLoader.add_implicit_resolver( + "tag:yaml.org,2002:float", + re.compile( + r"^[-+]?(?:(?:[0-9][0-9_]*)?\.[0-9_]+" + r"|[0-9][0-9_]*\.(?:[0-9_]*)" + r"|[0-9][0-9_]*(?:[eE][-+]?[0-9]+))$" + ), + list("-+0123456789."), +) + + +def _construct_yaml12_integer(loader: _Yaml12SafeLoader, node: ScalarNode) -> int: + """Construct YAML 1.2 decimal, octal, or hexadecimal integer syntax.""" + value = loader.construct_scalar(node).replace("_", "") + sign = 1 + if value.startswith("-"): + sign = -1 + value = value[1:] + elif value.startswith("+"): + value = value[1:] + if value.startswith("0o"): + return sign * int(value[2:], 8) + if value.startswith("0x"): + return sign * int(value[2:], 16) + return sign * int(value, 10) + + +_Yaml12SafeLoader.add_constructor( + "tag:yaml.org,2002:int", + _construct_yaml12_integer, +) + + +def load_comparison_configuration(path: Path) -> EffectiveConfiguration: + """Load, structurally validate, and semantically resolve one custom YAML file.""" + try: + content = path.read_bytes() + except OSError as error: + raise ConfigurationError( + code="unreadable", + path="$", + message="comparison configuration could not be read", + ) from error + try: + text = content.decode("utf-8", errors="strict") + except UnicodeDecodeError as error: + raise ConfigurationError( + code="invalid_syntax", + path="$", + message="comparison configuration is not valid UTF-8", + ) from error + + document = _load_yaml_document(text) + if ( + isinstance(document, dict) + and isinstance(document.get("schema_version"), str) + and document["schema_version"] != SUPPORTED_SCHEMA_VERSION + ): + raise ConfigurationError( + code="unsupported_version", + path="$.schema_version", + message="comparison-configuration schema version is not supported", + ) + _validate_schema(document) + if not isinstance(document, dict): + raise AssertionError("schema accepted a non-object configuration") + _validate_semantics(document) + return _build_effective_configuration(document) + + +def _load_yaml_document(text: str) -> object: + """Load exactly one safe YAML 1.2 document with forbidden features disabled.""" + try: + for token in yaml.scan(text, Loader=_Yaml12SafeLoader): + if isinstance(token, (AliasToken, AnchorToken, TagToken)): + raise ConfigurationError( + code="invalid_syntax", + path="$", + message=( + "YAML aliases, anchors, and explicit tags are not permitted" + ), + ) + return yaml.load(text, Loader=_Yaml12SafeLoader) + except ConfigurationError: + raise + except yaml.YAMLError as error: + raise ConfigurationError( + code="invalid_syntax", + path="$", + message="comparison configuration is not permitted YAML syntax", + ) from error + + +def _configuration_validator() -> Draft202012Validator: + """Load the authoritative packaged schema through the resource API.""" + resource = files(CONFIGURATION_SCHEMA_PACKAGE).joinpath(CONFIGURATION_SCHEMA_FILE) + schema = json.loads(resource.read_text(encoding="utf-8")) + return Draft202012Validator(schema, format_checker=FormatChecker()) + + +def _validate_schema(document: object) -> None: + """Raise the most deterministic actionable structural validation error.""" + errors = list(_configuration_validator().iter_errors(document)) + if not errors: + return + error = min(errors, key=_schema_error_sort_key) + path_components = list(error.absolute_path) + if error.validator == "required" and isinstance(error.instance, dict): + missing = sorted(set(error.validator_value) - set(error.instance)) + if missing: + path_components.append(missing[0]) + elif error.validator == "additionalProperties" and isinstance( + error.instance, dict + ): + permitted = set(error.schema.get("properties", {})) + extras = sorted(set(error.instance) - permitted) + if extras: + path_components.append(extras[0]) + path = _configuration_path(path_components) + raise ConfigurationError( + code="schema_violation", + path=path, + message=_schema_error_message(error), + ) + + +def _schema_error_sort_key(error: ValidationError) -> tuple[str, str]: + """Return a stable ordering key for competing schema failures.""" + return ( + _configuration_path(list(error.absolute_path)), + "/".join(str(component) for component in error.absolute_schema_path), + ) + + +def _schema_error_message(error: ValidationError) -> str: + """Describe a schema rule without reproducing configuration values.""" + messages = { + "required": "required property is missing", + "additionalProperties": "unknown property is not permitted", + "const": "value does not identify the required schema version or form", + "enum": "value is not one of the permitted choices", + "minItems": "list must contain at least one item", + "minLength": "string must not be empty", + "minProperties": "mapping must contain at least one property", + "uniqueItems": "list items must be unique", + "type": "value has the wrong type", + } + return messages.get( + str(error.validator), + f"value violates schema rule {error.validator}", + ) + + +def _configuration_path(components: list[object]) -> str: + """Render one unambiguous JSONPath-like configuration location.""" + path = "$" + for component in components: + if isinstance(component, int): + path += f"[{component}]" + elif re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*", str(component)): + path += f".{component}" + else: + path += f"[{json.dumps(str(component), ensure_ascii=False)}]" + return path + + +def _semantic_error(path: str, message: str) -> None: + """Raise one semantic configuration failure.""" + raise ConfigurationError( + code="semantic_violation", + path=path, + message=message, + ) + + +def _validate_semantics(document: JsonObject) -> None: + """Apply cross-property rules not reliably expressible in JSON Schema.""" + fields = document["fields"] + definitions = fields["definitions"] + in_scope = fields["in_scope"] + + for index, field_name in enumerate(fields["record_key"]): + path = f"$.fields.record_key[{index}]" + if field_name not in definitions: + _semantic_error(path, "record-key field has no field definition") + if field_name not in in_scope: + _semantic_error(path, "record-key field is not included in fields.in_scope") + _validate_key_field(field_name, definitions[field_name]) + + for index, field_name in enumerate(in_scope): + if field_name not in definitions: + _semantic_error( + f"$.fields.in_scope[{index}]", + "in-scope field has no field definition", + ) + + source_names: dict[str, dict[str, str]] = {"reference": {}, "query": {}} + for field_name, field in definitions.items(): + field_path = _field_path(field_name) + _validate_field_semantics(field_path, field) + for side in ("reference", "query"): + source_name = field["source_fields"][side] + previous = source_names[side].get(source_name) + if previous is not None: + _semantic_error( + f"{field_path}.source_fields.{side}", + f"source field is already used by canonical field {previous!r}", + ) + source_names[side][source_name] = field_name + + if document["metadata"]["id"] == AUTOMATIC_CONFIGURATION_ID: + _semantic_error( + "$.metadata.id", + "identifier is reserved for automatically generated configurations", + ) + + +def _validate_key_field(field_name: str, field: JsonObject) -> None: + """Validate the mandatory contract for one record-key field.""" + path = _field_path(field_name) + for side in ("reference", "query"): + if not field["required"][side]: + _semantic_error( + f"{path}.required.{side}", + "record-key field must be required on both sides", + ) + if field["null_policy"]["allowed"]: + _semantic_error( + f"{path}.null_policy.allowed", + "record-key field must not allow null", + ) + if not isinstance(field.get("logical_type"), str): + _semantic_error( + f"{path}.logical_type", + "record-key field must have a scalar logical type", + ) + if field["comparator"]["name"] != "typed_exact": + _semantic_error( + f"{path}.comparator.name", + "record-key field must use typed_exact", + ) + + +def _validate_field_semantics(path: str, field: JsonObject) -> None: + """Validate comparator, operation, and mapping relationships for one field.""" + logical_type = field.get("logical_type") + comparator_name = field["comparator"]["name"] + if logical_type is None: + if comparator_name != "preserved_exact": + _semantic_error( + f"{path}.comparator.name", + "field without logical_type must use preserved_exact", + ) + elif comparator_name == "preserved_exact": + _semantic_error( + f"{path}.comparator.name", + "preserved_exact field must omit logical_type", + ) + + _validate_comparator_compatibility(path, logical_type, field) + _validate_typed_normalisation(path, logical_type, field) + _validate_side_mappings(path, field) + _validate_collection_mappings(path, field) + + +def _validate_comparator_compatibility( + path: str, + logical_type: object, + field: JsonObject, +) -> None: + """Require the field comparator to match its resolved logical type.""" + comparator = field["comparator"]["name"] + if logical_type is None: + permitted = {"preserved_exact"} + elif isinstance(logical_type, str): + permitted_by_type = { + "string": {"typed_exact", "normalised_string"}, + "integer": {"typed_exact", "numeric_exact", "numeric_tolerance"}, + "decimal": {"typed_exact", "numeric_exact", "numeric_tolerance"}, + "boolean": {"typed_exact"}, + "date": {"typed_exact", "date_exact"}, + "datetime": {"typed_exact", "datetime_exact"}, + } + permitted = permitted_by_type[logical_type] + else: + permitted = {f"{logical_type['collection_type']}_exact"} + if comparator not in permitted: + _semantic_error( + f"{path}.comparator.name", + "comparator is incompatible with the field logical type", + ) + if comparator == "normalised_string": + source_operations = field.get("source_normalisation", {}) + if not source_operations.get("reference") and not source_operations.get( + "query" + ): + _semantic_error( + f"{path}.comparator.name", + "normalised_string requires a configured string normalisation", + ) + + +def _validate_typed_normalisation( + path: str, + logical_type: object, + field: JsonObject, +) -> None: + """Require each typed operation to match the scalar destination type.""" + compatible_type = { + "decimal_quantise": "decimal", + "datetime_truncate": "datetime", + } + for index, operation in enumerate(field.get("typed_normalisation", [])): + if logical_type != compatible_type[operation["operation"]]: + _semantic_error( + f"{path}.typed_normalisation[{index}].operation", + "typed normalisation is incompatible with the field logical type", + ) + + +def _validate_side_mappings(path: str, field: JsonObject) -> None: + """Reject duplicate, colliding, and missing-token-unreachable field mappings.""" + mappings = field.get("value_mapping", {}) + source_operations = field.get("source_normalisation", {}) + missing_tokens = field["null_policy"].get("missing_tokens", {}) + for side in ("reference", "query"): + operations = source_operations.get(side, []) + missing = { + _descriptor_key(descriptor, operations) + for descriptor in missing_tokens.get(side, []) + } + seen: set[tuple[str, object]] = set() + for index, entry in enumerate(mappings.get(side, [])): + entry_path = f"{path}.value_mapping.{side}[{index}].from" + key = _descriptor_key(entry["from"], operations) + if key in seen: + _semantic_error( + entry_path, + "mapping source duplicates another source after normalisation", + ) + if key in missing: + _semantic_error( + entry_path, + "mapping source is unreachable because it is a missing token", + ) + seen.add(key) + + +def _validate_collection_mappings(path: str, field: JsonObject) -> None: + """Reject element mapping sources that collide after element normalisation.""" + for side, representation in field.get("collection_representation", {}).items(): + operations = representation.get("element_normalisation", []) + seen: set[tuple[str, object]] = set() + for index, entry in enumerate(representation.get("element_mapping", [])): + entry_path = ( + f"{path}.collection_representation.{side}.element_mapping" + f"[{index}].from" + ) + key = _descriptor_key(entry["from"], operations) + if key in seen: + _semantic_error( + entry_path, + "element mapping source duplicates another source after " + "normalisation", + ) + seen.add(key) + + +def _descriptor_key( + descriptor: JsonObject, + operations: list[JsonObject], +) -> tuple[str, object]: + """Return parser-native descriptor equality after configured text operations.""" + category = descriptor["category"] + value = descriptor.get("value") + if category == "integer": + value = int(value) + elif category == "decimal": + value = Decimal(value) + elif category == "string": + for operation in operations: + value = _normalise_configuration_text(value, operation) + return category, value + + +def _normalise_configuration_text(value: str, operation: JsonObject) -> str: + """Apply source-text operation semantics needed for mapping reachability.""" + name = operation["operation"] + parameters = operation.get("parameters", {}) + if name == "unicode_normalise": + return unicodedata.normalize(parameters["form"], value) + if name == "trim": + side = parameters["side"] + if side == "leading": + return value.lstrip() + if side == "trailing": + return value.rstrip() + return value.strip() + if name == "case_fold": + return value.casefold() + raise AssertionError(f"unexpected validated source operation: {name}") + + +def _field_path(field_name: str) -> str: + """Return the unambiguous configuration path for a canonical field.""" + return _configuration_path(["fields", "definitions", field_name]) + + +def _build_effective_configuration(document: JsonObject) -> EffectiveConfiguration: + """Resolve optional declarations into immutable application values.""" + metadata = document["metadata"] + fields = tuple( + (name, _build_field(name, field)) + for name, field in document["fields"]["definitions"].items() + ) + return EffectiveConfiguration( + schema_version=document["schema_version"], + metadata=ConfigurationMetadata( + id=metadata["id"], + revision=metadata["revision"], + title=metadata.get("title"), + description=metadata.get("description"), + ), + record_key=tuple(document["fields"]["record_key"]), + in_scope=tuple(document["fields"]["in_scope"]), + _fields=fields, + _document_json=json.dumps( + document, + ensure_ascii=False, + separators=(",", ":"), + allow_nan=False, + ), + ) + + +def _build_field(name: str, field: JsonObject) -> FieldDefinition: + """Build one immutable field definition with resolved omissions.""" + logical_document = field.get("logical_type") + if isinstance(logical_document, dict): + logical_type: str | CollectionLogicalType | None = CollectionLogicalType( + collection_type=logical_document["collection_type"], + element_type=logical_document["element_type"], + element_comparator=_build_comparator( + logical_document["element_comparator"] + ), + ) + else: + logical_type = logical_document + + null_policy = field["null_policy"] + missing_tokens = null_policy.get("missing_tokens", {}) + source_normalisation = field.get("source_normalisation", {}) + value_mapping = field.get("value_mapping", {}) + collection_document = field.get("collection_representation", {}) + if isinstance(logical_type, CollectionLogicalType): + collection_representation: DatasetSides[ + CollectionRepresentation | None + ] = DatasetSides( + reference=_build_collection_representation( + collection_document.get("reference") + ), + query=_build_collection_representation(collection_document.get("query")), + ) + else: + collection_representation = DatasetSides(reference=None, query=None) + + return FieldDefinition( + name=name, + source_fields=DatasetSides(**field["source_fields"]), + required=DatasetSides(**field["required"]), + logical_type=logical_type, + null_policy=NullPolicy( + allowed=null_policy["allowed"], + nulls_equal=null_policy["nulls_equal"], + missing_tokens=DatasetSides( + reference=tuple( + _build_descriptor(item) + for item in missing_tokens.get("reference", []) + ), + query=tuple( + _build_descriptor(item) for item in missing_tokens.get("query", []) + ), + ), + ), + source_normalisation=DatasetSides( + reference=tuple( + _build_operation(item) + for item in source_normalisation.get("reference", []) + ), + query=tuple( + _build_operation(item) + for item in source_normalisation.get("query", []) + ), + ), + collection_representation=collection_representation, + value_mapping=DatasetSides( + reference=tuple( + _build_mapping(item) for item in value_mapping.get("reference", []) + ), + query=tuple( + _build_mapping(item) for item in value_mapping.get("query", []) + ), + ), + typed_normalisation=tuple( + _build_operation(item) for item in field.get("typed_normalisation", []) + ), + comparator=_build_comparator(field["comparator"]), + review_priority=field.get("review_priority", "normal"), + ) + + +def _build_operation(document: JsonObject) -> Operation: + """Build one immutable processing operation.""" + return Operation( + name=document["operation"], + parameters=tuple(document.get("parameters", {}).items()), + ) + + +def _build_comparator(document: JsonObject) -> Comparator: + """Build one immutable comparator declaration.""" + return Comparator( + name=document["name"], + parameters=tuple(document.get("parameters", {}).items()), + ) + + +def _build_descriptor(document: JsonObject) -> SourceValueDescriptor: + """Build one immutable source-value descriptor.""" + return SourceValueDescriptor( + category=document["category"], + value=document.get("value"), + ) + + +def _build_mapping(document: JsonObject) -> ValueMappingEntry: + """Build one immutable value-mapping entry.""" + return ValueMappingEntry( + source=_build_descriptor(document["from"]), + destination=_build_descriptor(document["to"]), + ) + + +def _build_collection_representation( + document: JsonObject | None, +) -> CollectionRepresentation: + """Build one side's collection representation, applying omission defaults.""" + if document is None: + return CollectionRepresentation( + format="json_array", + delimiter=None, + element_normalisation=(), + element_mapping=(), + ) + return CollectionRepresentation( + format=document["format"], + delimiter=document.get("delimiter"), + element_normalisation=tuple( + _build_operation(item) + for item in document.get("element_normalisation", []) + ), + element_mapping=tuple( + _build_mapping(item) for item in document.get("element_mapping", []) + ), + ) diff --git a/packages/adviso-compare/tests/test_configuration.py b/packages/adviso-compare/tests/test_configuration.py new file mode 100644 index 0000000..14d337d --- /dev/null +++ b/packages/adviso-compare/tests/test_configuration.py @@ -0,0 +1,661 @@ +"""Behavioural tests for custom comparison-configuration loading.""" + +from __future__ import annotations + +from pathlib import Path +from typing import Callable + +import pytest +import yaml + +from adviso_compare.application import prepare_comparison +from adviso_compare.configuration import ( + ConfigurationError, + load_comparison_configuration, +) +from adviso_compare.inputs import InputSide, ParsedDataset +from adviso_compare.invocation import ComparisonInvocation +from tests.support.documents import JsonObject, minimal_configuration + + +ConfigurationMutation = Callable[[JsonObject], None] + + +def _write_yaml(path: Path, document: JsonObject) -> Path: + """Write a controlled YAML configuration and return its path.""" + path.write_text(yaml.safe_dump(document, sort_keys=False), encoding="utf-8") + return path + + +def _load_document(tmp_path: Path, document: JsonObject): + """Load one mapping through the production YAML boundary.""" + return load_comparison_configuration(_write_yaml(tmp_path / "rules.yaml", document)) + + +def _add_mapping_field(document: JsonObject) -> JsonObject: + """Return a configuration containing one mapped string field.""" + field = document["fields"]["definitions"]["measured_value"] + field["logical_type"] = "string" + field["comparator"] = {"name": "typed_exact"} + field["value_mapping"] = { + "reference": [ + { + "from": {"category": "string", "value": "positive"}, + "to": {"category": "string", "value": "detected"}, + } + ] + } + return document + + +def _add_collection_field(document: JsonObject) -> JsonObject: + """Return a configuration containing one set-valued field.""" + field = document["fields"]["definitions"]["measured_value"] + field["logical_type"] = { + "collection_type": "set", + "element_type": "string", + "element_comparator": {"name": "typed_exact"}, + } + field["comparator"] = {"name": "set_exact"} + field["collection_representation"] = { + "query": { + "format": "delimited_text", + "delimiter": "/", + "element_normalisation": [{"operation": "case_fold"}], + "element_mapping": [ + { + "from": {"category": "string", "value": "wild_type"}, + "to": {"category": "string", "value": "WT"}, + } + ], + } + } + return document + + +def test_valid_configuration_resolves_immutable_defaults_without_altering_document( + tmp_path: Path, +) -> None: + """Omitted processing state resolves without adding embedded placeholders.""" + original = minimal_configuration() + + configuration = _load_document(tmp_path, original) + + assert configuration.schema_version == "1.0" + assert configuration.metadata.id == "org.example.synthetic-exact-comparison" + assert configuration.record_key == ("record_id",) + assert configuration.in_scope == ("record_id", "measured_value") + measured = configuration.field("measured_value") + assert measured.review_priority == "normal" + assert measured.source_normalisation.reference == () + assert measured.source_normalisation.query == () + assert measured.value_mapping.reference == () + assert measured.value_mapping.query == () + assert measured.typed_normalisation == () + assert measured.null_policy.missing_tokens.reference == () + assert measured.null_policy.missing_tokens.query == () + assert "review_priority" not in configuration.document["fields"]["definitions"][ + "measured_value" + ] + + returned_document = configuration.document + returned_document["metadata"]["id"] = "changed" + assert configuration.document == original + + +def test_collection_representation_resolves_omitted_side_to_json_array( + tmp_path: Path, +) -> None: + """Collection-side defaults are explicit in immutable application state.""" + configuration = _load_document( + tmp_path, _add_collection_field(minimal_configuration()) + ) + + field = configuration.field("measured_value") + assert field.collection_representation.reference.format == "json_array" + assert field.collection_representation.reference.delimiter is None + assert field.collection_representation.reference.element_normalisation == () + assert field.collection_representation.query.format == "delimited_text" + assert field.collection_representation.query.delimiter == "/" + assert field.collection_representation.query.element_normalisation[0].name == ( + "case_fold" + ) + + +def test_valid_mapping_and_normalisation_declarations_resolve_in_order( + tmp_path: Path, +) -> None: + """A representative scalar adaptor becomes ordered immutable application state.""" + document = _add_mapping_field(minimal_configuration()) + field = document["fields"]["definitions"]["measured_value"] + field["source_normalisation"] = { + "reference": [ + {"operation": "trim", "parameters": {"side": "both"}}, + {"operation": "case_fold"}, + ] + } + field["review_priority"] = "high" + + configuration = _load_document(tmp_path, document) + + resolved = configuration.field("measured_value") + operation_names = tuple( + operation.name for operation in resolved.source_normalisation.reference + ) + assert operation_names == ("trim", "case_fold") + assert resolved.value_mapping.reference[0].source.value == "positive" + assert resolved.value_mapping.reference[0].destination.value == "detected" + assert resolved.value_mapping.query == () + assert resolved.review_priority == "high" + + +def test_yaml_12_core_scalars_do_not_apply_yaml_11_boolean_or_timestamp_rules( + tmp_path: Path, +) -> None: + """Common YAML 1.1 words and dates remain strings under YAML 1.2 rules.""" + text = yaml.safe_dump(minimal_configuration(), sort_keys=False) + text = text.replace( + " revision: 1.0.0\n", + " revision: 1.0.0\n title: yes\n description: 2026-11-06\n", + ) + path = tmp_path / "rules.yaml" + path.write_text(text, encoding="utf-8") + + configuration = load_comparison_configuration(path) + + assert configuration.document["metadata"]["title"] == "yes" + assert configuration.document["metadata"]["description"] == "2026-11-06" + + +def test_yaml_12_leading_zero_integer_is_decimal_not_yaml_11_octal( + tmp_path: Path, +) -> None: + """YAML 1.2 decimal construction is used after implicit scalar resolution.""" + document = minimal_configuration() + field = document["fields"]["definitions"]["measured_value"] + field["typed_normalisation"] = [ + { + "operation": "decimal_quantise", + "parameters": {"places": 1, "rounding": "half_even"}, + } + ] + text = yaml.safe_dump(document, sort_keys=False).replace("places: 1", "places: 01") + path = tmp_path / "rules.yaml" + path.write_text(text, encoding="utf-8") + + configuration = load_comparison_configuration(path) + + operation = configuration.field("measured_value").typed_normalisation[0] + assert operation.parameters == (("places", 1), ("rounding", "half_even")) + + +@pytest.mark.parametrize( + "text", + [ + 'schema_version: "1.0"\nschema_version: "1.0"\n', + 'metadata: &metadata\n id: "example"\ncopy: *metadata\n', + 'metadata: !!map {id: "example"}\n', + 'metadata:\n <<: {id: "example"}\n', + '---\nschema_version: "1.0"\n---\nschema_version: "1.0"\n', + 'payload: !!python/object/apply:os.system ["false"]\n', + ], + ids=["duplicate", "anchor-alias", "explicit-tag", "merge", "multiple", "unsafe"], +) +def test_forbidden_yaml_features_are_invalid_syntax( + tmp_path: Path, + text: str, +) -> None: + """The loader rejects ambiguous or executable YAML features before validation.""" + path = tmp_path / "rules.yaml" + path.write_text(text, encoding="utf-8") + + with pytest.raises(ConfigurationError) as raised: + load_comparison_configuration(path) + + assert raised.value.code == "invalid_syntax" + assert raised.value.path == "$" + assert raised.value.diagnostic.startswith( + "ERROR [configuration.invalid_syntax]: $: " + ) + + +def _unknown_source_operation(document: JsonObject) -> None: + document["fields"]["definitions"]["measured_value"][ + "source_normalisation" + ] = {"reference": [{"operation": "invented"}]} + + +def _invalid_numeric_tolerance(document: JsonObject) -> None: + document["fields"]["definitions"]["measured_value"]["comparator"] = { + "name": "numeric_tolerance", + "parameters": { + "absolute_tolerance": "0.1", + "relative_tolerance": "0.1", + }, + } + + +def _null_mapping_source(document: JsonObject) -> None: + field = _add_mapping_field(document)["fields"]["definitions"]["measured_value"] + field["value_mapping"]["reference"][0]["from"] = {"category": "null"} + + +def _preserved_field_with_processing(document: JsonObject) -> None: + field = document["fields"]["definitions"]["measured_value"] + field.pop("logical_type") + field["comparator"] = {"name": "preserved_exact"} + field["source_normalisation"] = { + "reference": [{"operation": "case_fold"}] + } + + +def _collection_with_field_processing(document: JsonObject) -> None: + field = _add_collection_field(document)["fields"]["definitions"][ + "measured_value" + ] + field["source_normalisation"] = { + "reference": [{"operation": "case_fold"}] + } + + +@pytest.mark.parametrize( + ("mutation", "expected_path"), + [ + (lambda document: document.update({"unknown": True}), "$.unknown"), + (lambda document: document["metadata"].pop("revision"), "$.metadata.revision"), + (lambda document: document["metadata"].update({"id": None}), "$.metadata.id"), + ( + lambda document: document["fields"]["definitions"][ + "measured_value" + ].update({"source_normalisation": {}}), + "$.fields.definitions.measured_value.source_normalisation", + ), + ( + lambda document: document["fields"]["definitions"][ + "measured_value" + ].update({"review_priority": "urgent"}), + "$.fields.definitions.measured_value.review_priority", + ), + ( + _unknown_source_operation, + "$.fields.definitions.measured_value.source_normalisation.reference[0]", + ), + ( + _invalid_numeric_tolerance, + "$.fields.definitions.measured_value.comparator", + ), + ( + _null_mapping_source, + "$.fields.definitions.measured_value.value_mapping.reference[0].from", + ), + ( + _preserved_field_with_processing, + "$.fields.definitions.measured_value", + ), + ( + _collection_with_field_processing, + "$.fields.definitions.measured_value", + ), + ], + ids=[ + "unknown", + "required", + "null", + "empty-optional", + "enum", + "unknown-operation", + "tolerance-parameters", + "null-mapping-source", + "preserved-processing", + "collection-field-processing", + ], +) +def test_schema_violations_report_the_relevant_configuration_path( + tmp_path: Path, + mutation: ConfigurationMutation, + expected_path: str, +) -> None: + """Structural failures are distinct and point to the affected property.""" + document = minimal_configuration() + mutation(document) + + with pytest.raises(ConfigurationError) as raised: + _load_document(tmp_path, document) + + assert raised.value.code == "schema_violation" + assert raised.value.path == expected_path + assert expected_path in raised.value.diagnostic + + +def _key_names_unknown(document: JsonObject) -> None: + document["fields"]["record_key"] = ["unknown"] + + +def _scope_name_unknown(document: JsonObject) -> None: + document["fields"]["in_scope"].append("unknown") + + +def _key_not_in_scope(document: JsonObject) -> None: + document["fields"]["in_scope"].remove("record_id") + + +def _key_not_required(document: JsonObject) -> None: + document["fields"]["definitions"]["record_id"]["required"]["query"] = False + + +def _key_allows_null(document: JsonObject) -> None: + field = document["fields"]["definitions"]["record_id"] + field["null_policy"] = {"allowed": True, "nulls_equal": True} + + +def _key_is_collection(document: JsonObject) -> None: + field = document["fields"]["definitions"]["record_id"] + field["logical_type"] = { + "collection_type": "set", + "element_type": "string", + "element_comparator": {"name": "typed_exact"}, + } + field["comparator"] = {"name": "set_exact"} + + +def _key_comparator_not_typed_exact(document: JsonObject) -> None: + field = document["fields"]["definitions"]["record_id"] + field["logical_type"] = "integer" + field["comparator"] = {"name": "numeric_exact"} + + +def _incompatible_comparator(document: JsonObject) -> None: + document["fields"]["definitions"]["measured_value"]["comparator"] = { + "name": "date_exact" + } + + +def _incompatible_normalisation(document: JsonObject) -> None: + document["fields"]["definitions"]["measured_value"][ + "typed_normalisation" + ] = [ + { + "operation": "datetime_truncate", + "parameters": {"precision": "second"}, + } + ] + + +def _normalised_string_without_operation(document: JsonObject) -> None: + field = document["fields"]["definitions"]["measured_value"] + field["logical_type"] = "string" + field["comparator"] = {"name": "normalised_string"} + + +def _duplicate_reference_source(document: JsonObject) -> None: + fields = document["fields"]["definitions"] + fields["measured_value"]["source_fields"]["reference"] = "reference_id" + + +def _duplicate_query_source(document: JsonObject) -> None: + fields = document["fields"]["definitions"] + fields["measured_value"]["source_fields"]["query"] = "query_id" + + +def _reserved_identifier(document: JsonObject) -> None: + document["metadata"]["id"] = "org.adviso.automatic-strict-exact" + + +def _duplicate_mapping_source(document: JsonObject) -> None: + field = _add_mapping_field(document)["fields"]["definitions"]["measured_value"] + field["value_mapping"]["reference"].append( + { + "from": {"category": "string", "value": "positive"}, + "to": {"category": "string", "value": "present"}, + } + ) + + +def _mapping_sources_collide_after_normalisation(document: JsonObject) -> None: + field = _add_mapping_field(document)["fields"]["definitions"]["measured_value"] + field["source_normalisation"] = { + "reference": [{"operation": "case_fold"}] + } + field["value_mapping"]["reference"].append( + { + "from": {"category": "string", "value": "POSITIVE"}, + "to": {"category": "string", "value": "present"}, + } + ) + + +def _numeric_mapping_sources_are_equal(document: JsonObject) -> None: + field = document["fields"]["definitions"]["measured_value"] + field["value_mapping"] = { + "reference": [ + { + "from": {"category": "decimal", "value": "1.0"}, + "to": {"category": "decimal", "value": "1.0"}, + }, + { + "from": {"category": "decimal", "value": "1.00"}, + "to": {"category": "decimal", "value": "2.0"}, + }, + ] + } + + +def _mapping_source_is_missing_token(document: JsonObject) -> None: + field = _add_mapping_field(document)["fields"]["definitions"]["measured_value"] + field["null_policy"] = { + "allowed": True, + "nulls_equal": True, + "missing_tokens": { + "reference": [{"category": "string", "value": "positive"}] + }, + } + + +def _element_mapping_sources_collide(document: JsonObject) -> None: + field = _add_collection_field(document)["fields"]["definitions"][ + "measured_value" + ] + field["collection_representation"]["query"]["element_mapping"].append( + { + "from": {"category": "string", "value": "WILD_TYPE"}, + "to": {"category": "string", "value": "wild"}, + } + ) + + +@pytest.mark.parametrize( + ("mutation", "expected_path", "expected_reason"), + [ + ( + _key_names_unknown, + "$.fields.record_key[0]", + "record-key field has no field definition", + ), + ( + _scope_name_unknown, + "$.fields.in_scope[2]", + "in-scope field has no field definition", + ), + ( + _key_not_in_scope, + "$.fields.record_key[0]", + "record-key field is not included in fields.in_scope", + ), + ( + _key_not_required, + "$.fields.definitions.record_id.required.query", + "record-key field must be required on both sides", + ), + ( + _key_allows_null, + "$.fields.definitions.record_id.null_policy.allowed", + "record-key field must not allow null", + ), + ( + _key_is_collection, + "$.fields.definitions.record_id.logical_type", + "record-key field must have a scalar logical type", + ), + ( + _key_comparator_not_typed_exact, + "$.fields.definitions.record_id.comparator.name", + "record-key field must use typed_exact", + ), + ( + _incompatible_comparator, + "$.fields.definitions.measured_value.comparator.name", + "comparator is incompatible with the field logical type", + ), + ( + _incompatible_normalisation, + "$.fields.definitions.measured_value.typed_normalisation[0].operation", + "typed normalisation is incompatible with the field logical type", + ), + ( + _normalised_string_without_operation, + "$.fields.definitions.measured_value.comparator.name", + "normalised_string requires a configured string normalisation", + ), + ( + _duplicate_reference_source, + "$.fields.definitions.measured_value.source_fields.reference", + "source field is already used by canonical field", + ), + ( + _duplicate_query_source, + "$.fields.definitions.measured_value.source_fields.query", + "source field is already used by canonical field", + ), + ( + _reserved_identifier, + "$.metadata.id", + "identifier is reserved for automatically generated configurations", + ), + ( + _duplicate_mapping_source, + "$.fields.definitions.measured_value.value_mapping.reference[1].from", + "mapping source duplicates another source after normalisation", + ), + ( + _mapping_sources_collide_after_normalisation, + "$.fields.definitions.measured_value.value_mapping.reference[1].from", + "mapping source duplicates another source after normalisation", + ), + ( + _numeric_mapping_sources_are_equal, + "$.fields.definitions.measured_value.value_mapping.reference[1].from", + "mapping source duplicates another source after normalisation", + ), + ( + _mapping_source_is_missing_token, + "$.fields.definitions.measured_value.value_mapping.reference[0].from", + "mapping source is unreachable because it is a missing token", + ), + ( + _element_mapping_sources_collide, + "$.fields.definitions.measured_value.collection_representation.query." + "element_mapping[1].from", + "element mapping source duplicates another source after normalisation", + ), + ], + ids=[ + "key-reference", + "scope-reference", + "key-scope", + "key-contract", + "key-null", + "key-collection", + "key-comparator", + "comparator-type", + "normalisation-type", + "normalised-string", + "source-reuse", + "source-reuse-query", + "reserved-id", + "mapping-duplicate", + "mapping-normalised-duplicate", + "mapping-numeric-duplicate", + "mapping-unreachable", + "element-mapping-normalised-duplicate", + ], +) +def test_semantic_violations_report_the_specific_configuration_path( + tmp_path: Path, + mutation: ConfigurationMutation, + expected_path: str, + expected_reason: str, +) -> None: + """Cross-property failures are detected only after structural validation.""" + document = minimal_configuration() + mutation(document) + + with pytest.raises(ConfigurationError) as raised: + _load_document(tmp_path, document) + + assert raised.value.code == "semantic_violation" + assert raised.value.path == expected_path + assert expected_reason in raised.value.message + assert expected_path in raised.value.diagnostic + assert expected_reason in raised.value.diagnostic + + +def test_unsupported_version_and_unreadable_file_have_distinct_errors( + tmp_path: Path, +) -> None: + """Version negotiation and filesystem access retain their taxonomy.""" + unsupported = minimal_configuration() + unsupported["schema_version"] = "2.0" + + with pytest.raises(ConfigurationError) as version_error: + _load_document(tmp_path, unsupported) + with pytest.raises(ConfigurationError) as read_error: + load_comparison_configuration(tmp_path / "missing.yaml") + + assert version_error.value.code == "unsupported_version" + assert version_error.value.path == "$.schema_version" + assert read_error.value.code == "unreadable" + assert read_error.value.path == "$" + assert str(tmp_path) not in read_error.value.diagnostic + + +def test_invalid_utf8_and_non_mapping_root_are_classified_separately( + tmp_path: Path, +) -> None: + """Byte/YAML failures precede schema validation of a parsed root value.""" + invalid_utf8 = tmp_path / "encoding.yaml" + invalid_utf8.write_bytes(b"\xff") + sequence_root = tmp_path / "sequence.yaml" + sequence_root.write_text("- item\n", encoding="utf-8") + + with pytest.raises(ConfigurationError) as encoding_error: + load_comparison_configuration(invalid_utf8) + with pytest.raises(ConfigurationError) as root_error: + load_comparison_configuration(sequence_root) + + assert encoding_error.value.code == "invalid_syntax" + assert root_error.value.code == "schema_violation" + assert root_error.value.path == "$" + + +def test_custom_configuration_failure_precedes_all_dataset_access( + tmp_path: Path, +) -> None: + """Orchestration cannot inspect an input before custom rules are trustworthy.""" + document = minimal_configuration() + _reserved_identifier(document) + configuration_path = _write_yaml(tmp_path / "invalid.yaml", document) + invocation = ComparisonInvocation( + reference=tmp_path / "missing-reference.csv", + query=tmp_path / "missing-query.json", + output=tmp_path / "evidence", + configuration=configuration_path, + ) + + def unexpected_dataset_loader(path: Path, side: InputSide) -> ParsedDataset: + raise AssertionError(f"dataset loader accessed {side.value}: {path!r}") + + with pytest.raises(ConfigurationError) as raised: + prepare_comparison(invocation, dataset_loader=unexpected_dataset_loader) + + assert raised.value.code == "semantic_violation" + assert raised.value.path == "$.metadata.id" From 68dfd858e522190c633942078a31e6ab86ebde72 Mon Sep 17 00:00:00 2001 From: Kevin Howe Date: Fri, 6 Nov 2026 13:09:17 +0000 Subject: [PATCH 19/45] Refactoring/simplificaition after review --- .../src/adviso_compare/configuration.py | 138 ++++++------------ .../src/adviso_compare/processing.py | 77 ++++++++++ .../tests/test_configuration.py | 25 ++-- packages/adviso-compare/tests/test_inputs.py | 16 +- 4 files changed, 148 insertions(+), 108 deletions(-) create mode 100644 packages/adviso-compare/src/adviso_compare/processing.py diff --git a/packages/adviso-compare/src/adviso_compare/configuration.py b/packages/adviso-compare/src/adviso_compare/configuration.py index 1252462..56b0e73 100644 --- a/packages/adviso-compare/src/adviso_compare/configuration.py +++ b/packages/adviso-compare/src/adviso_compare/configuration.py @@ -5,12 +5,12 @@ import copy import json import re -import unicodedata -from dataclasses import dataclass +from dataclasses import dataclass, field as dataclass_field from decimal import Decimal from importlib.resources import files from pathlib import Path -from typing import Any, Generic, Iterator, Mapping, TypeVar +from types import MappingProxyType +from typing import Any, Generic, Mapping, TypeVar import yaml from jsonschema import Draft202012Validator, FormatChecker, ValidationError @@ -18,6 +18,12 @@ from yaml.nodes import MappingNode, ScalarNode from yaml.tokens import AliasToken, AnchorToken, TagToken +from adviso_compare.processing import ( + apply_source_text_operation, + field_comparator_is_compatible, + typed_normalisation_is_compatible, +) + CONFIGURATION_SCHEMA_FILE = "comparison-configuration-v1.0.schema.json" CONFIGURATION_SCHEMA_PACKAGE = "adviso_compare.schemas" @@ -41,7 +47,9 @@ class Operation: """One registered processing operation and its immutable parameters.""" name: str - parameters: tuple[tuple[str, object], ...] = () + parameters: Mapping[str, object] = dataclass_field( + default_factory=lambda: MappingProxyType({}) + ) @dataclass(frozen=True, slots=True) @@ -65,7 +73,9 @@ class Comparator: """A comparator declaration and its immutable parameters.""" name: str - parameters: tuple[tuple[str, object], ...] = () + parameters: Mapping[str, object] = dataclass_field( + default_factory=lambda: MappingProxyType({}) + ) @dataclass(frozen=True, slots=True) @@ -106,7 +116,7 @@ class FieldDefinition: logical_type: str | CollectionLogicalType | None null_policy: NullPolicy source_normalisation: DatasetSides[tuple[Operation, ...]] - collection_representation: DatasetSides[CollectionRepresentation | None] + collection_representation: DatasetSides[CollectionRepresentation] | None value_mapping: DatasetSides[tuple[ValueMappingEntry, ...]] typed_normalisation: tuple[Operation, ...] comparator: Comparator @@ -131,21 +141,13 @@ class EffectiveConfiguration: metadata: ConfigurationMetadata record_key: tuple[str, ...] in_scope: tuple[str, ...] - _fields: tuple[tuple[str, FieldDefinition], ...] - _document_json: str + fields: Mapping[str, FieldDefinition] + _document: JsonObject = dataclass_field(repr=False, compare=False) @property def document(self) -> JsonObject: """Return a fresh JSON-compatible copy of the validated source document.""" - document = json.loads(self._document_json) - if not isinstance(document, dict): - raise AssertionError("validated configuration document is not an object") - return document - - @property - def fields(self) -> Mapping[str, FieldDefinition]: - """Return the resolved field definitions as a read-only mapping.""" - return _TupleMapping(self._fields) + return copy.deepcopy(self._document) def field(self, name: str) -> FieldDefinition: """Return one resolved canonical field definition.""" @@ -155,25 +157,6 @@ def field(self, name: str) -> FieldDefinition: raise KeyError(f"unknown canonical field: {name}") from error -class _TupleMapping(Mapping[str, T]): - """A minimal immutable mapping backed by ordered key-value tuples.""" - - def __init__(self, items: tuple[tuple[str, T], ...]) -> None: - self._items = items - - def __getitem__(self, key: str) -> T: - for candidate, value in self._items: - if candidate == key: - return value - raise KeyError(key) - - def __iter__(self) -> Iterator[str]: - return (key for key, _ in self._items) - - def __len__(self) -> int: - return len(self._items) - - @dataclass(frozen=True, slots=True) class ConfigurationError(Exception): """A classified expected failure while loading a custom configuration.""" @@ -541,21 +524,17 @@ def _validate_comparator_compatibility( ) -> None: """Require the field comparator to match its resolved logical type.""" comparator = field["comparator"]["name"] - if logical_type is None: - permitted = {"preserved_exact"} - elif isinstance(logical_type, str): - permitted_by_type = { - "string": {"typed_exact", "normalised_string"}, - "integer": {"typed_exact", "numeric_exact", "numeric_tolerance"}, - "decimal": {"typed_exact", "numeric_exact", "numeric_tolerance"}, - "boolean": {"typed_exact"}, - "date": {"typed_exact", "date_exact"}, - "datetime": {"typed_exact", "datetime_exact"}, - } - permitted = permitted_by_type[logical_type] + if isinstance(logical_type, dict): + compatible = field_comparator_is_compatible( + comparator, + collection_type=logical_type["collection_type"], + ) else: - permitted = {f"{logical_type['collection_type']}_exact"} - if comparator not in permitted: + compatible = field_comparator_is_compatible( + comparator, + scalar_type=logical_type, + ) + if not compatible: _semantic_error( f"{path}.comparator.name", "comparator is incompatible with the field logical type", @@ -577,12 +556,11 @@ def _validate_typed_normalisation( field: JsonObject, ) -> None: """Require each typed operation to match the scalar destination type.""" - compatible_type = { - "decimal_quantise": "decimal", - "datetime_truncate": "datetime", - } for index, operation in enumerate(field.get("typed_normalisation", [])): - if logical_type != compatible_type[operation["operation"]]: + if not typed_normalisation_is_compatible( + operation["operation"], + logical_type if isinstance(logical_type, str) else None, + ): _semantic_error( f"{path}.typed_normalisation[{index}].operation", "typed normalisation is incompatible with the field logical type", @@ -650,28 +628,14 @@ def _descriptor_key( value = Decimal(value) elif category == "string": for operation in operations: - value = _normalise_configuration_text(value, operation) + value = apply_source_text_operation( + value, + operation["operation"], + operation.get("parameters", {}), + ) return category, value -def _normalise_configuration_text(value: str, operation: JsonObject) -> str: - """Apply source-text operation semantics needed for mapping reachability.""" - name = operation["operation"] - parameters = operation.get("parameters", {}) - if name == "unicode_normalise": - return unicodedata.normalize(parameters["form"], value) - if name == "trim": - side = parameters["side"] - if side == "leading": - return value.lstrip() - if side == "trailing": - return value.rstrip() - return value.strip() - if name == "case_fold": - return value.casefold() - raise AssertionError(f"unexpected validated source operation: {name}") - - def _field_path(field_name: str) -> str: """Return the unambiguous configuration path for a canonical field.""" return _configuration_path(["fields", "definitions", field_name]) @@ -680,10 +644,10 @@ def _field_path(field_name: str) -> str: def _build_effective_configuration(document: JsonObject) -> EffectiveConfiguration: """Resolve optional declarations into immutable application values.""" metadata = document["metadata"] - fields = tuple( - (name, _build_field(name, field)) + fields = { + name: _build_field(name, field) for name, field in document["fields"]["definitions"].items() - ) + } return EffectiveConfiguration( schema_version=document["schema_version"], metadata=ConfigurationMetadata( @@ -694,13 +658,8 @@ def _build_effective_configuration(document: JsonObject) -> EffectiveConfigurati ), record_key=tuple(document["fields"]["record_key"]), in_scope=tuple(document["fields"]["in_scope"]), - _fields=fields, - _document_json=json.dumps( - document, - ensure_ascii=False, - separators=(",", ":"), - allow_nan=False, - ), + fields=MappingProxyType(fields), + _document=copy.deepcopy(document), ) @@ -724,16 +683,15 @@ def _build_field(name: str, field: JsonObject) -> FieldDefinition: value_mapping = field.get("value_mapping", {}) collection_document = field.get("collection_representation", {}) if isinstance(logical_type, CollectionLogicalType): - collection_representation: DatasetSides[ - CollectionRepresentation | None - ] = DatasetSides( + collection_representation: DatasetSides[CollectionRepresentation] | None + collection_representation = DatasetSides( reference=_build_collection_representation( collection_document.get("reference") ), query=_build_collection_representation(collection_document.get("query")), ) else: - collection_representation = DatasetSides(reference=None, query=None) + collection_representation = None return FieldDefinition( name=name, @@ -784,7 +742,7 @@ def _build_operation(document: JsonObject) -> Operation: """Build one immutable processing operation.""" return Operation( name=document["operation"], - parameters=tuple(document.get("parameters", {}).items()), + parameters=MappingProxyType(dict(document.get("parameters", {}))), ) @@ -792,7 +750,7 @@ def _build_comparator(document: JsonObject) -> Comparator: """Build one immutable comparator declaration.""" return Comparator( name=document["name"], - parameters=tuple(document.get("parameters", {}).items()), + parameters=MappingProxyType(dict(document.get("parameters", {}))), ) diff --git a/packages/adviso-compare/src/adviso_compare/processing.py b/packages/adviso-compare/src/adviso_compare/processing.py new file mode 100644 index 0000000..d360b4c --- /dev/null +++ b/packages/adviso-compare/src/adviso_compare/processing.py @@ -0,0 +1,77 @@ +"""Shared runtime semantics for configured value processing.""" + +from __future__ import annotations + +import unicodedata +from types import MappingProxyType +from typing import Mapping + + +_SCALAR_COMPARATORS: Mapping[str, frozenset[str]] = MappingProxyType( + { + "string": frozenset({"typed_exact", "normalised_string"}), + "integer": frozenset( + {"typed_exact", "numeric_exact", "numeric_tolerance"} + ), + "decimal": frozenset( + {"typed_exact", "numeric_exact", "numeric_tolerance"} + ), + "boolean": frozenset({"typed_exact"}), + "date": frozenset({"typed_exact", "date_exact"}), + "datetime": frozenset({"typed_exact", "datetime_exact"}), + } +) + +_TYPED_NORMALISATION_TYPES: Mapping[str, str] = MappingProxyType( + { + "decimal_quantise": "decimal", + "datetime_truncate": "datetime", + } +) + + +def field_comparator_is_compatible( + comparator: str, + *, + scalar_type: str | None = None, + collection_type: str | None = None, +) -> bool: + """Return whether a comparator can assess the identified field type.""" + if collection_type is not None: + return comparator == f"{collection_type}_exact" + if scalar_type is None: + return comparator == "preserved_exact" + return comparator in _SCALAR_COMPARATORS[scalar_type] + + +def typed_normalisation_is_compatible( + operation: str, + logical_type: str | None, +) -> bool: + """Return whether a typed operation accepts the identified scalar type.""" + return _TYPED_NORMALISATION_TYPES[operation] == logical_type + + +def apply_source_text_operation( + value: str, + operation: str, + parameters: Mapping[str, object], +) -> str: + """Apply one registered source-text operation to a string value.""" + if operation == "unicode_normalise": + form = parameters["form"] + if not isinstance(form, str): + raise AssertionError("validated Unicode form is not a string") + return unicodedata.normalize(form, value) + if operation == "trim": + side = parameters["side"] + if side == "leading": + return value.lstrip() + if side == "trailing": + return value.rstrip() + if side == "both": + return value.strip() + raise AssertionError("validated trim side is not registered") + if operation == "case_fold": + return value.casefold() + raise AssertionError(f"validated source operation is not registered: {operation}") diff --git a/packages/adviso-compare/tests/test_configuration.py b/packages/adviso-compare/tests/test_configuration.py index 14d337d..745fee1 100644 --- a/packages/adviso-compare/tests/test_configuration.py +++ b/packages/adviso-compare/tests/test_configuration.py @@ -112,12 +112,14 @@ def test_collection_representation_resolves_omitted_side_to_json_array( ) field = configuration.field("measured_value") - assert field.collection_representation.reference.format == "json_array" - assert field.collection_representation.reference.delimiter is None - assert field.collection_representation.reference.element_normalisation == () - assert field.collection_representation.query.format == "delimited_text" - assert field.collection_representation.query.delimiter == "/" - assert field.collection_representation.query.element_normalisation[0].name == ( + representations = field.collection_representation + assert representations is not None + assert representations.reference.format == "json_array" + assert representations.reference.delimiter is None + assert representations.reference.element_normalisation == () + assert representations.query.format == "delimited_text" + assert representations.query.delimiter == "/" + assert representations.query.element_normalisation[0].name == ( "case_fold" ) @@ -186,7 +188,7 @@ def test_yaml_12_leading_zero_integer_is_decimal_not_yaml_11_octal( configuration = load_comparison_configuration(path) operation = configuration.field("measured_value").typed_normalisation[0] - assert operation.parameters == (("places", 1), ("rounding", "half_even")) + assert operation.parameters == {"places": 1, "rounding": "half_even"} @pytest.mark.parametrize( @@ -413,12 +415,17 @@ def _duplicate_mapping_source(document: JsonObject) -> None: def _mapping_sources_collide_after_normalisation(document: JsonObject) -> None: field = _add_mapping_field(document)["fields"]["definitions"]["measured_value"] + field["value_mapping"]["reference"][0]["from"]["value"] = " É " field["source_normalisation"] = { - "reference": [{"operation": "case_fold"}] + "reference": [ + {"operation": "unicode_normalise", "parameters": {"form": "NFC"}}, + {"operation": "trim", "parameters": {"side": "both"}}, + {"operation": "case_fold"}, + ] } field["value_mapping"]["reference"].append( { - "from": {"category": "string", "value": "POSITIVE"}, + "from": {"category": "string", "value": "e\u0301"}, "to": {"category": "string", "value": "present"}, } ) diff --git a/packages/adviso-compare/tests/test_inputs.py b/packages/adviso-compare/tests/test_inputs.py index 54f525e..be3f5f7 100644 --- a/packages/adviso-compare/tests/test_inputs.py +++ b/packages/adviso-compare/tests/test_inputs.py @@ -285,23 +285,21 @@ def test_invalid_utf8_is_rejected_before_format_parsing( assert raised.value.code == "invalid_encoding" -@pytest.mark.parametrize("token", ["NaN", "Infinity", "-Infinity"]) @pytest.mark.parametrize( - "document_template", + "document", [ - "{}", - '[{{"value":{}}}]', - '[{{"value":{{"nested":{}}}}}]', - '[{{"value":[{}]}}]', + b"NaN", + b'[{"value":Infinity}]', + b'[{"value":{"nested":-Infinity}}]', + b'[{"value":[NaN]}]', ], + ids=["root", "field", "nested-object", "nested-array"], ) def test_non_finite_json_tokens_are_rejected_in_every_structural_context( tmp_path: Path, - token: str, - document_template: str, + document: bytes, ) -> None: """No parser location enables non-standard JSON numerical constants.""" - document = document_template.format(token).encode() path = _write_bytes(tmp_path / "records.json", document) with pytest.raises(InputError) as raised: From ba2e4c2fa44f2ca0ef18b3869485d559abac7a78 Mon Sep 17 00:00:00 2001 From: Kevin Howe Date: Mon, 9 Nov 2026 15:43:09 +0000 Subject: [PATCH 20/45] Tightened up requirements relating to parsing of numerical values (including dates) --- packages/adviso-compare/README.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/packages/adviso-compare/README.md b/packages/adviso-compare/README.md index 6d4fe27..148975b 100644 --- a/packages/adviso-compare/README.md +++ b/packages/adviso-compare/README.md @@ -8,9 +8,11 @@ The project is intended to support bioinformatics validation and verification ac Version 1 implementation is in progress. The schema-validation baseline, command-line invocation layer, deterministic CSV, TSV, and JSON input parser, -and custom comparison-configuration loader are implemented. Automatic -configuration generation, the comparison kernel, and evidence generation are -not yet available. +custom comparison-configuration loader, and source-to-canonical field +translation pipeline are implemented. Automatic strict-exact configuration +generation and unambiguous custom or automatic record identity are also +implemented. Contract assessment, record alignment, value comparison, and +evidence generation are not yet available. The current normative artefacts are: From 6bea75a105d94ebf50755ad0f0f0c9544cb97344 Mon Sep 17 00:00:00 2001 From: Kevin Howe Date: Mon, 9 Nov 2026 15:44:57 +0000 Subject: [PATCH 21/45] Tightened up requirements relating to parsing and representation of numerical values --- packages/adviso-compare/docs/SPECIFICATION.md | 45 +++++++++++++++++-- 1 file changed, 41 insertions(+), 4 deletions(-) diff --git a/packages/adviso-compare/docs/SPECIFICATION.md b/packages/adviso-compare/docs/SPECIFICATION.md index b8a56b0..0b298be 100644 --- a/packages/adviso-compare/docs/SPECIFICATION.md +++ b/packages/adviso-compare/docs/SPECIFICATION.md @@ -214,6 +214,11 @@ For CSV and TSV: - **REQ-INP-037.** An invalid-root error MUST identify the invalid root value or array item. - **REQ-INP-038.** An invalid-root error MUST explain the required structure. - **REQ-INP-070.** Duplicate member names in any JSON object MUST cause an input error, including duplicates in record objects and nested object-valued fields. A parser MUST detect duplicates before constructing a mapping that could overwrite an earlier member. +- **REQ-INP-072.** A top-level JSON record object MUST NOT contain an empty + member name, because record member names participate in dataset-wide field + discovery and configuration. An empty member name in a nested object-valued + field remains valid data content. An empty record member name MUST produce an + `input.invalid_structure` execution error. ### 5.6 Parser-native value model @@ -590,6 +595,32 @@ The first-release typed-value operation identifiers are: - `decimal_quantise`, requiring a non-negative integer `places` and `rounding` with one of `half_even`, `half_up`, or `down`; and - `datetime_truncate`, requiring `precision` with one of `second`, `millisecond`, or `microsecond`. +**REQ-CFG-146.** For `decimal_quantise`, `places` is the number of +digits after the decimal point to which the value is quantised. `half_even` +MUST round to the nearest value, with an exact tie resolved to the value whose +final retained digit is even. `half_up` MUST round to the nearest value, with +an exact tie resolved away from zero. `down` MUST discard excess fractional +digits towards zero. + +**REQ-CFG-147.** The `trim` operation MUST recognise exactly the characters +with the Unicode `White_Space` property in the Unicode version recorded in the +results. It MUST remove the maximal sequence of those characters from the +configured leading end, trailing end, or both ends. It MUST NOT remove other +characters merely because the implementation language treats them as +whitespace. + +**REQ-CFG-148.** `datetime_truncate` MUST operate on the canonical UTC instant +after offset normalisation. `second` MUST discard the complete fractional-second +component, `millisecond` MUST retain at most three fractional digits, and +`microsecond` MUST retain at most six fractional digits. Discarded digits MUST +be removed without rounding. + +**REQ-CFG-149.** `case_fold` MUST apply locale-independent Unicode Default Case +Folding using the Unicode version recorded in the results. It MUST use full +case folding, including mappings that expand one character into multiple +characters. It MUST NOT apply Unicode normalisation implicitly; normalisation +occurs only through a separately configured `unicode_normalise` operation. + **REQ-CFG-066.** A source-text operation MUST appear only in `source_normalisation.reference`, `source_normalisation.query`, or a side's `collection_representation..element_normalisation`. It applies only to values or decoded elements, respectively, from the corresponding dataset side. **REQ-CFG-123.** A source-text operation MUST apply only when the current parser-native value is a string. A non-string value MUST pass through the source-normalisation stage unchanged; it MUST NOT be stringified, transformed, or rejected merely because a source-text operation is configured. @@ -792,6 +823,12 @@ The accepted parser-native inputs for each declared logical type are fixed as fo - **REQ-VAL-034.** Equivalent instants expressed with different offsets MUST have the same canonical value. - **REQ-VAL-035.** Supplied fractional-second precision MUST remain available in the source lexical evidence. The canonical datetime representation MUST remove unnecessary trailing fractional-second zeros and omit a zero-valued fractional component. Any reduction that changes the represented instant, rather than only its redundant lexical form, MUST require explicit precision normalisation. - **REQ-VAL-036.** Leap-second representations MUST be invalid in the first release. +- **REQ-VAL-092.** Datetime input MUST accept uppercase or lowercase `T` and + `Z`, but MUST NOT accept a space in place of `T`. Numeric offsets from + `+00:00` through `+23:59` and from `-00:01` through `-23:59` MUST be + accepted. `-00:00` MUST be invalid because it denotes an unknown local + offset rather than a known UTC instant. Canonical output continues to use + uppercase `T` and `Z`. #### 6.5.7 Collections @@ -1620,8 +1657,8 @@ This catalogue summarises minimum executable scenarios for the first release. It - Verifies: `REQ-CFG-053`, `REQ-CFG-054`, `REQ-CFG-055`, `REQ-CFG-056`, `REQ-CFG-057`, `REQ-CFG-122`, `REQ-VAL-014`, `REQ-VAL-047`, `REQ-VAL-048`, `REQ-VAL-049`, `REQ-VAL-050`, `REQ-VAL-051`, `REQ-VAL-052`, `REQ-VAL-053`, `REQ-VAL-054`, `REQ-VAL-055`, `REQ-VAL-086`, `REQ-INP-028` - **AT-CON-005:** Unknown, conflicting, and incomplete value mappings fail clearly; mapping to null differs from a missing map entry, and mapping from null is rejected. This consolidated parameterized scenario also covers: Source-value descriptors distinguish strings, numbers, booleans, and null in their permitted positions; mapping to null remains distinct from an absent mapping entry. Parameterized mapping fixtures verify reference/query-to-canonical field mapping, required-field reporting, exclusion of unknown source fields, declared source and destination types, permitted intentional many-to-one value mappings, and rejection of reused source column names or any other non-one-to-one column mapping within either dataset side. - Verifies: `REQ-CFG-065`, `REQ-CFG-071`, `REQ-CFG-072`, `REQ-CFG-073`, `REQ-CFG-074`, `REQ-CFG-127`, `REQ-VAL-058`, `REQ-VAL-065`, `REQ-VAL-067`, `REQ-CFG-058`, `REQ-CFG-059`, `REQ-VAL-068`, `REQ-VAL-056`, `REQ-VAL-057`, `REQ-VAL-059`, `REQ-VAL-060`, `REQ-VAL-064`, `REQ-VAL-066` -- **AT-CON-006:** Malformed syntax or structure, non-array JSON roots (including a single record object), JSON arrays containing non-object items, duplicate JSON member names at any depth, empty or duplicate tabular column names, and irregular tabular rows receive their specified error classification without silent data loss. This consolidated parameterized scenario also covers: A table-driven parser suite verifies the fixed CSV and TSV dialects, UTF-8-only decoding, CRLF/LF equivalence, quoted delimiters, doubled quotes, embedded line breaks, retention of one-based data-record and starting physical line numbers, and initial string preservation. - - Verifies: `REQ-INP-021`, `REQ-INP-022`, `REQ-INP-023`, `REQ-INP-024`, `REQ-INP-025`, `REQ-INP-034`, `REQ-INP-035`, `REQ-INP-036`, `REQ-INP-037`, `REQ-INP-038`, `REQ-INP-070`, `REQ-CMP-028`, `REQ-CMP-029`, `REQ-PRN-002`, `REQ-PRN-008`, `REQ-INP-010`, `REQ-INP-011`, `REQ-INP-012`, `REQ-INP-014`, `REQ-INP-015`, `REQ-INP-016`, `REQ-INP-017`, `REQ-INP-018`, `REQ-INP-019`, `REQ-INP-020`, `REQ-INP-026`, `REQ-INP-027`, `REQ-INP-063`, `REQ-INP-064` +- **AT-CON-006:** Malformed syntax or structure, non-array JSON roots (including a single record object), JSON arrays containing non-object items, duplicate JSON member names at any depth, empty top-level JSON record member names, empty or duplicate tabular column names, and irregular tabular rows receive their specified error classification without silent data loss. Empty member names in nested JSON object values remain valid data content. This consolidated parameterized scenario also covers: A table-driven parser suite verifies the fixed CSV and TSV dialects, UTF-8-only decoding, CRLF/LF equivalence, quoted delimiters, doubled quotes, embedded line breaks, retention of one-based data-record and starting physical line numbers, and initial string preservation. + - Verifies: `REQ-INP-021`, `REQ-INP-022`, `REQ-INP-023`, `REQ-INP-024`, `REQ-INP-025`, `REQ-INP-034`, `REQ-INP-035`, `REQ-INP-036`, `REQ-INP-037`, `REQ-INP-038`, `REQ-INP-070`, `REQ-INP-072`, `REQ-CMP-028`, `REQ-CMP-029`, `REQ-PRN-002`, `REQ-PRN-008`, `REQ-INP-010`, `REQ-INP-011`, `REQ-INP-012`, `REQ-INP-014`, `REQ-INP-015`, `REQ-INP-016`, `REQ-INP-017`, `REQ-INP-018`, `REQ-INP-019`, `REQ-INP-020`, `REQ-INP-026`, `REQ-INP-027`, `REQ-INP-063`, `REQ-INP-064` - **AT-CON-007:** JSON `[]` and header-only CSV/TSV are valid zero-record datasets, while zero-byte inputs receive their format-specific input errors. With a custom configuration, two valid zero-record datasets have complete records and values phases with no record differences or value opportunities. In automatic mode, two header-only tabular datasets containing `record_id` behave likewise, while two empty JSON arrays produce `input.invalid_record_key` because they expose no selectable key. Per-record requirements produce neither findings nor not-assessable classifications; available tabular header structure still produces applicable dataset-level contract findings. A one-sided zero-record comparison reports every record as missing or extra when a valid key is available and otherwise receives the specified automatic-key error. - Verifies: `REQ-INP-052`, `REQ-INP-053`, `REQ-INP-054`, `REQ-INP-055`, `REQ-INP-056`, `REQ-INP-057`, `REQ-INP-058`, `REQ-INP-059`, `REQ-INP-066`, `REQ-INP-067`, `REQ-INP-068`, `REQ-INP-069`, `REQ-INP-071`, `REQ-CMP-009`, `REQ-CMP-010`, `REQ-FND-022` - **AT-CON-008:** Comparison-configuration YAML scalar resolution and cross-format typed dataset parsing are deterministic across supported environments. @@ -1674,11 +1711,11 @@ This catalogue summarises minimum executable scenarios for the first release. It - **AT-VAL-005:** Boolean parsing accepts native booleans and lowercase textual `true` or `false`; other representations require a mapping. - Verifies: `REQ-VAL-024`, `REQ-VAL-025`, `REQ-VAL-026`, `REQ-VAL-027` - **AT-VAL-006:** Dates reject invalid or non-ISO full dates; datetimes require offsets and compare as instants. - - Verifies: `REQ-VAL-028`, `REQ-VAL-029`, `REQ-VAL-030`, `REQ-VAL-031`, `REQ-VAL-032`, `REQ-VAL-033`, `REQ-VAL-034`, `REQ-VAL-035`, `REQ-VAL-036`, `REQ-CMP-024` + - Verifies: `REQ-VAL-028`, `REQ-VAL-029`, `REQ-VAL-030`, `REQ-VAL-031`, `REQ-VAL-032`, `REQ-VAL-033`, `REQ-VAL-034`, `REQ-VAL-035`, `REQ-VAL-036`, `REQ-VAL-092`, `REQ-CMP-024` - **AT-VAL-007:** Lists preserve order and duplicates, sets reject typed-canonical duplicates and ignore order, and multisets ignore order while preserving occurrence counts. List elements support positional typed-exact comparison and numeric tolerance for numeric element types; sets and multisets accept only typed-exact element comparison, and every other element-comparator combination is rejected. This consolidated parameterized scenario also covers: the default and explicit `json_array` representations accept JSON arrays in JSON and JSON-array cell syntax in CSV/TSV; ordinary delimited text is rejected unless `delimited_text` is explicitly configured for that side; literal single- and multi-character delimiters, empty input, empty elements, lack of collection-level escaping, element normalisation, closed element mapping, and element typed parsing follow the defined order; and null collection elements produce the specified finding and `not_assessable` evidence. The suite covers every supported typed collection kind and its required element declaration. - Verifies: `REQ-CFG-124`, `REQ-CFG-125`, `REQ-CFG-126`, `REQ-CFG-137`, `REQ-CFG-138`, `REQ-CFG-139`, `REQ-CFG-140`, `REQ-CFG-141`, `REQ-CFG-142`, `REQ-VAL-037`, `REQ-VAL-038`, `REQ-VAL-039`, `REQ-VAL-040`, `REQ-VAL-041`, `REQ-VAL-042`, `REQ-CMP-025`, `REQ-CMP-040`, `REQ-CMP-041`, `REQ-VAL-043`, `REQ-VAL-044`, `REQ-VAL-045`, `REQ-VAL-046`, `REQ-VAL-006`, `REQ-VAL-081`, `REQ-VAL-082`, `REQ-VAL-087`, `REQ-VAL-088`, `REQ-VAL-089`, `REQ-VAL-090`, `REQ-VAL-091` - **AT-VAL-008:** Typed transformations follow Section 6.8 and retain evidence of the attributable source value. This consolidated parameterized scenario also covers: Parameterized normalisation fixtures exercise every registered source-text and typed-value operation, verify that source-text operations affect strings but pass non-strings through unchanged, and cover Unicode normalization, whitespace trimming, case folding, decimal quantisation, and datetime truncation. Rounding or precision reduction occurs only when explicitly configured. Deterministic set and multiset output ordering is verified as canonical serialization rather than configured normalisation. The suite also verifies retention of source lexical evidence. - - Verifies: `REQ-CFG-123`, `REQ-VAL-061`, `REQ-VAL-062`, `REQ-VAL-063`, `REQ-VAL-069`, `REQ-VAL-070`, `REQ-VAL-072`, `REQ-VAL-073`, `REQ-VAL-074`, `REQ-VAL-075`, `REQ-VAL-071`, `REQ-CMP-023`, `REQ-VAL-010` + - Verifies: `REQ-CFG-123`, `REQ-CFG-146`, `REQ-CFG-147`, `REQ-CFG-148`, `REQ-CFG-149`, `REQ-VAL-061`, `REQ-VAL-062`, `REQ-VAL-063`, `REQ-VAL-069`, `REQ-VAL-070`, `REQ-VAL-072`, `REQ-VAL-073`, `REQ-VAL-074`, `REQ-VAL-075`, `REQ-VAL-071`, `REQ-CMP-023`, `REQ-VAL-010` - **AT-VAL-009:** Numeric tolerance accepts exactly one absolute or relative tolerance and rejects configurations containing neither or both. Each form uses exact decimal arithmetic and an inclusive boundary, with below, at, and above boundary cases; relative tolerance is reference-directed and permits no non-zero difference when the reference is zero. - Verifies: `REQ-CFG-081`, `REQ-CFG-082`, `REQ-CFG-083`, `REQ-CFG-084`, `REQ-CMP-018`, `REQ-CMP-019`, `REQ-CMP-020`, `REQ-CMP-021`, `REQ-CMP-022` - **AT-VAL-010:** All assessable differences are collected in one run. From fdb9039f1a2ef6c1cb86493cab3454bb38ef366c Mon Sep 17 00:00:00 2001 From: Kevin Howe Date: Mon, 9 Nov 2026 15:46:17 +0000 Subject: [PATCH 22/45] Tightening of parsing --- .../src/adviso_compare/inputs.py | 153 ++++++++++-------- packages/adviso-compare/tests/test_inputs.py | 23 +++ 2 files changed, 108 insertions(+), 68 deletions(-) diff --git a/packages/adviso-compare/src/adviso_compare/inputs.py b/packages/adviso-compare/src/adviso_compare/inputs.py index a81bc65..061baa8 100644 --- a/packages/adviso-compare/src/adviso_compare/inputs.py +++ b/packages/adviso-compare/src/adviso_compare/inputs.py @@ -161,6 +161,15 @@ class _NonFiniteNumber(ValueError): """Signal a non-standard non-finite JSON numerical token.""" +@dataclass(frozen=True, slots=True) +class JsonValueError(ValueError): + """A syntax or structural failure while decoding one JSON value.""" + + kind: str + line: int | None = None + column: int | None = None + + def load_dataset(path: Path, side: InputSide) -> ParsedDataset: """Discover and parse one supported dataset from its exact bytes.""" input_format = _discover_format(path, side) @@ -374,85 +383,65 @@ def _parse_json( ) -> tuple[tuple[str, ...], tuple[ParsedRecord, ...]]: """Parse a record-oriented JSON array without lossy numerical conversion.""" - def parse_integer(lexical: str) -> NativeValue: - return NativeValue(NativeCategory.INTEGER, int(lexical), lexical) - - def parse_decimal(lexical: str) -> NativeValue: - return NativeValue(NativeCategory.DECIMAL, Decimal(lexical), lexical) - - def reject_constant(_lexical: str) -> object: - raise _NonFiniteNumber - - def parse_object(pairs: list[tuple[str, object]]) -> NativeValue: - names = [name for name, _ in pairs] - if len(set(names)) != len(names): - raise _DuplicateMember - members = tuple( - sorted( - ((name, _to_native(value)) for name, value in pairs), - key=lambda item: item[0], - ) - ) - return NativeValue(NativeCategory.OBJECT, members) - try: - root = json.loads( - text, - parse_int=parse_integer, - parse_float=parse_decimal, - parse_constant=reject_constant, - object_pairs_hook=parse_object, - ) - except _NonFiniteNumber as error: - raise InputError( - code="invalid_syntax", - side=side, - message=f"{side.value} JSON input contains a non-finite number", - ) from error - except json.JSONDecodeError as error: - raise InputError( - code="invalid_syntax", - side=side, - message=( + root = decode_json_value(text) + except JsonValueError as error: + if error.kind == "non_finite": + message = f"{side.value} JSON input contains a non-finite number" + elif error.kind == "duplicate_member": + message = f"{side.value} JSON input contains a duplicate object member" + else: + message = ( f"{side.value} JSON input has invalid syntax at line " - f"{error.lineno}, column {error.colno}" - ), - ) from error - except _DuplicateMember as error: + f"{error.line}, column {error.column}" + ) raise InputError( - code="invalid_structure", + code=( + "invalid_structure" + if error.kind == "duplicate_member" + else "invalid_syntax" + ), side=side, - message=f"{side.value} JSON input contains a duplicate object member", + message=message, ) from error - if not isinstance(root, list): + if root.category is not NativeCategory.ARRAY: raise InputError( code="invalid_structure", side=side, message=( - f"{side.value} JSON root is {_json_category(root)}; it must be " + f"{side.value} JSON root is {root.category.value}; it must be " "one array of record objects" ), ) + elements = root.value + if not isinstance(elements, tuple): + raise AssertionError("JSON array elements were not normalized") records: list[ParsedRecord] = [] all_field_names: set[str] = set() - for index, item in enumerate(root): - if ( - not isinstance(item, NativeValue) - or item.category is not NativeCategory.OBJECT - ): + for index, item in enumerate(elements): + if item.category is not NativeCategory.OBJECT: raise InputError( code="invalid_structure", side=side, message=( f"{side.value} JSON array item {index} is " - f"{_json_category(item)}; every item must be a record object" + f"{item.category.value}; every item must be a record object" ), ) members = item.value if not isinstance(members, tuple): raise AssertionError("JSON object members were not normalized") + if any(name == "" for name, _ in members): + raise InputError( + code="invalid_structure", + side=side, + message=( + f"{side.value} JSON record at array item {index} " + "contains an empty field name" + ), + ) all_field_names.update(name for name, _ in members) records.append( ParsedRecord( @@ -463,6 +452,49 @@ def parse_object(pairs: list[tuple[str, object]]) -> NativeValue: return tuple(sorted(all_field_names)), tuple(records) +def decode_json_value(text: str) -> NativeValue: + """Decode one JSON value while preserving exact parser-native categories.""" + + def parse_integer(lexical: str) -> NativeValue: + return NativeValue(NativeCategory.INTEGER, int(lexical), lexical) + + def parse_decimal(lexical: str) -> NativeValue: + return NativeValue(NativeCategory.DECIMAL, Decimal(lexical), lexical) + + def reject_constant(_lexical: str) -> object: + raise _NonFiniteNumber + + def parse_object(pairs: list[tuple[str, object]]) -> NativeValue: + names = [name for name, _ in pairs] + if len(set(names)) != len(names): + raise _DuplicateMember + members = tuple( + sorted( + ((name, _to_native(value)) for name, value in pairs), + key=lambda item: item[0], + ) + ) + return NativeValue(NativeCategory.OBJECT, members) + + try: + root = json.loads( + text, + parse_int=parse_integer, + parse_float=parse_decimal, + parse_constant=reject_constant, + object_pairs_hook=parse_object, + ) + except _NonFiniteNumber as error: + raise JsonValueError(kind="non_finite") from error + except json.JSONDecodeError as error: + raise JsonValueError( + kind="invalid_syntax", line=error.lineno, column=error.colno + ) from error + except _DuplicateMember as error: + raise JsonValueError(kind="duplicate_member") from error + return _to_native(root) + + def _to_native(value: object) -> NativeValue: """Recursively assign parser-native categories to JSON library values.""" if isinstance(value, NativeValue): @@ -484,18 +516,3 @@ def _to_native(value: object) -> NativeValue: def _escape_json_pointer(member_name: str) -> str: """Escape one JSON object member for use in an RFC 6901 pointer.""" return member_name.replace("~", "~0").replace("/", "~1") - - -def _json_category(value: object) -> str: - """Describe a parsed JSON value without reproducing its content.""" - if isinstance(value, NativeValue): - return value.category.value - if isinstance(value, list): - return "array" - if value is None: - return "null" - if isinstance(value, bool): - return "boolean" - if isinstance(value, str): - return "string" - return type(value).__name__ diff --git a/packages/adviso-compare/tests/test_inputs.py b/packages/adviso-compare/tests/test_inputs.py index be3f5f7..adfd5b4 100644 --- a/packages/adviso-compare/tests/test_inputs.py +++ b/packages/adviso-compare/tests/test_inputs.py @@ -159,6 +159,29 @@ def test_json_preserves_native_categories_lexical_numbers_and_pointers( assert tuple(name for name, _ in record.value("object").value) == ("a", "z") +def test_json_record_field_names_are_nonempty_but_nested_empty_names_are_data( + tmp_path: Path, +) -> None: + """Only names that participate in dataset-wide field discovery are nonempty.""" + invalid = _write_bytes( + tmp_path / "invalid.json", + b'[{"record_id":"S1","":"value"}]', + ) + nested = _write_bytes( + tmp_path / "nested.json", + b'[{"record_id":"S1","value":{"":"retained"}}]', + ) + + with pytest.raises(InputError) as raised: + load_dataset(invalid, InputSide.REFERENCE) + + assert raised.value.code == "invalid_structure" + assert "empty field name" in raised.value.message + dataset = load_dataset(nested, InputSide.QUERY) + nested_members = dataset.records[0].value("value").value + assert nested_members == (("", NativeValue(NativeCategory.STRING, "retained")),) + + def test_parser_native_equality_is_recursive_and_category_preserving( tmp_path: Path, ) -> None: From 2ac11ebdadc5d48de019ef8161a1747357a267f5 Mon Sep 17 00:00:00 2001 From: Kevin Howe Date: Mon, 9 Nov 2026 15:48:29 +0000 Subject: [PATCH 23/45] Added support for custom keys, side-specific source-field aliases --- .../src/adviso_compare/application.py | 23 +- .../src/adviso_compare/configuration.py | 26 +- .../src/adviso_compare/processing.py | 24 +- .../src/adviso_compare/records.py | 399 ++++++++++ .../src/adviso_compare/translation.py | 655 ++++++++++++++++ packages/adviso-compare/tests/test_records.py | 475 ++++++++++++ .../adviso-compare/tests/test_translation.py | 702 ++++++++++++++++++ 7 files changed, 2297 insertions(+), 7 deletions(-) create mode 100644 packages/adviso-compare/src/adviso_compare/records.py create mode 100644 packages/adviso-compare/src/adviso_compare/translation.py create mode 100644 packages/adviso-compare/tests/test_records.py create mode 100644 packages/adviso-compare/tests/test_translation.py diff --git a/packages/adviso-compare/src/adviso_compare/application.py b/packages/adviso-compare/src/adviso_compare/application.py index 3aea15a..8784014 100644 --- a/packages/adviso-compare/src/adviso_compare/application.py +++ b/packages/adviso-compare/src/adviso_compare/application.py @@ -12,6 +12,11 @@ ) from adviso_compare.inputs import InputSide, ParsedDataset, load_dataset from adviso_compare.invocation import ComparisonInvocation +from adviso_compare.records import ( + RecordModel, + build_record_model, + generate_automatic_configuration, +) @dataclass(frozen=True, slots=True) @@ -19,9 +24,11 @@ class PreparedComparison: """Validated configuration selection and losslessly parsed input datasets.""" invocation: ComparisonInvocation - configuration: EffectiveConfiguration | None + configuration: EffectiveConfiguration reference: ParsedDataset query: ParsedDataset + reference_records: RecordModel + query_records: RecordModel def prepare_comparison( @@ -38,11 +45,25 @@ def prepare_comparison( configuration = configuration_loader(invocation.configuration) reference = dataset_loader(invocation.reference, InputSide.REFERENCE) query = dataset_loader(invocation.query, InputSide.QUERY) + if configuration is None: + configuration = generate_automatic_configuration(reference, query) + reference_records = build_record_model( + reference, + configuration, + InputSide.REFERENCE, + ) + query_records = build_record_model( + query, + configuration, + InputSide.QUERY, + ) return PreparedComparison( invocation=invocation, configuration=configuration, reference=reference, query=query, + reference_records=reference_records, + query_records=query_records, ) diff --git a/packages/adviso-compare/src/adviso_compare/configuration.py b/packages/adviso-compare/src/adviso_compare/configuration.py index 56b0e73..d3ce7fd 100644 --- a/packages/adviso-compare/src/adviso_compare/configuration.py +++ b/packages/adviso-compare/src/adviso_compare/configuration.py @@ -1,4 +1,4 @@ -"""Safe loading and validation of custom comparison configurations.""" +"""Safely validate and resolve custom or generated comparison configurations.""" from __future__ import annotations @@ -322,6 +322,19 @@ def load_comparison_configuration(path: Path) -> EffectiveConfiguration: return _build_effective_configuration(document) +def resolve_automatic_configuration(document: JsonObject) -> EffectiveConfiguration: + """Validate and resolve the internally generated automatic configuration.""" + _validate_schema(document) + metadata = document.get("metadata") + if ( + not isinstance(metadata, dict) + or metadata.get("id") != AUTOMATIC_CONFIGURATION_ID + ): + raise AssertionError("automatic configuration has an unexpected identifier") + _validate_semantics(document, allow_automatic_identifier=True) + return _build_effective_configuration(document) + + def _load_yaml_document(text: str) -> object: """Load exactly one safe YAML 1.2 document with forbidden features disabled.""" try: @@ -427,7 +440,11 @@ def _semantic_error(path: str, message: str) -> None: ) -def _validate_semantics(document: JsonObject) -> None: +def _validate_semantics( + document: JsonObject, + *, + allow_automatic_identifier: bool = False, +) -> None: """Apply cross-property rules not reliably expressible in JSON Schema.""" fields = document["fields"] definitions = fields["definitions"] @@ -462,7 +479,10 @@ def _validate_semantics(document: JsonObject) -> None: ) source_names[side][source_name] = field_name - if document["metadata"]["id"] == AUTOMATIC_CONFIGURATION_ID: + if ( + document["metadata"]["id"] == AUTOMATIC_CONFIGURATION_ID + and not allow_automatic_identifier + ): _semantic_error( "$.metadata.id", "identifier is reserved for automatically generated configurations", diff --git a/packages/adviso-compare/src/adviso_compare/processing.py b/packages/adviso-compare/src/adviso_compare/processing.py index d360b4c..7083540 100644 --- a/packages/adviso-compare/src/adviso_compare/processing.py +++ b/packages/adviso-compare/src/adviso_compare/processing.py @@ -7,6 +7,24 @@ from typing import Mapping +_UNICODE_WHITE_SPACE_CHARACTERS = "".join( + chr(code_point) + for code_point in ( + *range(0x0009, 0x000E), + 0x0020, + 0x0085, + 0x00A0, + 0x1680, + *range(0x2000, 0x200B), + 0x2028, + 0x2029, + 0x202F, + 0x205F, + 0x3000, + ) +) + + _SCALAR_COMPARATORS: Mapping[str, frozenset[str]] = MappingProxyType( { "string": frozenset({"typed_exact", "normalised_string"}), @@ -66,11 +84,11 @@ def apply_source_text_operation( if operation == "trim": side = parameters["side"] if side == "leading": - return value.lstrip() + return value.lstrip(_UNICODE_WHITE_SPACE_CHARACTERS) if side == "trailing": - return value.rstrip() + return value.rstrip(_UNICODE_WHITE_SPACE_CHARACTERS) if side == "both": - return value.strip() + return value.strip(_UNICODE_WHITE_SPACE_CHARACTERS) raise AssertionError("validated trim side is not registered") if operation == "case_fold": return value.casefold() diff --git a/packages/adviso-compare/src/adviso_compare/records.py b/packages/adviso-compare/src/adviso_compare/records.py new file mode 100644 index 0000000..e99db74 --- /dev/null +++ b/packages/adviso-compare/src/adviso_compare/records.py @@ -0,0 +1,399 @@ +"""Resolve unambiguous record identity for custom and automatic comparisons.""" + +from __future__ import annotations + +import json +from dataclasses import dataclass +from types import MappingProxyType +from typing import Mapping + +from adviso_compare.configuration import ( + AUTOMATIC_CONFIGURATION_ID, + EffectiveConfiguration, + FieldDefinition, + JsonObject, + resolve_automatic_configuration, +) +from adviso_compare.inputs import ( + InputSide, + JsonLocation, + NativeCategory, + NativeValue, + ParsedDataset, + ParsedRecord, + SourceLocation, + TabularLocation, +) +from adviso_compare.translation import ( + ValueDisposition, + render_canonical_scalar, + translate_field_value, +) + + +@dataclass(frozen=True, slots=True) +class CanonicalKeyComponent: + """One typed, uniquely rendered component of a canonical record key.""" + + logical_type: str + text: str + + +RecordKey = tuple[CanonicalKeyComponent, ...] + + +@dataclass(frozen=True, slots=True) +class KeyedRecord: + """A source record accompanied by its source and canonical identities.""" + + record: ParsedRecord + key: RecordKey + source_identity: tuple[str, ...] + canonical_identity: tuple[str, ...] + + +@dataclass(frozen=True, slots=True) +class RecordModel: + """All records from one dataset indexed without silent replacement.""" + + side: InputSide + records: tuple[KeyedRecord, ...] + by_key: Mapping[RecordKey, KeyedRecord] + + +@dataclass(frozen=True, slots=True) +class RecordKeyError(Exception): + """An expected input failure that prevents unambiguous record identity.""" + + message: str + side: InputSide | None = None + key_fields: tuple[str, ...] = () + candidate_fields: tuple[str, ...] = () + locations: tuple[SourceLocation, ...] = () + code: str = "invalid_record_key" + + @property + def category(self) -> str: + """Return the stable execution-error category.""" + return "input" + + @property + def diagnostic(self) -> str: + """Render a value-free, single-line execution diagnostic.""" + return f"ERROR [input.{self.code}]: {self.message}" + + def __str__(self) -> str: + """Return the actionable value-free message.""" + return self.message + + +def generate_automatic_configuration( + reference: ParsedDataset, + query: ParsedDataset, +) -> EffectiveConfiguration: + """Select one conservative key and build strict preserved-exact rules.""" + key_name = _select_automatic_key(reference, query) + common_fields = sorted(set(reference.field_names) & set(query.field_names)) + ordered_fields = [key_name] + ordered_fields.extend(name for name in common_fields if name != key_name) + + definitions: JsonObject = { + key_name: { + "source_fields": { + "reference": key_name, + "query": key_name, + }, + "required": {"reference": True, "query": True}, + "logical_type": "string", + "null_policy": {"allowed": False, "nulls_equal": False}, + "comparator": {"name": "typed_exact"}, + } + } + for field_name in ordered_fields[1:]: + definitions[field_name] = { + "source_fields": { + "reference": field_name, + "query": field_name, + }, + "required": {"reference": False, "query": False}, + "null_policy": {"allowed": True, "nulls_equal": True}, + "comparator": {"name": "preserved_exact"}, + } + + document: JsonObject = { + "schema_version": "1.0", + "metadata": { + "id": AUTOMATIC_CONFIGURATION_ID, + "revision": "1.0", + }, + "fields": { + "record_key": [key_name], + "in_scope": ordered_fields, + "definitions": definitions, + }, + } + return resolve_automatic_configuration(document) + + +def build_record_model( + dataset: ParsedDataset, + configuration: EffectiveConfiguration, + side: InputSide, +) -> RecordModel: + """Translate configured key fields and reject every ambiguous identity.""" + keyed_records: list[KeyedRecord] = [] + index: dict[RecordKey, KeyedRecord] = {} + source_key_fields = tuple( + _source_field(configuration.field(name), side) + for name in configuration.record_key + ) + + for record in dataset.records: + components: list[CanonicalKeyComponent] = [] + source_identity: list[str] = [] + for canonical_name, source_name in zip( + configuration.record_key, + source_key_fields, + ): + field = configuration.field(canonical_name) + if not record.has_field(source_name): + raise _key_value_error( + side, + source_name, + "is missing", + record.location, + source_key_fields, + ) + source = record.value(source_name) + translated = translate_field_value(source, field, side) + if translated.is_null or source.category is NativeCategory.NULL: + raise _key_value_error( + side, + source_name, + "is null", + record.location_for(source_name), + source_key_fields, + ) + if ( + translated.disposition is not ValueDisposition.VALID + or translated.canonical is None + ): + raise _key_value_error( + side, + source_name, + "is invalid", + record.location_for(source_name), + source_key_fields, + ) + logical_type = field.logical_type + if not isinstance(logical_type, str): + raise AssertionError("validated key field is not scalar") + canonical_text = render_canonical_scalar( + translated.canonical.value, + logical_type, + ) + components.append(CanonicalKeyComponent(logical_type, canonical_text)) + source_identity.append(_source_identity_text(source)) + + key = tuple(components) + keyed = KeyedRecord( + record=record, + key=key, + source_identity=tuple(source_identity), + canonical_identity=tuple(component.text for component in key), + ) + previous = index.get(key) + if previous is not None: + locations = (previous.record.location, record.location) + fields_text = ", ".join(repr(name) for name in source_key_fields) + raise RecordKeyError( + message=( + f"{side.value} duplicate record key using {fields_text} at " + f"{_format_location(locations[0])} and " + f"{_format_location(locations[1])}" + ), + side=side, + key_fields=source_key_fields, + locations=locations, + ) + index[key] = keyed + keyed_records.append(keyed) + + return RecordModel( + side=side, + records=tuple(keyed_records), + by_key=MappingProxyType(index), + ) + + +def _select_automatic_key( + reference: ParsedDataset, + query: ParsedDataset, +) -> str: + """Apply the fixed record_id preference and candidate criteria.""" + shared_fields = sorted(set(reference.field_names) & set(query.field_names)) + if "record_id" in shared_fields: + _validate_automatic_record_id(reference, InputSide.REFERENCE) + _validate_automatic_record_id(query, InputSide.QUERY) + return "record_id" + + candidates = tuple( + field_name + for field_name in shared_fields + if _is_automatic_candidate(field_name, reference, query) + ) + if len(candidates) == 1: + return candidates[0] + if not candidates: + raise RecordKeyError( + message=( + "no field satisfied the automatic record-key criteria after " + "considering both inputs" + ) + ) + names = ", ".join(repr(name) for name in candidates) + raise RecordKeyError( + message=( + "automatic record-key selection is ambiguous after considering both " + f"inputs; candidate fields are {names}" + ), + candidate_fields=candidates, + ) + + +def _validate_automatic_record_id( + dataset: ParsedDataset, + side: InputSide, +) -> None: + """Require the preferred automatic key to be complete, string, and unique.""" + seen: dict[str, ParsedRecord] = {} + for record in dataset.records: + if not record.has_field("record_id"): + raise _key_value_error( + side, + "record_id", + "is missing", + record.location, + ("record_id",), + ) + value = record.value("record_id") + if value.category is not NativeCategory.STRING: + reason = ( + "is null" + if value.category is NativeCategory.NULL + else "is not a parser-native string" + ) + raise _key_value_error( + side, + "record_id", + reason, + record.location_for("record_id"), + ("record_id",), + ) + text = value.value + if not isinstance(text, str): + raise AssertionError("parser-native string does not contain text") + previous = seen.get(text) + if previous is not None: + locations = (previous.location, record.location) + raise RecordKeyError( + message=( + f"{side.value} duplicate record key using 'record_id' at " + f"{_format_location(locations[0])} and " + f"{_format_location(locations[1])}" + ), + side=side, + key_fields=("record_id",), + locations=locations, + ) + seen[text] = record + + +def _is_automatic_candidate( + field_name: str, + reference: ParsedDataset, + query: ParsedDataset, +) -> bool: + """Return whether one shared field satisfies every automatic-key criterion.""" + reference_values = _complete_unique_strings(reference, field_name) + if reference_values is None: + return False + query_values = _complete_unique_strings(query, field_name) + if query_values is None: + return False + return not reference_values.isdisjoint(query_values) + + +def _complete_unique_strings( + dataset: ParsedDataset, + field_name: str, +) -> set[str] | None: + """Return unique source strings, or None when the candidate is invalid.""" + values: set[str] = set() + for record in dataset.records: + if not record.has_field(field_name): + return None + value = record.value(field_name) + if value.category is not NativeCategory.STRING: + return None + text = value.value + if not isinstance(text, str): + raise AssertionError("parser-native string does not contain text") + if text in values: + return None + values.add(text) + return values + + +def _key_value_error( + side: InputSide, + source_name: str, + reason: str, + location: SourceLocation, + key_fields: tuple[str, ...], +) -> RecordKeyError: + """Construct one attributable, value-free key error.""" + return RecordKeyError( + message=( + f"{side.value} key field {source_name!r} {reason} at " + f"{_format_location(location)}" + ), + side=side, + key_fields=key_fields, + locations=(location,), + ) + + +def _source_field(field: FieldDefinition, side: InputSide) -> str: + """Select a field's source name for one dataset side.""" + if side is InputSide.REFERENCE: + return field.source_fields.reference + return field.source_fields.query + + +def _source_identity_text(value: NativeValue) -> str: + """Render a scalar source key using its retained lexical representation.""" + if value.category is NativeCategory.STRING: + if not isinstance(value.value, str): + raise AssertionError("parser-native string does not contain text") + return value.value + if value.category in {NativeCategory.INTEGER, NativeCategory.DECIMAL}: + if value.lexical is not None: + return value.lexical + return str(value.value) + if value.category is NativeCategory.BOOLEAN: + return "true" if value.value else "false" + raise AssertionError("validated key source is not a non-null scalar") + + +def _format_location(location: SourceLocation) -> str: + """Render a non-sensitive source location for an execution diagnostic.""" + if isinstance(location, TabularLocation): + return ( + f"record {location.record_number}, physical line " + f"{location.line_number}" + ) + if isinstance(location, JsonLocation): + pointer = json.dumps(location.pointer, ensure_ascii=False) + return f"JSON pointer {pointer}" + raise AssertionError("unknown source-location type") diff --git a/packages/adviso-compare/src/adviso_compare/translation.py b/packages/adviso-compare/src/adviso_compare/translation.py new file mode 100644 index 0000000..3f682f9 --- /dev/null +++ b/packages/adviso-compare/src/adviso_compare/translation.py @@ -0,0 +1,655 @@ +"""Translate configured source field values into canonical typed values.""" + +from __future__ import annotations + +import re +from dataclasses import dataclass +from datetime import date, datetime, timedelta, timezone +from decimal import Decimal, ROUND_DOWN, ROUND_HALF_EVEN, ROUND_HALF_UP, localcontext +from enum import Enum +from typing import TypeVar + +from adviso_compare.configuration import ( + CollectionLogicalType, + CollectionRepresentation, + DatasetSides, + FieldDefinition, + Operation, + SourceValueDescriptor, + ValueMappingEntry, +) +from adviso_compare.inputs import ( + InputSide, + JsonValueError, + NativeCategory, + NativeValue, + decode_json_value, +) +from adviso_compare.processing import apply_source_text_operation + + +_INTEGER_PATTERN = re.compile(r"^[+-]?[0-9]+$") +_DECIMAL_PATTERN = re.compile( + r"^[+-]?[0-9]+(?:\.[0-9]+)?(?:[eE][+-]?[0-9]+)?$" +) +_DATE_PATTERN = re.compile(r"^[0-9]{4}-[0-9]{2}-[0-9]{2}$") +_DATETIME_PATTERN = re.compile( + r"^(?P[0-9]{4}-[0-9]{2}-[0-9]{2})[Tt]" + r"(?P[0-9]{2}):(?P[0-9]{2}):(?P[0-9]{2})" + r"(?:\.(?P[0-9]+))?" + r"(?P[Zz]|[+-][0-9]{2}:[0-9]{2})$" +) +T = TypeVar("T") + + +class ValueDisposition(str, Enum): + """The assessment state of one configured field value.""" + + VALID = "valid" + ABSENT = "absent" + NOT_ASSESSABLE = "not_assessable" + NOT_ASSESSED = "not_assessed" + + +class TranslationIssueCode(str, Enum): + """The causal finding code for an unsuccessful field translation.""" + + MISSING_REQUIRED_FIELD = "missing_required_field" + TYPE_MISMATCH = "type_mismatch" + INVALID_FIELD_VALUE = "invalid_field_value" + NULL_POLICY_VIOLATION = "null_policy_violation" + MAPPING_FAILURE = "mapping_failure" + + +class TranslationStage(str, Enum): + """The processing stage at which an attributable value became unusable.""" + + PRESENCE = "presence" + NULL_POLICY = "null_policy" + VALUE_MAPPING = "value_mapping" + TYPE_PARSING = "type_parsing" + TYPED_NORMALISATION = "typed_normalisation" + COLLECTION_DECODING = "collection_decoding" + ELEMENT_MAPPING = "element_mapping" + ELEMENT_PARSING = "element_parsing" + COLLECTION_VALIDATION = "collection_validation" + + +@dataclass(frozen=True, slots=True) +class TranslationIssue: + """A structured cause for one not-assessable translated value.""" + + code: TranslationIssueCode + stage: TranslationStage + element_index: int | None = None + + +@dataclass(frozen=True, slots=True) +class CanonicalValue: + """One scalar, null, or collection value in its declared logical type.""" + + logical_type: str + value: object + element_type: str | None = None + + +@dataclass(frozen=True, slots=True) +class TranslatedValue: + """The complete source-to-canonical outcome for one field on one side.""" + + disposition: ValueDisposition + source: NativeValue | None + canonical: CanonicalValue | None = None + preserved: NativeValue | None = None + is_null: bool = False + issue: TranslationIssue | None = None + + +@dataclass(frozen=True, slots=True) +class _ExactDateTime: + """A UTC whole second plus an exact non-negative fractional second.""" + + utc_second: datetime + fraction: Decimal + + +class _TypeMismatch(ValueError): + """The parser-native category is incompatible with a logical type.""" + + +class _InvalidValue(ValueError): + """The parser-native category is accepted but its content is invalid.""" + + +def translate_field_value( + source: NativeValue | None, + field: FieldDefinition, + side: InputSide, + *, + in_scope: bool = True, +) -> TranslatedValue: + """Apply the configured translation pipeline to one attributable field value.""" + if not in_scope: + return TranslatedValue(ValueDisposition.NOT_ASSESSED, source) + if source is None: + if _for_side(field.required, side): + return _failure( + source, + TranslationIssueCode.MISSING_REQUIRED_FIELD, + TranslationStage.PRESENCE, + ) + return TranslatedValue(ValueDisposition.ABSENT, source) + + if field.logical_type is None: + if source.category is NativeCategory.NULL: + return _apply_null_policy(source, source, field) + return TranslatedValue( + ValueDisposition.VALID, + source, + preserved=source, + ) + + source_operations = _for_side(field.source_normalisation, side) + working = _apply_source_operations(source, source_operations) + if working.category is NativeCategory.NULL or _matches_any_descriptor( + working, + _for_side(field.null_policy.missing_tokens, side), + source_operations, + ): + return _apply_null_policy(source, working, field) + + mappings = _for_side(field.value_mapping, side) + if mappings: + mapped = _mapped_value(working, mappings, source_operations) + if mapped is None: + return _failure( + source, + TranslationIssueCode.MAPPING_FAILURE, + TranslationStage.VALUE_MAPPING, + ) + working = mapped + if working.category is NativeCategory.NULL: + return _apply_null_policy(source, working, field) + + if isinstance(field.logical_type, CollectionLogicalType): + return _translate_collection(source, working, field, side) + + try: + value = _parse_scalar(working, field.logical_type) + for operation in field.typed_normalisation: + value = _apply_typed_operation(value, operation) + value = _finalize_scalar(value, field.logical_type) + except _TypeMismatch: + return _failure( + source, + TranslationIssueCode.TYPE_MISMATCH, + TranslationStage.TYPE_PARSING, + ) + except _InvalidValue: + return _failure( + source, + TranslationIssueCode.INVALID_FIELD_VALUE, + TranslationStage.TYPE_PARSING, + ) + return TranslatedValue( + ValueDisposition.VALID, + source, + canonical=CanonicalValue(field.logical_type, value), + ) + + +def _apply_null_policy( + source: NativeValue, + _working: NativeValue, + field: FieldDefinition, +) -> TranslatedValue: + """Return the configured outcome for a recognised or mapped null value.""" + if not field.null_policy.allowed: + return _failure( + source, + TranslationIssueCode.NULL_POLICY_VIOLATION, + TranslationStage.NULL_POLICY, + ) + if field.logical_type is None: + return TranslatedValue( + ValueDisposition.VALID, + source, + preserved=source, + is_null=True, + ) + if isinstance(field.logical_type, CollectionLogicalType): + canonical = CanonicalValue( + field.logical_type.collection_type, + None, + element_type=field.logical_type.element_type, + ) + else: + canonical = CanonicalValue(field.logical_type, None) + return TranslatedValue( + ValueDisposition.VALID, + source, + canonical=canonical, + is_null=True, + ) + + +def _translate_collection( + source: NativeValue, + working: NativeValue, + field: FieldDefinition, + side: InputSide, +) -> TranslatedValue: + """Decode and translate every element of one declared collection value.""" + logical_type = field.logical_type + if not isinstance(logical_type, CollectionLogicalType): + raise AssertionError("collection translation received a scalar field") + representations = field.collection_representation + if representations is None: + raise AssertionError("resolved collection field has no representations") + representation = _for_side(representations, side) + try: + elements = _decode_collection(working, representation) + except _TypeMismatch: + return _failure( + source, + TranslationIssueCode.TYPE_MISMATCH, + TranslationStage.COLLECTION_DECODING, + ) + except _InvalidValue: + return _failure( + source, + TranslationIssueCode.INVALID_FIELD_VALUE, + TranslationStage.COLLECTION_DECODING, + ) + + canonical_elements: list[object] = [] + for index, element in enumerate(elements): + if element.category is NativeCategory.NULL: + return _failure( + source, + TranslationIssueCode.INVALID_FIELD_VALUE, + TranslationStage.ELEMENT_PARSING, + element_index=index, + ) + element = _apply_source_operations( + element, + representation.element_normalisation, + ) + if representation.element_mapping: + mapped = _mapped_value( + element, + representation.element_mapping, + representation.element_normalisation, + ) + if mapped is None: + return _failure( + source, + TranslationIssueCode.MAPPING_FAILURE, + TranslationStage.ELEMENT_MAPPING, + element_index=index, + ) + element = mapped + try: + parsed = _parse_scalar(element, logical_type.element_type) + canonical_elements.append( + _finalize_scalar(parsed, logical_type.element_type) + ) + except _TypeMismatch: + return _failure( + source, + TranslationIssueCode.TYPE_MISMATCH, + TranslationStage.ELEMENT_PARSING, + element_index=index, + ) + except _InvalidValue: + return _failure( + source, + TranslationIssueCode.INVALID_FIELD_VALUE, + TranslationStage.ELEMENT_PARSING, + element_index=index, + ) + + if logical_type.collection_type == "set": + if len(set(canonical_elements)) != len(canonical_elements): + return _failure( + source, + TranslationIssueCode.INVALID_FIELD_VALUE, + TranslationStage.COLLECTION_VALIDATION, + ) + canonical_elements.sort( + key=lambda value: render_canonical_scalar( + value, logical_type.element_type + ) + ) + elif logical_type.collection_type == "multiset": + canonical_elements.sort( + key=lambda value: render_canonical_scalar( + value, logical_type.element_type + ) + ) + + return TranslatedValue( + ValueDisposition.VALID, + source, + canonical=CanonicalValue( + logical_type.collection_type, + tuple(canonical_elements), + element_type=logical_type.element_type, + ), + ) + + +def _decode_collection( + value: NativeValue, + representation: CollectionRepresentation, +) -> tuple[NativeValue, ...]: + """Decode one side's declared source representation into native elements.""" + if representation.format == "delimited_text": + if value.category is not NativeCategory.STRING: + raise _TypeMismatch + if not isinstance(value.value, str): + raise AssertionError("parser-native string does not contain text") + if value.value == "": + return () + return tuple( + NativeValue(NativeCategory.STRING, element) + for element in value.value.split(representation.delimiter) + ) + + if value.category is NativeCategory.STRING: + if not isinstance(value.value, str): + raise AssertionError("parser-native string does not contain text") + try: + decoded = decode_json_value(value.value) + except JsonValueError as error: + raise _InvalidValue from error + elif value.category is NativeCategory.ARRAY: + decoded = value + else: + raise _TypeMismatch + if decoded.category is not NativeCategory.ARRAY: + raise _InvalidValue + if not isinstance(decoded.value, tuple): + raise AssertionError("parser-native array does not contain a tuple") + return decoded.value + + +def _apply_source_operations( + value: NativeValue, + operations: tuple[Operation, ...], +) -> NativeValue: + """Apply ordered source-text operations only to parser-native strings.""" + if value.category is not NativeCategory.STRING: + return value + text = value.value + if not isinstance(text, str): + raise AssertionError("parser-native string does not contain text") + for operation in operations: + text = apply_source_text_operation( + text, + operation.name, + operation.parameters, + ) + return NativeValue(NativeCategory.STRING, text) + + +def _mapped_value( + value: NativeValue, + mappings: tuple[ValueMappingEntry, ...], + source_operations: tuple[Operation, ...], +) -> NativeValue | None: + """Return the destination of one closed mapping or signal no match.""" + for mapping in mappings: + if _matches_descriptor(value, mapping.source, source_operations): + return _descriptor_value(mapping.destination) + return None + + +def _matches_any_descriptor( + value: NativeValue, + descriptors: tuple[SourceValueDescriptor, ...], + source_operations: tuple[Operation, ...], +) -> bool: + """Return whether a value equals any configured source descriptor.""" + return any( + _matches_descriptor(value, descriptor, source_operations) + for descriptor in descriptors + ) + + +def _matches_descriptor( + value: NativeValue, + descriptor: SourceValueDescriptor, + source_operations: tuple[Operation, ...], +) -> bool: + """Compare a native value with a typed descriptor by category and content.""" + expected = _apply_source_operations( + _descriptor_value(descriptor), + source_operations, + ) + return value.category is expected.category and value.value == expected.value + + +def _descriptor_value(descriptor: SourceValueDescriptor) -> NativeValue: + """Convert a validated descriptor into its parser-native representation.""" + category = NativeCategory(descriptor.category) + if category is NativeCategory.NULL: + return NativeValue(category) + if category is NativeCategory.INTEGER: + lexical = str(descriptor.value) + return NativeValue(category, int(lexical), lexical) + if category is NativeCategory.DECIMAL: + lexical = str(descriptor.value) + return NativeValue(category, Decimal(lexical), lexical) + return NativeValue(category, descriptor.value) + + +def _parse_scalar(value: NativeValue, logical_type: str) -> object: + """Parse one non-null native scalar into its declared logical type.""" + if logical_type == "string": + if value.category is not NativeCategory.STRING: + raise _TypeMismatch + return value.value + if logical_type == "integer": + if value.category is NativeCategory.INTEGER: + return value.value + if value.category is not NativeCategory.STRING: + raise _TypeMismatch + if not isinstance(value.value, str) or not _INTEGER_PATTERN.fullmatch( + value.value + ): + raise _InvalidValue + return int(value.value) + if logical_type == "decimal": + if value.category is NativeCategory.INTEGER: + return Decimal(value.value) + if value.category is NativeCategory.DECIMAL: + return value.value + if value.category is not NativeCategory.STRING: + raise _TypeMismatch + if not isinstance(value.value, str) or not _DECIMAL_PATTERN.fullmatch( + value.value + ): + raise _InvalidValue + parsed = Decimal(value.value) + if not parsed.is_finite(): + raise _InvalidValue + return parsed + if logical_type == "boolean": + if value.category is NativeCategory.BOOLEAN: + return value.value + if value.category is not NativeCategory.STRING: + raise _TypeMismatch + if value.value == "true": + return True + if value.value == "false": + return False + raise _InvalidValue + if logical_type == "date": + if value.category is not NativeCategory.STRING: + raise _TypeMismatch + if not isinstance(value.value, str) or not _DATE_PATTERN.fullmatch(value.value): + raise _InvalidValue + try: + return date.fromisoformat(value.value).isoformat() + except ValueError as error: + raise _InvalidValue from error + if logical_type == "datetime": + if value.category is not NativeCategory.STRING: + raise _TypeMismatch + if not isinstance(value.value, str): + raise _InvalidValue + return _parse_datetime(value.value) + raise AssertionError(f"validated logical type is not registered: {logical_type}") + + +def _parse_datetime(value: str) -> _ExactDateTime: + """Parse one offset-bearing datetime without losing fractional precision.""" + match = _DATETIME_PATTERN.fullmatch(value) + if match is None: + raise _InvalidValue + second = int(match["second"]) + if second > 59: + raise _InvalidValue + offset_text = match["offset"] + if offset_text in {"Z", "z"}: + offset = timezone.utc + else: + if offset_text == "-00:00": + raise _InvalidValue + offset_hour = int(offset_text[1:3]) + offset_minute = int(offset_text[4:6]) + if offset_hour > 23 or offset_minute > 59: + raise _InvalidValue + delta = timedelta(hours=offset_hour, minutes=offset_minute) + if offset_text.startswith("-"): + delta = -delta + try: + offset = timezone(delta) + except ValueError as error: + raise _InvalidValue from error + try: + local = datetime( + *map(int, match["date"].split("-")), + int(match["hour"]), + int(match["minute"]), + second, + tzinfo=offset, + ) + except ValueError as error: + raise _InvalidValue from error + fraction_text = match["fraction"] + fraction = Decimal(f"0.{fraction_text}") if fraction_text else Decimal(0) + try: + utc_second = local.astimezone(timezone.utc) + except (OverflowError, ValueError) as error: + raise _InvalidValue from error + return _ExactDateTime(utc_second, fraction) + + +def _apply_typed_operation(value: object, operation: Operation) -> object: + """Apply one validated typed-value operation.""" + if operation.name == "decimal_quantise": + if not isinstance(value, Decimal): + raise AssertionError("decimal operation received a non-decimal value") + places = operation.parameters["places"] + rounding_name = operation.parameters["rounding"] + if not isinstance(places, int) or not isinstance(rounding_name, str): + raise AssertionError("validated decimal operation parameters are invalid") + rounding = { + "half_even": ROUND_HALF_EVEN, + "half_up": ROUND_HALF_UP, + "down": ROUND_DOWN, + }[rounding_name] + quantum = Decimal(1).scaleb(-places) + integer_digits = max(value.adjusted() + 1, 1) if value else 1 + with localcontext() as context: + context.prec = ( + max(len(value.as_tuple().digits), integer_digits + places) + 2 + ) + return value.quantize(quantum, rounding=rounding) + if operation.name == "datetime_truncate": + if not isinstance(value, _ExactDateTime): + raise AssertionError("datetime operation received a non-datetime value") + precision = operation.parameters["precision"] + places = {"second": 0, "millisecond": 3, "microsecond": 6}[precision] + if places == 0: + fraction = Decimal(0) + else: + fraction = value.fraction.quantize( + Decimal(1).scaleb(-places), + rounding=ROUND_DOWN, + ) + return _ExactDateTime(value.utc_second, fraction) + raise AssertionError( + f"validated typed operation is not registered: {operation.name}" + ) + + +def _finalize_scalar(value: object, logical_type: str) -> object: + """Convert internal exact temporal state to its unique canonical value.""" + if logical_type == "datetime": + if not isinstance(value, _ExactDateTime): + raise AssertionError("parsed datetime has unexpected internal type") + return _render_datetime(value) + return value + + +def _render_datetime(value: _ExactDateTime) -> str: + """Render an exact UTC datetime using the canonical results representation.""" + instant = value.utc_second + text = ( + f"{instant.year:04d}-{instant.month:02d}-{instant.day:02d}T" + f"{instant.hour:02d}:{instant.minute:02d}:{instant.second:02d}" + ) + if value.fraction: + fraction = format(value.fraction, "f").partition(".")[2].rstrip("0") + text += f".{fraction}" + return f"{text}Z" + + +def render_canonical_scalar(value: object, logical_type: str) -> str: + """Render a canonical scalar for deterministic collection ordering.""" + if logical_type == "boolean": + return "true" if value else "false" + if logical_type == "decimal": + if not isinstance(value, Decimal): + raise AssertionError("canonical decimal has unexpected internal type") + return _render_decimal(value) + return str(value) + + +def _render_decimal(value: Decimal) -> str: + """Render one finite decimal in the canonical normalized scientific form.""" + if value.is_zero(): + return "0" + normalized = value.normalize() + sign = "-" if normalized.is_signed() else "" + digits = "".join(str(digit) for digit in normalized.as_tuple().digits) + exponent = normalized.adjusted() + coefficient = digits[0] + if len(digits) > 1: + coefficient += f".{digits[1:]}" + if exponent: + coefficient += f"e{exponent}" + return f"{sign}{coefficient}" + + +def _failure( + source: NativeValue | None, + code: TranslationIssueCode, + stage: TranslationStage, + *, + element_index: int | None = None, +) -> TranslatedValue: + """Construct one attributable not-assessable translation outcome.""" + return TranslatedValue( + ValueDisposition.NOT_ASSESSABLE, + source, + issue=TranslationIssue(code, stage, element_index), + ) + + +def _for_side(sides: DatasetSides[T], side: InputSide) -> T: + """Select one resolved side-specific configuration value.""" + return sides.reference if side is InputSide.REFERENCE else sides.query diff --git a/packages/adviso-compare/tests/test_records.py b/packages/adviso-compare/tests/test_records.py new file mode 100644 index 0000000..c9f33c0 --- /dev/null +++ b/packages/adviso-compare/tests/test_records.py @@ -0,0 +1,475 @@ +"""Behavioural tests for record identity and automatic configuration.""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Callable + +import pytest +import yaml + +from adviso_compare.application import prepare_comparison +from adviso_compare.configuration import EffectiveConfiguration +from adviso_compare.inputs import InputSide, JsonLocation, TabularLocation +from adviso_compare.invocation import ComparisonInvocation +from adviso_compare.records import RecordKeyError +from tests.support.documents import JsonObject, minimal_configuration + + +def _write(path: Path, text: str) -> Path: + """Write one controlled UTF-8 test document.""" + path.write_text(text, encoding="utf-8") + return path + + +def _write_configuration(tmp_path: Path, document: JsonObject) -> Path: + """Write one controlled custom configuration.""" + return _write( + tmp_path / "configuration.yaml", + yaml.safe_dump(document, sort_keys=False), + ) + + +def _invocation( + tmp_path: Path, + reference: Path, + query: Path, + configuration: Path | None = None, +) -> ComparisonInvocation: + """Build an invocation whose unused output remains inside the test directory.""" + return ComparisonInvocation( + reference=reference, + query=query, + output=tmp_path / "evidence", + configuration=configuration, + ) + + +def _composite_configuration() -> JsonObject: + """Declare aliased, normalized string and parsed integer key components.""" + document = minimal_configuration() + fields = document["fields"] + fields["record_key"] = ["record_id", "batch"] + fields["in_scope"].insert(1, "batch") + key = fields["definitions"]["record_id"] + key["source_normalisation"] = { + "reference": [ + {"operation": "trim", "parameters": {"side": "both"}}, + {"operation": "case_fold"}, + ], + "query": [{"operation": "case_fold"}], + } + fields["definitions"]["batch"] = { + "source_fields": { + "reference": "reference_batch", + "query": "query_batch", + }, + "required": {"reference": True, "query": True}, + "logical_type": "integer", + "null_policy": {"allowed": False, "nulls_equal": False}, + "comparator": {"name": "typed_exact"}, + } + return document + + +def test_custom_aliases_and_composite_key_processing_create_matching_identities( + tmp_path: Path, +) -> None: + """Each custom key component uses its declared source name and translation.""" + reference = _write( + tmp_path / "reference.csv", + "reference_id,reference_batch\n S1 ,01\n", + ) + query = _write( + tmp_path / "query.json", + '[{"query_id":"s1","query_batch":"1"}]', + ) + configuration = _write_configuration(tmp_path, _composite_configuration()) + + prepared = prepare_comparison( + _invocation(tmp_path, reference, query, configuration) + ) + + reference_record = prepared.reference_records.records[0] + query_record = prepared.query_records.records[0] + assert reference_record.key == query_record.key + assert reference_record.source_identity == (" S1 ", "01") + assert query_record.source_identity == ("s1", "1") + assert reference_record.canonical_identity == ("s1", "1") + assert prepared.configuration.record_key == ("record_id", "batch") + + +def _missing_key(document: JsonObject) -> None: + """Leave the required reference alias absent from a JSON record.""" + + +def _null_key(document: JsonObject) -> None: + """No configuration mutation is needed for a prohibited source null.""" + + +def _invalid_key(document: JsonObject) -> None: + """Use the composite declaration so its integer component can be invalid.""" + composite = _composite_configuration() + document.clear() + document.update(composite) + + +@pytest.mark.parametrize( + ("mutation", "reference_text", "message_fragment", "location"), + [ + ( + _missing_key, + '[{"expected_value":"1"}]', + "reference key field 'reference_id' is missing", + JsonLocation("/0"), + ), + ( + _null_key, + '[{"reference_id":null}]', + "reference key field 'reference_id' is null", + JsonLocation("/0/reference_id"), + ), + ( + _invalid_key, + ( + "reference_id,reference_batch\n" + "S1,not-an-integer\n" + ), + "reference key field 'reference_batch' is invalid", + TabularLocation(1, 2, "reference_batch"), + ), + ], +) +def test_missing_null_and_invalid_custom_keys_are_attributable_input_errors( + tmp_path: Path, + mutation: Callable[[JsonObject], None], + reference_text: str, + message_fragment: str, + location: object, +) -> None: + """A custom key defect stops record-model construction at its source location.""" + document = minimal_configuration() + mutation(document) + suffix = "csv" if reference_text.startswith("reference_id,") else "json" + reference = _write(tmp_path / f"reference.{suffix}", reference_text) + query = _write(tmp_path / "query.csv", "query_id\nS1\n") + configuration = _write_configuration(tmp_path, document) + + with pytest.raises(RecordKeyError) as raised: + prepare_comparison( + _invocation(tmp_path, reference, query, configuration) + ) + + error = raised.value + assert error.code == "invalid_record_key" + assert error.side is InputSide.REFERENCE + assert message_fragment in error.message + assert error.locations == (location,) + assert "not-an-integer" not in error.diagnostic + + +@pytest.mark.parametrize( + "duplicate_side", + [InputSide.REFERENCE, InputSide.QUERY], +) +def test_duplicate_canonical_keys_report_both_locations_without_overwriting( + tmp_path: Path, + duplicate_side: InputSide, +) -> None: + """Explicit normalization can reveal a duplicate without losing either record.""" + document = minimal_configuration() + key = document["fields"]["definitions"]["record_id"] + key["source_normalisation"] = { + "reference": [{"operation": "case_fold"}], + "query": [{"operation": "case_fold"}], + } + duplicate = "S1\ns1\n" + singleton = "s1\n" + reference = _write( + tmp_path / "reference.csv", + "reference_id\n" + + (duplicate if duplicate_side is InputSide.REFERENCE else singleton), + ) + query = _write( + tmp_path / "query.csv", + "query_id\n" + + (duplicate if duplicate_side is InputSide.QUERY else singleton), + ) + configuration = _write_configuration(tmp_path, document) + + with pytest.raises(RecordKeyError) as raised: + prepare_comparison( + _invocation(tmp_path, reference, query, configuration) + ) + + error = raised.value + assert error.side is duplicate_side + assert "duplicate record key" in error.message + assert error.locations == ( + TabularLocation(1, 2), + TabularLocation(2, 3), + ) + assert "S1" not in error.diagnostic + assert "s1" not in error.diagnostic + + +def test_key_error_diagnostic_escapes_control_characters_in_json_pointer( + tmp_path: Path, +) -> None: + """An exact structured location cannot introduce a diagnostic line break.""" + document = minimal_configuration() + document["fields"]["definitions"]["record_id"]["source_fields"][ + "reference" + ] = "id\nline" + reference = _write( + tmp_path / "reference.json", + json.dumps([{"id\nline": None}]), + ) + query = _write(tmp_path / "query.csv", "query_id\nS1\n") + configuration = _write_configuration(tmp_path, document) + + with pytest.raises(RecordKeyError) as raised: + prepare_comparison( + _invocation(tmp_path, reference, query, configuration) + ) + + assert raised.value.locations == (JsonLocation("/0/id\nline"),) + assert "\n" not in raised.value.diagnostic + assert r"\n" in raised.value.diagnostic + + +def test_automatic_record_id_generates_strict_exact_shared_fields_and_exclusions( + tmp_path: Path, +) -> None: + """Shared record_id is preferred and only exact same-name fields are declared.""" + reference = _write( + tmp_path / "reference.json", + '[{"record_id":"S1","z":1,"a":"x","reference_note":"r"}]', + ) + query = _write( + tmp_path / "query.csv", + "record_id,a,z,query_note\nS1,x,1,q\n", + ) + + prepared = prepare_comparison(_invocation(tmp_path, reference, query)) + + configuration = prepared.configuration + assert configuration.metadata.id == "org.adviso.automatic-strict-exact" + assert configuration.metadata.revision == "1.0" + assert configuration.record_key == ("record_id",) + assert configuration.in_scope == ("record_id", "a", "z") + assert tuple(configuration.fields) == ("record_id", "a", "z") + key = configuration.field("record_id") + assert key.logical_type == "string" + assert key.required.reference and key.required.query + assert not key.null_policy.allowed + for field_name in ("a", "z"): + field = configuration.field(field_name) + assert field.logical_type is None + assert field.comparator.name == "preserved_exact" + assert not field.required.reference and not field.required.query + assert field.null_policy.allowed and field.null_policy.nulls_equal + assert "reference_note" not in configuration.fields + assert "query_note" not in configuration.fields + assert configuration.document["fields"]["definitions"]["a"] == { + "source_fields": {"reference": "a", "query": "a"}, + "required": {"reference": False, "query": False}, + "null_policy": {"allowed": True, "nulls_equal": True}, + "comparator": {"name": "preserved_exact"}, + } + + +def test_automatic_discovery_selects_the_only_exact_shared_string_candidate( + tmp_path: Path, +) -> None: + """A non-record_id candidate requires completeness, uniqueness, and overlap.""" + reference = _write( + tmp_path / "reference.json", + '[{"sample":"S1","value":"a"},{"sample":"S2","value":"b"}]', + ) + query = _write( + tmp_path / "query.json", + '[{"sample":"S1","value":"x"},{"sample":"S3","value":"y"}]', + ) + + prepared = prepare_comparison(_invocation(tmp_path, reference, query)) + + assert prepared.configuration.record_key == ("sample",) + assert prepared.reference_records.records[0].canonical_identity == ("S1",) + + +@pytest.mark.parametrize( + ("reference_text", "query_text", "message_fragment", "candidates"), + [ + ( + '[{"sample":"R1","value":1}]', + '[{"sample":"Q1","value":1}]', + "no field satisfied the automatic record-key criteria", + (), + ), + ( + '[{"sample":"S1","accession":"A1"}]', + '[{"sample":"S1","accession":"A1"}]', + "automatic record-key selection is ambiguous", + ("accession", "sample"), + ), + ( + ( + '[{"part_a":"A","part_b":"1"},' + '{"part_a":"A","part_b":"2"},' + '{"part_a":"B","part_b":"1"},' + '{"part_a":"B","part_b":"2"}]' + ), + ( + '[{"part_a":"A","part_b":"1"},' + '{"part_a":"A","part_b":"2"},' + '{"part_a":"B","part_b":"1"},' + '{"part_a":"B","part_b":"2"}]' + ), + "no field satisfied the automatic record-key criteria", + (), + ), + ], +) +def test_automatic_discovery_rejects_zero_or_multiple_candidates_without_values( + tmp_path: Path, + reference_text: str, + query_text: str, + message_fragment: str, + candidates: tuple[str, ...], +) -> None: + """Automatic discovery neither guesses nor ranks otherwise valid candidates.""" + reference = _write(tmp_path / "reference.json", reference_text) + query = _write(tmp_path / "query.json", query_text) + + with pytest.raises(RecordKeyError) as raised: + prepare_comparison(_invocation(tmp_path, reference, query)) + + error = raised.value + assert error.side is None + assert message_fragment in error.message + assert error.candidate_fields == candidates + assert "S1" not in error.diagnostic + assert "A1" not in error.diagnostic + + +def test_invalid_automatic_candidates_are_ignored_without_coercion( + tmp_path: Path, +) -> None: + """Only a complete, non-null, unique parser-native string can be selected.""" + reference = _write( + tmp_path / "reference.json", + ( + '[{"sample":"S1","numeric":1,"nullable":"x",' + '"sometimes":"p","duplicate":"d"},' + '{"sample":"S2","numeric":2,"nullable":null,' + '"duplicate":"d"}]' + ), + ) + query = _write( + tmp_path / "query.json", + ( + '[{"sample":"S1","numeric":1,"nullable":"x",' + '"sometimes":"p","duplicate":"d"},' + '{"sample":"S3","numeric":3,"nullable":"y",' + '"sometimes":"q","duplicate":"d"}]' + ), + ) + + prepared = prepare_comparison(_invocation(tmp_path, reference, query)) + + assert prepared.configuration.record_key == ("sample",) + + +@pytest.mark.parametrize( + ("query_text", "message_fragment"), + [ + ( + ( + '[{"record_id":"Q1","sample":"S1"},' + '{"sample":"S2"}]' + ), + "is missing", + ), + ('[{"record_id":null,"sample":"S1"}]', "is null"), + ('[{"record_id":1,"sample":"S1"}]', "is not a parser-native string"), + ( + ( + '[{"record_id":"Q1","sample":"S1"},' + '{"record_id":"Q1","sample":"S2"}]' + ), + "duplicate record key", + ), + ], +) +def test_invalid_shared_record_id_never_falls_back_to_another_candidate( + tmp_path: Path, + query_text: str, + message_fragment: str, +) -> None: + """Every preferred-key defect is fatal even when sample could identify records.""" + reference = _write( + tmp_path / "reference.json", + ( + '[{"record_id":"R1","sample":"S1"},' + '{"record_id":"R2","sample":"S2"}]' + ), + ) + query = _write(tmp_path / "query.json", query_text) + + with pytest.raises(RecordKeyError) as raised: + prepare_comparison(_invocation(tmp_path, reference, query)) + + error = raised.value + assert error.side is InputSide.QUERY + assert error.key_fields == ("record_id",) + assert message_fragment in error.message + assert "sample" not in error.message + + +def test_header_only_record_id_is_valid_but_empty_json_has_no_automatic_key( + tmp_path: Path, +) -> None: + """Tabular headers expose record_id while empty JSON exposes no fields.""" + reference_csv = _write(tmp_path / "reference.csv", "record_id,value\n") + query_tsv = _write(tmp_path / "query.tsv", "record_id\tvalue\n") + + prepared = prepare_comparison( + _invocation(tmp_path, reference_csv, query_tsv) + ) + + assert prepared.configuration.record_key == ("record_id",) + assert prepared.reference_records.records == () + assert prepared.query_records.records == () + + empty_reference = _write(tmp_path / "empty-reference.json", "[]") + empty_query = _write(tmp_path / "empty-query.json", "[]") + with pytest.raises(RecordKeyError) as raised: + prepare_comparison( + _invocation(tmp_path, empty_reference, empty_query) + ) + assert "no field satisfied" in raised.value.message + + +def test_generated_document_is_schema_valid_and_reserved_id_remains_internal( + tmp_path: Path, +) -> None: + """Automatic configuration is valid but cannot be supplied as custom YAML.""" + reference = _write(tmp_path / "reference.csv", "record_id\nS1\n") + query = _write(tmp_path / "query.csv", "record_id\nS1\n") + prepared = prepare_comparison(_invocation(tmp_path, reference, query)) + generated = prepared.configuration.document + custom_path = _write_configuration(tmp_path, generated) + + from adviso_compare.configuration import ( # local to emphasize the boundary + ConfigurationError, + load_comparison_configuration, + ) + + with pytest.raises(ConfigurationError) as raised: + load_comparison_configuration(custom_path) + + assert raised.value.code == "semantic_violation" + assert raised.value.path == "$.metadata.id" + assert isinstance(prepared.configuration, EffectiveConfiguration) diff --git a/packages/adviso-compare/tests/test_translation.py b/packages/adviso-compare/tests/test_translation.py new file mode 100644 index 0000000..5007f9a --- /dev/null +++ b/packages/adviso-compare/tests/test_translation.py @@ -0,0 +1,702 @@ +"""Behavioural tests for source-to-canonical field translation.""" + +from __future__ import annotations + +from decimal import Decimal +from pathlib import Path +from typing import Callable + +import pytest +import yaml + +from adviso_compare.configuration import FieldDefinition, load_comparison_configuration +from adviso_compare.inputs import InputSide, NativeCategory, NativeValue +from adviso_compare.translation import ( + TranslationIssueCode, + TranslationStage, + ValueDisposition, + translate_field_value, +) +from tests.support.documents import JsonObject, minimal_configuration + + +FieldMutation = Callable[[JsonObject], None] + + +def _field_definition( + tmp_path: Path, + mutation: FieldMutation | None = None, +) -> FieldDefinition: + """Load a measured-value definition through the configuration boundary.""" + document = minimal_configuration() + field = document["fields"]["definitions"]["measured_value"] + if mutation is not None: + mutation(field) + path = tmp_path / "configuration.yaml" + path.write_text(yaml.safe_dump(document, sort_keys=False), encoding="utf-8") + return load_comparison_configuration(path).field("measured_value") + + +def _scalar_field(logical_type: str) -> FieldMutation: + def mutate(field: JsonObject) -> None: + field["logical_type"] = logical_type + field["comparator"] = {"name": "typed_exact"} + + return mutate + + +@pytest.mark.parametrize( + ("logical_type", "source", "expected"), + [ + ("string", NativeValue(NativeCategory.STRING, "01"), "01"), + ("integer", NativeValue(NativeCategory.INTEGER, 7, "7"), 7), + ("integer", NativeValue(NativeCategory.STRING, "+001"), 1), + ("decimal", NativeValue(NativeCategory.INTEGER, 2, "2"), Decimal("2")), + ( + "decimal", + NativeValue(NativeCategory.DECIMAL, Decimal("1.00"), "1.00"), + Decimal("1.00"), + ), + ("decimal", NativeValue(NativeCategory.STRING, "+1.2"), Decimal("1.2")), + ("decimal", NativeValue(NativeCategory.STRING, "01.2"), Decimal("1.2")), + ("decimal", NativeValue(NativeCategory.STRING, "1"), Decimal("1")), + ("decimal", NativeValue(NativeCategory.STRING, "1E+2"), Decimal("1E+2")), + ("boolean", NativeValue(NativeCategory.BOOLEAN, True), True), + ("boolean", NativeValue(NativeCategory.STRING, "false"), False), + ("date", NativeValue(NativeCategory.STRING, "2024-02-29"), "2024-02-29"), + ( + "datetime", + NativeValue(NativeCategory.STRING, "2024-01-02T03:04:05.1200+01:00"), + "2024-01-02T02:04:05.12Z", + ), + ( + "datetime", + NativeValue(NativeCategory.STRING, "2024-01-02t03:04:05z"), + "2024-01-02T03:04:05Z", + ), + ( + "datetime", + NativeValue(NativeCategory.STRING, "2024-01-02T03:04:05+23:59"), + "2024-01-01T03:05:05Z", + ), + ( + "datetime", + NativeValue(NativeCategory.STRING, "2024-01-02T03:04:05-23:59"), + "2024-01-03T03:03:05Z", + ), + ], +) +def test_declared_scalar_types_parse_without_implicit_cross_type_conversion( + tmp_path: Path, + logical_type: str, + source: NativeValue, + expected: object, +) -> None: + """Each accepted parser-native representation produces its exact typed value.""" + field = _field_definition(tmp_path, _scalar_field(logical_type)) + + result = translate_field_value(source, field, InputSide.REFERENCE) + + assert result.disposition is ValueDisposition.VALID + assert result.source is source + assert result.canonical is not None + assert result.canonical.logical_type == logical_type + assert result.canonical.value == expected + assert result.issue is None + + +@pytest.mark.parametrize( + ("logical_type", "source", "issue_code"), + [ + ("string", NativeValue(NativeCategory.INTEGER, 1, "1"), "type_mismatch"), + ( + "integer", + NativeValue(NativeCategory.DECIMAL, Decimal("1"), "1.0"), + "type_mismatch", + ), + ("integer", NativeValue(NativeCategory.STRING, "1.0"), "invalid_field_value"), + ("integer", NativeValue(NativeCategory.STRING, "1e2"), "invalid_field_value"), + ("integer", NativeValue(NativeCategory.BOOLEAN, True), "type_mismatch"), + ("decimal", NativeValue(NativeCategory.BOOLEAN, True), "type_mismatch"), + ("decimal", NativeValue(NativeCategory.STRING, ".5"), "invalid_field_value"), + ("decimal", NativeValue(NativeCategory.STRING, "1."), "invalid_field_value"), + ("decimal", NativeValue(NativeCategory.STRING, "NaN"), "invalid_field_value"), + ( + "decimal", + NativeValue(NativeCategory.STRING, "Infinity"), + "invalid_field_value", + ), + ( + "decimal", + NativeValue(NativeCategory.STRING, "-Infinity"), + "invalid_field_value", + ), + ("boolean", NativeValue(NativeCategory.INTEGER, 1, "1"), "type_mismatch"), + ("boolean", NativeValue(NativeCategory.STRING, "True"), "invalid_field_value"), + ( + "date", + NativeValue(NativeCategory.STRING, "2023-02-29"), + "invalid_field_value", + ), + ( + "date", + NativeValue(NativeCategory.STRING, "2024-1-2"), + "invalid_field_value", + ), + ( + "date", + NativeValue(NativeCategory.STRING, "2024-01"), + "invalid_field_value", + ), + ( + "datetime", + NativeValue(NativeCategory.STRING, "2024-01-02T03:04:05"), + "invalid_field_value", + ), + ( + "datetime", + NativeValue(NativeCategory.STRING, "2024-01-02T03:04:60Z"), + "invalid_field_value", + ), + ( + "datetime", + NativeValue(NativeCategory.STRING, "2024-01-02 03:04:05Z"), + "invalid_field_value", + ), + ( + "datetime", + NativeValue(NativeCategory.STRING, "2024-01-02T03:04:05-00:00"), + "invalid_field_value", + ), + ( + "datetime", + NativeValue(NativeCategory.STRING, "2024-01-02T03:04:05+24:00"), + "invalid_field_value", + ), + ( + "datetime", + NativeValue(NativeCategory.STRING, "2024-01-02T03:04:05-24:00"), + "invalid_field_value", + ), + ("integer", NativeValue(NativeCategory.OBJECT, ()), "type_mismatch"), + ], +) +def test_scalar_failures_distinguish_category_mismatch_from_invalid_content( + tmp_path: Path, + logical_type: str, + source: NativeValue, + issue_code: str, +) -> None: + """Failed typed values remain attributable without acquiring a canonical value.""" + field = _field_definition(tmp_path, _scalar_field(logical_type)) + + result = translate_field_value(source, field, InputSide.QUERY) + + assert result.disposition is ValueDisposition.NOT_ASSESSABLE + assert result.source is source + assert result.canonical is None + assert result.issue is not None + assert result.issue.code.value == issue_code + assert result.issue.stage is TranslationStage.TYPE_PARSING + + +def test_presence_scope_and_null_states_remain_distinct(tmp_path: Path) -> None: + """Absence, exclusion, valid null, and prohibited null have distinct outcomes.""" + required = _field_definition(tmp_path, _scalar_field("decimal")) + missing = translate_field_value(None, required, InputSide.REFERENCE) + excluded = translate_field_value( + NativeValue(NativeCategory.STRING, "ignored"), + required, + InputSide.REFERENCE, + in_scope=False, + ) + + def optional_nullable(field: JsonObject) -> None: + _scalar_field("decimal")(field) + field["required"] = {"reference": False, "query": False} + field["null_policy"] = {"allowed": True, "nulls_equal": True} + + nullable = _field_definition(tmp_path, optional_nullable) + absent = translate_field_value(None, nullable, InputSide.REFERENCE) + null = translate_field_value( + NativeValue(NativeCategory.NULL), nullable, InputSide.REFERENCE + ) + prohibited = translate_field_value( + NativeValue(NativeCategory.NULL), required, InputSide.REFERENCE + ) + + assert missing.disposition is ValueDisposition.NOT_ASSESSABLE + assert missing.issue.code is TranslationIssueCode.MISSING_REQUIRED_FIELD + assert excluded.disposition is ValueDisposition.NOT_ASSESSED + assert absent.disposition is ValueDisposition.ABSENT + assert null.disposition is ValueDisposition.VALID + assert null.is_null + assert null.canonical.value is None + assert prohibited.disposition is ValueDisposition.NOT_ASSESSABLE + assert prohibited.issue.code is TranslationIssueCode.NULL_POLICY_VIOLATION + + +def test_source_processing_missing_tokens_and_closed_mapping_follow_fixed_order( + tmp_path: Path, +) -> None: + """Normalisation precedes missing-token recognition and closed value mapping.""" + + def mapped(field: JsonObject) -> None: + field["logical_type"] = "string" + field["comparator"] = {"name": "typed_exact"} + field["null_policy"] = { + "allowed": True, + "nulls_equal": True, + "missing_tokens": { + "reference": [{"category": "string", "value": " NA "}] + }, + } + field["source_normalisation"] = { + "reference": [ + {"operation": "trim", "parameters": {"side": "both"}}, + {"operation": "case_fold"}, + ] + } + field["value_mapping"] = { + "reference": [ + { + "from": {"category": "string", "value": " RESISTANT "}, + "to": {"category": "string", "value": "R"}, + }, + { + "from": {"category": "string", "value": " UNKNOWN "}, + "to": {"category": "null"}, + }, + ] + } + + field = _field_definition(tmp_path, mapped) + + missing = translate_field_value( + NativeValue(NativeCategory.STRING, " NA "), field, InputSide.REFERENCE + ) + mapped_value = translate_field_value( + NativeValue(NativeCategory.STRING, " RESISTANT "), + field, + InputSide.REFERENCE, + ) + mapped_null = translate_field_value( + NativeValue(NativeCategory.STRING, " UNKNOWN "), field, InputSide.REFERENCE + ) + unmapped = translate_field_value( + NativeValue(NativeCategory.STRING, "susceptible"), + field, + InputSide.REFERENCE, + ) + + assert missing.disposition is ValueDisposition.VALID + assert missing.is_null + assert mapped_value.canonical.value == "R" + assert mapped_null.disposition is ValueDisposition.VALID + assert mapped_null.is_null + assert unmapped.disposition is ValueDisposition.NOT_ASSESSABLE + assert unmapped.issue.code is TranslationIssueCode.MAPPING_FAILURE + assert unmapped.issue.stage is TranslationStage.VALUE_MAPPING + + +def test_source_text_operations_leave_non_string_native_values_unchanged( + tmp_path: Path, +) -> None: + """Configured text operations neither stringify nor reject native numbers.""" + + def configured(field: JsonObject) -> None: + _scalar_field("decimal")(field) + field["source_normalisation"] = { + "reference": [ + {"operation": "unicode_normalise", "parameters": {"form": "NFC"}}, + {"operation": "trim", "parameters": {"side": "both"}}, + {"operation": "case_fold"}, + ] + } + + field = _field_definition(tmp_path, configured) + source = NativeValue(NativeCategory.INTEGER, 7, "7") + + result = translate_field_value(source, field, InputSide.REFERENCE) + + assert result.disposition is ValueDisposition.VALID + assert result.canonical.value == Decimal(7) + assert result.source is source + + +@pytest.mark.parametrize( + ("trim_side", "source_text", "expected"), + [ + ("both", " \u00a0K\u00a0 ", "k"), + ("both", "\u001cK\u001c", "\u001ck\u001c"), + ("leading", " K ", "k "), + ("trailing", " K ", " k"), + ], +) +def test_source_text_operations_apply_to_strings_in_declared_order( + tmp_path: Path, + trim_side: str, + source_text: str, + expected: str, +) -> None: + """Unicode normalisation, trimming, and case folding compose predictably.""" + + def configured(field: JsonObject) -> None: + _scalar_field("string")(field) + field["source_normalisation"] = { + "query": [ + {"operation": "unicode_normalise", "parameters": {"form": "NFKC"}}, + {"operation": "trim", "parameters": {"side": trim_side}}, + {"operation": "case_fold"}, + ] + } + + field = _field_definition(tmp_path, configured) + + result = translate_field_value( + NativeValue(NativeCategory.STRING, source_text), + field, + InputSide.QUERY, + ) + + assert result.disposition is ValueDisposition.VALID + assert result.canonical.value == expected + + +def test_case_folding_is_full_locale_independent_and_does_not_normalise( + tmp_path: Path, +) -> None: + """Case folding expands sharp s while retaining decomposed Unicode form.""" + + def configured(field: JsonObject) -> None: + _scalar_field("string")(field) + field["source_normalisation"] = { + "reference": [{"operation": "case_fold"}] + } + + field = _field_definition(tmp_path, configured) + result = translate_field_value( + NativeValue(NativeCategory.STRING, "Straße A\u030A"), + field, + InputSide.REFERENCE, + ) + + assert result.disposition is ValueDisposition.VALID + assert result.canonical.value == "strasse a\u030a" + + +@pytest.mark.parametrize( + ("operation", "source", "expected"), + [ + ( + { + "operation": "decimal_quantise", + "parameters": {"places": 2, "rounding": "half_even"}, + }, + NativeValue(NativeCategory.STRING, "1.225"), + Decimal("1.22"), + ), + ( + { + "operation": "decimal_quantise", + "parameters": {"places": 2, "rounding": "half_up"}, + }, + NativeValue(NativeCategory.STRING, "1.225"), + Decimal("1.23"), + ), + ( + { + "operation": "decimal_quantise", + "parameters": {"places": 2, "rounding": "down"}, + }, + NativeValue(NativeCategory.STRING, "-1.239"), + Decimal("-1.23"), + ), + ( + {"operation": "datetime_truncate", "parameters": {"precision": "second"}}, + NativeValue( + NativeCategory.STRING, + "2024-01-02T03:04:05.999999+01:00", + ), + "2024-01-02T02:04:05Z", + ), + ( + { + "operation": "datetime_truncate", + "parameters": {"precision": "millisecond"}, + }, + NativeValue(NativeCategory.STRING, "2024-01-02T03:04:05.123956Z"), + "2024-01-02T03:04:05.123Z", + ), + ( + { + "operation": "datetime_truncate", + "parameters": {"precision": "microsecond"}, + }, + NativeValue(NativeCategory.STRING, "2024-01-02T03:04:05.1234569Z"), + "2024-01-02T03:04:05.123456Z", + ), + ], +) +def test_typed_normalisation_operations_are_explicit_and_deterministic( + tmp_path: Path, + operation: JsonObject, + source: NativeValue, + expected: object, +) -> None: + """Every registered typed operation transforms an already parsed value.""" + logical_type = "datetime" if "datetime" in operation["operation"] else "decimal" + + def configured(field: JsonObject) -> None: + _scalar_field(logical_type)(field) + field["typed_normalisation"] = [operation] + + field = _field_definition(tmp_path, configured) + result = translate_field_value(source, field, InputSide.QUERY) + + assert result.disposition is ValueDisposition.VALID + assert result.canonical.value == expected + + +def _collection_field( + collection_type: str, + element_type: str, + *, + query_representation: JsonObject | None = None, +) -> FieldMutation: + def mutate(field: JsonObject) -> None: + field["logical_type"] = { + "collection_type": collection_type, + "element_type": element_type, + "element_comparator": {"name": "typed_exact"}, + } + field["comparator"] = {"name": f"{collection_type}_exact"} + if query_representation is not None: + field["collection_representation"] = { + "query": query_representation + } + + return mutate + + +@pytest.mark.parametrize( + ("collection_type", "element_type", "source", "representation", "expected"), + [ + ( + "list", + "integer", + NativeValue( + NativeCategory.ARRAY, + ( + NativeValue(NativeCategory.STRING, "02"), + NativeValue(NativeCategory.STRING, "1"), + NativeValue(NativeCategory.STRING, "1"), + ), + ), + None, + (2, 1, 1), + ), + ( + "set", + "integer", + NativeValue(NativeCategory.STRING, '["10", "2"]'), + {"format": "json_array"}, + (10, 2), + ), + ( + "multiset", + "string", + NativeValue(NativeCategory.STRING, "b||a||b"), + {"format": "delimited_text", "delimiter": "||"}, + ("a", "b", "b"), + ), + ( + "list", + "string", + NativeValue(NativeCategory.STRING, ""), + {"format": "delimited_text", "delimiter": ","}, + (), + ), + ( + "list", + "string", + NativeValue(NativeCategory.STRING, ",a,"), + {"format": "delimited_text", "delimiter": ","}, + ("", "a", ""), + ), + ], +) +def test_collection_representations_preserve_declared_collection_semantics( + tmp_path: Path, + collection_type: str, + element_type: str, + source: NativeValue, + representation: JsonObject | None, + expected: tuple[object, ...], +) -> None: + """Lists preserve order while sets and multisets use canonical output order.""" + field = _field_definition( + tmp_path, + _collection_field( + collection_type, + element_type, + query_representation=representation, + ), + ) + + result = translate_field_value(source, field, InputSide.QUERY) + + assert result.disposition is ValueDisposition.VALID + assert result.canonical.logical_type == collection_type + assert result.canonical.element_type == element_type + assert result.canonical.value == expected + + +def test_collection_element_processing_runs_before_element_type_parsing( + tmp_path: Path, +) -> None: + """Element normalisation and closed mapping adapt a delimited source dialect.""" + representation = { + "format": "delimited_text", + "delimiter": ";", + "element_normalisation": [ + {"operation": "trim", "parameters": {"side": "both"}}, + {"operation": "case_fold"}, + ], + "element_mapping": [ + { + "from": {"category": "string", "value": "r"}, + "to": {"category": "string", "value": "resistant"}, + }, + { + "from": {"category": "string", "value": "s"}, + "to": {"category": "string", "value": "susceptible"}, + }, + ], + } + field = _field_definition( + tmp_path, + _collection_field("list", "string", query_representation=representation), + ) + + result = translate_field_value( + NativeValue(NativeCategory.STRING, " R ; s "), field, InputSide.QUERY + ) + + assert result.disposition is ValueDisposition.VALID + assert result.canonical.value == ("resistant", "susceptible") + + +def _collection_with_unmatched_element_mapping(field: JsonObject) -> None: + """Configure a closed element mapping with no entry for the test value.""" + _collection_field( + "list", + "string", + query_representation={ + "format": "json_array", + "element_mapping": [ + { + "from": {"category": "string", "value": "known"}, + "to": {"category": "string", "value": "canonical"}, + } + ], + }, + )(field) + + +@pytest.mark.parametrize( + ("source", "mutation", "issue_code", "stage"), + [ + ( + NativeValue(NativeCategory.INTEGER, 1, "1"), + _collection_field("list", "integer"), + TranslationIssueCode.TYPE_MISMATCH, + TranslationStage.COLLECTION_DECODING, + ), + ( + NativeValue(NativeCategory.STRING, "not-json"), + _collection_field( + "list", "integer", query_representation={"format": "json_array"} + ), + TranslationIssueCode.INVALID_FIELD_VALUE, + TranslationStage.COLLECTION_DECODING, + ), + ( + NativeValue( + NativeCategory.ARRAY, + (NativeValue(NativeCategory.NULL),), + ), + _collection_field("list", "integer"), + TranslationIssueCode.INVALID_FIELD_VALUE, + TranslationStage.ELEMENT_PARSING, + ), + ( + NativeValue( + NativeCategory.ARRAY, + ( + NativeValue(NativeCategory.STRING, "01"), + NativeValue(NativeCategory.STRING, "1"), + ), + ), + _collection_field("set", "integer"), + TranslationIssueCode.INVALID_FIELD_VALUE, + TranslationStage.COLLECTION_VALIDATION, + ), + ( + NativeValue( + NativeCategory.ARRAY, + (NativeValue(NativeCategory.STRING, "unknown"),), + ), + _collection_with_unmatched_element_mapping, + TranslationIssueCode.MAPPING_FAILURE, + TranslationStage.ELEMENT_MAPPING, + ), + ], +) +def test_collection_failures_identify_the_failed_processing_stage( + tmp_path: Path, + source: NativeValue, + mutation: FieldMutation, + issue_code: TranslationIssueCode, + stage: TranslationStage, +) -> None: + """A failed collection never exposes a partial canonical collection value.""" + field = _field_definition(tmp_path, mutation) + + result = translate_field_value(source, field, InputSide.QUERY) + + assert result.disposition is ValueDisposition.NOT_ASSESSABLE + assert result.canonical is None + assert result.issue.code is issue_code + assert result.issue.stage is stage + + +def test_preserved_exact_retains_recursive_parser_native_content( + tmp_path: Path, +) -> None: + """An untyped field bypasses transformation and typed canonical construction.""" + + def preserved(field: JsonObject) -> None: + field.pop("logical_type") + field["comparator"] = {"name": "preserved_exact"} + + field = _field_definition(tmp_path, preserved) + source = NativeValue( + NativeCategory.OBJECT, + ( + ( + "nested", + NativeValue( + NativeCategory.ARRAY, + (NativeValue(NativeCategory.INTEGER, 1, "1"),), + ), + ), + ), + ) + + result = translate_field_value(source, field, InputSide.REFERENCE) + + assert result.disposition is ValueDisposition.VALID + assert result.source is source + assert result.preserved is source + assert result.canonical is None From abe95cce8162c913ec3b4f37b92d03070359a44a Mon Sep 17 00:00:00 2001 From: Kevin Howe Date: Mon, 9 Nov 2026 15:53:31 +0000 Subject: [PATCH 24/45] Strengthened tests --- .../docs/IMPLEMENTATION_PLAN.md | 5 +- packages/adviso-compare/tests/test_records.py | 57 ++++++++++++++++++- 2 files changed, 58 insertions(+), 4 deletions(-) diff --git a/packages/adviso-compare/docs/IMPLEMENTATION_PLAN.md b/packages/adviso-compare/docs/IMPLEMENTATION_PLAN.md index 3e76808..462e510 100644 --- a/packages/adviso-compare/docs/IMPLEMENTATION_PLAN.md +++ b/packages/adviso-compare/docs/IMPLEMENTATION_PLAN.md @@ -195,8 +195,9 @@ Work in this step: dataset-wide field discovery; - apply the specified `record_id` preference and conservative single-field key discovery algorithm, including its no-fallback rule; -- generate same-name non-key field definitions and one-sided exclusions without - type inference, mapping, normalisation, or coercion; and +- generate same-name non-key field definitions without type inference, + mapping, normalisation, or coercion, while leaving one-sided fields undefined + so that the contract phase can produce their exclusion summaries; and - validate the generated configuration before comparison. **Approval demonstration:** custom aliases and composite keys work; duplicate or diff --git a/packages/adviso-compare/tests/test_records.py b/packages/adviso-compare/tests/test_records.py index c9f33c0..eeebc7e 100644 --- a/packages/adviso-compare/tests/test_records.py +++ b/packages/adviso-compare/tests/test_records.py @@ -60,6 +60,20 @@ def _composite_configuration() -> JsonObject: ], "query": [{"operation": "case_fold"}], } + key["value_mapping"] = { + "reference": [ + { + "from": {"category": "string", "value": " S1 "}, + "to": {"category": "string", "value": "sample-1"}, + } + ], + "query": [ + { + "from": {"category": "string", "value": "s1"}, + "to": {"category": "string", "value": "sample-1"}, + } + ], + } fields["definitions"]["batch"] = { "source_fields": { "reference": "reference_batch", @@ -96,7 +110,7 @@ def test_custom_aliases_and_composite_key_processing_create_matching_identities( assert reference_record.key == query_record.key assert reference_record.source_identity == (" S1 ", "01") assert query_record.source_identity == ("s1", "1") - assert reference_record.canonical_identity == ("s1", "1") + assert reference_record.canonical_identity == ("sample-1", "1") assert prepared.configuration.record_key == ("record_id", "batch") @@ -239,7 +253,7 @@ def test_key_error_diagnostic_escapes_control_characters_in_json_pointer( assert r"\n" in raised.value.diagnostic -def test_automatic_record_id_generates_strict_exact_shared_fields_and_exclusions( +def test_automatic_record_id_declares_shared_fields_and_omits_one_sided_fields( tmp_path: Path, ) -> None: """Shared record_id is preferred and only exact same-name fields are declared.""" @@ -280,6 +294,45 @@ def test_automatic_record_id_generates_strict_exact_shared_fields_and_exclusions } +def test_automatic_configuration_is_invariant_to_input_and_field_order( + tmp_path: Path, +) -> None: + """Equivalent record, member, and column orderings produce identical rules.""" + first_reference = _write( + tmp_path / "first-reference.json", + ( + '[{"record_id":"S2","b":2,"a":"y"},' + '{"record_id":"S1","a":"x","b":1}]' + ), + ) + first_query = _write( + tmp_path / "first-query.csv", + "record_id,b,a\nS1,1,x\nS2,2,y\n", + ) + second_reference = _write( + tmp_path / "second-reference.json", + ( + '[{"a":"x","b":1,"record_id":"S1"},' + '{"a":"y","record_id":"S2","b":2}]' + ), + ) + second_query = _write( + tmp_path / "second-query.csv", + "a,record_id,b\ny,S2,2\nx,S1,1\n", + ) + + first = prepare_comparison( + _invocation(tmp_path, first_reference, first_query) + ) + second = prepare_comparison( + _invocation(tmp_path, second_reference, second_query) + ) + + assert first.configuration.document == second.configuration.document + assert set(first.reference_records.by_key) == set(second.reference_records.by_key) + assert set(first.query_records.by_key) == set(second.query_records.by_key) + + def test_automatic_discovery_selects_the_only_exact_shared_string_candidate( tmp_path: Path, ) -> None: From a569f2c49ab9398e50cd9621136a538e2bfb7bf3 Mon Sep 17 00:00:00 2001 From: Kevin Howe Date: Mon, 9 Nov 2026 16:11:30 +0000 Subject: [PATCH 25/45] Implemented contract phase of comparison --- packages/adviso-compare/README.md | 4 +- .../src/adviso_compare/contract.py | 325 +++++++++++++ .../adviso-compare/tests/test_contract.py | 444 ++++++++++++++++++ 3 files changed, 772 insertions(+), 1 deletion(-) create mode 100644 packages/adviso-compare/src/adviso_compare/contract.py create mode 100644 packages/adviso-compare/tests/test_contract.py diff --git a/packages/adviso-compare/README.md b/packages/adviso-compare/README.md index 148975b..7f25888 100644 --- a/packages/adviso-compare/README.md +++ b/packages/adviso-compare/README.md @@ -11,7 +11,9 @@ command-line invocation layer, deterministic CSV, TSV, and JSON input parser, custom comparison-configuration loader, and source-to-canonical field translation pipeline are implemented. Automatic strict-exact configuration generation and unambiguous custom or automatic record identity are also -implemented. Contract assessment, record alignment, value comparison, and +implemented. The contract phase now assesses configured fields independently on +both sides, retains causal assessment issues, and consolidates field exclusions +without blocking unaffected content. Record alignment, value comparison, and evidence generation are not yet available. The current normative artefacts are: diff --git a/packages/adviso-compare/src/adviso_compare/contract.py b/packages/adviso-compare/src/adviso_compare/contract.py new file mode 100644 index 0000000..3a84571 --- /dev/null +++ b/packages/adviso-compare/src/adviso_compare/contract.py @@ -0,0 +1,325 @@ +"""Assess the configured data contract without blocking unaffected content.""" + +from __future__ import annotations + +from dataclasses import dataclass +from enum import Enum +from types import MappingProxyType +from typing import Mapping, TypeAlias + +from adviso_compare.configuration import EffectiveConfiguration, FieldDefinition +from adviso_compare.inputs import ( + InputFormat, + InputSide, + JsonLocation, + ParsedDataset, + SourceLocation, + TabularLocation, +) +from adviso_compare.records import KeyedRecord, RecordKey, RecordModel +from adviso_compare.translation import ( + TranslatedValue, + TranslationIssueCode, + ValueDisposition, + translate_field_value, +) + + +class ContractPhaseStatus(str, Enum): + """Completion state of the contract phase.""" + + COMPLETE = "complete" + + +@dataclass(frozen=True, slots=True) +class FieldAssessment: + """Translation outcome for one configured field on one source record.""" + + side: InputSide + record: KeyedRecord + canonical_field: str + source_field: str + translated: TranslatedValue + + @property + def disposition(self) -> ValueDisposition: + """Return whether this field is valid, absent, or not assessable.""" + return self.translated.disposition + + @property + def location(self) -> SourceLocation: + """Return the field location, or containing record when absent.""" + if self.record.record.has_field(self.source_field): + return self.record.record.location_for(self.source_field) + return self.record.record.location + + +@dataclass(frozen=True, slots=True) +class ContractFinding: + """One causal contract issue retained for later evidence construction.""" + + code: str + side: InputSide + canonical_field: str + review_priority: str + record: KeyedRecord | None + assessment: FieldAssessment | None + + @property + def classification(self) -> str: + """Return the fixed classification for contract translation issues.""" + return "assessment_issue" + + @property + def phase(self) -> str: + """Return the phase in which this issue was detected.""" + return "contract" + + @property + def location(self) -> SourceLocation | None: + """Return the attributable source location when a record exists.""" + if self.assessment is None: + return None + return self.assessment.location + + +@dataclass(frozen=True, slots=True) +class ConfiguredFieldExclusion: + """One deliberately declared but out-of-scope canonical field.""" + + canonical_field: str + + @property + def code(self) -> str: + """Return the stable exclusion code.""" + return "configured_field_excluded" + + +@dataclass(frozen=True, slots=True) +class UndeclaredFieldExclusion: + """One observed source field not represented by a field definition.""" + + side: InputSide + source_field: str + record_count: int + + @property + def code(self) -> str: + """Return the stable exclusion code.""" + return "undeclared_field_excluded" + + +FieldExclusion: TypeAlias = ConfiguredFieldExclusion | UndeclaredFieldExclusion +AssessmentIndex: TypeAlias = Mapping[ + tuple[InputSide, RecordKey, str], FieldAssessment +] + + +@dataclass(frozen=True, slots=True) +class ContractResult: + """Complete contract outcomes needed by later comparison phases.""" + + status: ContractPhaseStatus + assessments: tuple[FieldAssessment, ...] + findings: tuple[ContractFinding, ...] + exclusions: tuple[FieldExclusion, ...] + _assessment_index: AssessmentIndex + + def assessment( + self, + side: InputSide, + record_key: RecordKey, + canonical_field: str, + ) -> FieldAssessment: + """Return one field assessment by side, record key, and canonical name.""" + return self._assessment_index[(side, record_key, canonical_field)] + + +def assess_contract( + configuration: EffectiveConfiguration, + *, + reference: ParsedDataset, + query: ParsedDataset, + reference_records: RecordModel, + query_records: RecordModel, +) -> ContractResult: + """Assess every applicable field contract and consolidate exclusions.""" + assessments: list[FieldAssessment] = [] + findings: list[ContractFinding] = [] + + for side, model in ( + (InputSide.REFERENCE, reference_records), + (InputSide.QUERY, query_records), + ): + for keyed_record in sorted( + model.records, + key=lambda item: item.canonical_identity, + ): + for canonical_field in sorted(configuration.in_scope): + field = configuration.field(canonical_field) + source_field = _for_side(field, side) + source = ( + keyed_record.record.value(source_field) + if keyed_record.record.has_field(source_field) + else None + ) + translated = translate_field_value(source, field, side) + assessment = FieldAssessment( + side=side, + record=keyed_record, + canonical_field=canonical_field, + source_field=source_field, + translated=translated, + ) + assessments.append(assessment) + if translated.disposition is ValueDisposition.NOT_ASSESSABLE: + findings.append(_finding_for(assessment, field)) + + findings.extend( + _zero_record_header_findings( + configuration, + reference if side is InputSide.REFERENCE else query, + side, + model, + ) + ) + + findings.sort(key=_finding_sort_key) + exclusions = _collect_exclusions(configuration, reference, query) + index = { + (assessment.side, assessment.record.key, assessment.canonical_field): assessment + for assessment in assessments + } + return ContractResult( + status=ContractPhaseStatus.COMPLETE, + assessments=tuple(assessments), + findings=tuple(findings), + exclusions=exclusions, + _assessment_index=MappingProxyType(index), + ) + + +def _finding_for( + assessment: FieldAssessment, + field: FieldDefinition, +) -> ContractFinding: + """Build the one causal finding for a failed field translation.""" + issue = assessment.translated.issue + if issue is None: + raise AssertionError("not-assessable translation has no causal issue") + return ContractFinding( + code=issue.code.value, + side=assessment.side, + canonical_field=assessment.canonical_field, + review_priority=field.review_priority, + record=assessment.record, + assessment=assessment, + ) + + +def _zero_record_header_findings( + configuration: EffectiveConfiguration, + dataset: ParsedDataset, + side: InputSide, + model: RecordModel, +) -> list[ContractFinding]: + """Assess required fields exposed by an empty tabular dataset's header.""" + if model.records: + return [] + if dataset.provenance.format is InputFormat.JSON: + return [] + + findings: list[ContractFinding] = [] + for canonical_field in sorted(configuration.in_scope): + field = configuration.field(canonical_field) + source_field = _for_side(field, side) + required = ( + field.required.reference + if side is InputSide.REFERENCE + else field.required.query + ) + if required and source_field not in dataset.field_names: + findings.append( + ContractFinding( + code=TranslationIssueCode.MISSING_REQUIRED_FIELD.value, + side=side, + canonical_field=canonical_field, + review_priority=field.review_priority, + record=None, + assessment=None, + ) + ) + return findings + + +def _collect_exclusions( + configuration: EffectiveConfiguration, + reference: ParsedDataset, + query: ParsedDataset, +) -> tuple[FieldExclusion, ...]: + """Return configured and undeclared exclusions in their specified order.""" + exclusions: list[FieldExclusion] = [ + ConfiguredFieldExclusion(canonical_field=name) + for name in sorted(set(configuration.fields) - set(configuration.in_scope)) + ] + for side, dataset in ( + (InputSide.REFERENCE, reference), + (InputSide.QUERY, query), + ): + represented = { + _for_side(field, side) for field in configuration.fields.values() + } + for source_field in sorted(set(dataset.field_names) - represented): + exclusions.append( + UndeclaredFieldExclusion( + side=side, + source_field=source_field, + record_count=sum( + record.has_field(source_field) for record in dataset.records + ), + ) + ) + return tuple(exclusions) + + +def _finding_sort_key(finding: ContractFinding) -> tuple[object, ...]: + """Apply the relevant prefix of the canonical total finding order.""" + code_order = { + "missing_required_field": 2, + "type_mismatch": 3, + "invalid_field_value": 4, + "null_policy_violation": 5, + "mapping_failure": 6, + } + side_order = {InputSide.REFERENCE: 0, InputSide.QUERY: 1} + return ( + finding.record is not None, + finding.record.canonical_identity if finding.record is not None else (), + finding.canonical_field, + code_order[finding.code], + side_order[finding.side], + _location_sort_key(finding.location), + ) + + +def _location_sort_key(location: SourceLocation | None) -> tuple[object, ...]: + """Return a deterministic order key for a possibly absent location.""" + if location is None: + return (0,) + if isinstance(location, TabularLocation): + return ( + 1, + location.record_number, + location.line_number, + location.field_name or "", + ) + if isinstance(location, JsonLocation): + return (1, location.pointer) + raise AssertionError("unknown source-location type") + + +def _for_side(field: FieldDefinition, side: InputSide) -> str: + """Return one field's source name for the selected dataset side.""" + if side is InputSide.REFERENCE: + return field.source_fields.reference + return field.source_fields.query diff --git a/packages/adviso-compare/tests/test_contract.py b/packages/adviso-compare/tests/test_contract.py new file mode 100644 index 0000000..a95c3b3 --- /dev/null +++ b/packages/adviso-compare/tests/test_contract.py @@ -0,0 +1,444 @@ +"""Behavioural tests for contract assessment and exclusions.""" + +from __future__ import annotations + +from pathlib import Path + +import yaml + +from adviso_compare.application import PreparedComparison, prepare_comparison +from adviso_compare.contract import ( + ConfiguredFieldExclusion, + ContractPhaseStatus, + ContractResult, + UndeclaredFieldExclusion, + assess_contract, +) +from adviso_compare.inputs import InputSide +from adviso_compare.invocation import ComparisonInvocation +from adviso_compare.translation import ValueDisposition +from tests.support.documents import JsonObject, minimal_configuration + + +def _write(path: Path, text: str) -> Path: + """Write one controlled UTF-8 document.""" + path.write_text(text, encoding="utf-8") + return path + + +def _field( + reference_name: str, + query_name: str, + *, + logical_type: str = "decimal", + required_reference: bool = True, + required_query: bool = True, + review_priority: str | None = None, +) -> JsonObject: + """Create one typed field declaration for a contract fixture.""" + field: JsonObject = { + "source_fields": { + "reference": reference_name, + "query": query_name, + }, + "required": { + "reference": required_reference, + "query": required_query, + }, + "logical_type": logical_type, + "null_policy": {"allowed": False, "nulls_equal": False}, + "comparator": {"name": "typed_exact"}, + } + if review_priority is not None: + field["review_priority"] = review_priority + return field + + +def _mixed_configuration() -> JsonObject: + """Declare every contract outcome alongside unaffected and excluded fields.""" + document = minimal_configuration() + fields = document["fields"] + definitions = fields["definitions"] + fields["in_scope"] = [ + "record_id", + "measured_value", + "required_field", + "typed_field", + "invalid_field", + "null_field", + "mapped_field", + "optional_field", + ] + definitions["required_field"] = _field( + "required_reference", + "required_query", + review_priority="high", + ) + definitions["typed_field"] = _field( + "typed_reference", + "typed_query", + review_priority="low", + ) + definitions["invalid_field"] = _field( + "invalid_reference", + "invalid_query", + ) + definitions["null_field"] = _field( + "null_reference", + "null_query", + review_priority="high", + ) + definitions["mapped_field"] = _field( + "mapped_reference", + "mapped_query", + logical_type="string", + review_priority="low", + ) + definitions["mapped_field"]["value_mapping"] = { + "reference": [ + { + "from": {"category": "string", "value": "known"}, + "to": {"category": "string", "value": "canonical"}, + } + ], + "query": [ + { + "from": {"category": "string", "value": "known"}, + "to": {"category": "string", "value": "canonical"}, + } + ], + } + definitions["optional_field"] = _field( + "optional_reference", + "optional_query", + required_reference=False, + required_query=False, + ) + definitions["excluded_field"] = _field( + "excluded_reference", + "excluded_query", + ) + return document + + +def _prepare_mixed_comparison(tmp_path: Path) -> PreparedComparison: + """Create two aligned heterogeneous datasets with independent defects.""" + reference = _write( + tmp_path / "reference.json", + ( + '[{"reference_id":"S1","expected_value":1,' + '"required_reference":1,"typed_reference":true,' + '"invalid_reference":"NaN","null_reference":1,' + '"mapped_reference":"unknown","excluded_reference":1,' + '"common_extra":"r1","reference_extra":"only"},' + '{"reference_id":"S2","expected_value":2,' + '"required_reference":2,"typed_reference":2,' + '"invalid_reference":"2","null_reference":2,' + '"mapped_reference":"known","excluded_reference":2,' + '"common_extra":"r2"}]' + ), + ) + query = _write( + tmp_path / "query.json", + ( + '[{"query_id":"S1","observed_value":1,' + '"typed_query":[],"invalid_query":"1",' + '"null_query":null,"mapped_query":"known",' + '"excluded_query":1,"common_extra":"q1",' + '"query_extra":"first"},' + '{"query_id":"S2","observed_value":2,' + '"required_query":2,"typed_query":2,"invalid_query":"2",' + '"null_query":2,"mapped_query":"known","excluded_query":2,' + '"query_extra":"second"}]' + ), + ) + configuration = _write( + tmp_path / "configuration.yaml", + yaml.safe_dump(_mixed_configuration(), sort_keys=False), + ) + return prepare_comparison( + ComparisonInvocation( + reference=reference, + query=query, + output=tmp_path / "evidence", + configuration=configuration, + ) + ) + + +def _assess(prepared: PreparedComparison) -> ContractResult: + """Pass a prepared comparison's domain inputs to the contract boundary.""" + return assess_contract( + prepared.configuration, + reference=prepared.reference, + query=prepared.query, + reference_records=prepared.reference_records, + query_records=prepared.query_records, + ) + + +def test_mixed_contract_collects_every_causal_finding_and_continues( + tmp_path: Path, +) -> None: + """Independent defects block only their values and retain valid assessments.""" + prepared = _prepare_mixed_comparison(tmp_path) + + result = _assess(prepared) + + assert result.status is ContractPhaseStatus.COMPLETE + observed = [ + ( + finding.code, + finding.side, + finding.canonical_field, + finding.review_priority, + finding.record.canonical_identity if finding.record else None, + ) + for finding in result.findings + ] + assert observed == [ + ( + "invalid_field_value", + InputSide.REFERENCE, + "invalid_field", + "normal", + ("S1",), + ), + ("mapping_failure", InputSide.REFERENCE, "mapped_field", "low", ("S1",)), + ("null_policy_violation", InputSide.QUERY, "null_field", "high", ("S1",)), + ("missing_required_field", InputSide.QUERY, "required_field", "high", ("S1",)), + ("type_mismatch", InputSide.REFERENCE, "typed_field", "low", ("S1",)), + ("type_mismatch", InputSide.QUERY, "typed_field", "low", ("S1",)), + ] + assert all( + finding.classification == "assessment_issue" for finding in result.findings + ) + assert all(finding.phase == "contract" for finding in result.findings) + assert all(finding.location is not None for finding in result.findings) + + key = prepared.reference_records.records[0].key + assert result.assessment(InputSide.REFERENCE, key, "invalid_field").disposition is ( + ValueDisposition.NOT_ASSESSABLE + ) + assert result.assessment(InputSide.QUERY, key, "required_field").disposition is ( + ValueDisposition.NOT_ASSESSABLE + ) + measured = result.assessment(InputSide.REFERENCE, key, "measured_value") + assert measured.disposition is ValueDisposition.VALID + second_key = prepared.reference_records.records[1].key + mapped = result.assessment(InputSide.REFERENCE, second_key, "mapped_field") + optional = result.assessment(InputSide.QUERY, second_key, "optional_field") + assert mapped.disposition is ValueDisposition.VALID + assert optional.disposition is ValueDisposition.ABSENT + + assessment_identities = [ + ( + assessment.side, + assessment.record.canonical_identity, + assessment.canonical_field, + ) + for assessment in result.assessments + ] + assert len(assessment_identities) == 32 + assert len(set(assessment_identities)) == 32 + + blocked = { + ( + assessment.side, + assessment.record.key, + assessment.canonical_field, + ) + for assessment in result.assessments + if assessment.disposition is ValueDisposition.NOT_ASSESSABLE + } + causal_findings = { + (finding.side, finding.record.key, finding.canonical_field) + for finding in result.findings + if finding.record is not None + } + assert blocked == causal_findings + assert len(blocked) == len(result.findings) == 6 + + unaffected = [ + assessment + for assessment in result.assessments + if assessment.record.canonical_identity == ("S2",) + ] + assert len(unaffected) == 16 + assert {assessment.disposition for assessment in unaffected} == { + ValueDisposition.VALID, + ValueDisposition.ABSENT, + } + + +def test_contract_consolidates_configured_and_undeclared_exclusions( + tmp_path: Path, +) -> None: + """Out-of-scope fields produce ordered summaries rather than per-record findings.""" + prepared = _prepare_mixed_comparison(tmp_path) + + result = _assess(prepared) + + assert result.exclusions == ( + ConfiguredFieldExclusion(canonical_field="excluded_field"), + UndeclaredFieldExclusion( + side=InputSide.REFERENCE, + source_field="common_extra", + record_count=2, + ), + UndeclaredFieldExclusion( + side=InputSide.REFERENCE, + source_field="reference_extra", + record_count=1, + ), + UndeclaredFieldExclusion( + side=InputSide.QUERY, + source_field="common_extra", + record_count=1, + ), + UndeclaredFieldExclusion( + side=InputSide.QUERY, + source_field="query_extra", + record_count=2, + ), + ) + assert all( + assessment.canonical_field != "excluded_field" + for assessment in result.assessments + ) + assert len(result.findings) == 6 + + +def test_zero_record_headers_support_dataset_contract_checks_and_zero_counts( + tmp_path: Path, +) -> None: + """Tabular structure remains assessable without inventing record opportunities.""" + document = minimal_configuration() + document["fields"]["definitions"]["header_only_excluded"] = _field( + "excluded_reference", + "excluded_query", + ) + reference = _write( + tmp_path / "reference.csv", + "reference_id,header_extra\n", + ) + query = _write( + tmp_path / "query.tsv", + "query_id\tobserved_value\tquery_extra\n", + ) + configuration = _write( + tmp_path / "configuration.yaml", + yaml.safe_dump(document, sort_keys=False), + ) + prepared = prepare_comparison( + ComparisonInvocation( + reference=reference, + query=query, + output=tmp_path / "evidence", + configuration=configuration, + ) + ) + + result = _assess(prepared) + + assert result.status is ContractPhaseStatus.COMPLETE + assert result.assessments == () + assert [ + (finding.code, finding.side, finding.canonical_field, finding.record) + for finding in result.findings + ] == [ + ("missing_required_field", InputSide.REFERENCE, "measured_value", None), + ] + assert result.findings[0].location is None + assert result.exclusions == ( + ConfiguredFieldExclusion(canonical_field="header_only_excluded"), + UndeclaredFieldExclusion( + side=InputSide.REFERENCE, + source_field="header_extra", + record_count=0, + ), + UndeclaredFieldExclusion( + side=InputSide.QUERY, + source_field="query_extra", + record_count=0, + ), + ) + + +def test_empty_json_exposes_no_structure_for_dataset_missing_field_findings( + tmp_path: Path, +) -> None: + """Custom configuration permits empty JSON without inferring field absence.""" + reference = _write(tmp_path / "reference.json", "[]") + query = _write(tmp_path / "query.json", "[]") + configuration = _write( + tmp_path / "configuration.yaml", + yaml.safe_dump(minimal_configuration(), sort_keys=False), + ) + prepared = prepare_comparison( + ComparisonInvocation( + reference=reference, + query=query, + output=tmp_path / "evidence", + configuration=configuration, + ) + ) + + result = _assess(prepared) + + assert result.status is ContractPhaseStatus.COMPLETE + assert result.assessments == () + assert result.findings == () + assert result.exclusions == () + + +def test_automatic_configuration_drives_contract_assessment_and_exclusions( + tmp_path: Path, +) -> None: + """Generated correspondence assesses common fields and excludes side-only ones.""" + reference = _write( + tmp_path / "reference.json", + '[{"record_id":"S1","shared":1,"reference_only":"context"}]', + ) + query = _write( + tmp_path / "query.json", + '[{"record_id":"S1","shared":1,"query_only":"context"}]', + ) + prepared = prepare_comparison( + ComparisonInvocation( + reference=reference, + query=query, + output=tmp_path / "evidence", + ) + ) + + result = _assess(prepared) + + assert result.status is ContractPhaseStatus.COMPLETE + assert result.findings == () + assert [ + ( + assessment.side, + assessment.record.canonical_identity, + assessment.canonical_field, + assessment.disposition, + ) + for assessment in result.assessments + ] == [ + (InputSide.REFERENCE, ("S1",), "record_id", ValueDisposition.VALID), + (InputSide.REFERENCE, ("S1",), "shared", ValueDisposition.VALID), + (InputSide.QUERY, ("S1",), "record_id", ValueDisposition.VALID), + (InputSide.QUERY, ("S1",), "shared", ValueDisposition.VALID), + ] + assert result.exclusions == ( + UndeclaredFieldExclusion( + side=InputSide.REFERENCE, + source_field="reference_only", + record_count=1, + ), + UndeclaredFieldExclusion( + side=InputSide.QUERY, + source_field="query_only", + record_count=1, + ), + ) From f3ff19f9ec3015881da0bce454d4ed5824c49a98 Mon Sep 17 00:00:00 2001 From: Kevin Howe Date: Mon, 9 Nov 2026 17:32:00 +0000 Subject: [PATCH 26/45] Implemented records phase of comparison --- packages/adviso-compare/README.md | 5 +- .../src/adviso_compare/alignment.py | 191 +++++++++++ .../adviso-compare/tests/test_alignment.py | 307 ++++++++++++++++++ 3 files changed, 501 insertions(+), 2 deletions(-) create mode 100644 packages/adviso-compare/src/adviso_compare/alignment.py create mode 100644 packages/adviso-compare/tests/test_alignment.py diff --git a/packages/adviso-compare/README.md b/packages/adviso-compare/README.md index 7f25888..1dc24b2 100644 --- a/packages/adviso-compare/README.md +++ b/packages/adviso-compare/README.md @@ -13,8 +13,9 @@ translation pipeline are implemented. Automatic strict-exact configuration generation and unambiguous custom or automatic record identity are also implemented. The contract phase now assesses configured fields independently on both sides, retains causal assessment issues, and consolidates field exclusions -without blocking unaffected content. Record alignment, value comparison, and -evidence generation are not yet available. +without blocking unaffected content. The records phase deterministically aligns +canonical identities and retains every matched, missing, and extra record. Value +comparison and evidence generation are not yet available. The current normative artefacts are: diff --git a/packages/adviso-compare/src/adviso_compare/alignment.py b/packages/adviso-compare/src/adviso_compare/alignment.py new file mode 100644 index 0000000..0d1b41f --- /dev/null +++ b/packages/adviso-compare/src/adviso_compare/alignment.py @@ -0,0 +1,191 @@ +"""Align validated reference and query records by canonical identity.""" + +from __future__ import annotations + +from dataclasses import dataclass +from enum import Enum +from types import MappingProxyType +from typing import Mapping + +from adviso_compare.inputs import InputSide, SourceLocation +from adviso_compare.records import KeyedRecord, RecordKey, RecordModel + + +class RecordsPhaseStatus(str, Enum): + """Completion state of the records phase.""" + + COMPLETE = "complete" + + +class RecordDisposition(str, Enum): + """Relationship of one canonical key across the two datasets.""" + + MATCHED = "matched" + MISSING = "missing" + EXTRA = "extra" + + +@dataclass(frozen=True, slots=True) +class RecordAlignment: + """Reference and query records sharing one canonical key, where present.""" + + key: RecordKey + reference: KeyedRecord | None + query: KeyedRecord | None + disposition: RecordDisposition + + @property + def canonical_identity(self) -> tuple[str, ...]: + """Return the canonical textual key components in configured order.""" + record = self.reference if self.reference is not None else self.query + if record is None: + raise AssertionError("record alignment has neither side") + return record.canonical_identity + + +@dataclass(frozen=True, slots=True) +class RecordFinding: + """One missing- or extra-record difference for later evidence construction.""" + + alignment: RecordAlignment + + @property + def code(self) -> str: + """Return the stable finding code implied by the alignment.""" + if self.alignment.disposition is RecordDisposition.MISSING: + return "missing_record" + if self.alignment.disposition is RecordDisposition.EXTRA: + return "extra_record" + raise AssertionError("matched alignment cannot produce a record finding") + + @property + def classification(self) -> str: + """Return the fixed classification for record differences.""" + return "comparison_difference" + + @property + def phase(self) -> str: + """Return the phase in which this difference was detected.""" + return "records" + + @property + def affected_side(self) -> InputSide: + """Return the side from which the aligned record is absent.""" + if self.alignment.disposition is RecordDisposition.MISSING: + return InputSide.QUERY + if self.alignment.disposition is RecordDisposition.EXTRA: + return InputSide.REFERENCE + raise AssertionError("matched alignment cannot affect a side") + + @property + def reference_location(self) -> SourceLocation | None: + """Return the reference record location, or None when it is absent.""" + if self.alignment.reference is None: + return None + return self.alignment.reference.record.location + + @property + def query_location(self) -> SourceLocation | None: + """Return the query record location, or None when it is absent.""" + if self.alignment.query is None: + return None + return self.alignment.query.record.location + + +@dataclass(frozen=True, slots=True) +class RecordCounts: + """Reconciled record counts for later canonical summary generation.""" + + reference: int + query: int + matched: int + missing: int + extra: int + + +@dataclass(frozen=True, slots=True) +class RecordsResult: + """Complete deterministic alignment of two valid record models.""" + + status: RecordsPhaseStatus + alignments: tuple[RecordAlignment, ...] + findings: tuple[RecordFinding, ...] + counts: RecordCounts + _alignment_index: Mapping[RecordKey, RecordAlignment] + + def alignment(self, record_key: RecordKey) -> RecordAlignment: + """Return the alignment for one canonical record key.""" + return self._alignment_index[record_key] + + @property + def matched(self) -> tuple[RecordAlignment, ...]: + """Return safely aligned records available to the values phase.""" + return tuple( + alignment + for alignment in self.alignments + if alignment.disposition is RecordDisposition.MATCHED + ) + + +def align_records( + reference: RecordModel, + query: RecordModel, +) -> RecordsResult: + """Classify the union of two valid record-key indexes.""" + keys = set(reference.by_key) | set(query.by_key) + ordered_keys = sorted( + keys, + key=lambda key: tuple(component.text for component in key), + ) + alignments: list[RecordAlignment] = [] + findings: list[RecordFinding] = [] + + for key in ordered_keys: + reference_record = reference.by_key.get(key) + query_record = query.by_key.get(key) + if reference_record is not None and query_record is not None: + disposition = RecordDisposition.MATCHED + elif reference_record is not None: + disposition = RecordDisposition.MISSING + elif query_record is not None: + disposition = RecordDisposition.EXTRA + else: + raise AssertionError("record-key union contains no record") + alignment = RecordAlignment( + key=key, + reference=reference_record, + query=query_record, + disposition=disposition, + ) + alignments.append(alignment) + if disposition is not RecordDisposition.MATCHED: + findings.append(RecordFinding(alignment)) + + matched = sum( + alignment.disposition is RecordDisposition.MATCHED + for alignment in alignments + ) + missing = sum( + alignment.disposition is RecordDisposition.MISSING + for alignment in alignments + ) + extra = sum( + alignment.disposition is RecordDisposition.EXTRA + for alignment in alignments + ) + alignment_tuple = tuple(alignments) + return RecordsResult( + status=RecordsPhaseStatus.COMPLETE, + alignments=alignment_tuple, + findings=tuple(findings), + counts=RecordCounts( + reference=len(reference.records), + query=len(query.records), + matched=matched, + missing=missing, + extra=extra, + ), + _alignment_index=MappingProxyType( + {alignment.key: alignment for alignment in alignment_tuple} + ), + ) diff --git a/packages/adviso-compare/tests/test_alignment.py b/packages/adviso-compare/tests/test_alignment.py new file mode 100644 index 0000000..ed55b96 --- /dev/null +++ b/packages/adviso-compare/tests/test_alignment.py @@ -0,0 +1,307 @@ +"""Behavioural tests for deterministic record alignment.""" + +from __future__ import annotations + +from pathlib import Path + +import pytest +import yaml + +from adviso_compare.alignment import ( + RecordDisposition, + RecordsResult, + RecordsPhaseStatus, + align_records, +) +from adviso_compare.application import PreparedComparison, prepare_comparison +from adviso_compare.inputs import InputSide, JsonLocation +from adviso_compare.invocation import ComparisonInvocation +from tests.support.documents import JsonObject, minimal_configuration + + +def _write(path: Path, text: str) -> Path: + """Write one controlled UTF-8 document.""" + path.write_text(text, encoding="utf-8") + return path + + +def _prepare( + tmp_path: Path, + reference_text: str, + query_text: str, + configuration: JsonObject | None = None, +) -> PreparedComparison: + """Prepare two JSON datasets with an optional custom configuration.""" + reference = _write(tmp_path / "reference.json", reference_text) + query = _write(tmp_path / "query.json", query_text) + configuration_path = None + if configuration is not None: + configuration_path = _write( + tmp_path / "configuration.yaml", + yaml.safe_dump(configuration, sort_keys=False), + ) + return prepare_comparison( + ComparisonInvocation( + reference=reference, + query=query, + output=tmp_path / "evidence", + configuration=configuration_path, + ) + ) + + +def _align(prepared: PreparedComparison) -> RecordsResult: + """Align the prepared record models through the records-phase boundary.""" + return align_records( + prepared.reference_records, + prepared.query_records, + ) + + +def test_reordered_inputs_have_the_same_complete_alignment(tmp_path: Path) -> None: + """Input occurrence order does not affect matches or canonical ordering.""" + first = _prepare( + tmp_path, + '[{"record_id":"S2","value":2},{"record_id":"S1","value":1}]', + '[{"record_id":"S1","value":1},{"record_id":"S2","value":2}]', + ) + second_path = tmp_path / "reordered" + second_path.mkdir() + second = _prepare( + second_path, + '[{"record_id":"S1","value":1},{"record_id":"S2","value":2}]', + '[{"record_id":"S2","value":2},{"record_id":"S1","value":1}]', + ) + + first_result = _align(first) + second_result = _align(second) + + assert first_result.status is RecordsPhaseStatus.COMPLETE + assert first_result.findings == second_result.findings == () + assert [ + (alignment.canonical_identity, alignment.disposition) + for alignment in first_result.alignments + ] == [ + (("S1",), RecordDisposition.MATCHED), + (("S2",), RecordDisposition.MATCHED), + ] + assert [ + (alignment.canonical_identity, alignment.disposition) + for alignment in second_result.alignments + ] == [ + (("S1",), RecordDisposition.MATCHED), + (("S2",), RecordDisposition.MATCHED), + ] + assert first_result.counts == second_result.counts + + +def test_composite_keys_preserve_multiple_records_for_one_sample( + tmp_path: Path, +) -> None: + """Ordered composite identity aligns distinct records sharing a sample ID.""" + configuration = minimal_configuration() + fields = configuration["fields"] + fields["record_key"] = ["record_id", "batch"] + fields["in_scope"].insert(1, "batch") + fields["definitions"]["batch"] = { + "source_fields": { + "reference": "reference_batch", + "query": "query_batch", + }, + "required": {"reference": True, "query": True}, + "logical_type": "integer", + "null_policy": {"allowed": False, "nulls_equal": False}, + "comparator": {"name": "typed_exact"}, + } + prepared = _prepare( + tmp_path, + ( + '[{"reference_id":"S1","reference_batch":"01"},' + '{"reference_id":"S1","reference_batch":"02"}]' + ), + ( + '[{"query_id":"S1","query_batch":"2"},' + '{"query_id":"S1","query_batch":"1"}]' + ), + configuration, + ) + + result = _align(prepared) + + assert [alignment.canonical_identity for alignment in result.alignments] == [ + ("S1", "1"), + ("S1", "2"), + ] + assert all( + alignment.disposition is RecordDisposition.MATCHED + for alignment in result.alignments + ) + assert [ + ( + alignment.reference.source_identity, + alignment.query.source_identity, + ) + for alignment in result.alignments + if alignment.reference is not None and alignment.query is not None + ] == [ + (("S1", "01"), ("S1", "1")), + (("S1", "02"), ("S1", "2")), + ] + + +def test_missing_and_extra_records_are_all_reported_deterministically( + tmp_path: Path, +) -> None: + """The union retains its matched intersection and every one-sided record.""" + prepared = _prepare( + tmp_path, + ( + '[{"record_id":"S3"},{"record_id":"S1"},' + '{"record_id":"S2"}]' + ), + ( + '[{"record_id":"S5"},{"record_id":"S1"},' + '{"record_id":"S4"}]' + ), + ) + + result = _align(prepared) + reordered_path = tmp_path / "reordered" + reordered_path.mkdir() + reordered = _prepare( + reordered_path, + ( + '[{"record_id":"S2"},{"record_id":"S3"},' + '{"record_id":"S1"}]' + ), + ( + '[{"record_id":"S4"},{"record_id":"S5"},' + '{"record_id":"S1"}]' + ), + ) + reordered_result = _align(reordered) + + assert result.status is RecordsPhaseStatus.COMPLETE + alignment_outcome = [ + (alignment.canonical_identity, alignment.disposition) + for alignment in result.alignments + ] + assert alignment_outcome == [ + (("S1",), RecordDisposition.MATCHED), + (("S2",), RecordDisposition.MISSING), + (("S3",), RecordDisposition.MISSING), + (("S4",), RecordDisposition.EXTRA), + (("S5",), RecordDisposition.EXTRA), + ] + assert [ + (alignment.canonical_identity, alignment.disposition) + for alignment in reordered_result.alignments + ] == alignment_outcome + finding_outcome = [ + ( + finding.code, + finding.affected_side, + finding.alignment.canonical_identity, + finding.alignment.reference.source_identity + if finding.alignment.reference is not None + else None, + finding.alignment.query.source_identity + if finding.alignment.query is not None + else None, + ) + for finding in result.findings + ] + assert finding_outcome == [ + ("missing_record", InputSide.QUERY, ("S2",), ("S2",), None), + ("missing_record", InputSide.QUERY, ("S3",), ("S3",), None), + ("extra_record", InputSide.REFERENCE, ("S4",), None, ("S4",)), + ("extra_record", InputSide.REFERENCE, ("S5",), None, ("S5",)), + ] + assert [ + ( + finding.code, + finding.affected_side, + finding.alignment.canonical_identity, + finding.alignment.reference.source_identity + if finding.alignment.reference is not None + else None, + finding.alignment.query.source_identity + if finding.alignment.query is not None + else None, + ) + for finding in reordered_result.findings + ] == finding_outcome + assert all( + finding.classification == "comparison_difference" + and finding.phase == "records" + for finding in result.findings + ) + assert result.findings[0].reference_location == JsonLocation("/2") + assert result.findings[0].query_location is None + assert result.findings[-1].reference_location is None + assert result.findings[-1].query_location == JsonLocation("/0") + assert [ + (finding.reference_location, finding.query_location) + for finding in reordered_result.findings + ] != [ + (finding.reference_location, finding.query_location) + for finding in result.findings + ] + assert result.matched == (result.alignments[0],) + assert result.alignment(result.matched[0].key) is result.matched[0] + assert reordered_result.counts == result.counts + assert result.counts.reference == result.counts.matched + result.counts.missing + assert result.counts.query == result.counts.matched + result.counts.extra + assert ( + result.counts.reference, + result.counts.query, + result.counts.matched, + result.counts.missing, + result.counts.extra, + ) == (3, 3, 1, 2, 2) + + +@pytest.mark.parametrize( + ("reference_text", "query_text", "expected_counts", "finding_codes"), + [ + ("[]", "[]", (0, 0, 0, 0, 0), ()), + ( + '[{"reference_id":"S1"}]', + "[]", + (1, 0, 0, 1, 0), + ("missing_record",), + ), + ( + "[]", + '[{"query_id":"S1"}]', + (0, 1, 0, 0, 1), + ("extra_record",), + ), + ], +) +def test_empty_and_one_sided_datasets_reconcile_without_special_cases( + tmp_path: Path, + reference_text: str, + query_text: str, + expected_counts: tuple[int, int, int, int, int], + finding_codes: tuple[str, ...], +) -> None: + """The same union rules cover empty, reference-only, and query-only inputs.""" + prepared = _prepare( + tmp_path, + reference_text, + query_text, + minimal_configuration(), + ) + + result = _align(prepared) + + assert result.status is RecordsPhaseStatus.COMPLETE + assert ( + result.counts.reference, + result.counts.query, + result.counts.matched, + result.counts.missing, + result.counts.extra, + ) == expected_counts + assert tuple(finding.code for finding in result.findings) == finding_codes From b61400cc2a03fd7c70e0a0bc675d4870fedb3362 Mon Sep 17 00:00:00 2001 From: Kevin Howe Date: Mon, 9 Nov 2026 17:45:14 +0000 Subject: [PATCH 27/45] Implemented compararors, and values phase of comparison --- packages/adviso-compare/README.md | 4 +- .../docs/IMPLEMENTATION_PLAN.md | 8 +- .../src/adviso_compare/comparators.py | 244 +++++++ .../src/adviso_compare/values.py | 218 +++++++ packages/adviso-compare/tests/test_values.py | 602 ++++++++++++++++++ 5 files changed, 1072 insertions(+), 4 deletions(-) create mode 100644 packages/adviso-compare/src/adviso_compare/comparators.py create mode 100644 packages/adviso-compare/src/adviso_compare/values.py create mode 100644 packages/adviso-compare/tests/test_values.py diff --git a/packages/adviso-compare/README.md b/packages/adviso-compare/README.md index 1dc24b2..4c5334e 100644 --- a/packages/adviso-compare/README.md +++ b/packages/adviso-compare/README.md @@ -15,7 +15,9 @@ implemented. The contract phase now assesses configured fields independently on both sides, retains causal assessment issues, and consolidates field exclusions without blocking unaffected content. The records phase deterministically aligns canonical identities and retains every matched, missing, and extra record. Value -comparison and evidence generation are not yet available. +comparison now applies every version 1 comparator and accounts for matched, +mismatched, and not-assessable opportunities. Canonical evidence generation is +not yet available. The current normative artefacts are: diff --git a/packages/adviso-compare/docs/IMPLEMENTATION_PLAN.md b/packages/adviso-compare/docs/IMPLEMENTATION_PLAN.md index 462e510..d0a034f 100644 --- a/packages/adviso-compare/docs/IMPLEMENTATION_PLAN.md +++ b/packages/adviso-compare/docs/IMPLEMENTATION_PLAN.md @@ -264,9 +264,11 @@ Work in this step: - preserve the independence of finding classification, phase, and review priority. -**Approval demonstration:** a matrix covering every comparator and its boundary -conditions produces the expected canonical outcomes, and one mixed comparison -accounts for every configured value opportunity. +**Approval demonstration:** a matrix covering representative semantics for +every comparator, together with focused numeric-tolerance boundary cases, +produces the expected canonical outcomes. One mixed custom comparison accounts +for every configured value opportunity, and an automatic-mode comparison +demonstrates that generated configuration flows through the values phase. ### Step 9 — Assemble and validate canonical results diff --git a/packages/adviso-compare/src/adviso_compare/comparators.py b/packages/adviso-compare/src/adviso_compare/comparators.py new file mode 100644 index 0000000..d9e8357 --- /dev/null +++ b/packages/adviso-compare/src/adviso_compare/comparators.py @@ -0,0 +1,244 @@ +"""Deterministic comparison of successfully translated field values.""" + +from __future__ import annotations + +from decimal import Decimal +from fractions import Fraction +from types import MappingProxyType +from typing import Callable, Mapping + +from adviso_compare.configuration import ( + CollectionLogicalType, + Comparator, + FieldDefinition, +) +from adviso_compare.inputs import NativeCategory, NativeValue +from adviso_compare.translation import ( + CanonicalValue, + TranslatedValue, + ValueDisposition, +) + + +ComparatorFunction = Callable[ + [FieldDefinition, TranslatedValue, TranslatedValue], bool +] + + +def compare_translated_values( + field: FieldDefinition, + reference: TranslatedValue, + query: TranslatedValue, +) -> bool: + """Apply one field's registered comparator to two valid translations.""" + if ( + reference.disposition is not ValueDisposition.VALID + or query.disposition is not ValueDisposition.VALID + ): + raise ValueError("comparators require two valid translated values") + if reference.is_null or query.is_null: + return ( + reference.is_null + and query.is_null + and field.null_policy.nulls_equal + ) + try: + comparator = _COMPARATORS[field.comparator.name] + except KeyError as error: + raise AssertionError("validated comparator is not registered") from error + return comparator(field, reference, query) + + +def _preserved_exact( + _field: FieldDefinition, + reference: TranslatedValue, + query: TranslatedValue, +) -> bool: + """Compare parser-native values recursively without lexical coercion.""" + if reference.preserved is None or query.preserved is None: + raise AssertionError("preserved comparator received no preserved value") + return _native_equal(reference.preserved, query.preserved) + + +def _native_equal(reference: NativeValue, query: NativeValue) -> bool: + """Apply parser-native category and recursive content equality.""" + if reference.category is not query.category: + return False + if reference.category is NativeCategory.ARRAY: + reference_values = _native_array(reference) + query_values = _native_array(query) + return len(reference_values) == len(query_values) and all( + _native_equal(reference_item, query_item) + for reference_item, query_item in zip(reference_values, query_values) + ) + if reference.category is NativeCategory.OBJECT: + reference_members = _native_object(reference) + query_members = _native_object(query) + if set(reference_members) != set(query_members): + return False + return all( + _native_equal(reference_members[name], query_members[name]) + for name in reference_members + ) + return reference.value == query.value + + +def _native_array(value: NativeValue) -> tuple[NativeValue, ...]: + """Return a validated parser-native array's elements.""" + if not isinstance(value.value, tuple) or not all( + isinstance(item, NativeValue) for item in value.value + ): + raise AssertionError("parser-native array has unexpected content") + return value.value + + +def _native_object(value: NativeValue) -> dict[str, NativeValue]: + """Return a validated parser-native object's members by name.""" + if not isinstance(value.value, tuple): + raise AssertionError("parser-native object has unexpected content") + members: dict[str, NativeValue] = {} + for member in value.value: + if ( + not isinstance(member, tuple) + or len(member) != 2 + or not isinstance(member[0], str) + or not isinstance(member[1], NativeValue) + ): + raise AssertionError("parser-native object has unexpected member") + if member[0] in members: + raise AssertionError("parser-native object has a duplicate member") + members[member[0]] = member[1] + return members + + +def _canonical_exact( + _field: FieldDefinition, + reference: TranslatedValue, + query: TranslatedValue, +) -> bool: + """Compare two canonical scalar values exactly.""" + return _canonical(reference) == _canonical(query) + + +def _numeric_tolerance( + field: FieldDefinition, + reference: TranslatedValue, + query: TranslatedValue, +) -> bool: + """Compare two canonical numbers with exact configured tolerance arithmetic.""" + reference_value = _canonical(reference).value + query_value = _canonical(query).value + return _numbers_within_tolerance( + reference_value, + query_value, + field.comparator, + ) + + +def _collection_exact( + field: FieldDefinition, + reference: TranslatedValue, + query: TranslatedValue, +) -> bool: + """Apply the declared ordered or unordered collection equality.""" + logical_type = field.logical_type + if not isinstance(logical_type, CollectionLogicalType): + raise AssertionError("collection comparator received a scalar field") + reference_value = _canonical(reference) + query_value = _canonical(query) + if reference_value.logical_type != logical_type.collection_type: + raise AssertionError("reference collection has unexpected logical type") + if query_value.logical_type != logical_type.collection_type: + raise AssertionError("query collection has unexpected logical type") + if not isinstance(reference_value.value, tuple) or not isinstance( + query_value.value, tuple + ): + raise AssertionError("non-null collection has unexpected canonical value") + + if logical_type.collection_type in {"set", "multiset"}: + return reference_value.value == query_value.value + if len(reference_value.value) != len(query_value.value): + return False + return all( + _compare_collection_element( + reference_element, + query_element, + logical_type.element_comparator, + ) + for reference_element, query_element in zip( + reference_value.value, + query_value.value, + ) + ) + + +def _compare_collection_element( + reference: object, + query: object, + comparator: Comparator, +) -> bool: + """Compare one positional list element under its declared comparator.""" + if comparator.name == "typed_exact": + return reference == query + if comparator.name == "numeric_tolerance": + return _numbers_within_tolerance(reference, query, comparator) + raise AssertionError("validated collection element comparator is not registered") + + +def _numbers_within_tolerance( + reference: object, + query: object, + comparator: Comparator, +) -> bool: + """Evaluate absolute or reference-directed relative tolerance exactly.""" + reference_fraction = _number_fraction(reference) + query_fraction = _number_fraction(query) + difference = abs(reference_fraction - query_fraction) + if "absolute_tolerance" in comparator.parameters: + allowed = _tolerance_fraction( + comparator.parameters["absolute_tolerance"] + ) + elif "relative_tolerance" in comparator.parameters: + allowed = _tolerance_fraction( + comparator.parameters["relative_tolerance"] + ) * abs(reference_fraction) + else: + raise AssertionError("numeric tolerance has no validated parameter") + return difference <= allowed + + +def _number_fraction(value: object) -> Fraction: + """Convert a canonical integer or decimal to an exact rational value.""" + if isinstance(value, bool) or not isinstance(value, (int, Decimal)): + raise AssertionError("numeric comparator received a non-numeric value") + return Fraction(value) + + +def _tolerance_fraction(value: object) -> Fraction: + """Convert a validated decimal-string tolerance without binary floating point.""" + if not isinstance(value, str): + raise AssertionError("numeric tolerance parameter is not a string") + return Fraction(Decimal(value)) + + +def _canonical(value: TranslatedValue) -> CanonicalValue: + """Return a successfully translated canonical value.""" + if value.canonical is None: + raise AssertionError("typed comparator received no canonical value") + return value.canonical + + +_COMPARATORS: Mapping[str, ComparatorFunction] = MappingProxyType( + { + "preserved_exact": _preserved_exact, + "typed_exact": _canonical_exact, + "normalised_string": _canonical_exact, + "numeric_exact": _canonical_exact, + "numeric_tolerance": _numeric_tolerance, + "date_exact": _canonical_exact, + "datetime_exact": _canonical_exact, + "list_exact": _collection_exact, + "set_exact": _collection_exact, + "multiset_exact": _collection_exact, + } +) diff --git a/packages/adviso-compare/src/adviso_compare/values.py b/packages/adviso-compare/src/adviso_compare/values.py new file mode 100644 index 0000000..bbe1071 --- /dev/null +++ b/packages/adviso-compare/src/adviso_compare/values.py @@ -0,0 +1,218 @@ +"""Compare every assessable non-key value in safely aligned records.""" + +from __future__ import annotations + +from dataclasses import dataclass +from enum import Enum +from typing import Literal + +from adviso_compare.alignment import RecordAlignment, RecordsResult +from adviso_compare.comparators import compare_translated_values +from adviso_compare.configuration import EffectiveConfiguration +from adviso_compare.contract import ContractResult, FieldAssessment +from adviso_compare.inputs import InputSide, SourceLocation +from adviso_compare.translation import ValueDisposition + + +class ValuesPhaseStatus(str, Enum): + """Completion state of the values phase.""" + + COMPLETE = "complete" + INCOMPLETE = "incomplete" + + +class ValueOpportunityDisposition(str, Enum): + """Outcome of one aligned, in-scope, non-key field opportunity.""" + + MATCHED = "matched" + MISMATCHED = "mismatched" + NOT_ASSESSABLE = "not_assessable" + + +@dataclass(frozen=True, slots=True) +class ValueOpportunity: + """One field comparison opportunity across an aligned record pair.""" + + alignment: RecordAlignment + canonical_field: str + reference: FieldAssessment + query: FieldAssessment + disposition: ValueOpportunityDisposition + + +@dataclass(frozen=True, slots=True) +class ValueFinding: + """One post-translation value difference for later evidence construction.""" + + opportunity: ValueOpportunity + review_priority: str + + @property + def code(self) -> str: + """Return the stable value-difference finding code.""" + return "value_mismatch" + + @property + def classification(self) -> str: + """Return the fixed classification for post-translation differences.""" + return "comparison_difference" + + @property + def phase(self) -> str: + """Return the phase in which the difference was detected.""" + return "values" + + @property + def affected_side(self) -> Literal["both"]: + """Return the fixed two-sided scope of a value comparison.""" + return "both" + + @property + def canonical_field(self) -> str: + """Return the compared canonical field name.""" + return self.opportunity.canonical_field + + @property + def reference_location(self) -> SourceLocation: + """Return the reference field or containing-record location.""" + return self.opportunity.reference.location + + @property + def query_location(self) -> SourceLocation: + """Return the query field or containing-record location.""" + return self.opportunity.query.location + + +@dataclass(frozen=True, slots=True) +class ValueCounts: + """Complete counts of in-scope non-key value opportunities.""" + + matched: int + mismatched: int + not_assessable: int + + +@dataclass(frozen=True, slots=True) +class ValuesResult: + """All deterministic values-phase outcomes for safely aligned records.""" + + status: ValuesPhaseStatus + opportunities: tuple[ValueOpportunity, ...] + findings: tuple[ValueFinding, ...] + counts: ValueCounts + + +def compare_values( + configuration: EffectiveConfiguration, + contract: ContractResult, + records: RecordsResult, +) -> ValuesResult: + """Classify every aligned in-scope non-key value opportunity exactly once.""" + compared_fields = sorted( + set(configuration.in_scope) - set(configuration.record_key) + ) + opportunities: list[ValueOpportunity] = [] + findings: list[ValueFinding] = [] + + for alignment in records.matched: + if alignment.reference is None or alignment.query is None: + raise AssertionError("matched alignment does not contain both records") + for canonical_field in compared_fields: + reference = contract.assessment( + InputSide.REFERENCE, + alignment.key, + canonical_field, + ) + query = contract.assessment( + InputSide.QUERY, + alignment.key, + canonical_field, + ) + disposition = _compare_opportunity( + configuration, + canonical_field, + reference, + query, + ) + opportunity = ValueOpportunity( + alignment=alignment, + canonical_field=canonical_field, + reference=reference, + query=query, + disposition=disposition, + ) + opportunities.append(opportunity) + if disposition is ValueOpportunityDisposition.MISMATCHED: + findings.append( + ValueFinding( + opportunity=opportunity, + review_priority=configuration.field( + canonical_field + ).review_priority, + ) + ) + + matched = sum( + item.disposition is ValueOpportunityDisposition.MATCHED + for item in opportunities + ) + mismatched = sum( + item.disposition is ValueOpportunityDisposition.MISMATCHED + for item in opportunities + ) + not_assessable = sum( + item.disposition is ValueOpportunityDisposition.NOT_ASSESSABLE + for item in opportunities + ) + status = ( + ValuesPhaseStatus.INCOMPLETE + if not_assessable + else ValuesPhaseStatus.COMPLETE + ) + return ValuesResult( + status=status, + opportunities=tuple(opportunities), + findings=tuple(findings), + counts=ValueCounts( + matched=matched, + mismatched=mismatched, + not_assessable=not_assessable, + ), + ) + + +def _compare_opportunity( + configuration: EffectiveConfiguration, + canonical_field: str, + reference: FieldAssessment, + query: FieldAssessment, +) -> ValueOpportunityDisposition: + """Classify one opportunity from its two contract assessments.""" + if ( + reference.disposition is ValueDisposition.NOT_ASSESSABLE + or query.disposition is ValueDisposition.NOT_ASSESSABLE + ): + return ValueOpportunityDisposition.NOT_ASSESSABLE + if ( + reference.disposition is ValueDisposition.ABSENT + and query.disposition is ValueDisposition.ABSENT + ): + return ValueOpportunityDisposition.MATCHED + if ( + reference.disposition is ValueDisposition.ABSENT + or query.disposition is ValueDisposition.ABSENT + ): + return ValueOpportunityDisposition.MISMATCHED + if ( + reference.disposition is not ValueDisposition.VALID + or query.disposition is not ValueDisposition.VALID + ): + raise AssertionError("value opportunity has an unexpected contract state") + matches = compare_translated_values( + configuration.field(canonical_field), + reference.translated, + query.translated, + ) + if matches: + return ValueOpportunityDisposition.MATCHED + return ValueOpportunityDisposition.MISMATCHED diff --git a/packages/adviso-compare/tests/test_values.py b/packages/adviso-compare/tests/test_values.py new file mode 100644 index 0000000..d51f96b --- /dev/null +++ b/packages/adviso-compare/tests/test_values.py @@ -0,0 +1,602 @@ +"""Behavioural tests for comparators and the values phase.""" + +from __future__ import annotations + +from decimal import Decimal +from pathlib import Path + +import pytest +import yaml + +from adviso_compare.alignment import align_records +from adviso_compare.application import PreparedComparison, prepare_comparison +from adviso_compare.comparators import compare_translated_values +from adviso_compare.configuration import FieldDefinition, load_comparison_configuration +from adviso_compare.contract import ContractResult, assess_contract +from adviso_compare.inputs import InputSide, NativeCategory, NativeValue +from adviso_compare.invocation import ComparisonInvocation +from adviso_compare.translation import ValueDisposition, translate_field_value +from adviso_compare.values import ( + ValueOpportunityDisposition, + ValuesPhaseStatus, + compare_values, +) +from tests.support.documents import JsonObject, minimal_configuration + + +def _write(path: Path, text: str) -> Path: + """Write one controlled UTF-8 document.""" + path.write_text(text, encoding="utf-8") + return path + + +def _scalar_field( + logical_type: str | None, + comparator: str, + *, + parameters: JsonObject | None = None, + source_normalisation: JsonObject | None = None, +) -> JsonObject: + """Create one scalar field declaration for comparator tests.""" + field: JsonObject = { + "source_fields": { + "reference": "expected_value", + "query": "observed_value", + }, + "required": {"reference": True, "query": True}, + "null_policy": {"allowed": False, "nulls_equal": False}, + "comparator": {"name": comparator}, + } + if logical_type is not None: + field["logical_type"] = logical_type + if parameters is not None: + field["comparator"]["parameters"] = parameters + if source_normalisation is not None: + field["source_normalisation"] = source_normalisation + return field + + +def _collection_field( + collection_type: str, + element_type: str, + *, + element_comparator: str = "typed_exact", + element_parameters: JsonObject | None = None, +) -> JsonObject: + """Create one collection field declaration for comparator tests.""" + comparator: JsonObject = {"name": element_comparator} + if element_parameters is not None: + comparator["parameters"] = element_parameters + return { + "source_fields": { + "reference": "expected_value", + "query": "observed_value", + }, + "required": {"reference": True, "query": True}, + "logical_type": { + "collection_type": collection_type, + "element_type": element_type, + "element_comparator": comparator, + }, + "null_policy": {"allowed": False, "nulls_equal": False}, + "comparator": {"name": f"{collection_type}_exact"}, + } + + +def _array(*values: str) -> NativeValue: + """Create a parser-native array of string elements.""" + return NativeValue( + NativeCategory.ARRAY, + tuple(NativeValue(NativeCategory.STRING, value) for value in values), + ) + + +def _object(*members: tuple[str, NativeValue]) -> NativeValue: + """Create one parser-native object retaining the supplied member order.""" + return NativeValue(NativeCategory.OBJECT, members) + + +def _resolved_field(tmp_path: Path, declaration: JsonObject) -> FieldDefinition: + """Resolve one test field through the normative configuration boundary.""" + document = minimal_configuration() + document["fields"]["definitions"]["measured_value"] = declaration + path = _write( + tmp_path / "configuration.yaml", + yaml.safe_dump(document, sort_keys=False), + ) + return load_comparison_configuration(path).field("measured_value") + + +@pytest.mark.parametrize( + ( + "declaration", + "reference_source", + "query_source", + "expected", + ), + [ + ( + _scalar_field(None, "preserved_exact"), + _object( + ("a", NativeValue(NativeCategory.DECIMAL, Decimal("1.0"), "1.0")), + ("b", _array("x", "y")), + ), + _object( + ("b", _array("x", "y")), + ( + "a", + NativeValue( + NativeCategory.DECIMAL, + Decimal("1.00"), + "1.00", + ), + ), + ), + True, + ), + ( + _scalar_field(None, "preserved_exact"), + _array("x", "y"), + _array("y", "x"), + False, + ), + ( + _scalar_field(None, "preserved_exact"), + NativeValue(NativeCategory.STRING, "1"), + NativeValue(NativeCategory.INTEGER, 1, "1"), + False, + ), + ( + _scalar_field("string", "typed_exact"), + NativeValue(NativeCategory.STRING, "same"), + NativeValue(NativeCategory.STRING, "same"), + True, + ), + ( + _scalar_field( + "string", + "normalised_string", + source_normalisation={ + "reference": [{"operation": "case_fold"}], + }, + ), + NativeValue(NativeCategory.STRING, "Alpha"), + NativeValue(NativeCategory.STRING, "alpha"), + True, + ), + ( + _scalar_field("decimal", "numeric_exact"), + NativeValue(NativeCategory.STRING, "1.00"), + NativeValue(NativeCategory.STRING, "1e0"), + True, + ), + ( + _scalar_field("integer", "numeric_exact"), + NativeValue(NativeCategory.STRING, "01"), + NativeValue(NativeCategory.STRING, "1"), + True, + ), + ( + _scalar_field("date", "date_exact"), + NativeValue(NativeCategory.STRING, "2026-11-09"), + NativeValue(NativeCategory.STRING, "2026-11-09"), + True, + ), + ( + _scalar_field("datetime", "datetime_exact"), + NativeValue(NativeCategory.STRING, "2026-11-09T12:00:00Z"), + NativeValue(NativeCategory.STRING, "2026-11-09T13:00:00+01:00"), + True, + ), + ( + _collection_field("list", "integer"), + _array("1", "2"), + _array("2", "1"), + False, + ), + ( + _collection_field("list", "integer"), + _array("1", "2"), + _array("1"), + False, + ), + ( + _collection_field( + "list", + "decimal", + element_comparator="numeric_tolerance", + element_parameters={"absolute_tolerance": "0.1"}, + ), + _array("1.0", "2.0"), + _array("1.1", "2.1"), + True, + ), + ( + _collection_field("set", "integer"), + _array("10", "2"), + _array("2", "10"), + True, + ), + ( + _collection_field("multiset", "string"), + _array("a", "b", "b"), + _array("b", "a", "b"), + True, + ), + ( + _collection_field("multiset", "string"), + _array("a", "b", "b"), + _array("a", "b"), + False, + ), + ], + ids=[ + "preserved-object-member-order", + "preserved-array-order", + "preserved-native-category", + "typed-exact", + "normalised-string", + "numeric-exact", + "integer-numeric-exact", + "date-exact", + "datetime-exact", + "list-order", + "list-length", + "list-element-tolerance", + "set-order", + "multiset-order", + "multiset-counts", + ], +) +def test_fixed_comparator_matrix( + tmp_path: Path, + declaration: JsonObject, + reference_source: NativeValue, + query_source: NativeValue, + expected: bool, +) -> None: + """Every registered comparator applies its specified canonical semantics.""" + field = _resolved_field(tmp_path, declaration) + reference = translate_field_value(reference_source, field, InputSide.REFERENCE) + query = translate_field_value(query_source, field, InputSide.QUERY) + + assert reference.disposition is ValueDisposition.VALID + assert query.disposition is ValueDisposition.VALID + assert compare_translated_values(field, reference, query) is expected + + +@pytest.mark.parametrize( + ("parameters", "reference", "query", "expected"), + [ + ({"absolute_tolerance": "0.1"}, "1", "1.1", True), + ({"absolute_tolerance": "0.1"}, "1", "0.9", True), + ({"absolute_tolerance": "0.1"}, "1", "1.1000000000000000001", False), + ({"relative_tolerance": "0.05"}, "100", "105", True), + ({"relative_tolerance": "0.05"}, "100", "95", True), + ({"relative_tolerance": "0.05"}, "100", "105.0000000000000001", False), + ({"relative_tolerance": "0.05"}, "105.1", "100", True), + ({"relative_tolerance": "0.05"}, "-100", "-105", True), + ({"relative_tolerance": "0.05"}, "0", "0", True), + ({"relative_tolerance": "0.05"}, "0", "0.0000000000000001", False), + ], +) +def test_numeric_tolerance_is_exact_inclusive_and_reference_directed( + tmp_path: Path, + parameters: JsonObject, + reference: str, + query: str, + expected: bool, +) -> None: + """Tolerance arithmetic remains exact at boundaries and around zero.""" + field = _resolved_field( + tmp_path, + _scalar_field("decimal", "numeric_tolerance", parameters=parameters), + ) + reference_value = translate_field_value( + NativeValue(NativeCategory.STRING, reference), + field, + InputSide.REFERENCE, + ) + query_value = translate_field_value( + NativeValue(NativeCategory.STRING, query), + field, + InputSide.QUERY, + ) + + assert compare_translated_values(field, reference_value, query_value) is expected + + +def _comparison_field( + reference_name: str, + query_name: str, + *, + required: bool = True, + nullable: bool = False, + nulls_equal: bool = False, + review_priority: str | None = None, +) -> JsonObject: + """Create one decimal field for a mixed values-phase fixture.""" + field = _scalar_field("decimal", "numeric_exact") + field["source_fields"] = { + "reference": reference_name, + "query": query_name, + } + field["required"] = {"reference": required, "query": required} + field["null_policy"] = { + "allowed": nullable, + "nulls_equal": nulls_equal, + } + if review_priority is not None: + field["review_priority"] = review_priority + return field + + +def _mixed_configuration() -> JsonObject: + """Declare every values-phase disposition plus one excluded field.""" + document = minimal_configuration() + fields = document["fields"] + fields["in_scope"] = [ + "record_id", + "exact_match", + "difference", + "blocked", + "optional_both_absent", + "optional_one_absent", + "nullable_both", + "nullable_mixed", + "nullable_not_equal", + ] + definitions = fields["definitions"] + definitions.pop("measured_value") + definitions["exact_match"] = _comparison_field("r_match", "q_match") + definitions["difference"] = _comparison_field( + "r_difference", + "q_difference", + review_priority="high", + ) + definitions["blocked"] = _comparison_field("r_blocked", "q_blocked") + definitions["optional_both_absent"] = _comparison_field( + "r_optional_both", + "q_optional_both", + required=False, + ) + definitions["optional_one_absent"] = _comparison_field( + "r_optional_one", + "q_optional_one", + required=False, + review_priority="low", + ) + definitions["nullable_both"] = _comparison_field( + "r_nullable_both", + "q_nullable_both", + nullable=True, + nulls_equal=True, + ) + definitions["nullable_mixed"] = _comparison_field( + "r_nullable_mixed", + "q_nullable_mixed", + nullable=True, + nulls_equal=True, + ) + definitions["nullable_not_equal"] = _comparison_field( + "r_nullable_not_equal", + "q_nullable_not_equal", + nullable=True, + nulls_equal=False, + ) + definitions["excluded"] = _comparison_field("r_excluded", "q_excluded") + return document + + +def _prepare_mixed(tmp_path: Path) -> PreparedComparison: + """Prepare one aligned record containing every value opportunity state.""" + reference = _write( + tmp_path / "reference.json", + ( + '[{"reference_id":"S1","r_match":1,"r_difference":1,' + '"r_blocked":1,"r_optional_one":1,"r_nullable_both":null,' + '"r_nullable_mixed":null,"r_nullable_not_equal":null,' + '"r_excluded":1},' + '{"reference_id":"S2","r_match":1,"r_difference":1,' + '"r_blocked":1,"r_nullable_both":null,' + '"r_nullable_mixed":null,"r_nullable_not_equal":null,' + '"r_excluded":1}]' + ), + ) + query = _write( + tmp_path / "query.json", + ( + '[{"query_id":"S1","q_match":1,"q_difference":2,' + '"q_nullable_both":null,"q_nullable_mixed":1,' + '"q_nullable_not_equal":null,"q_excluded":1},' + '{"query_id":"S3","q_match":1,"q_difference":1,' + '"q_blocked":1,"q_nullable_both":null,"q_nullable_mixed":1,' + '"q_nullable_not_equal":null,"q_excluded":1}]' + ), + ) + configuration = _write( + tmp_path / "configuration.yaml", + yaml.safe_dump(_mixed_configuration(), sort_keys=False), + ) + return prepare_comparison( + ComparisonInvocation( + reference=reference, + query=query, + output=tmp_path / "evidence", + configuration=configuration, + ) + ) + + +def _contract(prepared: PreparedComparison) -> ContractResult: + """Assess a prepared comparison through the contract boundary.""" + return assess_contract( + prepared.configuration, + reference=prepared.reference, + query=prepared.query, + reference_records=prepared.reference_records, + query_records=prepared.query_records, + ) + + +def test_mixed_values_phase_accounts_for_every_configured_opportunity( + tmp_path: Path, +) -> None: + """Blocked values coexist with all assessable matches and differences.""" + prepared = _prepare_mixed(tmp_path) + contract = _contract(prepared) + records = align_records(prepared.reference_records, prepared.query_records) + + result = compare_values(prepared.configuration, contract, records) + + assert result.status is ValuesPhaseStatus.INCOMPLETE + assert ( + records.counts.reference, + records.counts.query, + records.counts.matched, + records.counts.missing, + records.counts.extra, + ) == (2, 2, 1, 1, 1) + assert { + opportunity.alignment.canonical_identity + for opportunity in result.opportunities + } == {("S1",)} + assert [ + (opportunity.canonical_field, opportunity.disposition) + for opportunity in result.opportunities + ] == [ + ("blocked", ValueOpportunityDisposition.NOT_ASSESSABLE), + ("difference", ValueOpportunityDisposition.MISMATCHED), + ("exact_match", ValueOpportunityDisposition.MATCHED), + ("nullable_both", ValueOpportunityDisposition.MATCHED), + ("nullable_mixed", ValueOpportunityDisposition.MISMATCHED), + ("nullable_not_equal", ValueOpportunityDisposition.MISMATCHED), + ("optional_both_absent", ValueOpportunityDisposition.MATCHED), + ("optional_one_absent", ValueOpportunityDisposition.MISMATCHED), + ] + assert ( + result.counts.matched, + result.counts.mismatched, + result.counts.not_assessable, + ) == (3, 4, 1) + assert len(result.opportunities) == sum( + ( + result.counts.matched, + result.counts.mismatched, + result.counts.not_assessable, + ) + ) + assert [ + (finding.canonical_field, finding.review_priority) + for finding in result.findings + ] == [ + ("difference", "high"), + ("nullable_mixed", "normal"), + ("nullable_not_equal", "normal"), + ("optional_one_absent", "low"), + ] + assert all( + finding.code == "value_mismatch" + and finding.classification == "comparison_difference" + and finding.phase == "values" + and finding.affected_side == "both" + for finding in result.findings + ) + assert len(contract.findings) == 1 + assert contract.findings[0].code == "missing_required_field" + assert "excluded" not in { + opportunity.canonical_field for opportunity in result.opportunities + } + + +def test_automatic_configuration_flows_through_the_values_phase( + tmp_path: Path, +) -> None: + """Automatic mode compares shared fields and excludes one-sided context.""" + reference = _write( + tmp_path / "reference.json", + ( + '[{"record_id":"S1","same":1,"different":"expected",' + '"reference_only":"context"}]' + ), + ) + query = _write( + tmp_path / "query.json", + ( + '[{"record_id":"S1","same":1,"different":"observed",' + '"query_only":"context"}]' + ), + ) + prepared = prepare_comparison( + ComparisonInvocation( + reference=reference, + query=query, + output=tmp_path / "evidence", + ) + ) + contract = _contract(prepared) + result = compare_values( + prepared.configuration, + contract, + align_records(prepared.reference_records, prepared.query_records), + ) + + assert prepared.configuration.in_scope == ( + "record_id", + "different", + "same", + ) + assert { + exclusion.source_field for exclusion in contract.exclusions + } == {"query_only", "reference_only"} + assert [ + (opportunity.canonical_field, opportunity.disposition) + for opportunity in result.opportunities + ] == [ + ("different", ValueOpportunityDisposition.MISMATCHED), + ("same", ValueOpportunityDisposition.MATCHED), + ] + assert result.status is ValuesPhaseStatus.COMPLETE + assert ( + result.counts.matched, + result.counts.mismatched, + result.counts.not_assessable, + ) == (1, 1, 0) + assert [finding.canonical_field for finding in result.findings] == [ + "different" + ] + + +def test_value_mismatches_do_not_make_the_values_phase_incomplete( + tmp_path: Path, +) -> None: + """A completed unequal comparison is a finding, not an incomplete assessment.""" + reference = _write( + tmp_path / "reference.json", + '[{"reference_id":"S1","expected_value":1}]', + ) + query = _write( + tmp_path / "query.json", + '[{"query_id":"S1","observed_value":2}]', + ) + configuration = _write( + tmp_path / "configuration.yaml", + yaml.safe_dump(minimal_configuration(), sort_keys=False), + ) + prepared = prepare_comparison( + ComparisonInvocation( + reference=reference, + query=query, + output=tmp_path / "evidence", + configuration=configuration, + ) + ) + result = compare_values( + prepared.configuration, + _contract(prepared), + align_records(prepared.reference_records, prepared.query_records), + ) + + assert result.status is ValuesPhaseStatus.COMPLETE + assert result.counts.mismatched == 1 + assert len(result.findings) == 1 From 2cc635d4513ea27026b1b2c61ed106e8e5d28bc7 Mon Sep 17 00:00:00 2001 From: Kevin Howe Date: Mon, 9 Nov 2026 18:16:47 +0000 Subject: [PATCH 28/45] Implemented assembly and validation of canonical results --- packages/adviso-compare/README.md | 5 +- .../docs/IMPLEMENTATION_PLAN.md | 3 +- packages/adviso-compare/docs/SPECIFICATION.md | 5 +- .../src/adviso_compare/results.py | 1143 +++++++++++++++++ packages/adviso-compare/tests/test_results.py | 718 +++++++++++ 5 files changed, 1870 insertions(+), 4 deletions(-) create mode 100644 packages/adviso-compare/src/adviso_compare/results.py create mode 100644 packages/adviso-compare/tests/test_results.py diff --git a/packages/adviso-compare/README.md b/packages/adviso-compare/README.md index 4c5334e..11c8d62 100644 --- a/packages/adviso-compare/README.md +++ b/packages/adviso-compare/README.md @@ -16,8 +16,9 @@ both sides, retains causal assessment issues, and consolidates field exclusions without blocking unaffected content. The records phase deterministically aligns canonical identities and retains every matched, missing, and extra record. Value comparison now applies every version 1 comparator and accounts for matched, -mismatched, and not-assessable opportunities. Canonical evidence generation is -not yet available. +mismatched, and not-assessable opportunities. The comparison core now assembles +and self-validates deterministic canonical JSON results. Evidence-package +publication and the HTML report are not yet available. The current normative artefacts are: diff --git a/packages/adviso-compare/docs/IMPLEMENTATION_PLAN.md b/packages/adviso-compare/docs/IMPLEMENTATION_PLAN.md index d0a034f..4e080ba 100644 --- a/packages/adviso-compare/docs/IMPLEMENTATION_PLAN.md +++ b/packages/adviso-compare/docs/IMPLEMENTATION_PLAN.md @@ -294,7 +294,8 @@ Work in this step: **Approval demonstration:** representative complete and incomplete documents validate structurally and semantically, summaries reconcile with findings and opportunities, and repeated/order-varied runs differ only in the explicitly -permitted runtime and location evidence. +permitted runtime metadata, byte provenance, source locations, +location-dependent messages, and report-local finding IDs. ### Step 10 — Produce the evidence package and HTML report diff --git a/packages/adviso-compare/docs/SPECIFICATION.md b/packages/adviso-compare/docs/SPECIFICATION.md index 0b298be..ca6fe92 100644 --- a/packages/adviso-compare/docs/SPECIFICATION.md +++ b/packages/adviso-compare/docs/SPECIFICATION.md @@ -1306,7 +1306,10 @@ No comparison-configuration digest or command-invocation object is included in v - **REQ-OUT-062.** record-specific missing-required-field findings contain presence states but do not reproduce the present side's value; the dataset-level zero-record form contains no value evidence; - **REQ-OUT-064.** type-mismatch findings contain source values for affected sides and canonical values only where parsing succeeded; - **REQ-OUT-065.** invalid-field-value and mapping-failure findings contain the affected source value and no failed canonical value; -- **REQ-OUT-066.** null-policy-violation findings contain the affected source and canonical null representation; +- **REQ-OUT-066.** A null-policy-violation finding contains the affected + parser-native source null. For a field with a declared logical type, it also + contains that type's canonical null representation. A `preserved_exact` + field has no logical type and therefore contains no canonical value. - **REQ-OUT-067.** value-mismatch findings MUST state the presence state of both sides. Each present side MUST contain its source value and, for a declared typed field, its canonical value. An absent side contains neither value. A mismatch of two present untyped values contains no typed canonical values. - **REQ-OUT-068.** A causal finding for a not-assessable value opportunity MUST contain only the values permitted for that finding code and necessary to explain the failed prerequisite. It MUST NOT contain separate or duplicated not-assessable evidence. diff --git a/packages/adviso-compare/src/adviso_compare/results.py b/packages/adviso-compare/src/adviso_compare/results.py new file mode 100644 index 0000000..297e9b5 --- /dev/null +++ b/packages/adviso-compare/src/adviso_compare/results.py @@ -0,0 +1,1143 @@ +"""Assemble, validate, and serialize canonical comparison results.""" + +from __future__ import annotations + +import json +import platform as runtime_platform +import unicodedata +from collections import Counter +from dataclasses import dataclass +from datetime import datetime, timezone +from functools import lru_cache +from importlib.resources import files +from typing import TYPE_CHECKING, Any, Mapping +from uuid import UUID, uuid4 + +from jsonschema import Draft202012Validator, FormatChecker, ValidationError +from referencing import Registry, Resource + +from adviso_compare._version import __version__ +from adviso_compare.alignment import RecordAlignment, RecordFinding, RecordsResult +from adviso_compare.configuration import EffectiveConfiguration, FieldDefinition +from adviso_compare.contract import ( + ConfiguredFieldExclusion, + ContractFinding, + ContractResult, + FieldAssessment, + UndeclaredFieldExclusion, +) +from adviso_compare.inputs import ( + InputSide, + JsonLocation, + NativeCategory, + NativeValue, + ParsedDataset, + SourceLocation, + TabularLocation, +) +from adviso_compare.translation import ( + CanonicalValue, + TranslatedValue, + ValueDisposition, + render_canonical_scalar, +) +from adviso_compare.values import ValueFinding, ValuesResult + +if TYPE_CHECKING: + from adviso_compare.application import PreparedComparison + + +JsonObject = dict[str, Any] +RESULTS_SCHEMA_FILE = "comparison-results-v1.0.schema.json" +CONFIGURATION_SCHEMA_FILE = "comparison-configuration-v1.0.schema.json" +SCHEMA_PACKAGE = "adviso_compare.schemas" +FINDING_CODES = ( + "missing_record", + "extra_record", + "missing_required_field", + "type_mismatch", + "invalid_field_value", + "null_policy_violation", + "mapping_failure", + "value_mismatch", +) +CLASSIFICATIONS = ("comparison_difference", "assessment_issue") +REVIEW_PRIORITIES = ("low", "normal", "high", "unassigned") + + +@dataclass(frozen=True, slots=True) +class RuntimeIdentity: + """Non-identifying software-runtime provenance.""" + + python: str + platform: str + unicode_version: str + + +@dataclass(frozen=True, slots=True) +class ResultValidationError(ValueError): + """An internally generated result violates a structural or semantic rule.""" + + message: str + + def __str__(self) -> str: + """Return the concise invariant failure.""" + return self.message + + +def current_runtime_identity() -> RuntimeIdentity: + """Return normalized, non-identifying runtime provenance.""" + implementation = runtime_platform.python_implementation().casefold() + version = runtime_platform.python_version() + operating_system = runtime_platform.system().casefold() + operating_system = {"darwin": "macos"}.get( + operating_system, + operating_system, + ) + architecture = runtime_platform.machine().casefold() + architecture = { + "amd64": "x86_64", + "arm64": "aarch64", + }.get(architecture, architecture) + return RuntimeIdentity( + python=f"{implementation}-{version}", + platform=f"{operating_system}-{architecture}", + unicode_version=unicodedata.unidata_version, + ) + + +def assemble_result( + prepared: PreparedComparison, + contract: ContractResult, + records: RecordsResult, + values: ValuesResult, + *, + run_id: UUID | None = None, + completed_at: datetime | None = None, + software_version: str = __version__, + runtime: RuntimeIdentity | None = None, +) -> JsonObject: + """Build and self-validate one complete canonical result document.""" + identifier = run_id or uuid4() + completion = completed_at or datetime.now(timezone.utc) + if identifier.version != 4: + raise ValueError("run_id must be a UUID version 4") + if completion.tzinfo is None or completion.utcoffset() is None: + raise ValueError("completed_at must be timezone-aware") + completion_text = completion.astimezone(timezone.utc).replace( + microsecond=0 + ).strftime("%Y-%m-%dT%H:%M:%SZ") + runtime_identity = runtime or current_runtime_identity() + + findings = _assemble_findings(prepared.configuration, contract, records, values) + phase_statuses = { + "contract": contract.status.value, + "records": records.status.value, + "values": values.status.value, + } + run_status = ( + "complete" + if all(status == "complete" for status in phase_statuses.values()) + else "incomplete" + ) + document: JsonObject = { + "results_schema_version": "1.0", + "run": { + "id": str(identifier), + "status": run_status, + "completed_at": completion_text, + }, + "phases": phase_statuses, + "software": {"name": "adviso-compare", "version": software_version}, + "runtime": { + "python": runtime_identity.python, + "platform": runtime_identity.platform, + "unicode_version": runtime_identity.unicode_version, + }, + "inputs": { + "reference": _input_document(prepared.reference), + "query": _input_document(prepared.query), + }, + "record_key_fields": _record_key_fields(prepared.configuration), + "comparison_configuration": { + "document": _ordered_configuration(prepared.configuration.document) + }, + "summary": _summary_document(records, values, findings), + "exclusions": [_exclusion_document(item) for item in contract.exclusions], + "findings": findings, + } + validate_result( + document, + prepared=prepared, + contract=contract, + records=records, + values=values, + ) + return document + + +def serialize_result(document: Mapping[str, object]) -> bytes: + """Serialize a validated result as canonical UTF-8 JSON.""" + return ( + json.dumps( + document, + ensure_ascii=False, + indent=2, + allow_nan=False, + ) + + "\n" + ).encode("utf-8") + + +def validate_result( + document: JsonObject, + *, + prepared: PreparedComparison | None = None, + contract: ContractResult | None = None, + records: RecordsResult | None = None, + values: ValuesResult | None = None, +) -> None: + """Apply schema and semantic validation to a generated result.""" + try: + _results_validator().validate(document) + except ValidationError as error: + raise ResultValidationError( + "canonical result failed schema validation" + ) from error + _validate_semantics(document, prepared, contract, records, values) + + +@lru_cache(maxsize=1) +def _results_validator() -> Draft202012Validator: + """Load the packaged schemas with format assertion and local resolution.""" + package = files(SCHEMA_PACKAGE) + results = json.loads( + package.joinpath(RESULTS_SCHEMA_FILE).read_text(encoding="utf-8") + ) + configuration = json.loads( + package.joinpath(CONFIGURATION_SCHEMA_FILE).read_text(encoding="utf-8") + ) + registry = Registry().with_resources( + ( + (results["$id"], Resource.from_contents(results)), + (configuration["$id"], Resource.from_contents(configuration)), + ) + ) + return Draft202012Validator( + results, + registry=registry, + format_checker=FormatChecker(), + ) + + +def _input_document(dataset: ParsedDataset) -> JsonObject: + """Render exact-byte input provenance without exposing a path.""" + provenance = dataset.provenance + return { + "basename": provenance.basename, + "format": provenance.format.value, + "encoding": provenance.encoding, + "size_bytes": provenance.size_bytes, + "digest": {"algorithm": "sha256", "value": provenance.sha256}, + } + + +def _record_key_fields(configuration: EffectiveConfiguration) -> JsonObject: + """Render ordered source and canonical key-field declarations.""" + return { + "reference": [ + configuration.field(name).source_fields.reference + for name in configuration.record_key + ], + "query": [ + configuration.field(name).source_fields.query + for name in configuration.record_key + ], + "canonical": list(configuration.record_key), + } + + +def _ordered_configuration(document: JsonObject) -> JsonObject: + """Return the configuration in fixed-schema and dynamic-name order.""" + metadata = document["metadata"] + fields = document["fields"] + definitions = fields["definitions"] + return { + "schema_version": document["schema_version"], + "metadata": _ordered_selected( + metadata, + ("id", "revision", "title", "description"), + ), + "fields": { + "record_key": fields["record_key"], + "in_scope": fields["in_scope"], + "definitions": { + name: _ordered_field_definition(definitions[name]) + for name in sorted(definitions) + }, + }, + } + + +def _ordered_field_definition(document: JsonObject) -> JsonObject: + """Order one embedded field declaration without applying defaults.""" + ordered: JsonObject = {} + for name in ( + "source_fields", + "required", + "logical_type", + "null_policy", + "source_normalisation", + "collection_representation", + "value_mapping", + "typed_normalisation", + "comparator", + "review_priority", + ): + if name not in document: + continue + value = document[name] + if name in {"source_fields", "required", "source_normalisation"}: + ordered[name] = _ordered_sides(value) + elif name == "logical_type" and isinstance(value, dict): + logical = _ordered_selected( + value, + ("collection_type", "element_type", "element_comparator"), + ) + if "element_comparator" in logical: + logical["element_comparator"] = _ordered_named_parameters( + logical["element_comparator"] + ) + ordered[name] = logical + elif name == "null_policy": + null_policy = _ordered_selected( + value, + ("allowed", "nulls_equal", "missing_tokens"), + ) + if "missing_tokens" in null_policy: + null_policy["missing_tokens"] = _ordered_sides( + null_policy["missing_tokens"] + ) + ordered[name] = null_policy + elif name in {"collection_representation", "value_mapping"}: + ordered[name] = _ordered_sides(value) + elif name == "comparator": + ordered[name] = _ordered_named_parameters(value) + else: + ordered[name] = _ordered_nested(value) + return ordered + + +def _ordered_sides(document: object) -> object: + """Order a reference/query mapping and recursively order its content.""" + if not isinstance(document, dict): + return _ordered_nested(document) + return { + name: _ordered_nested(document[name]) + for name in ("reference", "query") + if name in document + } + + +def _ordered_named_parameters(document: JsonObject) -> JsonObject: + """Order an operation or comparator declaration.""" + return _ordered_selected(document, ("name", "operation", "parameters")) + + +def _ordered_selected( + document: Mapping[str, object], + names: tuple[str, ...], +) -> JsonObject: + """Copy selected fixed properties in documented schema order.""" + return { + name: _ordered_nested(document[name]) + for name in names + if name in document + } + + +def _ordered_nested(value: object) -> object: + """Order known fixed objects and otherwise use Unicode member order.""" + if isinstance(value, list): + return [_ordered_nested(item) for item in value] + if not isinstance(value, dict): + return value + keys = set(value) + if keys <= {"reference", "query"}: + return _ordered_sides(value) + if "operation" in value: + return _ordered_selected(value, ("operation", "parameters")) + if keys <= {"category", "value"} and "category" in value: + return _ordered_selected(value, ("category", "value")) + if keys <= {"from", "to"}: + return _ordered_selected(value, ("from", "to")) + if "format" in value: + return _ordered_selected( + value, + ("format", "delimiter", "element_normalisation", "element_mapping"), + ) + if keys <= {"name", "parameters"} and "name" in value: + return _ordered_named_parameters(value) + return {name: _ordered_nested(value[name]) for name in sorted(value)} + + +def _summary_document( + records: RecordsResult, + values: ValuesResult, + findings: list[JsonObject], +) -> JsonObject: + """Build fixed-shape, zero-inclusive reconciled summary counts.""" + code_counts = Counter(item["code"] for item in findings) + classification_counts = Counter(item["classification"] for item in findings) + priority_counts = Counter( + item.get("review_priority", "unassigned") for item in findings + ) + return { + "records": { + "reference": records.counts.reference, + "query": records.counts.query, + "matched": records.counts.matched, + "missing": records.counts.missing, + "extra": records.counts.extra, + }, + "values": { + "matched": values.counts.matched, + "mismatched": values.counts.mismatched, + "not_assessable": values.counts.not_assessable, + }, + "findings": { + "total": len(findings), + "by_classification": { + name: classification_counts[name] for name in CLASSIFICATIONS + }, + "by_code": {name: code_counts[name] for name in FINDING_CODES}, + "by_review_priority": { + name: priority_counts[name] for name in REVIEW_PRIORITIES + }, + }, + } + + +def _exclusion_document( + exclusion: ConfiguredFieldExclusion | UndeclaredFieldExclusion, +) -> JsonObject: + """Render one of the two closed exclusion forms.""" + if isinstance(exclusion, ConfiguredFieldExclusion): + return { + "code": exclusion.code, + "canonical_field": exclusion.canonical_field, + } + return { + "code": exclusion.code, + "side": exclusion.side.value, + "source_field": exclusion.source_field, + "record_count": exclusion.record_count, + } + + +def _assemble_findings( + configuration: EffectiveConfiguration, + contract: ContractResult, + records: RecordsResult, + values: ValuesResult, +) -> list[JsonObject]: + """Convert, coalesce, sort, and identify every finding candidate.""" + candidates = [ + *( + _record_finding_document(finding) + for finding in records.findings + ), + *( + _contract_finding_document(configuration, records, contract, finding) + for finding in contract.findings + ), + *( + _value_finding_document(configuration, finding) + for finding in values.findings + ), + ] + unique: dict[tuple[object, ...], JsonObject] = {} + for candidate in candidates: + key = _finding_sort_key(candidate) + unique.setdefault(key, candidate) + ordered = [unique[key] for key in sorted(unique)] + return [ + _identified_finding(index, candidate) + for index, candidate in enumerate(ordered, start=1) + ] + + +def _record_finding_document(finding: RecordFinding) -> JsonObject: + """Build structured evidence for one missing or extra record.""" + return { + "classification": finding.classification, + "code": finding.code, + "phase": finding.phase, + "affected_side": finding.affected_side.value, + "record_identity": _record_identity(finding.alignment), + "source_locations": { + "reference": _location_document(finding.reference_location), + "query": _location_document(finding.query_location), + }, + } + + +def _contract_finding_document( + configuration: EffectiveConfiguration, + records: RecordsResult, + contract: ContractResult, + finding: ContractFinding, +) -> JsonObject: + """Build code-specific evidence for one failed contract prerequisite.""" + base: JsonObject = { + "classification": finding.classification, + "code": finding.code, + "phase": finding.phase, + "affected_side": finding.side.value, + } + if finding.record is None: + base["field"] = finding.canonical_field + base["review_priority"] = finding.review_priority + return base + + alignment = records.alignment(finding.record.key) + field = configuration.field(finding.canonical_field) + assessments = { + side: _assessment_for_alignment( + contract, + alignment, + side, + finding.canonical_field, + ) + for side in (InputSide.REFERENCE, InputSide.QUERY) + } + base["record_identity"] = _record_identity(alignment) + base["source_locations"] = _field_locations(field, alignment) + base["field"] = finding.canonical_field + base["review_priority"] = finding.review_priority + base["values"] = _contract_values(finding, field, assessments) + return base + + +def _value_finding_document( + configuration: EffectiveConfiguration, + finding: ValueFinding, +) -> JsonObject: + """Build complete two-sided evidence for one post-translation difference.""" + opportunity = finding.opportunity + field = configuration.field(finding.canonical_field) + return { + "classification": finding.classification, + "code": finding.code, + "phase": finding.phase, + "affected_side": finding.affected_side, + "record_identity": _record_identity(opportunity.alignment), + "source_locations": { + "reference": _location_document(finding.reference_location), + "query": _location_document(finding.query_location), + }, + "field": finding.canonical_field, + "review_priority": finding.review_priority, + "values": { + "reference": _field_evidence( + opportunity.reference.translated, + include_source=True, + include_canonical=field.logical_type is not None, + ), + "query": _field_evidence( + opportunity.query.translated, + include_source=True, + include_canonical=field.logical_type is not None, + ), + }, + } + + +def _assessment_for_alignment( + contract: ContractResult, + alignment: RecordAlignment, + side: InputSide, + canonical_field: str, +) -> FieldAssessment | None: + """Return the field assessment when the aligned side contains a record.""" + record = alignment.reference if side is InputSide.REFERENCE else alignment.query + if record is None: + return None + return contract.assessment(side, record.key, canonical_field) + + +def _contract_values( + finding: ContractFinding, + field: FieldDefinition, + assessments: Mapping[InputSide, FieldAssessment | None], +) -> JsonObject: + """Apply the finding-code-specific source-value minimisation rules.""" + values: JsonObject = {} + for side in (InputSide.REFERENCE, InputSide.QUERY): + assessment = assessments[side] + if assessment is None: + values[side.value] = {"state": "absent"} + continue + translated = assessment.translated + if finding.code == "missing_required_field": + values[side.value] = { + "state": ( + "absent" + if translated.disposition is ValueDisposition.NOT_ASSESSABLE + and translated.source is None + else "present" + ) + } + continue + affected = side is finding.side + evidence = _field_evidence( + translated, + include_source=affected, + include_canonical=False, + ) + if ( + affected + and finding.code == "null_policy_violation" + and field.logical_type is not None + ): + evidence["canonical"] = _null_canonical_document(field) + values[side.value] = evidence + return values + + +def _field_evidence( + translated: TranslatedValue, + *, + include_source: bool, + include_canonical: bool, +) -> JsonObject: + """Render one side's presence and permitted source/canonical evidence.""" + if translated.disposition is ValueDisposition.ABSENT or translated.source is None: + return {"state": "absent"} + evidence: JsonObject = {"state": "present"} + if include_source: + evidence["source"] = _source_value_document(translated.source) + if include_canonical and translated.canonical is not None: + evidence["canonical"] = _canonical_value_document(translated.canonical) + return evidence + + +def _null_canonical_document(field: FieldDefinition) -> JsonObject: + """Render the declared typed-null representation for a null violation.""" + logical_type = field.logical_type + if logical_type is None: + raise AssertionError("untyped null violation has no canonical null") + if isinstance(logical_type, str): + return {"logical_type": logical_type, "value": None} + return { + "logical_type": logical_type.collection_type, + "element_type": logical_type.element_type, + "value": None, + } + + +def _source_value_document(value: NativeValue) -> JsonObject: + """Render a recursively typed parser-native source value.""" + document: JsonObject = {"category": value.category.value} + if value.category is NativeCategory.NULL: + return document + if value.category is NativeCategory.STRING: + document["value"] = value.value + elif value.category in {NativeCategory.INTEGER, NativeCategory.DECIMAL}: + document["value"] = ( + value.lexical if value.lexical is not None else str(value.value) + ) + elif value.category is NativeCategory.BOOLEAN: + document["value"] = value.value + elif value.category is NativeCategory.ARRAY: + if not isinstance(value.value, tuple): + raise AssertionError("parser-native array does not contain a tuple") + document["value"] = [_source_value_document(item) for item in value.value] + elif value.category is NativeCategory.OBJECT: + if not isinstance(value.value, tuple): + raise AssertionError("parser-native object does not contain members") + document["value"] = { + name: _source_value_document(member) + for name, member in sorted(value.value, key=lambda item: item[0]) + } + else: + raise AssertionError("unregistered parser-native category") + return document + + +def _canonical_value_document(value: CanonicalValue) -> JsonObject: + """Render one typed canonical value in its unique evidence form.""" + if value.element_type is not None: + rendered = None + if value.value is not None: + if not isinstance(value.value, tuple): + raise AssertionError("canonical collection does not contain a tuple") + rendered = [ + _compact_canonical_scalar(item, value.element_type) + for item in value.value + ] + return { + "logical_type": value.logical_type, + "element_type": value.element_type, + "value": rendered, + } + rendered_value: object = value.value + if value.value is not None: + rendered_value = _compact_canonical_scalar(value.value, value.logical_type) + return {"logical_type": value.logical_type, "value": rendered_value} + + +def _compact_canonical_scalar(value: object, logical_type: str) -> object: + """Render a canonical scalar using the schema's compact scalar form.""" + if logical_type == "boolean": + return value + return render_canonical_scalar(value, logical_type) + + +def _record_identity(alignment: RecordAlignment) -> JsonObject: + """Render source-side and canonical ordered record identities.""" + return { + "reference": ( + list(alignment.reference.source_identity) + if alignment.reference is not None + else None + ), + "query": ( + list(alignment.query.source_identity) + if alignment.query is not None + else None + ), + "canonical": list(alignment.canonical_identity), + } + + +def _field_locations( + field: FieldDefinition, + alignment: RecordAlignment, +) -> JsonObject: + """Render each side's present-field or containing-record location.""" + return { + "reference": _location_document( + alignment.reference.record.location_for(field.source_fields.reference) + if alignment.reference is not None + else None + ), + "query": _location_document( + alignment.query.record.location_for(field.source_fields.query) + if alignment.query is not None + else None + ), + } + + +def _location_document(location: SourceLocation | None) -> JsonObject | None: + """Render one tabular or JSON source location.""" + if location is None: + return None + if isinstance(location, TabularLocation): + document: JsonObject = { + "record_number": location.record_number, + "line_number": location.line_number, + } + if location.field_name is not None: + document["field"] = location.field_name + return document + if isinstance(location, JsonLocation): + return {"json_pointer": location.pointer} + raise AssertionError("unknown source-location type") + + +def _identified_finding(index: int, candidate: JsonObject) -> JsonObject: + """Assign a sequential ID and deterministic presentation message.""" + document: JsonObject = { + "id": f"F{index:06d}", + "classification": candidate["classification"], + "code": candidate["code"], + "phase": candidate["phase"], + "message": _finding_message(candidate), + "affected_side": candidate["affected_side"], + } + for name in ( + "record_identity", + "source_locations", + "field", + "review_priority", + "values", + ): + if name in candidate: + document[name] = candidate[name] + return document + + +def _finding_message(finding: Mapping[str, object]) -> str: + """Derive concise English presentation text from structured properties.""" + code = finding["code"] + side = str(finding["affected_side"]) + field = finding.get("field") + if code == "missing_record": + return "The query record is missing." + if code == "extra_record": + return "The query contains an extra record." + if code == "missing_required_field": + return f"Required {side} field {field!r} is absent." + if code == "type_mismatch": + return f"The {side} value for field {field!r} has an incompatible type." + if code == "invalid_field_value": + return f"The {side} value for field {field!r} is invalid." + if code == "null_policy_violation": + return f"The {side} value for field {field!r} violates its null policy." + if code == "mapping_failure": + return f"The {side} value for field {field!r} has no configured mapping." + if code == "value_mismatch": + return f"Reference and query values for field {field!r} differ." + raise AssertionError("unregistered finding code") + + +def _finding_sort_key(finding: Mapping[str, object]) -> tuple[object, ...]: + """Return the complete canonical total-order and coalescence key.""" + classification_order = {name: index for index, name in enumerate(CLASSIFICATIONS)} + phase_order = {"contract": 0, "records": 1, "values": 2} + code_order = {name: index for index, name in enumerate(FINDING_CODES)} + side_order = {"reference": 0, "query": 1, "both": 2} + identity = finding.get("record_identity") + canonical_identity: tuple[str, ...] = () + if isinstance(identity, dict): + canonical_identity = tuple(identity["canonical"]) + remaining = { + key: value + for key, value in finding.items() + if key not in {"id", "message"} + } + return ( + classification_order[str(finding["classification"])], + phase_order[str(finding["phase"])], + identity is not None, + canonical_identity, + str(finding.get("field", "")), + code_order[str(finding["code"])], + side_order[str(finding["affected_side"])], + _locations_sort_key(finding.get("source_locations")), + json.dumps( + remaining, + ensure_ascii=False, + sort_keys=True, + separators=(",", ":"), + ), + ) + + +def _locations_sort_key(locations: object) -> tuple[object, ...]: + """Return the specified two-sided source-location ordering key.""" + if not isinstance(locations, dict): + return ((0,), (0,)) + return ( + _location_sort_key(locations.get("reference")), + _location_sort_key(locations.get("query")), + ) + + +def _location_sort_key(location: object) -> tuple[object, ...]: + """Return the specified order for one serialized location.""" + if location is None: + return (0,) + if not isinstance(location, dict): + raise ResultValidationError("source location is not an object or null") + if "record_number" in location: + return ( + 1, + location["record_number"], + location["line_number"], + location.get("field", ""), + ) + return (1, location["json_pointer"]) + + +def _validate_semantics( + document: JsonObject, + prepared: PreparedComparison | None, + contract: ContractResult | None, + domain_records: RecordsResult | None, + domain_values: ValuesResult | None, +) -> None: + """Verify invariants that JSON Schema cannot express.""" + phases = document["phases"] + expected_status = ( + "complete" + if all(value == "complete" for value in phases.values()) + else "incomplete" + ) + _require( + document["run"]["status"] == expected_status, + "run status disagrees with phases", + ) + + records = document["summary"]["records"] + _require( + records["reference"] == records["matched"] + records["missing"], + "reference record counts do not reconcile", + ) + _require( + records["query"] == records["matched"] + records["extra"], + "query record counts do not reconcile", + ) + configuration_fields = document["comparison_configuration"]["document"][ + "fields" + ] + compared_field_count = len( + set(configuration_fields["in_scope"]) + - set(configuration_fields["record_key"]) + ) + value_summary = document["summary"]["values"] + classified_opportunities = sum(value_summary.values()) + expected_opportunities = records["matched"] * compared_field_count + _require( + classified_opportunities == expected_opportunities, + "value opportunity counts do not reconcile", + ) + expected_values_status = ( + "incomplete" if value_summary["not_assessable"] else "complete" + ) + _require( + phases["values"] == expected_values_status, + "values phase status disagrees with not-assessable opportunities", + ) + if domain_records is not None: + _require( + records + == { + "reference": domain_records.counts.reference, + "query": domain_records.counts.query, + "matched": domain_records.counts.matched, + "missing": domain_records.counts.missing, + "extra": domain_records.counts.extra, + }, + "record summary disagrees with the records phase", + ) + if domain_values is not None: + _require( + document["summary"]["values"] + == { + "matched": domain_values.counts.matched, + "mismatched": domain_values.counts.mismatched, + "not_assessable": domain_values.counts.not_assessable, + }, + "value summary disagrees with the values phase", + ) + if ( + contract is not None + and domain_records is not None + and domain_values is not None + ): + expected_phases = { + "contract": contract.status.value, + "records": domain_records.status.value, + "values": domain_values.status.value, + } + _require(phases == expected_phases, "phase status disagrees with phase results") + if prepared is None: + raise ResultValidationError( + "phase-result validation requires prepared comparison context" + ) + expected_findings = _assemble_findings( + prepared.configuration, + contract, + domain_records, + domain_values, + ) + _require( + document["findings"] == expected_findings, + "findings disagree with the comparison phase results", + ) + + findings = document["findings"] + summary = document["summary"]["findings"] + _require(summary["total"] == len(findings), "finding total does not reconcile") + expected_classifications = Counter(item["classification"] for item in findings) + expected_codes = Counter(item["code"] for item in findings) + expected_priorities = Counter( + item.get("review_priority", "unassigned") for item in findings + ) + _require( + summary["by_classification"] + == {name: expected_classifications[name] for name in CLASSIFICATIONS}, + "finding classification counts do not reconcile", + ) + _require( + summary["by_code"] == {name: expected_codes[name] for name in FINDING_CODES}, + "finding code counts do not reconcile", + ) + _require( + summary["by_review_priority"] + == {name: expected_priorities[name] for name in REVIEW_PRIORITIES}, + "finding review-priority counts do not reconcile", + ) + + expected_ids = [f"F{index:06d}" for index in range(1, len(findings) + 1)] + _require( + [item["id"] for item in findings] == expected_ids, + "finding IDs are not sequential", + ) + finding_keys = [_finding_sort_key(item) for item in findings] + _require( + finding_keys == sorted(finding_keys), + "findings are not canonically ordered", + ) + _require( + len(finding_keys) == len(set(finding_keys)), + "duplicate findings are present", + ) + + key_fields = document["record_key_fields"] + key_length = len(key_fields["canonical"]) + _require( + len(key_fields["reference"]) == key_length + and len(key_fields["query"]) == key_length, + "record-key field arrays have inconsistent lengths", + ) + for finding in findings: + identity = finding.get("record_identity") + if identity is None: + continue + _require( + len(identity["canonical"]) == key_length + and ( + identity["reference"] is None + or len(identity["reference"]) == key_length + ) + and (identity["query"] is None or len(identity["query"]) == key_length), + "record identity has an inconsistent component count", + ) + + exclusion_keys = [_exclusion_sort_key(item) for item in document["exclusions"]] + _require( + exclusion_keys == sorted(exclusion_keys), + "exclusions are not canonically ordered", + ) + _require( + len(exclusion_keys) == len(set(exclusion_keys)), + "duplicate exclusions are present", + ) + _validate_configuration_derivations(document, prepared) + + +def _validate_configuration_derivations( + document: JsonObject, + prepared: PreparedComparison | None, +) -> None: + """Cross-check configuration-derived evidence and available source context.""" + configuration = document["comparison_configuration"]["document"] + if prepared is not None: + _require( + configuration == prepared.configuration.document, + "embedded configuration disagrees with the effective configuration", + ) + _require( + document["inputs"] + == { + "reference": _input_document(prepared.reference), + "query": _input_document(prepared.query), + }, + "input provenance disagrees with the parsed inputs", + ) + fields = configuration["fields"] + definitions = fields["definitions"] + record_key = fields["record_key"] + expected_key_fields = { + "reference": [ + definitions[name]["source_fields"]["reference"] for name in record_key + ], + "query": [ + definitions[name]["source_fields"]["query"] for name in record_key + ], + "canonical": record_key, + } + _require( + document["record_key_fields"] == expected_key_fields, + "record-key metadata disagrees with the embedded configuration", + ) + for finding in document["findings"]: + field = finding.get("field") + if field is None: + continue + _require(field in definitions, "finding names an undeclared canonical field") + expected_priority = definitions[field].get("review_priority", "normal") + _require( + finding["review_priority"] == expected_priority, + "finding review priority disagrees with the embedded configuration", + ) + + expected_configured = [ + {"code": "configured_field_excluded", "canonical_field": name} + for name in sorted(set(definitions) - set(fields["in_scope"])) + ] + actual_configured = [ + item + for item in document["exclusions"] + if item["code"] == "configured_field_excluded" + ] + _require( + actual_configured == expected_configured, + "configured-field exclusions disagree with the embedded configuration", + ) + if prepared is None: + return + expected_undeclared = _expected_undeclared_exclusions( + prepared, + definitions, + ) + actual_undeclared = [ + item + for item in document["exclusions"] + if item["code"] == "undeclared_field_excluded" + ] + _require( + actual_undeclared == expected_undeclared, + "undeclared-field exclusions disagree with observed source fields", + ) + + +def _expected_undeclared_exclusions( + prepared: PreparedComparison, + definitions: Mapping[str, JsonObject], +) -> list[JsonObject]: + """Derive expected source-field exclusions from parsed datasets.""" + expected: list[JsonObject] = [] + for side, dataset in ( + ("reference", prepared.reference), + ("query", prepared.query), + ): + represented = { + definition["source_fields"][side] + for definition in definitions.values() + } + for source_field in sorted(set(dataset.field_names) - represented): + expected.append( + { + "code": "undeclared_field_excluded", + "side": side, + "source_field": source_field, + "record_count": sum( + record.has_field(source_field) for record in dataset.records + ), + } + ) + return expected + + +def _exclusion_sort_key(exclusion: Mapping[str, object]) -> tuple[object, ...]: + """Return the complete specified exclusion order and identity.""" + if exclusion["code"] == "configured_field_excluded": + return (0, exclusion["canonical_field"]) + side_order = {"reference": 0, "query": 1} + return ( + 1, + side_order[str(exclusion["side"])], + exclusion["source_field"], + exclusion["record_count"], + ) + + +def _require(condition: bool, message: str) -> None: + """Raise the internal validation exception for a failed invariant.""" + if not condition: + raise ResultValidationError(message) diff --git a/packages/adviso-compare/tests/test_results.py b/packages/adviso-compare/tests/test_results.py new file mode 100644 index 0000000..d25740e --- /dev/null +++ b/packages/adviso-compare/tests/test_results.py @@ -0,0 +1,718 @@ +"""Behavioural tests for canonical result assembly and validation.""" + +from __future__ import annotations + +import json +from copy import deepcopy +from dataclasses import replace +from datetime import datetime, timezone +from pathlib import Path +from uuid import UUID + +import pytest +import yaml + +from adviso_compare.alignment import align_records +from adviso_compare.application import PreparedComparison, prepare_comparison +from adviso_compare.contract import assess_contract +from adviso_compare.invocation import ComparisonInvocation +from adviso_compare.results import ( + ResultValidationError, + RuntimeIdentity, + assemble_result, + current_runtime_identity, + serialize_result, + validate_result, +) +from adviso_compare.values import compare_values +from tests.support.documents import JsonObject, minimal_configuration +from tests.support.schemas import results_validator + + +RUN_ID = UUID("123e4567-e89b-42d3-a456-426614174000") +COMPLETED_AT = datetime(2026, 11, 9, 12, 34, 56, tzinfo=timezone.utc) +RUNTIME = RuntimeIdentity( + python="cpython-3.12.7", + platform="linux-x86_64", + unicode_version="15.1.0", +) + + +def _write(path: Path, text: str) -> Path: + """Write one controlled UTF-8 document.""" + path.write_text(text, encoding="utf-8") + return path + + +def _prepare( + tmp_path: Path, + reference_text: str, + query_text: str, + configuration: JsonObject | None = None, +) -> PreparedComparison: + """Prepare one custom or automatic comparison.""" + tmp_path.mkdir(parents=True, exist_ok=True) + reference = _write(tmp_path / "reference.json", reference_text) + query = _write(tmp_path / "query.json", query_text) + configuration_path = None + if configuration is not None: + configuration_path = _write( + tmp_path / "configuration.yaml", + yaml.safe_dump(configuration, sort_keys=False), + ) + return prepare_comparison( + ComparisonInvocation( + reference=reference, + query=query, + output=tmp_path / "evidence", + configuration=configuration_path, + ) + ) + + +def _assemble(prepared: PreparedComparison) -> JsonObject: + """Run the three phases and assemble deterministic test evidence.""" + contract = assess_contract( + prepared.configuration, + reference=prepared.reference, + query=prepared.query, + reference_records=prepared.reference_records, + query_records=prepared.query_records, + ) + records = align_records(prepared.reference_records, prepared.query_records) + values = compare_values(prepared.configuration, contract, records) + return assemble_result( + prepared, + contract, + records, + values, + run_id=RUN_ID, + completed_at=COMPLETED_AT, + software_version="1.2.3", + runtime=RUNTIME, + ) + + +def test_complete_result_reconciles_findings_summaries_and_provenance( + tmp_path: Path, +) -> None: + """A complete mixed comparison becomes coherent canonical evidence.""" + prepared = _prepare( + tmp_path, + ( + '[{"reference_id":"S2","expected_value":2,"reference_note":"r"},' + '{"reference_id":"S1","expected_value":1}]' + ), + ( + '[{"query_id":"S3","observed_value":3,"query_note":"q"},' + '{"query_id":"S1","observed_value":2}]' + ), + minimal_configuration(), + ) + + result = _assemble(prepared) + + results_validator().validate(result) + assert result["run"] == { + "id": str(RUN_ID), + "status": "complete", + "completed_at": "2026-11-09T12:34:56Z", + } + assert result["phases"] == { + "contract": "complete", + "records": "complete", + "values": "complete", + } + assert result["summary"]["records"] == { + "reference": 2, + "query": 2, + "matched": 1, + "missing": 1, + "extra": 1, + } + assert result["summary"]["values"] == { + "matched": 0, + "mismatched": 1, + "not_assessable": 0, + } + assert [finding["code"] for finding in result["findings"]] == [ + "missing_record", + "extra_record", + "value_mismatch", + ] + assert [finding["id"] for finding in result["findings"]] == [ + "F000001", + "F000002", + "F000003", + ] + assert result["findings"][2]["values"] == { + "reference": { + "state": "present", + "source": {"category": "integer", "value": "1"}, + "canonical": {"logical_type": "decimal", "value": "1"}, + }, + "query": { + "state": "present", + "source": {"category": "integer", "value": "2"}, + "canonical": {"logical_type": "decimal", "value": "2"}, + }, + } + assert result["summary"]["findings"]["by_classification"] == { + "comparison_difference": 3, + "assessment_issue": 0, + } + assert result["summary"]["findings"]["by_review_priority"] == { + "low": 0, + "normal": 1, + "high": 0, + "unassigned": 2, + } + assert result["exclusions"] == [ + { + "code": "undeclared_field_excluded", + "side": "reference", + "source_field": "reference_note", + "record_count": 1, + }, + { + "code": "undeclared_field_excluded", + "side": "query", + "source_field": "query_note", + "record_count": 1, + }, + ] + assert result["inputs"]["reference"]["basename"] == "reference.json" + assert result["inputs"]["reference"]["digest"]["value"] == ( + prepared.reference.provenance.sha256 + ) + + +def test_incomplete_result_retains_causal_evidence_without_duplicate_finding( + tmp_path: Path, +) -> None: + """A failed prerequisite makes values incomplete and remains causal evidence.""" + prepared = _prepare( + tmp_path, + '[{"reference_id":"S1","expected_value":1}]', + '[{"query_id":"S1"}]', + minimal_configuration(), + ) + + result = _assemble(prepared) + + results_validator().validate(result) + assert result["run"]["status"] == "incomplete" + assert result["phases"]["values"] == "incomplete" + assert result["summary"]["values"] == { + "matched": 0, + "mismatched": 0, + "not_assessable": 1, + } + assert [finding["code"] for finding in result["findings"]] == [ + "missing_required_field" + ] + finding = result["findings"][0] + assert finding["values"] == { + "reference": {"state": "present"}, + "query": {"state": "absent"}, + } + assert finding["record_identity"] == { + "reference": ["S1"], + "query": ["S1"], + "canonical": ["S1"], + } + + +@pytest.mark.parametrize( + ("case", "query_value", "expected_code", "source_category", "has_canonical"), + [ + ("ordinary", "true", "type_mismatch", "boolean", False), + ("ordinary", '".5"', "invalid_field_value", "string", False), + ("ordinary", "null", "null_policy_violation", "null", True), + ("untyped_null", "null", "null_policy_violation", "null", False), + ("mapping", '"unknown"', "mapping_failure", "string", False), + ], +) +def test_contract_finding_registry_renders_code_specific_value_evidence( + tmp_path: Path, + case: str, + query_value: str, + expected_code: str, + source_category: str, + has_canonical: bool, +) -> None: + """Contract finding forms retain only their permitted causal evidence.""" + configuration = minimal_configuration() + reference_value = "1" + if case == "mapping": + reference_value = '"known"' + field = configuration["fields"]["definitions"]["measured_value"] + field["logical_type"] = "string" + field["comparator"] = {"name": "typed_exact"} + field["value_mapping"] = { + "query": [ + { + "from": {"category": "string", "value": "known"}, + "to": {"category": "string", "value": "known"}, + } + ] + } + elif case == "untyped_null": + field = configuration["fields"]["definitions"]["measured_value"] + field.pop("logical_type") + field["comparator"] = {"name": "preserved_exact"} + prepared = _prepare( + tmp_path, + f'[{{"reference_id":"S1","expected_value":{reference_value}}}]', + f'[{{"query_id":"S1","observed_value":{query_value}}}]', + configuration, + ) + + result = _assemble(prepared) + + assert [finding["code"] for finding in result["findings"]] == [expected_code] + evidence = result["findings"][0]["values"]["query"] + assert evidence["state"] == "present" + assert evidence["source"]["category"] == source_category + assert ("canonical" in evidence) is has_canonical + if has_canonical: + assert evidence["canonical"] == { + "logical_type": "decimal", + "value": None, + } + + +def test_dataset_level_missing_field_uses_the_distinct_compact_finding_form( + tmp_path: Path, +) -> None: + """An empty table can report a missing required column without a record.""" + reference = _write(tmp_path / "reference.csv", "reference_id\n") + query = _write(tmp_path / "query.tsv", "query_id\tobserved_value\n") + configuration = _write( + tmp_path / "configuration.yaml", + yaml.safe_dump(minimal_configuration(), sort_keys=False), + ) + prepared = prepare_comparison( + ComparisonInvocation( + reference=reference, + query=query, + output=tmp_path / "evidence", + configuration=configuration, + ) + ) + + result = _assemble(prepared) + + assert result["run"]["status"] == "complete" + assert result["summary"]["values"] == { + "matched": 0, + "mismatched": 0, + "not_assessable": 0, + } + assert result["findings"] == [ + { + "id": "F000001", + "classification": "assessment_issue", + "code": "missing_required_field", + "phase": "contract", + "message": "Required reference field 'measured_value' is absent.", + "affected_side": "reference", + "field": "measured_value", + "review_priority": "normal", + } + ] + + +@pytest.mark.parametrize( + "mutation", + [ + lambda result: result["run"].update(status="incomplete"), + lambda result: result["summary"]["records"].update(reference=99), + lambda result: result["summary"]["findings"].update(total=99), + lambda result: result["findings"][0].update(id="F000009"), + lambda result: result["record_key_fields"].update(canonical=["wrong"]), + ], +) +def test_semantic_validation_rejects_internally_incoherent_results( + tmp_path: Path, + mutation: object, +) -> None: + """Schema-valid-looking contradictions cannot pass self-validation.""" + prepared = _prepare( + tmp_path, + '[{"reference_id":"S1","expected_value":1}]', + '[{"query_id":"S1","observed_value":2}]', + minimal_configuration(), + ) + result = _assemble(prepared) + invalid = deepcopy(result) + mutation(invalid) # type: ignore[operator] + + with pytest.raises(ResultValidationError): + validate_result(invalid, prepared=prepared) + + +@pytest.mark.parametrize( + "mutation", + [ + lambda result: result["summary"]["values"].update(matched=2), + lambda result: ( + result["run"].update(status="incomplete"), + result["phases"].update(values="incomplete"), + ), + ], +) +def test_semantic_validation_reconciles_opportunities_and_values_status( + tmp_path: Path, + mutation: object, +) -> None: + """Value totals and completion status agree without phase-domain context.""" + prepared = _prepare( + tmp_path, + '[{"reference_id":"S1","expected_value":1}]', + '[{"query_id":"S1","observed_value":1}]', + minimal_configuration(), + ) + invalid = deepcopy(_assemble(prepared)) + mutation(invalid) # type: ignore[operator] + + with pytest.raises(ResultValidationError): + validate_result(invalid, prepared=prepared) + + +def test_finding_total_order_coalesces_duplicates_before_assigning_ids( + tmp_path: Path, +) -> None: + """Classification, scope, identity, and coalescence determine local IDs.""" + reference = _write(tmp_path / "reference.csv", "reference_id\n") + query = _write( + tmp_path / "query.json", + ( + '[{"query_id":"S2","observed_value":"invalid"},' + '{"query_id":"S1","observed_value":"invalid"}]' + ), + ) + configuration = _write( + tmp_path / "configuration.yaml", + yaml.safe_dump(minimal_configuration(), sort_keys=False), + ) + prepared = prepare_comparison( + ComparisonInvocation( + reference=reference, + query=query, + output=tmp_path / "evidence", + configuration=configuration, + ) + ) + contract = assess_contract( + prepared.configuration, + reference=prepared.reference, + query=prepared.query, + reference_records=prepared.reference_records, + query_records=prepared.query_records, + ) + contract_with_duplicate = replace( + contract, + findings=contract.findings + (contract.findings[-1],), + ) + records = align_records(prepared.reference_records, prepared.query_records) + values = compare_values(prepared.configuration, contract, records) + + result = assemble_result( + prepared, + contract_with_duplicate, + records, + values, + run_id=RUN_ID, + completed_at=COMPLETED_AT, + software_version="1.2.3", + runtime=RUNTIME, + ) + + assert [ + ( + finding["id"], + finding["classification"], + finding["code"], + finding.get("record_identity", {}).get("canonical"), + ) + for finding in result["findings"] + ] == [ + ("F000001", "comparison_difference", "extra_record", ["S1"]), + ("F000002", "comparison_difference", "extra_record", ["S2"]), + ("F000003", "assessment_issue", "missing_required_field", None), + ("F000004", "assessment_issue", "invalid_field_value", ["S1"]), + ("F000005", "assessment_issue", "invalid_field_value", ["S2"]), + ] + assert result["summary"]["findings"]["total"] == 5 + + +@pytest.mark.parametrize( + ( + "logical_type", + "comparator", + "reference_value", + "query_value", + "expected_query_canonical", + ), + [ + ( + "integer", + "typed_exact", + '"01"', + '"2"', + {"logical_type": "integer", "value": "2"}, + ), + ( + "string", + "typed_exact", + '"A"', + '"B"', + {"logical_type": "string", "value": "B"}, + ), + ( + "boolean", + "typed_exact", + "true", + "false", + {"logical_type": "boolean", "value": False}, + ), + ( + "date", + "date_exact", + '"2026-01-01"', + '"2026-01-02"', + {"logical_type": "date", "value": "2026-01-02"}, + ), + ( + "datetime", + "datetime_exact", + '"2026-01-01T00:00:00Z"', + '"2026-01-01T01:00:00Z"', + {"logical_type": "datetime", "value": "2026-01-01T01:00:00Z"}, + ), + ( + { + "collection_type": "list", + "element_type": "decimal", + "element_comparator": {"name": "typed_exact"}, + }, + "list_exact", + "[1,2]", + "[1,3]", + { + "logical_type": "list", + "element_type": "decimal", + "value": ["1", "3"], + }, + ), + ], +) +def test_typed_value_evidence_uses_each_canonical_rendering_form( + tmp_path: Path, + logical_type: object, + comparator: str, + reference_value: str, + query_value: str, + expected_query_canonical: JsonObject, +) -> None: + """Scalar and collection evidence retains its unique canonical form.""" + configuration = minimal_configuration() + field = configuration["fields"]["definitions"]["measured_value"] + field["logical_type"] = logical_type + field["comparator"] = {"name": comparator} + if isinstance(logical_type, dict): + field["collection_representation"] = { + "reference": {"format": "json_array"}, + "query": {"format": "json_array"}, + } + prepared = _prepare( + tmp_path, + f'[{{"reference_id":"S1","expected_value":{reference_value}}}]', + f'[{{"query_id":"S1","observed_value":{query_value}}}]', + configuration, + ) + + result = _assemble(prepared) + + assert result["findings"][0]["values"]["query"]["canonical"] == ( + expected_query_canonical + ) + + +def test_untyped_recursive_source_evidence_is_lossless_and_ordered( + tmp_path: Path, +) -> None: + """Preserved arrays and objects retain recursively typed source evidence.""" + prepared = _prepare( + tmp_path, + ( + '[{"record_id":"S1","payload":' + '{"z":[1,null],"a":true}}]' + ), + ( + '[{"record_id":"S1","payload":' + '{"a":false,"z":[1,null]}}]' + ), + ) + + result = _assemble(prepared) + + query_evidence = result["findings"][0]["values"]["query"] + assert "canonical" not in query_evidence + assert query_evidence["source"] == { + "category": "object", + "value": { + "a": {"category": "boolean", "value": False}, + "z": { + "category": "array", + "value": [ + {"category": "integer", "value": "1"}, + {"category": "null"}, + ], + }, + }, + } + + +def test_default_run_and_runtime_metadata_use_live_normalized_values( + tmp_path: Path, +) -> None: + """Ordinary assembly creates a UUID, UTC timestamp, and runtime identity.""" + prepared = _prepare( + tmp_path, + '[{"reference_id":"S1","expected_value":1}]', + '[{"query_id":"S1","observed_value":1}]', + minimal_configuration(), + ) + contract = assess_contract( + prepared.configuration, + reference=prepared.reference, + query=prepared.query, + reference_records=prepared.reference_records, + query_records=prepared.query_records, + ) + records = align_records(prepared.reference_records, prepared.query_records) + values = compare_values(prepared.configuration, contract, records) + + result = assemble_result(prepared, contract, records, values) + + assert UUID(result["run"]["id"]).version == 4 + assert result["run"]["completed_at"].endswith("Z") + runtime = current_runtime_identity() + assert result["runtime"] == { + "python": runtime.python, + "platform": runtime.platform, + "unicode_version": runtime.unicode_version, + } + + +def test_canonical_serialization_is_utf8_ordered_and_newline_terminated( + tmp_path: Path, +) -> None: + """Canonical JSON has stable layout and emits non-ASCII text directly.""" + source_configuration = minimal_configuration() + source_configuration["metadata"]["title"] = "Café comparison" + source_fields = source_configuration["fields"] + source_definitions = source_fields["definitions"] + source_fields["definitions"] = { + "record_id": source_definitions["record_id"], + "measured_value": source_definitions["measured_value"], + } + configuration = { + "fields": source_fields, + "metadata": { + "revision": source_configuration["metadata"]["revision"], + "title": source_configuration["metadata"]["title"], + "id": source_configuration["metadata"]["id"], + }, + "schema_version": source_configuration["schema_version"], + } + prepared = _prepare( + tmp_path, + '[{"reference_id":"S1","expected_value":1}]', + '[{"query_id":"S1","observed_value":1}]', + configuration, + ) + result = _assemble(prepared) + + rendered = serialize_result(result) + + embedded = result["comparison_configuration"]["document"] + assert list(embedded) == ["schema_version", "metadata", "fields"] + assert list(embedded["metadata"]) == ["id", "revision", "title"] + assert list(embedded["fields"]["definitions"]) == [ + "measured_value", + "record_id", + ] + assert rendered.startswith(b'{\n "results_schema_version": "1.0",\n') + assert b'Caf\xc3\xa9 comparison' in rendered + assert b"\\u00e9" not in rendered + assert b"\r" not in rendered + assert rendered.endswith(b"\n") and not rendered.endswith(b"\n\n") + assert json.loads(rendered) == result + + +def test_repeated_and_record_reordered_runs_preserve_the_comparison_outcome( + tmp_path: Path, +) -> None: + """Only permitted byte provenance and location evidence vary with order.""" + first = _prepare( + tmp_path / "first", + ( + '[{"record_id":"S3","value":3},{"record_id":"S1","value":1},' + '{"record_id":"S2","value":2}]' + ), + ( + '[{"record_id":"S4","value":4},{"record_id":"S2","value":2},' + '{"record_id":"S1","value":9}]' + ), + ) + reordered = _prepare( + tmp_path / "reordered", + ( + '[{"record_id":"S2","value":2},{"record_id":"S1","value":1},' + '{"record_id":"S3","value":3}]' + ), + ( + '[{"record_id":"S1","value":9},{"record_id":"S2","value":2},' + '{"record_id":"S4","value":4}]' + ), + ) + + first_result = _assemble(first) + repeated_result = _assemble(first) + reordered_result = _assemble(reordered) + + assert serialize_result(first_result) == serialize_result(repeated_result) + assert ( + first_result["comparison_configuration"] + == reordered_result["comparison_configuration"] + ) + assert _comparison_outcome(first_result) == _comparison_outcome( + reordered_result + ) + assert first_result["inputs"] != reordered_result["inputs"] + assert [ + finding["source_locations"] for finding in first_result["findings"] + ] != [ + finding["source_locations"] for finding in reordered_result["findings"] + ] + + +def _comparison_outcome(result: JsonObject) -> JsonObject: + """Extract the specification's location-independent comparison outcome.""" + return { + "run_status": result["run"]["status"], + "phases": result["phases"], + "summary": result["summary"], + "exclusions": result["exclusions"], + "findings": [ + { + name: value + for name, value in finding.items() + if name not in {"id", "message", "source_locations"} + } + for finding in result["findings"] + ], + } From 2405fd8e36cf7d7c5fd17cd9f6ffa9314f0d00e5 Mon Sep 17 00:00:00 2001 From: Kevin Howe Date: Mon, 9 Nov 2026 18:40:18 +0000 Subject: [PATCH 29/45] Implemented reporting --- packages/adviso-compare/README.md | 6 +- .../src/adviso_compare/reporting.py | 656 ++++++++++++++++++ .../adviso-compare/tests/test_reporting.py | 468 +++++++++++++ 3 files changed, 1128 insertions(+), 2 deletions(-) create mode 100644 packages/adviso-compare/src/adviso_compare/reporting.py create mode 100644 packages/adviso-compare/tests/test_reporting.py diff --git a/packages/adviso-compare/README.md b/packages/adviso-compare/README.md index 11c8d62..0142fca 100644 --- a/packages/adviso-compare/README.md +++ b/packages/adviso-compare/README.md @@ -17,8 +17,10 @@ without blocking unaffected content. The records phase deterministically aligns canonical identities and retains every matched, missing, and extra record. Value comparison now applies every version 1 comparator and accounts for matched, mismatched, and not-assessable opportunities. The comparison core now assembles -and self-validates deterministic canonical JSON results. Evidence-package -publication and the HTML report are not yet available. +and self-validates deterministic canonical JSON results. It can also render the +self-contained HTML view and atomically publish the validated, two-file evidence +package with strict overwrite protection. End-to-end command-line integration +and execution-error handling remain to be implemented. The current normative artefacts are: diff --git a/packages/adviso-compare/src/adviso_compare/reporting.py b/packages/adviso-compare/src/adviso_compare/reporting.py new file mode 100644 index 0000000..e32f44e --- /dev/null +++ b/packages/adviso-compare/src/adviso_compare/reporting.py @@ -0,0 +1,656 @@ +"""Render and atomically publish the version 1 evidence package.""" + +from __future__ import annotations + +import ctypes +import errno +import html +import json +import os +import shutil +import stat +import sys +import tempfile +from dataclasses import dataclass +from pathlib import Path +from typing import TYPE_CHECKING, Any, Mapping +from uuid import uuid4 + +from adviso_compare.results import JsonObject, serialize_result, validate_result + +if TYPE_CHECKING: + from adviso_compare.alignment import RecordsResult + from adviso_compare.application import PreparedComparison + from adviso_compare.contract import ContractResult + from adviso_compare.values import ValuesResult + + +RESULTS_FILENAME = "comparison-results.json" +REPORT_FILENAME = "comparison-report.html" +PACKAGE_MEMBERS = frozenset({RESULTS_FILENAME, REPORT_FILENAME}) +AUTOMATIC_CONFIGURATION_ID = "org.adviso.automatic-strict-exact" +_AT_FDCWD = -100 +_RENAME_NOREPLACE = 1 +_RENAME_EXCHANGE = 2 +_UNSUPPORTED_RENAME_ERRORS = { + getattr(errno, name) + for name in ("EINVAL", "ENOSYS", "ENOTSUP", "EOPNOTSUPP") + if hasattr(errno, name) +} + + +@dataclass(frozen=True, slots=True) +class OutputError(Exception): + """An expected failure while selecting or publishing output.""" + + code: str + message: str + + @property + def category(self) -> str: + """Return the stable execution-error category.""" + return "output" + + @property + def diagnostic(self) -> str: + """Render the stable single-line non-sensitive diagnostic.""" + return f"ERROR [output.{self.code}]: {self.message}" + + def __str__(self) -> str: + """Return the concise actionable message.""" + return self.message + + +def render_html_report(document: Mapping[str, Any]) -> bytes: + """Render a self-contained HTML view using only canonical-result facts.""" + run = document["run"] + phases = document["phases"] + inputs = document["inputs"] + configuration = document["comparison_configuration"]["document"] + metadata = configuration["metadata"] + summary = document["summary"] + finding_summary = summary["findings"] + exclusions = document["exclusions"] + findings = document["findings"] + + status = str(run["status"]) + difference_count = finding_summary["by_classification"][ + "comparison_difference" + ] + issue_count = finding_summary["by_classification"]["assessment_issue"] + sections = [ + "", + '', + "", + '', + '', + "AdvISO Compare evidence report", + "", + "", + "", + '", + '
', + '

Run summary

', + '
', + _metric("Run completeness", status.title(), status), + _metric("Comparison differences", str(difference_count), "difference"), + _metric("Assessment issues", str(issue_count), "issue"), + "
", + "
", + _phase_section(phases), + _identity_section(inputs, metadata), + ] + if metadata["id"] == AUTOMATIC_CONFIGURATION_ID: + sections.append(_automatic_mode_notice()) + sections.extend( + ( + _counts_section(summary), + _findings_section(findings), + _limitations_section(exclusions, summary), + _provenance_section(document), + _review_section(), + "
", + "", + "", + "", + ) + ) + return "\n".join(sections).encode("utf-8") + + +def publish_evidence_package( + destination: Path, + document: JsonObject, + *, + prepared: PreparedComparison, + contract: ContractResult, + records: RecordsResult, + values: ValuesResult, + overwrite: bool = False, +) -> Path: + """Validate, stage, and publish one complete two-file evidence package.""" + validate_result( + document, + prepared=prepared, + contract=contract, + records=records, + values=values, + ) + json_content = serialize_result(document) + published_document = json.loads(json_content.decode("utf-8")) + html_content = render_html_report(published_document) + destination = Path(destination) + existing = _inspect_destination(destination, overwrite=overwrite) + parent = destination.parent + if not parent.is_dir(): + raise OutputError( + code="write_failed", + message="the output parent directory does not exist or is not a directory", + ) + + staging_root: Path | None = None + try: + staging_root = Path( + tempfile.mkdtemp(prefix=f".{destination.name}.staging-", dir=parent) + ) + os.chmod(staging_root, 0o700) + stage = staging_root / "package" + stage.mkdir(mode=0o700) + _write_atomic_member(stage / RESULTS_FILENAME, json_content) + _write_atomic_member(stage / REPORT_FILENAME, html_content) + _fsync_directory(stage) + + if existing: + _inspect_destination(destination, overwrite=True) + _replace_existing_package(stage, destination) + else: + _publish_new_package(stage, destination) + _fsync_directory(parent) + except OutputError: + raise + except OSError as error: + raise OutputError( + code="write_failed", + message="the evidence package could not be written or published", + ) from error + finally: + if staging_root is not None: + _remove_temporary_directory(staging_root) + return destination + + +def _publish_new_package(stage: Path, destination: Path) -> None: + """Publish to an absent path without replacement where supported.""" + try: + supported = _renameat2(stage, destination, _RENAME_NOREPLACE) + except OSError as error: + if error.errno not in {errno.EEXIST, errno.ENOTEMPTY}: + raise + raise OutputError( + code="invalid_destination", + message="the selected output destination already exists", + ) from error + if supported: + return + _require_absent(destination) + os.rename(stage, destination) + + +def _replace_existing_package( + stage: Path, + destination: Path, +) -> None: + """Replace only through an atomic exchange that preserves the original.""" + if not _exchange_directories(stage, destination): + raise OutputError( + code="write_failed", + message=( + "atomic overwrite is not supported at this destination; " + "select a new output destination" + ), + ) + _remove_temporary_directory(stage) + + +def _exchange_directories(left: Path, right: Path) -> bool: + """Atomically exchange two directories when the host filesystem supports it.""" + return _renameat2(left, right, _RENAME_EXCHANGE) + + +def _renameat2(source: Path, destination: Path, flags: int) -> bool: + """Invoke Linux renameat2, returning false when the operation is unavailable.""" + if sys.platform != "linux": + return False + try: + renameat2 = ctypes.CDLL(None, use_errno=True).renameat2 + except AttributeError: + return False + renameat2.argtypes = ( + ctypes.c_int, + ctypes.c_char_p, + ctypes.c_int, + ctypes.c_char_p, + ctypes.c_uint, + ) + renameat2.restype = ctypes.c_int + result = renameat2( + _AT_FDCWD, + os.fsencode(source), + _AT_FDCWD, + os.fsencode(destination), + flags, + ) + if result == 0: + return True + error_number = ctypes.get_errno() + if error_number in _UNSUPPORTED_RENAME_ERRORS: + return False + raise OSError(error_number, os.strerror(error_number)) + + +def _inspect_destination(destination: Path, *, overwrite: bool) -> bool: + """Return whether a replaceable complete package occupies the destination.""" + try: + destination_stat = destination.lstat() + except FileNotFoundError: + return False + except OSError as error: + raise OutputError( + code="invalid_destination", + message="the selected output destination could not be inspected", + ) from error + if not overwrite: + raise OutputError( + code="invalid_destination", + message="the selected output destination already exists", + ) + if not stat.S_ISDIR(destination_stat.st_mode): + raise OutputError( + code="invalid_destination", + message=( + "overwrite requires an existing complete evidence-package " + "directory" + ), + ) + try: + entries = {entry.name: entry for entry in os.scandir(destination)} + if set(entries) != PACKAGE_MEMBERS: + raise OutputError( + code="invalid_destination", + message=( + "overwrite requires a directory containing exactly the two " + "version 1 evidence-package files" + ), + ) + if any(not entry.is_file(follow_symlinks=False) for entry in entries.values()): + raise OutputError( + code="invalid_destination", + message="evidence-package members must be regular files", + ) + result_bytes = (destination / RESULTS_FILENAME).read_bytes() + report_text = (destination / REPORT_FILENAME).read_text(encoding="utf-8") + existing_result = json.loads(result_bytes.decode("utf-8")) + validate_result(existing_result) + if not report_text.casefold().startswith(""): + raise ValueError("report is not recognizable HTML") + except OutputError: + raise + except (OSError, UnicodeError, json.JSONDecodeError, ValueError) as error: + raise OutputError( + code="invalid_destination", + message=( + "the existing directory is not a complete version 1 evidence " + "package" + ), + ) from error + return True + + +def _require_absent(destination: Path) -> None: + """Refuse a destination that appeared while the package was staged.""" + try: + destination.lstat() + except FileNotFoundError: + return + except OSError as error: + raise OutputError( + code="invalid_destination", + message="the selected output destination could not be inspected", + ) from error + raise OutputError( + code="invalid_destination", + message="the selected output destination already exists", + ) + + +def _write_atomic_member(path: Path, content: bytes) -> None: + """Write one private member and atomically give it its final staged name.""" + temporary = path.with_name(f".{path.name}.temporary-{uuid4().hex}") + descriptor: int | None = None + try: + descriptor = os.open( + temporary, + os.O_WRONLY | os.O_CREAT | os.O_EXCL, + 0o600, + ) + with os.fdopen(descriptor, "wb") as stream: + descriptor = None + stream.write(content) + stream.flush() + os.fsync(stream.fileno()) + os.chmod(temporary, 0o600) + os.replace(temporary, path) + finally: + if descriptor is not None: + os.close(descriptor) + try: + temporary.unlink() + except FileNotFoundError: + pass + + +def _remove_temporary_directory(path: Path) -> None: + """Best-effort removal for private staging or backup directories.""" + for _ in range(2): + try: + shutil.rmtree(path) + return + except FileNotFoundError: + return + except OSError: + continue + + +def _fsync_directory(path: Path) -> None: + """Request durable directory metadata where the platform supports it.""" + flags = os.O_RDONLY + if hasattr(os, "O_DIRECTORY"): + flags |= os.O_DIRECTORY + try: + descriptor = os.open(path, flags) + except OSError: + return + try: + os.fsync(descriptor) + except OSError: + pass + finally: + os.close(descriptor) + + +def _metric(label: str, value: str, kind: str) -> str: + """Render one labelled headline measure.""" + return ( + f'
{_escape(label)}' + f"{_escape(value)}
" + ) + + +def _phase_section(phases: Mapping[str, Any]) -> str: + """Render the ordered comparison-phase traceability table.""" + rows = "".join( + f"{_escape(name.title())}" + f"{_escape(str(phases[name]).title())}" + for name in ("contract", "records", "values") + ) + return ( + '

Comparison phases

' + "

The phases were performed in the order shown.

" + f"" + f"{rows}
PhaseCompletion status
" + ) + + +def _identity_section(inputs: Mapping[str, Any], metadata: Mapping[str, Any]) -> str: + """Render input and effective-configuration identities.""" + return ( + '

Compared inputs ' + "and configuration

" + f"
Reference
{_escape(inputs['reference']['basename'])}
" + f"
Query
{_escape(inputs['query']['basename'])}
" + f"
Configuration identifier
{_escape(metadata['id'])}
" + f"
Configuration revision
{_escape(metadata['revision'])}
" + "
" + ) + + +def _automatic_mode_notice() -> str: + """Render the required limitations warning for automatic mode.""" + return ( + '" + ) + + +def _counts_section(summary: Mapping[str, Any]) -> str: + """Render record, value, and finding metrics without deriving a verdict.""" + records = summary["records"] + values = summary["values"] + findings = summary["findings"] + return ( + '

Measured counts

' + "

Records

" + + _definition_list(records) + + "

Values

" + + _definition_list(values) + + "

Findings

Total findings: " + + f"{findings['total']}

" + + "

Findings by classification

" + + _definition_list(findings["by_classification"]) + + "

Findings by code

" + + _definition_list(findings["by_code"]) + + "

Findings by review priority

" + + _definition_list(findings["by_review_priority"]) + + "
" + ) + + +def _findings_section(findings: list[Mapping[str, Any]]) -> str: + """Render findings classification-first with phase as traceability.""" + groups = [] + labels = ( + ("comparison_difference", "Comparison differences"), + ("assessment_issue", "Assessment issues"), + ) + for classification, label in labels: + selected = [ + finding + for finding in findings + if finding["classification"] == classification + ] + items = "".join(_finding_card(finding) for finding in selected) + if not items: + items = "

None reported.

" + groups.append(f"

{label}

{items}") + return ( + '

Finding details

' + + "".join(groups) + + "
" + ) + + +def _finding_card(finding: Mapping[str, Any]) -> str: + """Render one finding without adding facts outside its JSON object.""" + priority = finding.get("review_priority", "unassigned") + structured = { + name: value + for name, value in finding.items() + if name not in {"id", "classification", "message", "phase", "review_priority"} + } + evidence = _escape(json.dumps(structured, ensure_ascii=False, indent=2)) + return ( + '
' + f"

{_escape(finding['id'])}: {_escape(finding['message'])}

" + f'

Phase: {_escape(finding["phase"])} ' + f'Priority: ' + f"{_escape(priority)}

" + "
Structured evidence" + f"
{evidence}
" + "
" + ) + + +def _limitations_section( + exclusions: list[Mapping[str, Any]], + summary: Mapping[str, Any], +) -> str: + """Render excluded and not-assessable content explicitly.""" + exclusion_items = "".join( + f"
  • {_escape(item['code'])}: " + f"{_escape(_exclusion_description(item))}
  • " + for item in exclusions + ) + if not exclusion_items: + exclusion_items = "
  • No field exclusions were reported.
  • " + not_assessable = summary["values"]["not_assessable"] + return ( + '

    Limitations, ' + "exclusions, and unassessed content

    " + f"

    Not-assessable value opportunities: {not_assessable}. " + "Related assessment-issue findings explain failed prerequisites.

    " + f"
      {exclusion_items}
    " + ) + + +def _exclusion_description(item: Mapping[str, Any]) -> str: + """Describe one fixed exclusion form from its structured properties.""" + if item["code"] == "configured_field_excluded": + return f"configured canonical field {item['canonical_field']} was out of scope" + return ( + f"{item['side']} source field {item['source_field']} was undeclared " + f"({item['record_count']} records)" + ) + + +def _provenance_section(document: Mapping[str, Any]) -> str: + """Render sufficient provenance and embedded reproduction material.""" + run = document["run"] + software = document["software"] + runtime = document["runtime"] + inputs = document["inputs"] + configuration = document["comparison_configuration"]["document"] + input_rows = "".join( + "" + f"{_escape(side.title())}" + f"{_escape(value['basename'])}" + f"{_escape(value['format'])}" + f"{_escape(value['digest']['value'])}" + "" + for side, value in ( + ("reference", inputs["reference"]), + ("query", inputs["query"]), + ) + ) + config_json = _escape(json.dumps(configuration, ensure_ascii=False, indent=2)) + return ( + '

    Provenance and ' + "reproduction

    " + f"
    Run identifier
    {_escape(run['id'])}
    " + f"
    Completed at
    {_escape(run['completed_at'])}
    " + f"
    Software
    {_escape(software['name'])} " + f"{_escape(software['version'])}
    " + f"
    Python
    {_escape(runtime['python'])}
    " + f"
    Platform
    {_escape(runtime['platform'])}
    " + f"
    Unicode data
    {_escape(runtime['unicode_version'])}
    " + f"
    Results schema
    {_escape(document['results_schema_version'])}
    " + "

    Input byte provenance

    " + "" + f"{input_rows}" + "
    SideBasenameFormatSHA-256
    Effective comparison configuration" + f"
    {config_json}
    " + ) + + +def _review_section() -> str: + """Provide generic space for governance outside the comparison tool.""" + return ( + '

    External review and ' + "approval

    Reviewer: ____________________

    " + "

    Date: ____________________

    Review notes:

    " + '
    ' + "
    " + ) + + +def _definition_list(values: Mapping[str, Any]) -> str: + """Render a compact labelled count list.""" + entries = "".join( + f"
    {_escape(name.replace('_', ' ').title())}
    {_escape(value)}
    " + for name, value in values.items() + ) + return f'
    {entries}
    ' + + +def _escape(value: object) -> str: + """Escape one canonical value for HTML text or attribute use.""" + return html.escape(str(value), quote=True) + + +_STYLES = """ +:root { color-scheme: light; font-family: system-ui, sans-serif; line-height: 1.5; } +body { color: #17202a; background: #fff; margin: 0; } +.page-header, main { max-width: 72rem; margin: 0 auto; padding: 1.25rem; } +.page-header { border-bottom: .35rem solid #244b6b; } +.warning, .notice { border: .18rem solid #8a4b08; padding: 1rem; background: #fff4df; } +section, aside { margin-block: 2rem; } +.headline-grid { + display: grid; + grid-template-columns: repeat(auto-fit, minmax(13rem, 1fr)); + gap: 1rem; +} +.metric { border: .15rem solid #59636e; padding: 1rem; } +.metric span, .metric strong { display: block; } +.metric strong { font-size: 1.6rem; } +table { border-collapse: collapse; width: 100%; } +th, td { + border: 1px solid #7b8792; + padding: .5rem; + text-align: left; + vertical-align: top; +} +dl { display: grid; grid-template-columns: minmax(12rem, 1fr) 3fr; gap: .3rem 1rem; } +dt { font-weight: 700; } +.counts { grid-template-columns: repeat(2, minmax(8rem, max-content)); } +.finding { + border-left: .35rem solid #59636e; + padding: .25rem 1rem 1rem; + margin-block: 1rem; + background: #f5f7f8; +} +.label { + display: inline-block; + border: 1px solid #59636e; + border-radius: .25rem; + padding: .15rem .4rem; + margin: .15rem; +} +.priority-high::before { content: "▲ "; } +.priority-normal::before { content: "● "; } +.priority-low::before { content: "▼ "; } +.priority-unassigned::before { content: "— "; } +pre { + white-space: pre-wrap; + overflow-wrap: anywhere; + background: #eef1f3; + padding: 1rem; +} +code { overflow-wrap: anywhere; } +.review-space { min-height: 8rem; border: 1px solid #59636e; } +@media print { .page-header, main { max-width: none; } details { display: block; } } +""".strip() diff --git a/packages/adviso-compare/tests/test_reporting.py b/packages/adviso-compare/tests/test_reporting.py new file mode 100644 index 0000000..094c123 --- /dev/null +++ b/packages/adviso-compare/tests/test_reporting.py @@ -0,0 +1,468 @@ +"""Behavioural tests for HTML rendering and evidence-package publication.""" + +from __future__ import annotations + +import json +import os +import stat +from copy import deepcopy +from dataclasses import dataclass +from datetime import datetime, timezone +from pathlib import Path +from typing import Any +from uuid import UUID + +import pytest +import yaml + +import adviso_compare.reporting as reporting +from adviso_compare.alignment import RecordsResult, align_records +from adviso_compare.application import PreparedComparison, prepare_comparison +from adviso_compare.contract import ContractResult, assess_contract +from adviso_compare.invocation import ComparisonInvocation +from adviso_compare.reporting import ( + OutputError, + publish_evidence_package, + render_html_report, +) +from adviso_compare.results import ( + ResultValidationError, + RuntimeIdentity, + assemble_result, +) +from adviso_compare.values import ValuesResult, compare_values +from tests.support.documents import JsonObject, minimal_configuration, minimal_result + + +@dataclass(frozen=True, slots=True) +class _Evidence: + """One generated result and the context required to revalidate it.""" + + document: JsonObject + prepared: PreparedComparison + contract: ContractResult + records: RecordsResult + values: ValuesResult + + +def _generated_evidence( + tmp_path: Path, + *, + incomplete: bool = False, + automatic: bool = False, +) -> _Evidence: + """Generate canonical evidence through all three comparison phases.""" + tmp_path.mkdir(parents=True, exist_ok=True) + if automatic: + reference_text = '[{"record_id":"S1","result":1}]' + query_text = '[{"record_id":"S1","result":2}]' + configuration = None + else: + reference_text = '[{"reference_id":"S1","expected_value":1}]' + query_text = ( + '[{"query_id":"S1"}]' + if incomplete + else '[{"query_id":"S1","observed_value":2}]' + ) + configuration = minimal_configuration() + reference = tmp_path / "reference.json" + query = tmp_path / "query.json" + reference.write_text(reference_text, encoding="utf-8") + query.write_text(query_text, encoding="utf-8") + configuration_path = None + if configuration is not None: + configuration_path = tmp_path / "configuration.yaml" + configuration_path.write_text( + yaml.safe_dump(configuration, sort_keys=False), + encoding="utf-8", + ) + prepared = prepare_comparison( + ComparisonInvocation( + reference=reference, + query=query, + output=tmp_path / "unused-evidence", + configuration=configuration_path, + ) + ) + contract = assess_contract( + prepared.configuration, + reference=prepared.reference, + query=prepared.query, + reference_records=prepared.reference_records, + query_records=prepared.query_records, + ) + records = align_records(prepared.reference_records, prepared.query_records) + values = compare_values(prepared.configuration, contract, records) + document = assemble_result( + prepared, + contract, + records, + values, + run_id=UUID("123e4567-e89b-42d3-a456-426614174000"), + completed_at=datetime(2026, 11, 9, 12, 34, 56, tzinfo=timezone.utc), + software_version="1.2.3", + runtime=RuntimeIdentity( + python="cpython-3.12.7", + platform="linux-x86_64", + unicode_version="15.1.0", + ), + ) + return _Evidence(document, prepared, contract, records, values) + + +def _publish( + destination: Path, + evidence: _Evidence, + *, + overwrite: bool = False, +) -> Path: + """Publish generated evidence with its mandatory validation context.""" + return publish_evidence_package( + destination, + evidence.document, + prepared=evidence.prepared, + contract=evidence.contract, + records=evidence.records, + values=evidence.values, + overwrite=overwrite, + ) + + +def test_html_report_is_a_self_contained_view_of_canonical_json() -> None: + """The report presents all interpretive sections using JSON facts alone.""" + result = minimal_result(status="incomplete") + result["exclusions"] = [ + { + "code": "undeclared_field_excluded", + "side": "query", + "source_field": "unreported