From f09c83fd6b2b47d2d67ac199ce08efe987c2ae7c Mon Sep 17 00:00:00 2001 From: zz_y Date: Tue, 8 Sep 2026 07:44:06 -0600 Subject: [PATCH 1/4] docs: propose family-neutral self-describing summary architecture --- docs/design_docs/README.md | 6 + docs/design_docs/self-describing-summary.md | 382 ++++++++++++++++++++ 2 files changed, 388 insertions(+) create mode 100644 docs/design_docs/self-describing-summary.md diff --git a/docs/design_docs/README.md b/docs/design_docs/README.md index 4cee362e..6b0080a6 100644 --- a/docs/design_docs/README.md +++ b/docs/design_docs/README.md @@ -8,6 +8,12 @@ Backend-specific implementation design notes are organized by component under [`../developer_docs`](../developer_docs/README.md). They explain current Rust internals and are subordinate to the shared system contracts. +Proposals for shared-contract review (not duplicate normative specifications): + +- [Self-Describing Summary (SDS)](self-describing-summary.md): general summary + semantics, Schema / Dictionary / Record, compatibility, guarantees and + high-level migration across Planner and downstream runtimes. + Backend-specific operating profiles: - [ASAPQuery compatibility profile](asapquery-compatibility-profile.md) defines diff --git a/docs/design_docs/self-describing-summary.md b/docs/design_docs/self-describing-summary.md new file mode 100644 index 00000000..4497d534 --- /dev/null +++ b/docs/design_docs/self-describing-summary.md @@ -0,0 +1,382 @@ +# Self-Describing Summary (SDS) + +Status: proposed system contract, not an implemented wire format. +Audience: ProjectASAP designers, architects and component maintainers. +This backend-hosted proposal extends the shared summary contracts; it does not +create a second authoritative specification beside the Collector system design. + +## Design decision + +**An SDS is summary state or a summary result together with a resolvable +description of its meaning, representation, permitted operations and guarantees.** +An aggregation is a computation that produces an SDS, not the SDS itself. + +The common model must accommodate exact accumulators, distribution summaries, +distinct-count summaries and keyed-frequency summaries. Item/weight is a +type-specific update signature, not a mandatory SDS-wide structure. + +

Figure 1. SDS across the ASAPPlanner–runtime boundary.

+ +```mermaid +flowchart TD + Inputs[Queries and workload requirements] --> Planner + subgraph PB["ASAPPlanner — semantic planning"] + Planner[Canonical expressions and legal alternatives] + Planner --> DAG[Selected post-ASAP DAG] + Rules[Summary operation and guarantee rules] --> Planner + end + subgraph RB["Downstream runtime — binding and execution"] + Bind[Bind selected summaries to supported implementations] + Schema[SDS Schema: semantic and representation contract] + Producer[Collector or backend-local producer] + Dictionary[SDS Dictionary: source and group identities] + Record[SDS Record: coverage and state or result] + Validate[Resolve descriptors and validate] + Store[Store or transport validated SDS] + Read[Execute selected merge and readout] + Bind --> Schema + Bind --> Producer + Producer --> Dictionary + Producer --> Record + Schema --> Validate + Dictionary --> Validate + Record --> Validate + Validate --> Store --> Read + end + DAG --> Bind + Bind -->|Capability and cost evidence| Planner + DAG -->|Selected result dependencies| Read + Read --> Results[Query results and scoped guarantees] +``` + +The post-ASAP DAG remains the authoritative semantic plan. SDS describes the +data flowing through its summary operations; it is neither another optimization +DAG nor a replacement for scheduling, deployment or query plans. + +## Problem, baseline and minimum outcome + +Today summary interpretation is spread across Planner types, backend +aggregation configuration, state schemas and algorithm codecs. A decoder can +understand bytes without knowing the input population or what a readout means. +Conversely, knowing the algorithm name does not establish merge compatibility. +The recent heap-update issue illustrates this split, but does not define the +general abstraction. + +The [Collector data model](https://github.com/ProjectASAP/ASAPCollector-public/blob/3745d7fca6d37b4f2da03889a0db3098d07416cd/docs/data_model.md) +separates configuration, recurring series identity and window payload into +Schema / Dictionary / Record. Its final section also describes self-framed +ASAPv1 sketch bytes and the lifetime of Arrow dictionary state. SDS retains +that separation and adds an explicit semantic contract; it does not assume +that a decodable sketch envelope alone explains the summarized computation. + +Backend baseline: +[`StateSchemaContract` and physical projections](../../control_plane/src/physical/compiler.rs), +[`BackendAggregation`](../../control_plane/src/physical/colored_dag/emitter.rs), +and the [summary-family extension boundary](../developer_docs/cross-component-adding-summary-family.md). +These are foundations to evolve, not duplicate with parallel registries. + +Minimum outcome: the same selected summary can be produced, transferred, +persisted and read without guessing its semantics from metric names, missing +options or receiver-local defaults. One receiver can reject an incompatible +operation and explain which contract failed. + +Non-goals: arbitrary executable code in metadata, a universal sketch algorithm, +new query optimization rules, a new wire codec, all-backend deployment coverage, +or automatic support for every future summary. This PR changes documentation +only; Rust types, Planner dependencies and deployed records remain unchanged. + +## Inputs, outputs and workflow + +Inputs are the selected summary operations, canonical input bindings and +requirements, plus runtime implementation/codec capabilities. Outputs are +validated SDS descriptors and records, followed by the readouts already +selected by the DAG. + +1. Planner establishes what computation is legal and what its guarantee means. +2. Binding checks the entire selected operation path and chooses concrete + state/encoding implementations. Unsupported candidates return to selection. +3. The producer receives an immutable SDS Schema and emits dictionary entries + and records consistent with it. +4. A receiver resolves descriptors, verifies generation/provenance and checks + payload compatibility before accepting state. +5. The executor performs only selected, supported operations. A transformation + that changes semantics produces an explicitly described output SDS. +6. Readouts retain coverage and the applicable guarantee. Missing state, + unknown descriptors or unmet assumptions cause explicit rejection/fallback, + not implicit summary substitution. + +## The data contract + +Schema, Dictionary and Record are logical roles, not a mandate to introduce +three new services or tables. Existing plan and transport structures can carry +these roles. + +### SDS Schema: stable meaning and representation + +| Common concern | Required meaning | +| --- | --- | +| Input domain | Canonical source/projection/filter bindings, input types, units where relevant, and null/NaN/duplicate semantics | +| Population structure | Group-key schema and reduction meaning; an ungrouped population is valid | +| Summary type | Versioned semantic type, algorithm or exact-state implementation, and type-specific parameters | +| State representation | Typed state layout, codec/version and any compatibility-critical configuration | +| Operations | Typed input/output contracts and preconditions for supported build/update/merge/readout; retract/subtract only when actually supported | +| Guarantee contract | Which readout quantity is bounded, error model, confidence scope, assumptions and composition rules | +| Coverage interpretation | Domain and boundary conventions for interpreting instance coverage; time windows are one profile | + +An input or rule reference must resolve to a versioned definition, rather than +a process-local pointer. Schemas describe known operations; they do not embed +an interpreter or authorize an operation merely by naming it. + +Type-specific contracts are tagged, versioned schemas, not unrestricted +key/value bags. For example: + +| Summary type | Type-specific input/state contract | Possible readouts | +| --- | --- | --- | +| Exact SUM/COUNT | Numeric observations, count qualification, sum/count fields and numeric arithmetic semantics | Sum, observation count, derived mean | +| DDSketch | Numeric observation; relative-accuracy parameter and supported value domain | Quantile with value-error semantics | +| KLL | Ordered observation; capacity and algorithm/version-specific compaction contract | Quantile with rank-error semantics | +| HLL | Element extraction/canonicalization, precision and hash configuration | Distinct-count estimate | +| CMS / CountSketch | Key extraction, increment semantics, dimensions and hash configuration | Point-frequency estimate | +| Heap-bearing frequency summary | Frequency contract plus candidate-maintenance state and capacity | Frequency and supported TopK readouts | + +These are proposed contract examples, not a capability claim for every current +runtime. A generic integer `sketch_size` cannot represent all these parameters. +Likewise, mandatory `item` and `weight` fields would misdescribe non-frequency +summaries. They belong only where the operation signature needs them. + +Grouping keys describe populations; they are not the same as the elements +inserted into a summary. For example, an HLL grouped by service can summarize +distinct user IDs within each service. + +### SDS Dictionary: reusable instance identity + +The dictionary associates a compact local ID with a Schema reference and +concrete source/group values. A time-series profile uses metric and labels; +a relational profile may use dataset identity and typed group tuples. +Field identities, absent values and nulls must not collapse into ambiguous +string concatenations. + +An `agg_id`-like identifier selects the aggregation recipe/Schema. +A `series_id`-like identifier selects a concrete population under that recipe. +Neither is a globally meaningful identity without its namespace/context. +Identical labels under different contracts must not alias accidentally. + +### SDS Record: one instance of state or result + +A record identifies its dictionary entry, actual coverage and a tagged payload: + +- Full state: a complete state snapshot for the declared coverage. +- State delta: a change requiring a particular base/checkpoint and supported + delta-application contract. +- Readout result: a typed scalar, tuple or collection, identifying the readout + and its arguments. It is not implicitly mergeable summary state. + +Thus a quantile value identifies its quantile level; a TopK result identifies +its selection/readout contract. A bare `value=42` is not self-describing. +Output shape is not restricted to `f64`. + +Actual window bounds and inclusion rules belong to coverage for a time-series +record. A batch summary instead identifies its dataset snapshot/partition. +Do not require every SDS to have a time window. Completeness, freshness and +instance-specific evidence travel with, or are durably referenced by, the +record; a Schema's mathematical guarantee does not certify missing input. + +Producer epoch, sequence, plan generation and checkpoint linkage remain in the +existing frame/provenance contract and accompany the SDS when necessary. +Do not create a conflicting second sequence or readiness protocol. + +## What self-describing requires + +The unit of self-description is the record plus its resolvable descriptor +closure, not necessarily one network row containing every field. + +- A connected stream may establish Schema once and extend a dictionary. +- A persisted/exported unit must include or durably reference all required + descriptors. A delta also needs its base chain to reconstruct state. +- On reconnect or transfer to another replica, restore the descriptor context + before interpreting dependent records. IDs must not resolve against an + unrelated previous session. +- Unknown references are rejected or buffered within explicit resource limits. + No fallback to a guessed algorithm, parameter set or update mode. +- Inline descriptions and control-plane-installed descriptions must agree; + conflicting definitions for one identity are errors. + +Wire framing answers how to decode bytes. Semantic self-description answers +what those bytes summarize and which operations are valid. Neither implies +that the receiver implements the algorithm or trusts the producer. + +## Reuse, merging and identity + +Three questions must remain separate: + +| Question | Required reasoning | +| --- | --- | +| Can two consumers reuse one producer? | Planner proves that one maintained state satisfies both selected computations and requirements | +| Can two states be merged directly? | The summary's operation rule checks representation, input semantics, coverage, provenance and assumptions | +| Can one summary be transformed for another consumer? | Planner selects a legal rollup, projection, conversion or parameter transformation, and binding supports it | + +Different fields do not universally prohibit reuse. Equal fields do not +universally permit a merge. HLL union may legitimately overlap populations, +whereas summing two overlapping SUM states double-counts observations. +Algebraic associativity does not promise bitwise-identical floating-point +results under every merge order; the numeric contract must state the policy. + +Use a semantic contract identity for meaning, a representation identity for +encoding compatibility, and existing materialization/generation identities +for maintained runtime state. These are distinct roles, not necessarily new +independent ID fields. Audit current fingerprints and schema IDs before +extending them. Compact stream IDs refer to these identities; they do not +replace them. Equality fingerprints are lookup aids, not equivalence proofs. + +## Examples beyond TopK + +### Exact state shared by several readouts + +For a fixed, complete snapshot of integer observations: + +| Group | Observations | SDS state | +| --- | --- | --- | +| service=api, region=east | 10 | sum=10, count=1 | +| service=api, region=west | 2, 4, 8 | sum=14, count=3 | + +The Schema specifies the input projection, group keys, integer sum/count +semantics and component-wise merge. The dictionary binds east/west identities; +each record carries its snapshot coverage and typed state. + +A selected rollup to service produces sum=24, count=4. SUM reads 24 and +sample-weighted mean reads 6 from that shared state. Averaging the two regional +means instead gives a different computation and is not a legal substitution. +Counts must use the same qualifying observations as the sum; COUNT(*) and +COUNT(nullable_column) cannot be conflated. Rollup checks disjoint input +coverage or an explicitly authorized duplicate policy. + +### Quantiles and distinct counts + +A DDSketch and a KLL may both answer a quantile request, but do not describe +the same error quantity or interchangeable state. Planner chooses a legal +alternative for the requested error contract; a receiver does not merge them +because both advertise a quantile readout. + +Two HLL states can support union when their versioned rule permits the +element encoding, hash configuration and precision combination. Adding their +cardinality estimates is not that operation. Precision conversion is allowed +only through an explicitly supported transformation and revised guarantee. + +### Frequency and TopK + +A frequency summary can use key/increment updates without any TopK readout. +A heap-bearing implementation may additionally maintain candidates. Its heap +capacity describes retained state; readout `k` describes the requested result. +Changing `k` need not create a new producer if capacity and guarantee suffice. + +For temporal counts, the increment is one per qualifying observation; for +temporal sums it is the selected value. These are distinct type-specific +update contracts. The compatibility adapter may derive `weight_mode` from +them, but new SDS interpretation must not rely on `heap_update_mode: None`. +TopK membership evidence is distinct from frequency-estimation error evidence. + +## Accuracy, evidence and scope + +Accuracy belongs to a supported readout under stated conditions, not merely +to an algorithm name or a generic `epsilon/delta` pair on opaque bytes. +Distinguish requested accuracy, the type's guarantee model and the +instance-specific evidence that permits claiming an achieved guarantee. + +Record the error quantity (for example, value-relative versus rank error), +failure event, scope, assumptions and evidence provenance. Deterministic +bounds need no invented delta; unknown guarantees remain unknown. Exact +arithmetic claims are separate from complete-input and freshness claims. + +For a fixed 20-group query result, simultaneous failure probability at most +0.05 follows if each valid row bound is at most 0.0025, by the union bound. +Independence is unnecessary. Twenty per-row 95% intervals do not establish +95% confidence for the entire result. Adaptive group selection, TopK membership, +or repeated evaluations need their own applicable bounds and scope. + +Planner owns guarantee composition and failure-budget allocation. SDS preserves +the selected scope and evidence references so the runtime cannot silently +reinterpret a per-row bound as a whole-query bound. Shared state does not make +errors independent or eliminate a consumer's guarantee obligations. + +## Ownership and minimal complexity + +| Owner | Authority | +| --- | --- | +| ASAPPlanner | Logical summary operations, legal rewrites/reuse and guarantee composition | +| Summary implementation libraries | Algorithm/state versions, codecs, implemented operations and associated mathematical contracts | +| Shared ProjectASAP data contract | SDS descriptor/reference/record meaning and cross-component conformance rules | +| Collector and backend adapters | Actual capability support, binding, production, validation, transport and execution | +| Runtime plans | Deployment, handles, materialization generations, lifecycle and readiness | + +Use the existing family-extension and schema/frame boundaries to implement +SDS. Do not add a generic plugin engine, new optimizer or centralized registry +service merely to name this contract. Once accepted, the shared normative +definition belongs with the existing cross-component contracts; downstream +documentation references it and describes only its own adapter. + +Rejected alternatives: a TopK-shaped universal struct; a flat arbitrary +parameter map with implicit defaults; copying every descriptor into every +record; and treating byte/schema equality as proof of semantic reuse. +Each either excludes other families, obscures compatibility or repeats data +without adding meaning. Type-specific contracts plus resolvable references +are the smallest design covering the demonstrated cases. + +## Acceptance before implementation + +These are proposed acceptance gates, not tests claimed to exist in this PR. +Test design here is by the design author, not an independent reviewer. + +| Gate | Observable acceptance | +| --- | --- | +| Generality | SUM/COUNT, quantile and HLL examples work without fake item/weight fields; frequency examples specify their own update signature | +| Shared execution | Exact example yields sum=24 and mean=6 with one maintained compatible producer per partition/generation | +| Compatibility | Reject incompatible hashes, unknown versions, ambiguous inputs, missing readout arguments and illegal overlapping SUM merges | +| Valid transformations | Planner-selected group rollup succeeds; invalid pooled-mean replacement fails | +| Persistence/transport | Decode after restart/replica transfer with restored descriptors; reject missing descriptor/base and conflicting ID reuse | +| Guarantees | Preserve rank/value distinction and per-row/whole-result scope; incomplete coverage cannot be labeled complete | +| Operational safety | Unknown-type/oversized metadata and unbounded dictionary growth are rejected or bounded; duplicates and wrong-generation frames follow existing rules | +| Traceability | A rejected record identifies contract, materialization, producer/generation and reason without logging sensitive source values | + +Measure descriptor bytes per unique schema/population and record bytes per +emitted state, including reconnect overhead. Do not claim constant total +metadata under unbounded group cardinality. Track validation failures and +descriptor misses separately from payload decoding and query readiness. +Extension effort is evaluated by adding a non-frequency summary without +changing the common fields; no fabricated throughput or delivery estimate is +part of this proposal. + +## High-level migration, risks and completion + +1. Map current Planner types, state schemas, fingerprints and frame fields to + the contract above. Agree on one normative owner and versioning policy. +2. Define typed contracts for existing exact, distribution, distinct and + frequency summaries. Unknown legacy semantics stay explicitly unsupported. +3. Add producer/consumer adapters using the existing carrier. Keep the old + runtime configuration as a derived compatibility projection, not a second + semantic authority. +4. Demonstrate descriptor resolution, persistence and the exact shared-state + example in backend-local execution. Validate distributed transport separately. +5. Dual-validate old/new descriptions during rollout; switch only when they + agree. Retire implicit defaults after all supported producers and stored + states have an explicit interpretation. + +Schema evolution creates new immutable identities; it must not reinterpret +old persisted records. Rollback remains possible while old representations +are retained and both readers understand the active version. Writing +new-only state requires capability negotiation and an explicit rollback or +rebuild path. No destructive data conversion is authorized by this proposal. + +Principal risks are semantic drift between descriptors and implementations, +unbounded descriptor retention, incorrect overlap assumptions and unsupported +cross-version state. Admission validation, conformance fixtures and durable +descriptor retention address these risks without trusting metadata as proof. + +Completion is scoped: the first supported profile passes the gates above and +no longer guesses summary semantics. Other engines, summary families and +workload lifecycles are ProjectASAP-wide extensions, not a prerequisite for +this backend's first SDS milestone. + +Before implementation, maintainers must settle the normative package/location, +canonical descriptor identity/version rules and supported compatibility matrix. +An implementation schedule depends on that audit; this design does not claim +the migration is already delivered. From 73cd4274cdd4bfbd8f65e30150865870bf2d175b Mon Sep 17 00:00:00 2001 From: zz_y Date: Tue, 8 Sep 2026 07:49:57 -0600 Subject: [PATCH 2/4] docs: limit SDS design to structure definitions --- docs/design_docs/README.md | 5 +- docs/design_docs/self-describing-summary.md | 533 ++++++-------------- 2 files changed, 165 insertions(+), 373 deletions(-) diff --git a/docs/design_docs/README.md b/docs/design_docs/README.md index 6b0080a6..6db50782 100644 --- a/docs/design_docs/README.md +++ b/docs/design_docs/README.md @@ -10,9 +10,8 @@ internals and are subordinate to the shared system contracts. Proposals for shared-contract review (not duplicate normative specifications): -- [Self-Describing Summary (SDS)](self-describing-summary.md): general summary - semantics, Schema / Dictionary / Record, compatibility, guarantees and - high-level migration across Planner and downstream runtimes. +- [Self-Describing Summary (SDS)](self-describing-summary.md): Schema, + Dictionary and Record structure definitions. Backend-specific operating profiles: diff --git a/docs/design_docs/self-describing-summary.md b/docs/design_docs/self-describing-summary.md index 4497d534..4e79ac89 100644 --- a/docs/design_docs/self-describing-summary.md +++ b/docs/design_docs/self-describing-summary.md @@ -1,382 +1,175 @@ # Self-Describing Summary (SDS) -Status: proposed system contract, not an implemented wire format. -Audience: ProjectASAP designers, architects and component maintainers. -This backend-hosted proposal extends the shared summary contracts; it does not -create a second authoritative specification beside the Collector system design. - -## Design decision - -**An SDS is summary state or a summary result together with a resolvable -description of its meaning, representation, permitted operations and guarantees.** -An aggregation is a computation that produces an SDS, not the SDS itself. - -The common model must accommodate exact accumulators, distribution summaries, -distinct-count summaries and keyed-frequency summaries. Item/weight is a -type-specific update signature, not a mandatory SDS-wide structure. - -

Figure 1. SDS across the ASAPPlanner–runtime boundary.

- -```mermaid -flowchart TD - Inputs[Queries and workload requirements] --> Planner - subgraph PB["ASAPPlanner — semantic planning"] - Planner[Canonical expressions and legal alternatives] - Planner --> DAG[Selected post-ASAP DAG] - Rules[Summary operation and guarantee rules] --> Planner - end - subgraph RB["Downstream runtime — binding and execution"] - Bind[Bind selected summaries to supported implementations] - Schema[SDS Schema: semantic and representation contract] - Producer[Collector or backend-local producer] - Dictionary[SDS Dictionary: source and group identities] - Record[SDS Record: coverage and state or result] - Validate[Resolve descriptors and validate] - Store[Store or transport validated SDS] - Read[Execute selected merge and readout] - Bind --> Schema - Bind --> Producer - Producer --> Dictionary - Producer --> Record - Schema --> Validate - Dictionary --> Validate - Record --> Validate - Validate --> Store --> Read - end - DAG --> Bind - Bind -->|Capability and cost evidence| Planner - DAG -->|Selected result dependencies| Read - Read --> Results[Query results and scoped guarantees] -``` - -The post-ASAP DAG remains the authoritative semantic plan. SDS describes the -data flowing through its summary operations; it is neither another optimization -DAG nor a replacement for scheduling, deployment or query plans. - -## Problem, baseline and minimum outcome - -Today summary interpretation is spread across Planner types, backend -aggregation configuration, state schemas and algorithm codecs. A decoder can -understand bytes without knowing the input population or what a readout means. -Conversely, knowing the algorithm name does not establish merge compatibility. -The recent heap-update issue illustrates this split, but does not define the -general abstraction. - -The [Collector data model](https://github.com/ProjectASAP/ASAPCollector-public/blob/3745d7fca6d37b4f2da03889a0db3098d07416cd/docs/data_model.md) -separates configuration, recurring series identity and window payload into -Schema / Dictionary / Record. Its final section also describes self-framed -ASAPv1 sketch bytes and the lifetime of Arrow dictionary state. SDS retains -that separation and adds an explicit semantic contract; it does not assume -that a decodable sketch envelope alone explains the summarized computation. - -Backend baseline: -[`StateSchemaContract` and physical projections](../../control_plane/src/physical/compiler.rs), -[`BackendAggregation`](../../control_plane/src/physical/colored_dag/emitter.rs), -and the [summary-family extension boundary](../developer_docs/cross-component-adding-summary-family.md). -These are foundations to evolve, not duplicate with parallel registries. - -Minimum outcome: the same selected summary can be produced, transferred, -persisted and read without guessing its semantics from metric names, missing -options or receiver-local defaults. One receiver can reject an incompatible -operation and explain which contract failed. - -Non-goals: arbitrary executable code in metadata, a universal sketch algorithm, -new query optimization rules, a new wire codec, all-backend deployment coverage, -or automatic support for every future summary. This PR changes documentation -only; Rust types, Planner dependencies and deployed records remain unchanged. - -## Inputs, outputs and workflow - -Inputs are the selected summary operations, canonical input bindings and -requirements, plus runtime implementation/codec capabilities. Outputs are -validated SDS descriptors and records, followed by the readouts already -selected by the DAG. - -1. Planner establishes what computation is legal and what its guarantee means. -2. Binding checks the entire selected operation path and chooses concrete - state/encoding implementations. Unsupported candidates return to selection. -3. The producer receives an immutable SDS Schema and emits dictionary entries - and records consistent with it. -4. A receiver resolves descriptors, verifies generation/provenance and checks - payload compatibility before accepting state. -5. The executor performs only selected, supported operations. A transformation - that changes semantics produces an explicitly described output SDS. -6. Readouts retain coverage and the applicable guarantee. Missing state, - unknown descriptors or unmet assumptions cause explicit rejection/fallback, - not implicit summary substitution. - -## The data contract - -Schema, Dictionary and Record are logical roles, not a mandate to introduce -three new services or tables. Existing plan and transport structures can carry -these roles. - -### SDS Schema: stable meaning and representation - -| Common concern | Required meaning | -| --- | --- | -| Input domain | Canonical source/projection/filter bindings, input types, units where relevant, and null/NaN/duplicate semantics | -| Population structure | Group-key schema and reduction meaning; an ungrouped population is valid | -| Summary type | Versioned semantic type, algorithm or exact-state implementation, and type-specific parameters | -| State representation | Typed state layout, codec/version and any compatibility-critical configuration | -| Operations | Typed input/output contracts and preconditions for supported build/update/merge/readout; retract/subtract only when actually supported | -| Guarantee contract | Which readout quantity is bounded, error model, confidence scope, assumptions and composition rules | -| Coverage interpretation | Domain and boundary conventions for interpreting instance coverage; time windows are one profile | - -An input or rule reference must resolve to a versioned definition, rather than -a process-local pointer. Schemas describe known operations; they do not embed -an interpreter or authorize an operation merely by naming it. - -Type-specific contracts are tagged, versioned schemas, not unrestricted -key/value bags. For example: - -| Summary type | Type-specific input/state contract | Possible readouts | +Proposed logical structures; not an implemented serialization format. + +## SDS + +| Field | Type | Definition | | --- | --- | --- | -| Exact SUM/COUNT | Numeric observations, count qualification, sum/count fields and numeric arithmetic semantics | Sum, observation count, derived mean | -| DDSketch | Numeric observation; relative-accuracy parameter and supported value domain | Quantile with value-error semantics | -| KLL | Ordered observation; capacity and algorithm/version-specific compaction contract | Quantile with rank-error semantics | -| HLL | Element extraction/canonicalization, precision and hash configuration | Distinct-count estimate | -| CMS / CountSketch | Key extraction, increment semantics, dimensions and hash configuration | Point-frequency estimate | -| Heap-bearing frequency summary | Frequency contract plus candidate-maintenance state and capacity | Frequency and supported TopK readouts | - -These are proposed contract examples, not a capability claim for every current -runtime. A generic integer `sketch_size` cannot represent all these parameters. -Likewise, mandatory `item` and `weight` fields would misdescribe non-frequency -summaries. They belong only where the operation signature needs them. - -Grouping keys describe populations; they are not the same as the elements -inserted into a summary. For example, an HLL grouped by service can summarize -distinct user IDs within each service. - -### SDS Dictionary: reusable instance identity - -The dictionary associates a compact local ID with a Schema reference and -concrete source/group values. A time-series profile uses metric and labels; -a relational profile may use dataset identity and typed group tuples. -Field identities, absent values and nulls must not collapse into ambiguous -string concatenations. - -An `agg_id`-like identifier selects the aggregation recipe/Schema. -A `series_id`-like identifier selects a concrete population under that recipe. -Neither is a globally meaningful identity without its namespace/context. -Identical labels under different contracts must not alias accidentally. - -### SDS Record: one instance of state or result - -A record identifies its dictionary entry, actual coverage and a tagged payload: - -- Full state: a complete state snapshot for the declared coverage. -- State delta: a change requiring a particular base/checkpoint and supported - delta-application contract. -- Readout result: a typed scalar, tuple or collection, identifying the readout - and its arguments. It is not implicitly mergeable summary state. - -Thus a quantile value identifies its quantile level; a TopK result identifies -its selection/readout contract. A bare `value=42` is not self-describing. -Output shape is not restricted to `f64`. - -Actual window bounds and inclusion rules belong to coverage for a time-series -record. A batch summary instead identifies its dataset snapshot/partition. -Do not require every SDS to have a time window. Completeness, freshness and -instance-specific evidence travel with, or are durably referenced by, the -record; a Schema's mathematical guarantee does not certify missing input. - -Producer epoch, sequence, plan generation and checkpoint linkage remain in the -existing frame/provenance contract and accompany the SDS when necessary. -Do not create a conflicting second sequence or readiness protocol. - -## What self-describing requires - -The unit of self-description is the record plus its resolvable descriptor -closure, not necessarily one network row containing every field. - -- A connected stream may establish Schema once and extend a dictionary. -- A persisted/exported unit must include or durably reference all required - descriptors. A delta also needs its base chain to reconstruct state. -- On reconnect or transfer to another replica, restore the descriptor context - before interpreting dependent records. IDs must not resolve against an - unrelated previous session. -- Unknown references are rejected or buffered within explicit resource limits. - No fallback to a guessed algorithm, parameter set or update mode. -- Inline descriptions and control-plane-installed descriptions must agree; - conflicting definitions for one identity are errors. - -Wire framing answers how to decode bytes. Semantic self-description answers -what those bytes summarize and which operations are valid. Neither implies -that the receiver implements the algorithm or trusts the producer. - -## Reuse, merging and identity - -Three questions must remain separate: - -| Question | Required reasoning | -| --- | --- | -| Can two consumers reuse one producer? | Planner proves that one maintained state satisfies both selected computations and requirements | -| Can two states be merged directly? | The summary's operation rule checks representation, input semantics, coverage, provenance and assumptions | -| Can one summary be transformed for another consumer? | Planner selects a legal rollup, projection, conversion or parameter transformation, and binding supports it | +| `schemas` | `SdsSchema[]` | Immutable summary descriptors | +| `dictionary` | `SdsIdentity[]` | Reusable population identities | +| `records` | `SdsRecord[]` | Summary states or readout results | + +References are namespace-qualified. Every record's descriptor references must +resolve within the SDS or a durably retained descriptor context. + +## SdsSchema + +| Field | Type | Definition | +| --- | --- | --- | +| `schema_id` | `QualifiedId` | Immutable descriptor identity | +| `schema_version` | `Version` | Descriptor format version | +| `input` | `InputContract` | Summarized input and its interpretation | +| `grouping` | `GroupingContract` | Population partitioning and group-key types | +| `summary` | `SummaryTypeContract` | Summary-specific state and operation semantics | +| `representation` | `RepresentationContract` | Payload layout and encoding | +| `guarantees` | `GuaranteeContract[]` | Guarantees available for specified operations | + +### InputContract + +| Field | Type | Definition | +| --- | --- | --- | +| `source` | `CanonicalSourceBinding` | Versioned source definition; concrete snapshot/partition is identified by record coverage | +| `fields` | `FieldDefinition[]` | Stable field IDs, data types, nullability and optional units | +| `projection` | `TypedExpression[]` | Expressions supplying the operation's inputs | +| `filter` | `Optional` | Input qualification; absent means no additional filter | +| `observation_semantics` | `VersionedSemanticContract` | Null, NaN, duplicate and ordering interpretation | + +### GroupingContract + +| Field | Type | Definition | +| --- | --- | --- | +| `mode` | `Global \| PerEntity \| ByKeys` | One population, preserved entity populations, or explicit grouping | +| `keys` | `TypedExpression[]` | Group-key expressions; empty for Global | +| `key_semantics` | `VersionedSemanticContract` | Equality, canonicalization and absent/null handling | + +### SummaryTypeContract -Different fields do not universally prohibit reuse. Equal fields do not -universally permit a merge. HLL union may legitimately overlap populations, -whereas summing two overlapping SUM states double-counts observations. -Algebraic associativity does not promise bitwise-identical floating-point -results under every merge order; the numeric contract must state the policy. +| Field | Type | Definition | +| --- | --- | --- | +| `type_id` | `QualifiedId` | Summary semantic type | +| `type_version` | `Version` | State and operation semantics version | +| `parameters` | `TypedParameterRecord` | Parameters validated against this summary type's parameter schema | +| `state_schema` | `TypeDefinition` | Logical state structure | +| `operations` | `OperationContract[]` | Supported state construction, modification, combination and readout | -Use a semantic contract identity for meaning, a representation identity for -encoding compatibility, and existing materialization/generation identities -for maintained runtime state. These are distinct roles, not necessarily new -independent ID fields. Audit current fingerprints and schema IDs before -extending them. Compact stream IDs refer to these identities; they do not -replace them. Equality fingerprints are lookup aids, not equivalence proofs. +`parameters` and operation signatures are type-specific; `item` and `weight` +are not common SDS fields. -## Examples beyond TopK +| Summary type | Type-specific parameter fields | Update input signature | +| --- | --- | --- | +| Exact aggregate | Aggregate components, numeric representation and overflow policy | Typed observations with component-specific qualification | +| DDSketch | Relative accuracy and supported value-domain policy | Numeric observation | +| KLL | Capacity and compaction configuration | Ordered observation | +| HLL | Precision, element encoding and hash configuration | Element | +| CMS | Width, depth, key encoding, hash configuration and increment-domain policy | Key, increment | +| CountSketch | Width, depth, key encoding, hash configuration and increment-domain policy | Key, increment | +| Heap-bearing frequency summary | Base frequency contract, candidate policy and heap capacity | Base frequency update input | -### Exact state shared by several readouts +### OperationContract -For a fixed, complete snapshot of integer observations: +| Field | Type | Definition | +| --- | --- | --- | +| `operation_id` | `QualifiedId` | Versioned operation definition | +| `kind` | `Build \| Update \| Merge \| Retract \| Subtract \| Readout` | Operation category; only supported operations are listed | +| `inputs` | `TypeDefinition[]` | Ordered input/state signatures | +| `arguments` | `ParameterSchema` | Typed operation arguments | +| `output` | `TypeDefinition` | Output state or result type | +| `preconditions` | `RuleRef[]` | Compatibility, coverage, ordering and provenance requirements | +| `guarantee_rules` | `RuleRef[]` | Applicable guarantee derivation/composition rules | -| Group | Observations | SDS state | +### RepresentationContract + +| Field | Type | Definition | | --- | --- | --- | -| service=api, region=east | 10 | sum=10, count=1 | -| service=api, region=west | 2, 4, 8 | sum=14, count=3 | - -The Schema specifies the input projection, group keys, integer sum/count -semantics and component-wise merge. The dictionary binds east/west identities; -each record carries its snapshot coverage and typed state. - -A selected rollup to service produces sum=24, count=4. SUM reads 24 and -sample-weighted mean reads 6 from that shared state. Averaging the two regional -means instead gives a different computation and is not a legal substitution. -Counts must use the same qualifying observations as the sum; COUNT(*) and -COUNT(nullable_column) cannot be conflated. Rollup checks disjoint input -coverage or an explicitly authorized duplicate policy. - -### Quantiles and distinct counts - -A DDSketch and a KLL may both answer a quantile request, but do not describe -the same error quantity or interchangeable state. Planner chooses a legal -alternative for the requested error contract; a receiver does not merge them -because both advertise a quantile readout. - -Two HLL states can support union when their versioned rule permits the -element encoding, hash configuration and precision combination. Adding their -cardinality estimates is not that operation. Precision conversion is allowed -only through an explicitly supported transformation and revised guarantee. - -### Frequency and TopK - -A frequency summary can use key/increment updates without any TopK readout. -A heap-bearing implementation may additionally maintain candidates. Its heap -capacity describes retained state; readout `k` describes the requested result. -Changing `k` need not create a new producer if capacity and guarantee suffice. - -For temporal counts, the increment is one per qualifying observation; for -temporal sums it is the selected value. These are distinct type-specific -update contracts. The compatibility adapter may derive `weight_mode` from -them, but new SDS interpretation must not rely on `heap_update_mode: None`. -TopK membership evidence is distinct from frequency-estimation error evidence. - -## Accuracy, evidence and scope - -Accuracy belongs to a supported readout under stated conditions, not merely -to an algorithm name or a generic `epsilon/delta` pair on opaque bytes. -Distinguish requested accuracy, the type's guarantee model and the -instance-specific evidence that permits claiming an achieved guarantee. - -Record the error quantity (for example, value-relative versus rank error), -failure event, scope, assumptions and evidence provenance. Deterministic -bounds need no invented delta; unknown guarantees remain unknown. Exact -arithmetic claims are separate from complete-input and freshness claims. - -For a fixed 20-group query result, simultaneous failure probability at most -0.05 follows if each valid row bound is at most 0.0025, by the union bound. -Independence is unnecessary. Twenty per-row 95% intervals do not establish -95% confidence for the entire result. Adaptive group selection, TopK membership, -or repeated evaluations need their own applicable bounds and scope. - -Planner owns guarantee composition and failure-budget allocation. SDS preserves -the selected scope and evidence references so the runtime cannot silently -reinterpret a per-row bound as a whole-query bound. Shared state does not make -errors independent or eliminate a consumer's guarantee obligations. - -## Ownership and minimal complexity - -| Owner | Authority | +| `representation_id` | `QualifiedId` | Concrete state-layout identity | +| `codec` | `QualifiedId` | Payload codec | +| `codec_version` | `Version` | Codec version | +| `layout` | `TypeDefinition` | Encoded payload layout | +| `payload_kinds` | `Set` | Supported payload forms | + +### GuaranteeContract + +| Field | Type | Definition | +| --- | --- | --- | +| `guarantee_id` | `QualifiedId` | Versioned guarantee definition | +| `operation` | `QualifiedId` | Operation/readout to which the guarantee applies | +| `kind` | `Exact \| DeterministicBound \| ProbabilisticBound \| Unknown` | Guarantee category | +| `error_quantity` | `Optional` | Quantity, units and normalization being bounded | +| `bound` | `Optional` | Bound or versioned bound derivation | +| `failure_probability` | `Optional` | Required for a probabilistic bound; not invented for other categories | +| `scope` | `GuaranteeScope` | Population/readout and evaluation set covered by the claim | +| `assumptions` | `RuleRef[]` | Required input, algorithm and evidence conditions | + +## SdsIdentity + +| Field | Type | Definition | +| --- | --- | --- | +| `identity_id` | `QualifiedId` | Population identity; may have a compact dictionary alias | +| `schema_id` | `QualifiedId` | Referenced SDS Schema | +| `source_identity` | `TypedRecord` | Concrete source identity, including metric name where applicable | +| `group_values` | `TypedTuple` | Values matching the grouping key/entity schema | + +## SdsRecord + +| Field | Type | Definition | +| --- | --- | --- | +| `record_id` | `QualifiedId` | Record identity | +| `identity_id` | `QualifiedId` | Referenced dictionary identity | +| `coverage` | `Coverage` | Actual summarized input extent and completeness | +| `payload` | `SdsPayload` | Exactly one state or result variant | +| `provenance` | `ProvenanceRef` | Resolvable producer, generation and sequence metadata | +| `guarantee_evidence` | `GuaranteeEvidence[]` | Instance-specific evidence for applicable guarantees | + +### SdsPayload + +| Variant | Fields | | --- | --- | -| ASAPPlanner | Logical summary operations, legal rewrites/reuse and guarantee composition | -| Summary implementation libraries | Algorithm/state versions, codecs, implemented operations and associated mathematical contracts | -| Shared ProjectASAP data contract | SDS descriptor/reference/record meaning and cross-component conformance rules | -| Collector and backend adapters | Actual capability support, binding, production, validation, transport and execution | -| Runtime plans | Deployment, handles, materialization generations, lifecycle and readiness | - -Use the existing family-extension and schema/frame boundaries to implement -SDS. Do not add a generic plugin engine, new optimizer or centralized registry -service merely to name this contract. Once accepted, the shared normative -definition belongs with the existing cross-component contracts; downstream -documentation references it and describes only its own adapter. - -Rejected alternatives: a TopK-shaped universal struct; a flat arbitrary -parameter map with implicit defaults; copying every descriptor into every -record; and treating byte/schema equality as proof of semantic reuse. -Each either excludes other families, obscures compatibility or repeats data -without adding meaning. Type-specific contracts plus resolvable references -are the smallest design covering the demonstrated cases. - -## Acceptance before implementation - -These are proposed acceptance gates, not tests claimed to exist in this PR. -Test design here is by the design author, not an independent reviewer. - -| Gate | Observable acceptance | +| `FullState` | `state: bytes` | +| `StateDelta` | `base_record: QualifiedId`, `apply_operation: QualifiedId`, `delta: bytes` | +| `ReadoutResult` | `operation: QualifiedId`, `arguments: TypedParameterRecord`, `value: TypedValue` | + +State bytes use the referenced representation contract. A readout result uses +its operation's output type and is not implicitly mergeable state. + +### Coverage + +| Field | Type | Definition | +| --- | --- | --- | +| `extent` | `TimeExtent \| DatasetExtent` | Time interval or dataset snapshot/partition extent | +| `completeness` | `Complete \| Partial \| Unknown` | Coverage status, separate from mathematical accuracy | +| `evidence` | `EvidenceRef[]` | Evidence establishing coverage/freshness | + +| Extent | Fields | +| --- | --- | +| `TimeExtent` | `clock: ClockDefinition`, `start: Timestamp`, `end: Timestamp`, `bounds: IntervalBounds` | +| `DatasetExtent` | `snapshot: QualifiedId`, `partitions: TypedSet`, `selection: Optional` | + +### GuaranteeEvidence + +| Field | Type | Definition | +| --- | --- | --- | +| `guarantee_id` | `QualifiedId` | Schema guarantee being evaluated | +| `scope` | `GuaranteeScope` | Concrete population/readout/evaluation scope | +| `status` | `Established \| Unverified \| Invalid` | Whether the conditions for this instance are established | +| `evidence` | `EvidenceRef[]` | Resolvable evidence and its validity/provenance | + +## Referenced types + +| Type | Definition | | --- | --- | -| Generality | SUM/COUNT, quantile and HLL examples work without fake item/weight fields; frequency examples specify their own update signature | -| Shared execution | Exact example yields sum=24 and mean=6 with one maintained compatible producer per partition/generation | -| Compatibility | Reject incompatible hashes, unknown versions, ambiguous inputs, missing readout arguments and illegal overlapping SUM merges | -| Valid transformations | Planner-selected group rollup succeeds; invalid pooled-mean replacement fails | -| Persistence/transport | Decode after restart/replica transfer with restored descriptors; reject missing descriptor/base and conflicting ID reuse | -| Guarantees | Preserve rank/value distinction and per-row/whole-result scope; incomplete coverage cannot be labeled complete | -| Operational safety | Unknown-type/oversized metadata and unbounded dictionary growth are rejected or bounded; duplicates and wrong-generation frames follow existing rules | -| Traceability | A rejected record identifies contract, materialization, producer/generation and reason without logging sensitive source values | - -Measure descriptor bytes per unique schema/population and record bytes per -emitted state, including reconnect overhead. Do not claim constant total -metadata under unbounded group cardinality. Track validation failures and -descriptor misses separately from payload decoding and query readiness. -Extension effort is evaluated by adding a non-frequency summary without -changing the common fields; no fabricated throughput or delivery estimate is -part of this proposal. - -## High-level migration, risks and completion - -1. Map current Planner types, state schemas, fingerprints and frame fields to - the contract above. Agree on one normative owner and versioning policy. -2. Define typed contracts for existing exact, distribution, distinct and - frequency summaries. Unknown legacy semantics stay explicitly unsupported. -3. Add producer/consumer adapters using the existing carrier. Keep the old - runtime configuration as a derived compatibility projection, not a second - semantic authority. -4. Demonstrate descriptor resolution, persistence and the exact shared-state - example in backend-local execution. Validate distributed transport separately. -5. Dual-validate old/new descriptions during rollout; switch only when they - agree. Retire implicit defaults after all supported producers and stored - states have an explicit interpretation. - -Schema evolution creates new immutable identities; it must not reinterpret -old persisted records. Rollback remains possible while old representations -are retained and both readers understand the active version. Writing -new-only state requires capability negotiation and an explicit rollback or -rebuild path. No destructive data conversion is authorized by this proposal. - -Principal risks are semantic drift between descriptors and implementations, -unbounded descriptor retention, incorrect overlap assumptions and unsupported -cross-version state. Admission validation, conformance fixtures and durable -descriptor retention address these risks without trusting metadata as proof. - -Completion is scoped: the first supported profile passes the gates above and -no longer guesses summary semantics. Other engines, summary families and -workload lifecycles are ProjectASAP-wide extensions, not a prerequisite for -this backend's first SDS milestone. - -Before implementation, maintainers must settle the normative package/location, -canonical descriptor identity/version rules and supported compatibility matrix. -An implementation schedule depends on that audit; this design does not claim -the migration is already delivered. +| `QualifiedId` | Namespace plus immutable identifier | +| `Version` | Version identifier with an explicit compatibility definition | +| `TypeDefinition` | Resolvable, versioned scalar/tuple/collection/state type | +| `TypedValue / TypedTuple / TypedRecord / TypedSet` | Values whose types and field identities resolve through a TypeDefinition | +| `ParameterSchema / TypedParameterRecord` | Versioned parameter definition and values validated against it | +| `CanonicalSourceBinding` | Resolvable source definition with stable field bindings | +| `FieldDefinition` | Field ID, name, type, nullability and optional unit | +| `TypedExpression / TypedPredicate` | Canonical expression with resolved input/output types | +| `VersionedSemanticContract / RuleRef` | Versioned semantic definition or compatibility rule; not executable code supplied by a record | +| `TypedErrorDefinition / TypedBound` | Error quantity and bound with defined types, units and interpretation | +| `Probability` | Finite number in [0, 1] | +| `GuaranteeScope` | Population selector, operation arguments and covered evaluation set; distinguishes per-row, whole-result and repeated-evaluation claims | +| `ClockDefinition / Timestamp / IntervalBounds` | Clock/time unit, time value and inclusive/exclusive interval boundaries | +| `ProvenanceRef` | Reference to existing frame/materialization metadata: producer, epoch, sequence and plan generation | +| `EvidenceRef` | Immutable evidence reference including issuer, observation time, validity and subject scope | From 01795a7e28dfd60ff0bb4195f8e12be1fc7505c2 Mon Sep 17 00:00:00 2001 From: zz_y Date: Tue, 8 Sep 2026 14:15:27 -0600 Subject: [PATCH 3/4] docs: define SDS as three reusable descriptor and instance layers --- docs/design_docs/self-describing-summary.md | 271 +++++++++----------- 1 file changed, 124 insertions(+), 147 deletions(-) diff --git a/docs/design_docs/self-describing-summary.md b/docs/design_docs/self-describing-summary.md index 4e79ac89..61313f49 100644 --- a/docs/design_docs/self-describing-summary.md +++ b/docs/design_docs/self-describing-summary.md @@ -1,175 +1,152 @@ # Self-Describing Summary (SDS) -Proposed logical structures; not an implemented serialization format. +This proposal defines three logical layers for summary producers and consumers. +It does not change the current runtime or wire format. -## SDS - -| Field | Type | Definition | -| --- | --- | --- | -| `schemas` | `SdsSchema[]` | Immutable summary descriptors | -| `dictionary` | `SdsIdentity[]` | Reusable population identities | -| `records` | `SdsRecord[]` | Summary states or readout results | - -References are namespace-qualified. Every record's descriptor references must -resolve within the SDS or a durably retained descriptor context. - -## SdsSchema - -| Field | Type | Definition | -| --- | --- | --- | -| `schema_id` | `QualifiedId` | Immutable descriptor identity | -| `schema_version` | `Version` | Descriptor format version | -| `input` | `InputContract` | Summarized input and its interpretation | -| `grouping` | `GroupingContract` | Population partitioning and group-key types | -| `summary` | `SummaryTypeContract` | Summary-specific state and operation semantics | -| `representation` | `RepresentationContract` | Payload layout and encoding | -| `guarantees` | `GuaranteeContract[]` | Guarantees available for specified operations | - -### InputContract - -| Field | Type | Definition | -| --- | --- | --- | -| `source` | `CanonicalSourceBinding` | Versioned source definition; concrete snapshot/partition is identified by record coverage | -| `fields` | `FieldDefinition[]` | Stable field IDs, data types, nullability and optional units | -| `projection` | `TypedExpression[]` | Expressions supplying the operation's inputs | -| `filter` | `Optional` | Input qualification; absent means no additional filter | -| `observation_semantics` | `VersionedSemanticContract` | Null, NaN, duplicate and ordering interpretation | - -### GroupingContract - -| Field | Type | Definition | +| Layer | Describes | Changes when | | --- | --- | --- | -| `mode` | `Global \| PerEntity \| ByKeys` | One population, preserved entity populations, or explicit grouping | -| `keys` | `TypedExpression[]` | Group-key expressions; empty for Global | -| `key_semantics` | `VersionedSemanticContract` | Equality, canonicalization and absent/null handling | +| **Summary Descriptor** | Summary operator and fidelity guarantees | Algorithm, configuration or guarantee contract changes | +| **Data Descriptor** | Summarized source and population | Source binding or population definition changes | +| **Summary Instance** | Instance metadata and summary state | A concrete materialization is created or updated | -### SummaryTypeContract +Separating these layers lets many materialized instances reuse the same operator +configuration and data scope. A new time interval creates a new instance without +copying or redefining either descriptor. -| Field | Type | Definition | -| --- | --- | --- | -| `type_id` | `QualifiedId` | Summary semantic type | -| `type_version` | `Version` | State and operation semantics version | -| `parameters` | `TypedParameterRecord` | Parameters validated against this summary type's parameter schema | -| `state_schema` | `TypeDefinition` | Logical state structure | -| `operations` | `OperationContract[]` | Supported state construction, modification, combination and readout | - -`parameters` and operation signatures are type-specific; `item` and `weight` -are not common SDS fields. - -| Summary type | Type-specific parameter fields | Update input signature | -| --- | --- | --- | -| Exact aggregate | Aggregate components, numeric representation and overflow policy | Typed observations with component-specific qualification | -| DDSketch | Relative accuracy and supported value-domain policy | Numeric observation | -| KLL | Capacity and compaction configuration | Ordered observation | -| HLL | Precision, element encoding and hash configuration | Element | -| CMS | Width, depth, key encoding, hash configuration and increment-domain policy | Key, increment | -| CountSketch | Width, depth, key encoding, hash configuration and increment-domain policy | Key, increment | -| Heap-bearing frequency summary | Base frequency contract, candidate policy and heap capacity | Base frequency update input | +## 1. Summary Descriptor -### OperationContract +A Summary Descriptor defines **how the data is summarized** and **which fidelity +claims the summary supports**. It does not identify a source population or a +particular time interval. | Field | Type | Definition | | --- | --- | --- | -| `operation_id` | `QualifiedId` | Versioned operation definition | -| `kind` | `Build \| Update \| Merge \| Retract \| Subtract \| Readout` | Operation category; only supported operations are listed | -| `inputs` | `TypeDefinition[]` | Ordered input/state signatures | -| `arguments` | `ParameterSchema` | Typed operation arguments | -| `output` | `TypeDefinition` | Output state or result type | -| `preconditions` | `RuleRef[]` | Compatibility, coverage, ordering and provenance requirements | -| `guarantee_rules` | `RuleRef[]` | Applicable guarantee derivation/composition rules | +| `summary_descriptor_id` | `QualifiedId` | Immutable descriptor identity | +| `operator` | `SummaryOperator` | Algorithm, semantic version, parameters and supported operations | +| `fidelity` | `FidelityGuarantee[]` | Exactness or error guarantees, with their scope and conditions | +| `state_representation` | `StateRepresentation` | State type, codec and codec version | -### RepresentationContract +`SummaryOperator` contains an algorithm identifier, versioned semantics, +type-specific parameters, and supported build/update/merge/readout signatures. +Parameters and operation arguments depend on the summary type; `item` and +`weight` are not mandatory common fields. -| Field | Type | Definition | -| --- | --- | --- | -| `representation_id` | `QualifiedId` | Concrete state-layout identity | -| `codec` | `QualifiedId` | Payload codec | -| `codec_version` | `Version` | Codec version | -| `layout` | `TypeDefinition` | Encoded payload layout | -| `payload_kinds` | `Set` | Supported payload forms | +For example, a KLL operator may specify `k: 200`. The value of `k` is an +algorithm parameter, **not itself a numerical error guarantee**. Its fidelity +contract separately identifies the supported rank-error bound or versioned +bound derivation, probability of failure, readout scope and required conditions. +If that guarantee is unavailable, fidelity is explicitly `Unknown`. -### GuaranteeContract +A `FidelityGuarantee` contains: -| Field | Type | Definition | -| --- | --- | --- | -| `guarantee_id` | `QualifiedId` | Versioned guarantee definition | -| `operation` | `QualifiedId` | Operation/readout to which the guarantee applies | -| `kind` | `Exact \| DeterministicBound \| ProbabilisticBound \| Unknown` | Guarantee category | -| `error_quantity` | `Optional` | Quantity, units and normalization being bounded | -| `bound` | `Optional` | Bound or versioned bound derivation | -| `failure_probability` | `Optional` | Required for a probabilistic bound; not invented for other categories | -| `scope` | `GuaranteeScope` | Population/readout and evaluation set covered by the claim | -| `assumptions` | `RuleRef[]` | Required input, algorithm and evidence conditions | +- The applicable operation and error quantity, such as quantile rank error. +- A category: `Exact`, `DeterministicBound`, `ProbabilisticBound` or `Unknown`. +- A bound or versioned bound derivation, and a failure probability when applicable. +- The population/readout/evaluation scope and required assumptions. -## SdsIdentity +A `StateRepresentation` identifies the logical state type and versioned encoding. +Compatible bytes alone do not establish that two operators have compatible +semantics or guarantees. -| Field | Type | Definition | -| --- | --- | --- | -| `identity_id` | `QualifiedId` | Population identity; may have a compact dictionary alias | -| `schema_id` | `QualifiedId` | Referenced SDS Schema | -| `source_identity` | `TypedRecord` | Concrete source identity, including metric name where applicable | -| `group_values` | `TypedTuple` | Values matching the grouping key/entity schema | +## 2. Data Descriptor -## SdsRecord +A Data Descriptor defines **which data is summarized**. It is independent of the +summary algorithm and of a particular materialized interval. | Field | Type | Definition | | --- | --- | --- | -| `record_id` | `QualifiedId` | Record identity | -| `identity_id` | `QualifiedId` | Referenced dictionary identity | -| `coverage` | `Coverage` | Actual summarized input extent and completeness | -| `payload` | `SdsPayload` | Exactly one state or result variant | -| `provenance` | `ProvenanceRef` | Resolvable producer, generation and sequence metadata | -| `guarantee_evidence` | `GuaranteeEvidence[]` | Instance-specific evidence for applicable guarantees | - -### SdsPayload +| `data_descriptor_id` | `QualifiedId` | Immutable data-scope identity | +| `source` | `SourceBinding` | Metric/series or dataset, including its versioned field definitions | +| `population` | `PopulationDefinition` | Selection predicate and grouping/entity scope | +| `observation_semantics` | `SemanticContract` | Value projection, units and handling of missing, duplicate or invalid observations | -| Variant | Fields | -| --- | --- | -| `FullState` | `state: bytes` | -| `StateDelta` | `base_record: QualifiedId`, `apply_operation: QualifiedId`, `delta: bytes` | -| `ReadoutResult` | `operation: QualifiedId`, `arguments: TypedParameterRecord`, `value: TypedValue` | +For example, the source can be the metric `cpu_usage`, and the summarized +population can be the series satisfying `container_type="login"`. -State bytes use the referenced representation contract. A readout result uses -its operation's output type and is not implicitly mergeable state. +`PopulationDefinition` records both selection and partitioning. It distinguishes +one summary over all selected observations, independent summaries per series, +and summaries grouped by specified label keys. Concrete group values belong in +the instance metadata when one descriptor describes a reusable grouping rule. -### Coverage - -| Field | Type | Definition | -| --- | --- | --- | -| `extent` | `TimeExtent \| DatasetExtent` | Time interval or dataset snapshot/partition extent | -| `completeness` | `Complete \| Partial \| Unknown` | Coverage status, separate from mathematical accuracy | -| `evidence` | `EvidenceRef[]` | Evidence establishing coverage/freshness | +A population predicate is a typed, resolved data-selection definition. It is not +an arbitrary executable program attached to a summary. -| Extent | Fields | -| --- | --- | -| `TimeExtent` | `clock: ClockDefinition`, `start: Timestamp`, `end: Timestamp`, `bounds: IntervalBounds` | -| `DatasetExtent` | `snapshot: QualifiedId`, `partitions: TypedSet`, `selection: Optional` | +## 3. Summary Instance -### GuaranteeEvidence +A Summary Instance combines **instance metadata** with **the actual summary +state**, referencing one Summary Descriptor and one Data Descriptor. | Field | Type | Definition | | --- | --- | --- | -| `guarantee_id` | `QualifiedId` | Schema guarantee being evaluated | -| `scope` | `GuaranteeScope` | Concrete population/readout/evaluation scope | -| `status` | `Established \| Unverified \| Invalid` | Whether the conditions for this instance are established | -| `evidence` | `EvidenceRef[]` | Resolvable evidence and its validity/provenance | - -## Referenced types - -| Type | Definition | -| --- | --- | -| `QualifiedId` | Namespace plus immutable identifier | -| `Version` | Version identifier with an explicit compatibility definition | -| `TypeDefinition` | Resolvable, versioned scalar/tuple/collection/state type | -| `TypedValue / TypedTuple / TypedRecord / TypedSet` | Values whose types and field identities resolve through a TypeDefinition | -| `ParameterSchema / TypedParameterRecord` | Versioned parameter definition and values validated against it | -| `CanonicalSourceBinding` | Resolvable source definition with stable field bindings | -| `FieldDefinition` | Field ID, name, type, nullability and optional unit | -| `TypedExpression / TypedPredicate` | Canonical expression with resolved input/output types | -| `VersionedSemanticContract / RuleRef` | Versioned semantic definition or compatibility rule; not executable code supplied by a record | -| `TypedErrorDefinition / TypedBound` | Error quantity and bound with defined types, units and interpretation | -| `Probability` | Finite number in [0, 1] | -| `GuaranteeScope` | Population selector, operation arguments and covered evaluation set; distinguishes per-row, whole-result and repeated-evaluation claims | -| `ClockDefinition / Timestamp / IntervalBounds` | Clock/time unit, time value and inclusive/exclusive interval boundaries | -| `ProvenanceRef` | Reference to existing frame/materialization metadata: producer, epoch, sequence and plan generation | -| `EvidenceRef` | Immutable evidence reference including issuer, observation time, validity and subject scope | +| `instance_id` | `QualifiedId` | Materialized instance identity | +| `summary_descriptor_id` | `QualifiedId` | Referenced operator/fidelity descriptor | +| `data_descriptor_id` | `QualifiedId` | Referenced source/population descriptor | +| `metadata` | `InstanceMetadata` | Concrete extent, population binding, completeness and provenance | +| `state` | `SummaryState` | Materialized state encoded according to the Summary Descriptor | + +`InstanceMetadata` contains the concrete time range or dataset extent, any group +values needed by the population rule, completeness (`Complete`, `Partial` or +`Unknown`), producer/generation/sequence provenance and instance-specific fidelity +evidence. Time ranges specify their clock, units and interval boundaries. +Completeness is separate from mathematical approximation error. + +`SummaryState` is the state itself, not a quantile readout or other query result. +If a transport carries a delta, it must identify its base instance/version and +the descriptor's supported apply operation; it cannot be interpreted as a full +state without that context. + +## Shared-descriptor example + +The following example summarizes `cpu_usage` observations from login containers +using KLL with `k=200`. All three instances reuse the same Summary Descriptor and +Data Descriptor; only the instance time range and state change. + +```yaml +summary_descriptor: + summary_descriptor_id: example:kll-200-v1 + operator: + algorithm: KLL + parameters: {k: 200} + semantics: example:kll-semantics-v1 + fidelity: + - operation: quantile + error_quantity: rank_error + category: Unknown # No numerical guarantee is inferred from k alone. + state_representation: example:kll-state-codec-v1 + +data_descriptor: + data_descriptor_id: example:login-cpu-v1 + source: {metric: cpu_usage} + population: + predicate: {container_type: {equals: login}} + grouping: global + observation_semantics: example:cpu-observations-v1 + +instances: + - instance_id: example:login-cpu-0 + summary_descriptor_id: example:kll-200-v1 + data_descriptor_id: example:login-cpu-v1 + metadata: {time_range: "[0,10)", clock: example:seconds} + state: S0 + - instance_id: example:login-cpu-1 + summary_descriptor_id: example:kll-200-v1 + data_descriptor_id: example:login-cpu-v1 + metadata: {time_range: "[10,20)", clock: example:seconds} + state: S1 + - instance_id: example:login-cpu-2 + summary_descriptor_id: example:kll-200-v1 + data_descriptor_id: example:login-cpu-v1 + metadata: {time_range: "[20,30)", clock: example:seconds} + state: S2 +``` + +`S0`, `S1` and `S2` denote separate encoded KLL states. The example omits concrete +payload bytes and producer evidence; it makes no completeness or numerical error +claim. Descriptor references must resolve within the supplied context or a +durably retained descriptor registry. + +Changing `k` creates a new Summary Descriptor. Changing the source or population +creates a new Data Descriptor. Advancing the time range creates a new Summary +Instance. Merge compatibility additionally requires the operator's merge rules, +compatible data scopes and valid instance coverage; sharing descriptors alone +does not authorize merging overlapping observations. From 65bc97f49d1e8e5280d26f060ae736d9c4f220c8 Mon Sep 17 00:00:00 2001 From: zz_y Date: Tue, 8 Sep 2026 14:16:38 -0600 Subject: [PATCH 4/4] docs: clarify proposed Planner and backend SDS ownership --- docs/design_docs/self-describing-summary.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/docs/design_docs/self-describing-summary.md b/docs/design_docs/self-describing-summary.md index 61313f49..73924149 100644 --- a/docs/design_docs/self-describing-summary.md +++ b/docs/design_docs/self-describing-summary.md @@ -13,6 +13,25 @@ Separating these layers lets many materialized instances reuse the same operator configuration and data scope. A new time interval creates a new instance without copying or redefining either descriptor. +## Proposed ownership + +The descriptor vocabulary belongs in shared semantic contracts, suitable for +Planner's shared types. Planner reasons about operators, fidelity, source and +population semantics. The backend binds those descriptions to actual series, +filters and grouping, and owns materialized instance state and its lifecycle. + +| Layer | Proposed responsibility | +| --- | --- | +| Summary Descriptor | Shared semantic definition used by Planner and backend | +| Data Descriptor | Shared source/population definition; backend resolves concrete runtime bindings | +| Summary Instance | Backend owns metadata, state, updates, storage and retirement | + +Planner may observe instance availability, covered time ranges and descriptor +references as planning evidence. It does not need the encoded summary state. +SDS describes summaries; an installed QueryPlan specifies how to execute a query +using them. This ownership split is a proposal, not a claim that SDS types or +interfaces already exist in either repository. + ## 1. Summary Descriptor A Summary Descriptor defines **how the data is summarized** and **which fidelity