diff --git a/CHANGELOG.md b/CHANGELOG.md index 5761d8f6..3e212abc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -540,6 +540,39 @@ AgentExperience.NET is a **preview**. It is not production ready, and public API ships, in which the MCP SDK lifts two library floors (DI abstractions 10.0.13, `Microsoft.Extensions.AI.Abstractions` 10.11.0) inside the tool only. Guide: [CLI and MCP](docs/guide/cli-and-mcp.md). +### One living lesson per situation (story 28.6) + +- **`ExperienceConsolidationService.ConsolidateAsync(authorization, scope, options, ct)`** (Core, registered by the + one-call setup; `provider.GetAgentExperienceLessons().ConsolidateAsync(identity)` from the lessons view) merges the + validated records of one scope that failed the same way and were fixed the same way into one consolidated record. + Only records the library finalized and can vouch for are sources: `Origin == Finalized`, a verifying provenance + signature when signing is on, verified, with a library-written, confirmed lesson; never a hand-written record, never + another scope's or a grant's. The grouping key is exact: the set of resolved failure signature keys (story 28.2) and + the working attempt's tool sequence, plus the values of the argument keys `ExperienceConsolidationOptions.ArgumentKeys` + names (the recipe's by default from the lessons view), so `retry(delay=0)` and `retry(delay=30)` stay two lessons. A + group merges once it holds records from `MinimumSources` (3) distinct runs. Deterministic and explicit: no model, + nothing runs it automatically. +- **The consolidated record** renders as its most recent source does (its content is copied from that source only), + with the lesson `Verified in N independent runs (latest yyyy-MM-dd). Failed first with from in K of + them. Worked: …` (template version 1). Its counters are verified evidence only, deduplicated by independence key: one + validation per source, plus the `Verified` updates in the sources' histories; host-trusted evidence and raised + counters are never carried. `ReadConfidenceAsync(VerifiedOnly)` counts them as verified when signing is on and the + record's signature (over the producer and the source list) verifies. Its ID is derived from its source IDs, so a + re-run is idempotent and a later verified run is folded in by a new record that supersedes the old one. It is marked + in its provenance (`ExperienceConsolidation.ProvenanceSource`; `Provenance.ExposedTo` lists its sources, a signed + claim), so no store, schema, port or contract changes. +- **Its sources are superseded** through the ordinary lifecycle: re-read right before the record is created (a changed + one is left out), then one compare-and-set commit each; a source that changed in between is left untouched and + reported. Until then the record and its sources coexist: `ExperienceConsolidationGroupResult.Complete` and the + result's `IncompleteGroups` say so, and the next pass finishes. A record already at the derived ID is adopted only when + its signature, producer, sources, lesson and starting counters match. Groups beyond 256 sources, or whose records kept + changing, are reported `Deferred`. +- **Erasure does not reach a copy**: `FindConsolidatedContainingAsync` (service and lessons view) finds the consolidated + records that list a source, to erase with it; the next pass leaves an erased source out. Contesting or revoking a + consolidated record works as for any record; its sources stay superseded. The lessons view shows + `ExperienceLesson.ConsolidatedFrom` ("consolidated from N" on the HTML page), and the benefit estimate treats a + consolidated lesson as a new lesson: holdout rows that named its sources stay as they are. + ## 0.1.0-preview.17 This preview measures whether injected lessons help. An opt-in holdout withholds the block on a deterministic share of diff --git a/README.md b/README.md index 46da61c8..f9bbe6cb 100644 --- a/README.md +++ b/README.md @@ -226,7 +226,9 @@ stored (`ListAsync(identity)` with your own identity in production): status, con whether retrieval would use it and why not, and the block an agent there would be shown for it. `ExperienceLessonsHtml.Render(page)` turns that into one self-contained HTML page. The library maps no endpoint; serve it behind your own authorization, because it discloses the lessons. See -[Seeing what it learned](docs/guide/deployment.md#seeing-what-it-learned). +[Seeing what it learned](docs/guide/deployment.md#seeing-what-it-learned). When many runs learned the same thing, +`ConsolidateAsync(identity)` on the same view merges them into one lesson that says how many runs confirmed it (see +[Consolidating repeated lessons](docs/guide/lifecycle.md#consolidating-repeated-lessons)). ## Choosing a reflector diff --git a/docs/guide/deletion-and-retention.md b/docs/guide/deletion-and-retention.md index 969dbe75..3bfe0364 100644 --- a/docs/guide/deletion-and-retention.md +++ b/docs/guide/deletion-and-retention.md @@ -13,6 +13,12 @@ the KL-2 boundary in [Known limits and documented boundaries](../known-limits.md ## Deleting a record +> **Consolidated lessons copy content.** A [consolidated lesson](lifecycle.md#consolidating-repeated-lessons) carries a +> copy of its most recent source's task, attempts, outcome and reflection, and lists every source's ID. Erasing a +> source does **not** reach a consolidated record made from it. Before erasing a record, call +> `FindConsolidatedContainingAsync` (on `ExperienceConsolidationService`, or `GetAgentExperienceLessons()` with the +> identity) and erase the records it returns too; the next consolidation pass leaves the erased source out. + ```csharp var store = new PostgresExperienceRecordStore(dataSource); @@ -353,6 +359,9 @@ below shrink to the derived search data, and that page says exactly what remains - **Exported telemetry.** Spans and metrics this library emitted carry record IDs; erasing a record does not retract them. The erasure's own telemetry adds nothing that was erased (see [below](#telemetry)). - **External artifacts a record merely named.** Tickets, logs, commits: the library never held them. +- **Consolidated records made from the erased one.** A consolidated record copies its latest source's content and + lists every source's ID; erase it too (`FindConsolidatedContainingAsync` finds it, see + [above](#deleting-a-record)). - **The dead heap tuple — until `VACUUM`, the erased text is still in this database.** The tombstone is written with an `UPDATE`, and an `UPDATE` in PostgreSQL writes a new row version and leaves the old one in the heap. Until `VACUUM` reclaims it, the previous version of the record row still carries the task summary, the lesson, diff --git a/docs/guide/deployment.md b/docs/guide/deployment.md index 689f58ea..1df8d0d7 100644 --- a/docs/guide/deployment.md +++ b/docs/guide/deployment.md @@ -277,7 +277,10 @@ each lesson, newest first, it gives: page's `InjectionsTracked` is `true`); and not tracked (`InjectionsTracked` is `false`: a store that does not track injections, or a PostgreSQL schema whose application role has not been granted `0026`'s table yet), which the HTML shows as "not tracked" rather than "never". .NET, Python and Go agents all record their injections (the ports from - [contract 1.4](../../contract/README-1.4.md)), so in a mixed fleet "never" means no agent in the scope was shown it. + [contract 1.4](../../contract/README-1.4.md)), so in a mixed fleet "never" means no agent in the scope was shown it; +- `ConsolidatedFrom`, for a lesson a consolidation pass merged (`ConsolidateAsync(identity)` on the same view): how + many source records it was merged from, each now superseded by it. See + [Consolidating repeated lessons](lifecycle.md#consolidating-repeated-lessons). `ExperienceLessonsQuery` narrows it: `Statuses`, `Limit` (the store's bounds, 1 to 500, default 50) and `IncludeIneligible` (default `true`). The page's `HasMore` says whether more lessons match than it shows (or may: the diff --git a/docs/guide/lifecycle.md b/docs/guide/lifecycle.md index bd19904a..47b79c36 100644 --- a/docs/guide/lifecycle.md +++ b/docs/guide/lifecycle.md @@ -224,6 +224,110 @@ cannot abort on a pre-`0006` `Superseded` event that has no replacement — one store never applied Core's table. New and updated rows are checked from that moment on. The script's header carries the reconciliation query and the `VALIDATE CONSTRAINT` statements to run once it comes back empty. +## Consolidating repeated lessons + +Every verified run becomes its own record, so fifty runs that hit the same failure and fixed it the same way would be +fifty lessons, each starting at confidence 2/3 and competing for the block's slots. A consolidation pass turns them +into one lesson that says how many runs confirmed it. It is explicit and deterministic: nothing runs it automatically, +no model is involved, and it is never on the request path. + +```csharp +// The one-call setup: the lessons view runs it in the identity's own scope. +var result = await provider.GetAgentExperienceLessons().ConsolidateAsync(identity); + +// Or Core directly, with your own options. +var result = await consolidation.ConsolidateAsync( + authorization, + scope, + new ExperienceConsolidationOptions { MinimumSources = 3, ArgumentKeys = new Dictionary> { ["retry"] = ["delay"] } }, + cancellationToken); +``` + +**Which records are sources.** Only records the library finalized and can still vouch for: in exactly the scope +asked for (never another scope's, never one a sharing grant lends), `Validated` or `Reinforced`, verified, +`Origin == Finalized`, and, when [provenance signing](confidence.md#signing-provenance) is on, carrying a signature that +verifies (or listed in `TrustUnsignedRecordIds`), with a library-written lesson whose content is confirmed. That is the +rule confidence verification applies to a run's record. A record written through the store port by hand is never a +source, whatever it claims. The pass reads at most 500 such records, newest first (`MayHaveMore` says when there were +more). + +**What is one situation.** Two sources are one situation when they hold the same set of *resolved* +[failure signatures](injection.md#recall-at-the-moment-of-failure) (the exact `ExperienceFailureSignature.Key`) and +their working attempt (the final one, ended without an error: the attempt the block's `Worked:` line names) called the +same tools in the same order, with equal values for the argument keys `ArgumentKeys` names. Name the keys your blocks +show (the lessons view defaults to the keys `options.Recipe` made visible), so `retry(delay=0)` and `retry(delay=30)` +stay two lessons. A group is merged once it holds at least two records from `MinimumSources` (default 3) distinct runs. + +**The consolidated record.** A new record whose ID is derived from its source IDs +(`ExperienceConsolidation.ConsolidatedIdFor`), so re-running a pass finds the record it made. Its task, summary, +attempts, outcome, environment and reflection fields are its most recent source's (nothing is copied from the older +ones), so its `Tried:` and `Worked:` lines and recipe render as that source's did. Its lesson reads `Verified in N +independent runs (latest 2026-10-07). Failed first with TimeoutException from run_tests in N of them. Worked: run_tests +→ run_tests.` (template version 1). It is marked in its provenance, so no store, schema, port or contract changes: +`Provenance.Source` is `AgentExperience.Consolidation`, its reflection's producer `AgentExperience.Consolidation/1` +(signed content), and `Provenance.ExposedTo` lists its source records (at most 256) at the revision the pass read them. +`ExperienceConsolidation.IsConsolidated` and `SourcesOf` read it back. Its `Origin` stays `HostWritten` and its +`SourceRunId` is a derived identifier no run has: it vouches for no run. With provenance signing it is signed like a +finalized record, and since the signed claims cover `ExposedTo`, its source list cannot be changed without the +signature failing. + +**Its counters are verified evidence only**, deduplicated by +[independence key](confidence.md#independence-is-keyed-and-the-database-owns-the-key) rather than summed by record: each +source's own validation, never more than the one finalization sets, under its run and closed round; and every counted +update admitted `Verified` in each source's history and in each merged consolidated record's own. Host-trusted evidence +and evidence with no recorded admission are never carried, and neither is any counter someone raised in the store. So a +run that produced one source and was also counted as reuse of another counts once. The record is created at the score +those counters explain, as finalization creates every record; your confidence engine scores it from its first evidence +on. Reading it back: `ReadConfidenceAsync` with `ExcludeHostTrusted` reports what is stored. With `VerifiedOnly`, a +consolidated record counts its initial counters as verified only when signing is configured and its signature verifies +(it then covers the producer and the source list), capped at one per listed source; without signing it is treated like +any record written without finalization, and its initial counters are left out. + +**Supersession.** Right before the record is created, the pass reads its sources again and leaves out any that changed +since it first read them (the group is re-evaluated; if they keep changing the group is reported `Deferred`). The +record is created as a `Candidate` and validated, and then each source is moved to `Superseded` by it through +`CommitAsync`, one commit per source, at the revision and status the pass read. A source that changed in between (a +contradiction, a revocation, a reinforcement) is left untouched and reported with its `StaleRevision` or +`StatusMismatch`; each commit is the store's own transaction, so a source is either superseded or untouched, never +half. Each source's history ends with the supersession that names the consolidated record and the producer +`AgentExperience.Consolidation/1`. + +**The window where both exist.** Between the record's validation and the last supersession, the consolidated record and +its sources are all eligible, and retrieval can return both. A pass that stops in that window (a crash, a refused +commit, a source that changed) leaves them coexisting until the next pass: its group's `Complete` is `false` and the +result's `IncompleteGroups` counts it. Every event ID and timestamp is derived, so an interrupted pass, or two passes at +once, converge on the same record, and the next pass finishes the supersessions (`ExperienceConsolidationGroupOutcome.Existing`). +A record already stored under the derived ID is adopted only when it is a consolidated record this library would +accept as a source (producer, signature, confirmed content) over the same sources, with the same lesson and the same +starting counters the pass computed; otherwise the group is `Refused` and nothing is superseded. + +**Reversal and later runs.** A consolidated record is an ordinary record: contest or revoke it like any other. Its +sources stay superseded (the transition table never moves a superseded record back), so withdrawing a consolidated +lesson withdraws the situation's lesson. A later verified run in the same situation is finalized into an ordinary +record; the next pass folds it in, creating a record over the larger source set (a new derived ID) that supersedes the +old consolidated record and the new one. A source the old record lists is carried over only when it is still a trusted +source of the same situation that the old record superseded: one revoked or erased since is left out of the list, the +run count and the counters. + +**Erasure does not reach a copy.** A consolidated record copies its most recent source's content and lists every +source's ID, so erasing a source leaves that copy in place. Before erasing a record, call +`FindConsolidatedContainingAsync` (on the service, or on `GetAgentExperienceLessons()` with the identity), which +follows the source's supersession chain and scans the scope's 500 most recent records, and erase what it returns too +(see [Deletion and retention](deletion-and-retention.md#deleting-a-record)). + +**Limits and deferral.** A consolidated record lists at most 256 sources. A group with more takes the consolidated +records' sources and the newest records that fit, and reports the rest in `Deferred`; once a consolidated record lists +256 sources, further runs of the same situation stay their own records, and their group is reported with the outcome +`Deferred`. + +**Measurement.** Exposure rows and holdout assignments that named a source stay as they are. The +[benefit estimate](injection.md#measuring-benefit-the-holdout) is per lesson, so a consolidated record is a new lesson +with no measurements of its own yet. The lessons view shows a consolidated lesson's `ConsolidatedFrom` ("consolidated +from N" on the HTML page). + +**Not done here.** The Python and Go ports read, render and move consolidated records like any other, but do not run a +pass. + ## In Python and Go Since contract 1.16 ([what it adds](../../contract/README-1.16.md)) both ports move a stored record through this diff --git a/src/AgentExperience.Core/Confidence/ProvenanceSigning.cs b/src/AgentExperience.Core/Confidence/ProvenanceSigning.cs index 6ba45f3d..d5763e3d 100644 --- a/src/AgentExperience.Core/Confidence/ProvenanceSigning.cs +++ b/src/AgentExperience.Core/Confidence/ProvenanceSigning.cs @@ -605,6 +605,19 @@ private static byte[] Guarded(Func encode) } } + /// + /// One tool call argument value's canonical encoding, exactly as the content digest encodes it: so two values that + /// sign alike (1.5 and 1.50, an enum and its name) encode alike here too. Consolidation keys an + /// allowlisted argument by it. + /// + /// The value has no JSON form, or is nested too deeply. + internal static byte[] EncodeArgumentValue(object? value) => Guarded(() => + { + using var buffer = new MemoryStream(); + WriteValue(buffer, ToJson(value), depth: 0); + return buffer.ToArray(); + }); + private static byte[] EncodeClaims(ExperienceRecord record) { using var buffer = new MemoryStream(); diff --git a/src/AgentExperience.Core/DependencyInjection/AgentExperienceCoreServiceCollectionExtensions.cs b/src/AgentExperience.Core/DependencyInjection/AgentExperienceCoreServiceCollectionExtensions.cs index 21964eff..2ae8a490 100644 --- a/src/AgentExperience.Core/DependencyInjection/AgentExperienceCoreServiceCollectionExtensions.cs +++ b/src/AgentExperience.Core/DependencyInjection/AgentExperienceCoreServiceCollectionExtensions.cs @@ -24,7 +24,8 @@ public static class AgentExperienceCoreServiceCollectionExtensions { /// /// Registers the sanitizer, the in-memory capture service, the default reflector, the lifecycle - /// service, and the finalization service as singletons. + /// service, the finalization service and the consolidation service (it runs only when the host + /// calls it) as singletons. /// /// /// @@ -160,6 +161,12 @@ internal static IServiceCollection AddAgentExperienceCoreWithIndexing( provider.GetService(), provider.GetService())); + // Story 28.6: on demand only; nothing resolves or runs it unless the host does. + services.TryAddSingleton(provider => new ExperienceConsolidationService( + provider.GetRequiredService(), + provider.GetRequiredService(), + provider.GetService())); + return services; } diff --git a/src/AgentExperience.Core/Lifecycle/ConsolidationResults.cs b/src/AgentExperience.Core/Lifecycle/ConsolidationResults.cs new file mode 100644 index 00000000..03984448 --- /dev/null +++ b/src/AgentExperience.Core/Lifecycle/ConsolidationResults.cs @@ -0,0 +1,325 @@ +using System.Buffers.Binary; +using System.Security.Cryptography; +using System.Text; +using AgentExperience.Abstractions; + +namespace AgentExperience.Core.Lifecycle; + +/// +/// What one consolidation pass () merges, and how. +/// +public sealed record ExperienceConsolidationOptions +{ + /// The default . + public const int DefaultMinimumSources = 3; + + /// + /// How many distinct source runs a group needs before it is merged into one consolidated record: from 2 to + /// ; (3) by default. Counted over + /// the runs the group's records came from, so a consolidated record in the group counts every run it was merged from. + /// + public int MinimumSources { get; init; } = DefaultMinimumSources; + + /// + /// Per tool name, the argument keys whose values on the working attempt's calls are part of the grouping key, so + /// retry(delay=0) and retry(delay=30) stay two lessons. (the default) groups on + /// tool names alone. Name the keys the Historical Reference block shows (the injection's approach-argument allowlist), + /// so a merged lesson never hides a difference its sources' blocks showed. Tool names and keys are matched ordinally; + /// values are compared by their canonical encoding (the one provenance signing uses), never by their text. + /// + public IReadOnlyDictionary>? ArgumentKeys { get; init; } + + /// Throws when the options cannot be used. + internal void Validate(string parameterName) + { + if (MinimumSources is < 2 or > ExperienceConsolidation.MaxSources) + { + throw new ArgumentOutOfRangeException( + parameterName, + MinimumSources, + $"{nameof(MinimumSources)} must be from 2 to {ExperienceConsolidation.MaxSources}."); + } + + if (ArgumentKeys is null) + { + return; + } + + foreach (var (tool, keys) in ArgumentKeys) + { + if (string.IsNullOrWhiteSpace(tool) || keys is null || keys.Any(string.IsNullOrWhiteSpace)) + { + throw new ArgumentException( + $"{nameof(ArgumentKeys)} must name non-blank tools, each with a non-null list of non-blank argument keys.", + parameterName); + } + } + } +} + +/// +/// The fixed names and rules of a consolidated record: how one is recognised, what it lists, and the ID a set of +/// sources derives. +/// +/// +/// +/// A consolidated record is an ordinary written by +/// . It is marked in its provenance rather than by a new +/// value, so no store, schema or port changes: its +/// is , its is +/// , its reflection's is , and its +/// lists its source records, each at the revision consolidation read it (at most +/// ). Its stays : +/// it was not derived from one captured run, so it vouches for none. Its is a +/// derived identifier no run has. +/// +/// +/// With provenance signing configured it is signed like a finalized record, and the signed claims include +/// : the source list cannot be changed without the signature failing. +/// +/// +public static class ExperienceConsolidation +{ + /// The of every consolidated record. + public const string ProvenanceSource = "AgentExperience.Consolidation"; + + /// The version of the lesson template (and of the record's shape), as its . + public const string TemplateVersion = "1"; + + /// The of every consolidated record's reflection, and the producer of the lifecycle events consolidation commits. + public const string Producer = ProvenanceSource + "/" + TemplateVersion; + + /// The most source records one consolidated record lists: , the bound on . + public const int MaxSources = RunExposure.MaxPerRun; + + /// The most preconditions a consolidated record's reflection carries: its most recent source's, cut to this many. + public const int MaxPreconditions = 16; + + /// The most tool names the lesson text's Worked: sentence names; the rest are counted. + public const int MaxLessonToolNames = 16; + + /// The namespace every consolidation identifier is derived under. + private static readonly Guid DerivationNamespace = Guid.Parse("6f1d2c0e-28a6-4c57-9b1e-0c0a5d1e2860"); + + private const byte RecordIdTag = 1; + private const byte RunIdTag = 2; + private const byte ReflectionIdTag = 3; + private const byte ValidateEventTag = 4; + private const byte SupersedeEventTag = 5; + + /// + /// Whether is a consolidated record: its provenance source is + /// and its reflection's producer starts with AgentExperience.Consolidation/. + /// + /// A record. + /// Whether consolidation wrote it. + /// is . + public static bool IsConsolidated(ExperienceRecord record) + { + ArgumentNullException.ThrowIfNull(record); + return string.Equals(record.Provenance?.Source, ProvenanceSource, StringComparison.Ordinal) + && record.Reflection?.Producer is { } producer + && producer.StartsWith(ProvenanceSource + "/", StringComparison.Ordinal); + } + + /// The source records a consolidated record lists, in ID order; empty for any other record. + /// A record. + /// The IDs of the records it was merged from. + /// is . + public static IReadOnlyList SourcesOf(ExperienceRecord record) => + IsConsolidated(record) + ? [.. (record.Provenance.ExposedTo ?? []).Where(exposure => exposure is not null).Select(exposure => exposure.ExperienceId).Distinct().Order()] + : []; + + /// + /// The ID of the consolidated record merging exactly in : a + /// SHA-256 of a fixed namespace, every scope field (length-prefixed) and the sorted, distinct source IDs, as a + /// version-8 GUID. The same sources always derive the same ID, so a pass re-run over them finds its own record; a + /// new source folded in derives a new one. + /// + /// The scope the sources lie in. + /// The source records. + /// The derived ID. + /// An argument is . + public static Guid ConsolidatedIdFor(Scope scope, IEnumerable sourceIds) + { + ArgumentNullException.ThrowIfNull(scope); + ArgumentNullException.ThrowIfNull(sourceIds); + var sorted = sourceIds.Distinct().Order().ToList(); + var suffix = new byte[sorted.Count * 16]; + for (var i = 0; i < sorted.Count; i++) + { + sorted[i].TryWriteBytes(suffix.AsSpan(i * 16, 16), bigEndian: true, out _); + } + + return Derive(RecordIdTag, scope, Guid.Empty, suffix); + } + + /// The of the consolidated record : derived, and no run's. + internal static Guid RunIdFor(Scope scope, Guid consolidatedId) => Derive(RunIdTag, scope, consolidatedId, []); + + /// The of the consolidated record . + internal static Guid ReflectionIdFor(Scope scope, Guid consolidatedId) => Derive(ReflectionIdTag, scope, consolidatedId, []); + + /// The event ID of the commit that validates the consolidated record . + internal static Guid ValidateEventIdFor(Scope scope, Guid consolidatedId) => Derive(ValidateEventTag, scope, consolidatedId, []); + + /// The event ID of the commit that supersedes by . + internal static Guid SupersedeEventIdFor(Scope scope, Guid consolidatedId, Guid sourceId) + { + var suffix = new byte[16]; + sourceId.TryWriteBytes(suffix, bigEndian: true, out _); + return Derive(SupersedeEventTag, scope, consolidatedId, suffix); + } + + private static Guid Derive(byte tag, Scope scope, Guid subject, byte[] suffix) + { + using var input = new MemoryStream(); + Span guid = stackalloc byte[16]; + Span length = stackalloc byte[4]; + DerivationNamespace.TryWriteBytes(guid, bigEndian: true, out _); + input.Write(guid); + input.WriteByte(tag); + subject.TryWriteBytes(guid, bigEndian: true, out _); + input.Write(guid); + foreach (var field in new[] { scope.TenantId, scope.ApplicationId, scope.ProjectId, scope.TeamId, scope.AgentId, scope.UserId }) + { + if (field is null) + { + input.WriteByte(0); + continue; + } + + var bytes = Encoding.UTF8.GetBytes(field); + BinaryPrimitives.WriteInt32BigEndian(length, bytes.Length); + input.WriteByte(1); + input.Write(length); + input.Write(bytes); + } + + input.Write(suffix); + var hash = SHA256.HashData(input.ToArray()); + var id = hash.AsSpan(0, 16); + id[6] = (byte)((id[6] & 0x0F) | 0x80); + id[8] = (byte)((id[8] & 0x3F) | 0x80); + return new Guid(id, bigEndian: true); + } +} + +/// What one consolidation pass did in one scope. +/// The scope consolidated: exactly the one asked for. +/// +/// The store's outcome for the pass's read: , or why nothing was read (the scope +/// lies outside the authorization, say). Per-group and per-source outcomes are in . +/// +/// Every group the pass acted on, in grouping-key order. Groups with nothing to do are not listed. +public sealed record ExperienceConsolidationResult( + Scope Scope, + ExperienceStoreOutcome Outcome, + IReadOnlyList Groups) +{ + /// How many validated or reinforced records the pass read. + public int RecordsRead { get; init; } + + /// + /// Whether the scope may hold more validated or reinforced records than one store read answers + /// (, newest first): the older ones were not considered this pass. + /// + public bool MayHaveMore { get; init; } + + /// The store's validation errors when is . + public IReadOnlyList Errors { get; init; } = []; + + /// + /// How many groups are incomplete: not (a consolidated record + /// coexists with sources it did not supersede, or the group was refused or deferred). Zero when the pass finished + /// everything it started; otherwise run it again. + /// + public int IncompleteGroups => Groups.Count(group => group is not null && !group.Complete); +} + +/// What happened to one group: records in one scope with the same resolved failure signature and the same working sequence. +/// The consolidated record the group's records are superseded by. +/// Whether that record was created now, already existed, or could not be used. +/// How many distinct source runs it was merged from. +/// Each record this pass tried to supersede by it, and the outcome; empty when is . +public sealed record ExperienceConsolidationGroupResult( + Guid ConsolidatedExperienceId, + ExperienceConsolidationGroupOutcome Outcome, + int IndependentRuns, + IReadOnlyList Sources) +{ + /// Why, in words, when is . + public string? Reason { get; init; } + + /// + /// Records of the group left for a later pass: because the consolidated record would have listed more than + /// sources, or, with , + /// because the group could not be merged this pass. + /// + public int Deferred { get; init; } + + /// + /// Whether the group is finished: its consolidated record exists and every record this pass tried to supersede by it + /// is superseded, and nothing was deferred. means the consolidated record and some of its + /// sources coexist (both can be retrieved) until a later pass finishes, or the group was refused or deferred. + /// + public bool Complete => + Outcome is ExperienceConsolidationGroupOutcome.Created or ExperienceConsolidationGroupOutcome.Existing + && Deferred == 0 + && Sources.All(source => source is { Superseded: true }); +} + +/// How a group's consolidated record came to be used. +public enum ExperienceConsolidationGroupOutcome +{ + /// This pass created and validated the consolidated record. + Created, + + /// + /// The consolidated record already existed (an earlier pass, or a concurrent one, created it from the same sources), + /// and this pass superseded what that pass had not. + /// + Existing, + + /// + /// The consolidated record could not be created or used (the store refused it, its ID names another record, or it is + /// no longer eligible), so nothing was superseded. See . + /// + Refused, + + /// + /// The group was not merged this pass: its sources would exceed , or + /// its records kept changing while the pass read them. counts + /// the records left; is . + /// + Deferred, +} + +/// What found. +/// The store's outcome for the read: , or why nothing was read. +/// The consolidated records that list the source, in ID order. +public sealed record ExperienceConsolidatedLookupResult(ExperienceStoreOutcome Outcome, IReadOnlyList ExperienceIds) +{ + /// + /// Whether the scope holds more records than one store read answers, so a consolidated record off the source's + /// supersession chain and older than the most recent may have been missed. + /// + public bool MayHaveMore { get; init; } +} + +/// One record a pass tried to supersede by a consolidated record. +/// The source record. +/// +/// The lifecycle commit's outcome: when it is now superseded; any other +/// value (a or +/// when it changed after the pass read it) when it was left untouched. when the commit threw. +/// +public sealed record ExperienceConsolidationSourceResult(Guid ExperienceId, LifecycleTransitionOutcome? Outcome) +{ + /// Whether the record is now superseded by the consolidated record. + public bool Superseded => Outcome == LifecycleTransitionOutcome.Committed; + + /// The exception's type name when the commit threw; the record is untouched or superseded as the store left it, and a later pass finishes it. + public string? ExceptionType { get; init; } +} diff --git a/src/AgentExperience.Core/Lifecycle/ExperienceConsolidationService.cs b/src/AgentExperience.Core/Lifecycle/ExperienceConsolidationService.cs new file mode 100644 index 00000000..a489a181 --- /dev/null +++ b/src/AgentExperience.Core/Lifecycle/ExperienceConsolidationService.cs @@ -0,0 +1,917 @@ +using System.Globalization; +using System.Security.Cryptography; +using System.Text; +using AgentExperience.Abstractions; +using AgentExperience.Core.Confidence; +using AgentExperience.Core.Finalization; +using AgentExperience.Core.Reflections; + +namespace AgentExperience.Core.Lifecycle; + +/// +/// One living lesson per situation: an on-demand, deterministic pass that merges the validated records of +/// one scope that failed the same way and were fixed the same way into one consolidated record, which supersedes them +/// through the ordinary lifecycle and carries their combined, verified, independent support. +/// +/// +/// +/// Never automatic, never a model. Nothing calls it on the request path or in the background; the host calls +/// . Every field of a consolidated record is derived from its sources by fixed rules. +/// +/// +/// Which records are sources. Only records the library itself finalized and can still vouch for: in exactly the +/// scope asked for (never another scope's, never one a grant lends), or +/// , verified, , with a provenance +/// signature that verifies (or listed as trusted unsigned) when signing is configured -- the rule confidence verification +/// applies to a run's record -- and a reflection that is not model-authored, with confirmed content. A hand-written record +/// is never a source. The pass reads at most records, newest first. +/// +/// +/// The grouping key. Two sources group together when they hold the same set of resolved failure signature +/// keys (the exact ) and their working attempt -- the final attempt, ended +/// without an error, the one the block's Worked: line names -- called the same tools in the same order, with the +/// same values for the argument keys names. A group is merged +/// when it has at least two records and they came from at least +/// distinct runs. A consolidated record in the group stands for the sources it lists that are still trusted sources of +/// the same key and that it superseded (a listed source revoked or erased since is left out). +/// +/// +/// The consolidated record. Its ID is over its sources. +/// Its task, summary, attempts, outcome, environment, approaches, preconditions, warnings and reuse guidance are its most +/// recent source's; its lesson is the template Verified in N independent runs (latest yyyy-MM-dd). Failed first with +/// <class> from <tool> in K of them. Worked: <tool> → <tool>. +/// (). Its counters are verified evidence only, deduplicated by +/// independence key: each source's own validation (at most +/// , under its run and closed round) and every +/// counted update admitted in each source's history and in each merged +/// consolidated record's own; host-trusted and unrecorded-admission evidence is never carried. +/// +/// +/// Supersession. Right before the record is created, the sources are read again and any that changed is left out +/// (the group is re-evaluated). The record is created as a and validated, then +/// each source is moved to by it, one compare-and-set commit per source. Until +/// those commits land the record and its sources coexist; a source that changed meanwhile is left untouched and +/// reported, and is . Every identifier and +/// timestamp is derived, so a re-run or a concurrent pass converges on the same record. A record already stored under +/// the derived ID is adopted only when it is a trusted consolidated record whose lesson and counters equal those the +/// pass computed. +/// +/// +public sealed class ExperienceConsolidationService +{ + private const int MaxRevalidations = 3; + + private const int MaxChainHops = 64; + + private static readonly ExperienceReadOptions ScopeCheckRead = new(ExperienceReadPurpose.ScopeCheck); + + private static readonly IReadOnlyList Mergeable = [ExperienceStatus.Validated, ExperienceStatus.Reinforced]; + + private readonly IExperienceRecordStore _store; + private readonly ExperienceLifecycleService _lifecycle; + private readonly TimeProvider _clock; + + /// Creates a consolidation service. + /// The record store the pass reads from and creates consolidated records in. + /// + /// The lifecycle service the supersessions are committed through. Its provenance signing, when configured, signs the + /// consolidated record and decides which records are vouched for. + /// + /// The clock a consolidated record's creation time is read from; for the system clock. + /// or is . + public ExperienceConsolidationService(IExperienceRecordStore store, ExperienceLifecycleService lifecycle, TimeProvider? clock = null) + { + ArgumentNullException.ThrowIfNull(store); + ArgumentNullException.ThrowIfNull(lifecycle); + _store = store; + _lifecycle = lifecycle; + _clock = clock ?? TimeProvider.System; + } + + private ProvenanceSigner? Signer => _lifecycle.Independence.Signer; + + /// + /// Runs one consolidation pass over : groups its trusted validated records, creates (or finds) + /// each group's consolidated record, and supersedes the group's records by it. See the type's remarks for every rule. + /// + /// What the host has established the caller may do; passed to the store unchanged. + /// The exact scope to consolidate. Never treated as authority. + /// What to merge. + /// Cancels the pass; commits already made stay made, and a re-run finishes the rest. + /// + /// What the pass did, per group and per source. A store refusal of the read (the scope outside the authorization, say) + /// is a result carrying that outcome, never an exception. + /// + /// An argument is . + /// is out of range. + /// names a blank tool or key. + /// Storage infrastructure failed while reading or creating; supersession failures are reported per source instead. + /// was cancelled. + public async Task ConsolidateAsync( + AuthorizationContext authorization, + Scope scope, + ExperienceConsolidationOptions options, + CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(authorization); + ArgumentNullException.ThrowIfNull(scope); + ArgumentNullException.ThrowIfNull(options); + options.Validate(nameof(options)); + + var read = await _store + .QueryAsync(authorization, new ExperienceRecordQuery(scope, Mergeable, ExperienceRecordQuery.MaxLimit), cancellationToken) + .ConfigureAwait(false); + if (read.Outcome != ExperienceStoreOutcome.Found) + { + return new(scope, read.Outcome, []) { Errors = read.Errors ?? [] }; + } + + var records = (read.Records ?? []).Where(record => record is not null).ToList(); + var groups = new SortedDictionary>(StringComparer.Ordinal); + foreach (var record in records) + { + if (record.Status is not (ExperienceStatus.Validated or ExperienceStatus.Reinforced) + || !(IsTrustedSource(record, scope) || IsTrustedConsolidated(record, scope)) + || GroupKey(record, options) is not { } key) + { + continue; + } + + if (!groups.TryGetValue(key.Value, out var members)) + { + groups[key.Value] = members = []; + } + + members.Add(record); + } + + var results = new List(); + foreach (var (key, members) in groups) + { + cancellationToken.ThrowIfCancellationRequested(); + if (await ConsolidateGroupAsync(authorization, scope, key, members, options, cancellationToken).ConfigureAwait(false) is { } result) + { + results.Add(result); + } + } + + return new(scope, ExperienceStoreOutcome.Found, results) + { + RecordsRead = records.Count, + MayHaveMore = records.Count >= ExperienceRecordQuery.MaxLimit, + }; + } + + /// + /// The consolidated records in that list as a source, so a host + /// erasing that source can erase them too: erasing a source never reaches a consolidated record that copied its content. + /// + /// + /// Found two ways: along the source's supersession chain (each replacement, and each record that replaced it in turn, + /// that lists the source), and among the most recent records of the scope in + /// any status. Call it before erasing the source: an erased record has no history left to follow. + /// + /// What the host has established the caller may do. + /// The exact scope the source lies in. + /// The source record. + /// Cancels the read. + /// The consolidated records found, and whether the scan may have missed older ones. + /// or is . + /// Storage infrastructure failed. + /// was cancelled. + public async Task FindConsolidatedContainingAsync( + AuthorizationContext authorization, + Scope scope, + Guid sourceId, + CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(authorization); + ArgumentNullException.ThrowIfNull(scope); + + var scan = await _store + .QueryAsync(authorization, new ExperienceRecordQuery(scope, Limit: ExperienceRecordQuery.MaxLimit), cancellationToken) + .ConfigureAwait(false); + if (scan.Outcome != ExperienceStoreOutcome.Found) + { + return new(scan.Outcome, []); + } + + var found = new SortedSet(); + var scanned = (scan.Records ?? []).Where(record => record is not null).ToList(); + foreach (var record in scanned) + { + if (record.Scope == scope && ExperienceConsolidation.SourcesOf(record).Contains(sourceId)) + { + found.Add(record.ExperienceId); + } + } + + var visited = new HashSet { sourceId }; + var current = sourceId; + for (var hop = 0; hop < MaxChainHops; hop++) + { + if (await ReplacementOfAsync(authorization, scope, current, cancellationToken).ConfigureAwait(false) is not { } next + || !visited.Add(next)) + { + break; + } + + var read = await _store.GetAsync(authorization, scope, next, ScopeCheckRead, cancellationToken).ConfigureAwait(false); + if (read is not { Outcome: ExperienceStoreOutcome.Found, SharedByGrant: false, Record: { } replacement }) + { + break; + } + + if (replacement.Scope == scope && ExperienceConsolidation.SourcesOf(replacement).Contains(sourceId)) + { + found.Add(replacement.ExperienceId); + } + + current = next; + } + + return new(ExperienceStoreOutcome.Found, [.. found]) { MayHaveMore = scanned.Count >= ExperienceRecordQuery.MaxLimit }; + } + + /// + /// Whether may be a source: finalized by the library, vouched for by its signature when + /// signing is configured, verified, with a library-written lesson whose content is confirmed. Status is not checked. + /// + private bool IsTrustedSource(ExperienceRecord record, Scope scope) => + record.Scope == scope + && !ExperienceConsolidation.IsConsolidated(record) + && record.Origin == ExperienceRecordOrigin.Finalized + && IsVouched(record) + && record.Outcome?.Status == TaskVerificationStatus.Verified + && record.Reflection is not null + && !RecordAuthorship.IsModelAuthored(record, Signer); + + /// + /// Whether is a consolidated record this library wrote: the consolidation producer exactly + /// (signed content), vouched for by its signature when signing is configured, content confirmed. Status is not checked. + /// + private bool IsTrustedConsolidated(ExperienceRecord record, Scope scope) => + record.Scope == scope + && ExperienceConsolidation.IsConsolidated(record) + && string.Equals(record.Reflection?.Producer, ExperienceConsolidation.Producer, StringComparison.Ordinal) + && IsVouched(record) + && record.Outcome?.Status == TaskVerificationStatus.Verified + && !RecordAuthorship.IsModelAuthored(record, Signer); + + /// The independence verifier's rule for a run's record: without signing, anything; with it, a signature that verifies, or a listed unsigned record. + private bool IsVouched(ExperienceRecord record) => + Signer is not { } signer || signer.Verify(record) is ProvenanceSignatureCheck.Valid or ProvenanceSignatureCheck.TrustedUnsigned; + + /// + /// The record's grouping key and what the lesson text says of it, or when it holds no resolved + /// failure signature or has no working attempt. + /// + internal static GroupingKey? GroupKey(ExperienceRecord record, ExperienceConsolidationOptions options) + { + var signatures = ExperienceFailureSignatures.Of(record) + .Where(entry => entry.Resolution == ExperienceSignatureResolution.Resolved) + .Select(entry => entry.Signature) + .DistinctBy(signature => signature.Key, StringComparer.Ordinal) + .OrderBy(signature => signature.Key, StringComparer.Ordinal) + .ToList(); + if (signatures.Count == 0 || WorkingCalls(record) is not { Count: > 0 } calls) + { + return null; + } + + var text = new StringBuilder(); + foreach (var signature in signatures) + { + Append(text, "s", signature.Key); + } + + foreach (var call in calls) + { + Append(text, "t", call.ToolName ?? string.Empty); + if (options.ArgumentKeys is { } allowlist && call.ToolName is { } tool && allowlist.TryGetValue(tool, out var keys)) + { + foreach (var argument in keys) + { + if (call.Arguments is { } arguments && arguments.TryGetValue(argument, out var value)) + { + byte[] encoded; + try + { + encoded = ProvenanceSigner.EncodeArgumentValue(value); + } + catch (ArgumentException) + { + // A value with no canonical encoding cannot be compared, so the record is never merged. + return null; + } + + Append(text, "a", argument + "=" + Convert.ToHexStringLower(SHA256.HashData(encoded))); + } + else + { + Append(text, "m", argument); + } + } + } + } + + return new(text.ToString(), signatures, [.. calls.Select(call => call.ToolName ?? string.Empty)]); + + static void Append(StringBuilder text, string kind, string value) => + text.Append(kind).Append(value.Length.ToString(CultureInfo.InvariantCulture)).Append(':').Append(value); + } + + /// + /// The working attempt's calls in call order: the final attempt by sequence number, when it ended without an error and + /// attempt sequence numbers are unique (the block's Worked: rule); otherwise . + /// + private static List? WorkingCalls(ExperienceRecord record) + { + var attempts = (record.Attempts ?? []).Where(attempt => attempt is not null).ToList(); + if (attempts.Count == 0 || attempts.Select(attempt => attempt.SequenceNumber).Distinct().Count() != attempts.Count) + { + return null; + } + + var worked = attempts.MaxBy(attempt => attempt.SequenceNumber)!; + return worked.Error is null + ? [.. (worked.ToolCalls ?? []).Where(call => call is not null).OrderBy(call => call.SequenceNumber)] + : null; + } + + private async Task ConsolidateGroupAsync( + AuthorizationContext authorization, + Scope scope, + string key, + List members, + ExperienceConsolidationOptions options, + CancellationToken cancellationToken) + { + var plain = members.Where(member => !ExperienceConsolidation.IsConsolidated(member)).ToList(); + var consolidated = members.Where(ExperienceConsolidation.IsConsolidated).ToList(); + + for (var attempt = 0; attempt < MaxRevalidations; attempt++) + { + // What each consolidated member stands for: the sources it lists that are still trusted sources of this key + // and that it superseded -- or that are in the group as live records, from an interrupted pass. + var plainIds = plain.Select(record => record.ExperienceId).ToHashSet(); + var leaves = new Dictionary>(); + foreach (var member in consolidated) + { + leaves[member.ExperienceId] = await LeavesOfAsync(authorization, scope, key, member, plainIds, options, cancellationToken).ConfigureAwait(false); + } + + // Resuming: a consolidated member that already lists every source the group stands for. The rest are superseded by it. + var standsFor = plainIds.Concat(leaves.Values.SelectMany(list => list.Select(record => record.ExperienceId))).ToHashSet(); + var covering = consolidated + .Where(record => ExperienceConsolidation.SourcesOf(record).ToHashSet().IsSupersetOf(standsFor)) + .OrderByDescending(record => record.CreatedAt) + .ThenByDescending(record => record.ExperienceId.ToString("D", CultureInfo.InvariantCulture), StringComparer.Ordinal) + .FirstOrDefault(); + if (covering is not null) + { + var rest = members.Where(member => member.ExperienceId != covering.ExperienceId).ToList(); + if (rest.Count == 0) + { + return null; + } + + var runs = leaves[covering.ExperienceId].Concat(plain).Select(record => record.SourceRunId).Distinct().Count(); + return Group( + covering.ExperienceId, + ExperienceConsolidationGroupOutcome.Existing, + runs, + await SupersedeAsync(authorization, scope, covering, rest, cancellationToken).ConfigureAwait(false), + deferred: 0); + } + + if (members.Count < 2) + { + return null; + } + + // Consolidated members always fit (they cannot be split); plain records fill what is left, newest first. + var sources = new Dictionary(); + foreach (var source in leaves.Values.SelectMany(list => list)) + { + sources.TryAdd(source.ExperienceId, source); + } + + if (sources.Count > ExperienceConsolidation.MaxSources) + { + return Group(Guid.Empty, ExperienceConsolidationGroupOutcome.Deferred, sources.Select(pair => pair.Value.SourceRunId).Distinct().Count(), [], members.Count) with + { + Reason = $"The group's consolidated records already list more than {ExperienceConsolidation.MaxSources} sources.", + }; + } + + var included = new List(consolidated); + var deferred = 0; + foreach (var record in plain + .OrderByDescending(record => record.CreatedAt) + .ThenByDescending(record => record.ExperienceId.ToString("D", CultureInfo.InvariantCulture), StringComparer.Ordinal)) + { + if (sources.ContainsKey(record.ExperienceId) || sources.Count < ExperienceConsolidation.MaxSources) + { + sources[record.ExperienceId] = record; + included.Add(record); + } + else + { + deferred++; + } + } + + var runIds = sources.Values.Select(source => source.SourceRunId).ToHashSet(); + if (included.Count < 2 || runIds.Count < options.MinimumSources) + { + return deferred > 0 + ? Group(Guid.Empty, ExperienceConsolidationGroupOutcome.Deferred, runIds.Count, [], deferred) with + { + Reason = $"The consolidated record would list more than {ExperienceConsolidation.MaxSources} sources.", + } + : null; + } + + // Right before creating: a member that changed since the pass read it is left out, and the group re-evaluated. + var changed = await ChangedAsync(authorization, scope, included, cancellationToken).ConfigureAwait(false); + if (changed.Count > 0) + { + plain.RemoveAll(record => changed.Contains(record.ExperienceId)); + consolidated.RemoveAll(record => changed.Contains(record.ExperienceId)); + members.RemoveAll(record => changed.Contains(record.ExperienceId)); + continue; + } + + var id = ExperienceConsolidation.ConsolidatedIdFor(scope, sources.Keys); + var target = await BuildAsync(authorization, scope, id, [.. sources.Values], consolidated, options, cancellationToken).ConfigureAwait(false); + var (record2, outcome, reason) = await CreateOrAdoptAsync(authorization, scope, target, cancellationToken).ConfigureAwait(false); + if (record2 is null) + { + return Group(id, ExperienceConsolidationGroupOutcome.Refused, runIds.Count, [], deferred) with { Reason = reason }; + } + + return Group( + id, + outcome, + runIds.Count, + await SupersedeAsync(authorization, scope, record2, included, cancellationToken).ConfigureAwait(false), + deferred); + } + + return Group(Guid.Empty, ExperienceConsolidationGroupOutcome.Deferred, 0, [], members.Count) with + { + Reason = "The group's records kept changing while the pass read them; a later pass will merge them.", + }; + + static ExperienceConsolidationGroupResult Group( + Guid id, + ExperienceConsolidationGroupOutcome outcome, + int runs, + IReadOnlyList sources, + int deferred) => new(id, outcome, runs, sources) { Deferred = deferred }; + } + + /// + /// The sources a consolidated record stands for: those it lists that are still readable in exactly this scope, are + /// trusted sources of the same grouping key, and were superseded by it (or are live members of the group). + /// + private async Task> LeavesOfAsync( + AuthorizationContext authorization, + Scope scope, + string key, + ExperienceRecord consolidated, + HashSet liveMembers, + ExperienceConsolidationOptions options, + CancellationToken cancellationToken) + { + var kept = new List(); + foreach (var source in await ReadSourcesAsync(authorization, scope, ExperienceConsolidation.SourcesOf(consolidated), cancellationToken).ConfigureAwait(false)) + { + if (!IsTrustedSource(source, scope) || GroupKey(source, options)?.Value != key) + { + continue; + } + + if (liveMembers.Contains(source.ExperienceId) + || (source.Status == ExperienceStatus.Superseded + && await ReplacementOfAsync(authorization, scope, source.ExperienceId, cancellationToken).ConfigureAwait(false) == consolidated.ExperienceId)) + { + kept.Add(source); + } + } + + return kept; + } + + /// The record that superseded , from its history, or . + private async Task ReplacementOfAsync(AuthorizationContext authorization, Scope scope, Guid experienceId, CancellationToken cancellationToken) + { + Guid? replacement = null; + long? after = null; + while (true) + { + var page = await _store + .GetHistoryAsync(authorization, new ExperienceRecordHistoryQuery(scope, experienceId, ExperienceRecordHistoryQuery.MaxLimit, after), cancellationToken) + .ConfigureAwait(false); + if (page.Outcome != ExperienceStoreOutcome.Found) + { + return null; + } + + foreach (var stored in page.Events ?? []) + { + if (stored?.Event is { CurrentStatus: ExperienceStatus.Superseded, ReplacementExperienceId: { } id }) + { + replacement = id; + } + } + + if (page.NextStartAfterRevision is not { } next || (after is { } previous && next <= previous)) + { + return replacement; + } + + after = next; + } + } + + /// The IDs of whose revision or status moved, or that are no longer readable here. + private async Task> ChangedAsync(AuthorizationContext authorization, Scope scope, List records, CancellationToken cancellationToken) + { + var fresh = (await ReadSourcesAsync(authorization, scope, [.. records.Select(record => record.ExperienceId)], cancellationToken).ConfigureAwait(false)) + .ToDictionary(record => record.ExperienceId); + return records + .Where(record => !fresh.TryGetValue(record.ExperienceId, out var now) || now.Revision != record.Revision || now.Status != record.Status) + .Select(record => record.ExperienceId) + .ToHashSet(); + } + + /// + /// The listed records still readable in exactly (never through a grant), read as a scope + /// check. An erased record is left out. + /// + private async Task> ReadSourcesAsync( + AuthorizationContext authorization, + Scope scope, + IReadOnlyList ids, + CancellationToken cancellationToken) + { + var found = new List(ids.Count); + foreach (var chunk in ids.Chunk(ExperienceRecordGetManyResult.MaxCount)) + { + var read = await _store.GetManyAsync(authorization, scope, chunk, ScopeCheckRead, cancellationToken).ConfigureAwait(false); + if (read.Outcome != ExperienceStoreOutcome.Found) + { + continue; + } + + foreach (var result in read.Results ?? []) + { + if (result is { Outcome: ExperienceStoreOutcome.Found, SharedByGrant: false, Record: { } record } + && record.Scope == scope + && chunk.Contains(record.ExperienceId)) + { + found.Add(record); + } + } + } + + return found; + } + + /// The consolidated record over , as it is created: a Candidate at revision 0, signed when signing is configured. + private async Task BuildAsync( + AuthorizationContext authorization, + Scope scope, + Guid id, + List sources, + List consolidated, + ExperienceConsolidationOptions options, + CancellationToken cancellationToken) + { + var latest = sources + .OrderByDescending(source => source.CreatedAt) + .ThenByDescending(source => source.ExperienceId.ToString("D", CultureInfo.InvariantCulture), StringComparer.Ordinal) + .First(); + var grouping = GroupKey(latest, options)!; + var runs = sources.Select(source => source.SourceRunId).Distinct().Count(); + var failedFirst = sources + .Where(source => ExperienceFailureSignatures.Of(source).Select(entry => entry.Signature.Key).ToHashSet(StringComparer.Ordinal) + .IsSupersetOf(grouping.Signatures.Select(signature => signature.Key))) + .Select(source => source.SourceRunId) + .Distinct() + .Count(); + + var (supporting, contradictions) = await CountAsync(authorization, scope, sources, consolidated, cancellationToken).ConfigureAwait(false); + var now = _clock.GetUtcNow(); + var reflection = latest.Reflection!; + var runId = ExperienceConsolidation.RunIdFor(scope, id); + + var record = new ExperienceRecord( + ExperienceId: id, + SourceRunId: runId, + Scope: scope, + TaskId: latest.TaskId, + TaskSummary: latest.TaskSummary, + Attempts: latest.Attempts, + Outcome: latest.Outcome, + CompletionScore: latest.CompletionScore, + Reflection: new Reflection( + ReflectionId: ExperienceConsolidation.ReflectionIdFor(scope, id), + ExperienceRunId: runId, + Lesson: LessonText(runs, latest.CreatedAt, grouping, failedFirst), + SuccessfulApproaches: reflection.SuccessfulApproaches ?? [], + FailedApproaches: reflection.FailedApproaches ?? [], + Preconditions: [.. (reflection.Preconditions ?? []).Where(value => value is not null).Take(ExperienceConsolidation.MaxPreconditions)], + Warnings: reflection.Warnings ?? [], + ReuseGuidance: reflection.ReuseGuidance, + EvidenceIds: reflection.EvidenceIds ?? [], + VerificationStatus: TaskVerificationStatus.Verified, + CompletionScore: reflection.CompletionScore, + VerificationRuleVersion: reflection.VerificationRuleVersion, + Producer: ExperienceConsolidation.Producer, + CreatedAt: now), + Environment: latest.Environment, + Provenance: new Provenance(ExperienceConsolidation.ProvenanceSource, ExperienceConsolidation.TemplateVersion, now, CorrelationId: null) + { + ExposedTo = [.. sources.OrderBy(source => source.ExperienceId).Select(source => new RunExposure(source.ExperienceId, source.Revision))], + }, + Status: ExperienceStatus.Candidate, + ReuseConfidence: 0, + SupportingValidations: supporting, + Contradictions: contradictions, + Revision: 0, + CreatedAt: now, + UpdatedAt: now); + // Created at the score its counters explain, as finalization creates every record (stores refuse any other at + // creation); the confidence engine scores it from the first evidence on, as any record. + record = record with { ReuseConfidence = ReuseConfidenceHeuristic.Score(supporting, contradictions) }; + return Signer is { } signer ? record with { ProvenanceSignature = signer.Sign(record) } : record; + } + + /// The lesson text, template version 1. + internal static string LessonText(int runs, DateTimeOffset latest, GroupingKey key, int failedFirst) + { + var classes = string.Join(" and ", key.Signatures.Select(signature => $"{signature.ErrorClass} from {signature.ToolName}")); + var names = key.ToolNames.Take(ExperienceConsolidation.MaxLessonToolNames).ToList(); + var worked = string.Join(" → ", names) + + (key.ToolNames.Count > names.Count + ? string.Create(CultureInfo.InvariantCulture, $" (+{key.ToolNames.Count - names.Count} more)") + : string.Empty); + return string.Create( + CultureInfo.InvariantCulture, + $"Verified in {runs} independent runs (latest {latest.UtcDateTime:yyyy-MM-dd}). Failed first with {classes} in {failedFirst} of them. Worked: {worked}."); + } + + /// + /// The merged counters, verified evidence only: every distinct independence key among the sources' own validations + /// (at most one each) and the counted updates admitted in their + /// histories and in each merged consolidated record's own. A source's contradictions it started with are counted too, + /// each on its own (finalization starts every record at none). + /// + private async Task<(int Supporting, int Contradictions)> CountAsync( + AuthorizationContext authorization, + Scope scope, + List sources, + List consolidated, + CancellationToken cancellationToken) + { + var supportingKeys = new HashSet(StringComparer.Ordinal); + var contradictionKeys = new HashSet(StringComparer.Ordinal); + long unkeyedContradictions = 0; + foreach (var source in sources) + { + var history = await CountedAsync(authorization, scope, source, cancellationToken).ConfigureAwait(false); + Add(history); + + // The source's own validation: finalization's, under its run and closed round -- the key the same run's reuse + // evidence on another record is counted under, so one run is one confirmation. Never more than finalization sets. + if (source.SupportingValidations - history.Supporting > 0 && ExperienceFinalizationService.InitialSupportingValidations > 0) + { + supportingKeys.Add(InitialKey(source)); + } + + unkeyedContradictions += Math.Max(0, source.Contradictions - history.Contradictions); + } + + foreach (var record in consolidated) + { + Add(await CountedAsync(authorization, scope, record, cancellationToken).ConfigureAwait(false)); + } + + return (Saturate(supportingKeys.Count), Saturate(contradictionKeys.Count + unkeyedContradictions)); + + void Add(CountedHistory history) + { + foreach (var (updateKey, supporting, contradicting) in history.Verified) + { + if (supporting > 0) + { + supportingKeys.Add(updateKey); + } + + if (contradicting > 0) + { + contradictionKeys.Add(updateKey); + } + } + } + + static int Saturate(long value) => (int)Math.Min(int.MaxValue, Math.Max(0, value)); + } + + /// The key a source's own validation is counted under. + private static string InitialKey(ExperienceRecord source) => + source.ClosedRoundId is { } round + ? ConfidenceIndependenceKey.ForMachine(source.SourceRunId, round).Value + : string.Create(CultureInfo.InvariantCulture, $"initial:{source.SourceRunId:D}"); + + /// + /// The counted confidence updates in 's history up to the revision read: their totals (every + /// admission, to tell what the record started with) and, separately, the verified ones with their keys. + /// + private async Task CountedAsync(AuthorizationContext authorization, Scope scope, ExperienceRecord record, CancellationToken cancellationToken) + { + var history = new CountedHistory(); + long? after = null; + while (true) + { + var page = await _store + .GetHistoryAsync(authorization, new ExperienceRecordHistoryQuery(scope, record.ExperienceId, ExperienceRecordHistoryQuery.MaxLimit, after), cancellationToken) + .ConfigureAwait(false); + if (page.Outcome != ExperienceStoreOutcome.Found) + { + return history; + } + + foreach (var stored in page.Events ?? []) + { + if (stored?.Event?.Confidence is not { Counted: true } update || stored.AppliedRevision > record.Revision) + { + continue; + } + + var supporting = update.NewSupportingValidations - update.PriorSupportingValidations; + var contradicting = update.NewContradictions - update.PriorContradictions; + history.Supporting += supporting; + history.Contradictions += contradicting; + if (update.Admission == ConfidenceEvidenceAdmission.Verified && KeyOf(update) is { } updateKey) + { + history.Verified.Add((updateKey, supporting, contradicting)); + } + } + + if (page.NextStartAfterRevision is not { } next || (after is { } previous && next <= previous)) + { + return history; + } + + after = next; + } + } + + private static string? KeyOf(ConfidenceUpdate update) + { + try + { + return ReuseConfidenceHeuristic.IndependenceKeyFor(update).Value; + } + catch (ArgumentException) + { + return null; + } + } + + /// + /// Creates and validates it, or adopts the record an earlier pass stored under the same ID -- + /// only when it is a trusted consolidated record over the same sources, with the same lesson and the counters it + /// started with equal to those computed now -- and finishes validating it. Returns the record to supersede by, or + /// and why not. + /// + private async Task<(ExperienceRecord? Record, ExperienceConsolidationGroupOutcome Outcome, string? Reason)> CreateOrAdoptAsync( + AuthorizationContext authorization, + Scope scope, + ExperienceRecord target, + CancellationToken cancellationToken) + { + var outcome = ExperienceConsolidationGroupOutcome.Created; + var created = await _store.CreateAsync(authorization, target, cancellationToken).ConfigureAwait(false); + var record = target; + if (created.Outcome == ExperienceStoreOutcome.Conflict) + { + outcome = ExperienceConsolidationGroupOutcome.Existing; + var existing = await _store.GetAsync(authorization, scope, target.ExperienceId, ScopeCheckRead, cancellationToken).ConfigureAwait(false); + if (existing is not { Outcome: ExperienceStoreOutcome.Found, SharedByGrant: false, Record: { } stored } + || !IsTrustedConsolidated(stored, scope) + || !ExperienceConsolidation.SourcesOf(stored).SequenceEqual(ExperienceConsolidation.SourcesOf(target)) + || !string.Equals(stored.Reflection?.Lesson, target.Reflection?.Lesson, StringComparison.Ordinal)) + { + return (null, ExperienceConsolidationGroupOutcome.Refused, + "A record stored under the consolidated record's derived ID is not the consolidated record this pass computed (its signature, producer, sources or lesson differ)."); + } + + var started = await CountedAsync(authorization, scope, stored, cancellationToken).ConfigureAwait(false); + if (stored.SupportingValidations - started.Supporting != target.SupportingValidations + || stored.Contradictions - started.Contradictions != target.Contradictions) + { + return (null, ExperienceConsolidationGroupOutcome.Refused, + "A record stored under the consolidated record's derived ID started with counters other than those computed from its sources."); + } + + record = stored; + } + else if (created.Outcome != ExperienceStoreOutcome.Created) + { + return (null, ExperienceConsolidationGroupOutcome.Refused, $"The store refused the consolidated record ({created.Outcome})."); + } + + if (record.Status == ExperienceStatus.Candidate) + { + var reason = string.Create( + CultureInfo.InvariantCulture, + $"Consolidated from {ExperienceConsolidation.SourcesOf(record).Count} verified records of the same failure and fix."); + var commit = await _lifecycle.CommitAsync( + authorization, + new CommitLifecycleTransitionRequest( + ExperienceConsolidation.ValidateEventIdFor(scope, record.ExperienceId), + record.ExperienceId, + scope, + ExperienceStatus.Candidate, + ExperienceStatus.Validated, + reason, + ExperienceConsolidation.Producer, + record.CreatedAt, + ExpectedRevision: record.Revision), + cancellationToken).ConfigureAwait(false); + if (commit.Outcome == LifecycleTransitionOutcome.Committed) + { + return (record with { Status = ExperienceStatus.Validated, Revision = commit.Revision }, outcome, null); + } + + var reread = await _store.GetAsync(authorization, scope, record.ExperienceId, ScopeCheckRead, cancellationToken).ConfigureAwait(false); + record = reread.Record ?? record; + } + + return ExperienceStatuses.IsEligibleForReuse(record.Status) + ? (record, outcome, null) + : (null, ExperienceConsolidationGroupOutcome.Refused, $"The consolidated record is {record.Status}, so nothing is superseded by it."); + } + + /// Supersedes each of by , at the revision and status read, reporting each. + private async Task> SupersedeAsync( + AuthorizationContext authorization, + Scope scope, + ExperienceRecord replacement, + IEnumerable sources, + CancellationToken cancellationToken) + { + var results = new List(); + foreach (var source in sources.OrderBy(source => source.ExperienceId)) + { + cancellationToken.ThrowIfCancellationRequested(); + try + { + var commit = await _lifecycle.CommitAsync( + authorization, + new CommitLifecycleTransitionRequest( + ExperienceConsolidation.SupersedeEventIdFor(scope, replacement.ExperienceId, source.ExperienceId), + source.ExperienceId, + scope, + source.Status, + ExperienceStatus.Superseded, + string.Create(CultureInfo.InvariantCulture, $"Consolidated into {replacement.ExperienceId:D}, one lesson for the same failure and fix."), + ExperienceConsolidation.Producer, + replacement.CreatedAt, + source.Revision, + replacement.ExperienceId), + cancellationToken).ConfigureAwait(false); + results.Add(new(source.ExperienceId, commit.Outcome)); + } + catch (OperationCanceledException) + { + throw; + } +#pragma warning disable CA1031 // One source's failure is reported, and the rest are still superseded. + catch (Exception ex) +#pragma warning restore CA1031 + { + results.Add(new(source.ExperienceId, null) { ExceptionType = ex.GetType().Name }); + } + } + + return results; + } + + /// A record's grouping key, with the signatures and tool names its lesson text names. + internal sealed record GroupingKey(string Value, IReadOnlyList Signatures, IReadOnlyList ToolNames); + + private sealed class CountedHistory + { + public int Supporting { get; set; } + + public int Contradictions { get; set; } + + public List<(string Key, int Supporting, int Contradicting)> Verified { get; } = []; + } +} diff --git a/src/AgentExperience.Core/Lifecycle/ExperienceLifecycleService.cs b/src/AgentExperience.Core/Lifecycle/ExperienceLifecycleService.cs index dc6ef9b5..ca763bfa 100644 --- a/src/AgentExperience.Core/Lifecycle/ExperienceLifecycleService.cs +++ b/src/AgentExperience.Core/Lifecycle/ExperienceLifecycleService.cs @@ -1294,15 +1294,29 @@ .. Enum.IsDefined(filter) ? Array.Empty() : [new StoreVali var initialContradicting = Math.Max(0, record.Contradictions - verifiedContradicting - trustedContradicting - unrecordedContradicting); // With provenance signing configured, a record marked finalized counts as finalized here only when its // signature verifies (or it is unsigned and listed in the cutover set), exactly as it must to vouch for a run. - var initialUnverified = record.Origin != ExperienceRecordOrigin.Finalized - || _independence.Signer?.Verify(record) is ProvenanceSignatureCheck.Missing or ProvenanceSignatureCheck.UnknownKey or ProvenanceSignatureCheck.Invalid; + // A consolidated record counts its initial counters as verified only when signing is configured and its signature, + // which covers the consolidation producer and the source list, verifies: consolidation builds them from verified + // evidence alone. Unsigned, it is a hand-written record like any other. + var vouchedConsolidated = _independence.Signer is { } consolidationSigner + && ExperienceConsolidation.IsConsolidated(record) + && string.Equals(record.Reflection?.Producer, ExperienceConsolidation.Producer, StringComparison.Ordinal) + && consolidationSigner.Verify(record) == ProvenanceSignatureCheck.Valid; + var initialUnverified = !vouchedConsolidated + && (record.Origin != ExperienceRecordOrigin.Finalized + || _independence.Signer?.Verify(record) is ProvenanceSignatureCheck.Missing or ProvenanceSignatureCheck.UnknownKey or ProvenanceSignatureCheck.Invalid); // What is excluded of the initial counters under VerifiedOnly. The counters are not signed claims, so with // signing configured a vouched-for record's verified share is capped at what finalization itself sets (one - // supporting validation, no contradiction); anything above that is its writer's statement. + // supporting validation, no contradiction); anything above that is its writer's statement. A vouched consolidated + // record's is capped at one per source its signed list names. int unverifiedInitialSupporting; int unverifiedInitialContradicting; - if (initialUnverified) + if (vouchedConsolidated) + { + unverifiedInitialSupporting = Math.Max(0, initialSupporting - ExperienceConsolidation.SourcesOf(record).Count); + unverifiedInitialContradicting = initialContradicting; + } + else if (initialUnverified) { (unverifiedInitialSupporting, unverifiedInitialContradicting) = (initialSupporting, initialContradicting); } diff --git a/src/AgentExperience.MicrosoftAgentFramework/AgentExperienceLessons.cs b/src/AgentExperience.MicrosoftAgentFramework/AgentExperienceLessons.cs index 46361b5a..c8b50306 100644 --- a/src/AgentExperience.MicrosoftAgentFramework/AgentExperienceLessons.cs +++ b/src/AgentExperience.MicrosoftAgentFramework/AgentExperienceLessons.cs @@ -1,5 +1,6 @@ using AgentExperience.Abstractions; using AgentExperience.Core.Holdout; +using AgentExperience.Core.Lifecycle; using AgentExperience.Core.Retrieval; using AgentExperience.MicrosoftAgentFramework.Export; using AgentExperience.MicrosoftAgentFramework.Injection; @@ -31,17 +32,20 @@ public sealed class AgentExperienceLessons private readonly ExperienceContextProvider _provider; private readonly TimeProvider _clock; private readonly AgentExperienceRegistration? _registration; + private readonly ExperienceConsolidationService? _consolidation; internal AgentExperienceLessons( IExperienceRecordStore store, ExperienceContextProvider provider, TimeProvider clock, - AgentExperienceRegistration? registration = null) + AgentExperienceRegistration? registration = null, + ExperienceConsolidationService? consolidation = null) { _store = store; _provider = provider; _clock = clock; _registration = registration; + _consolidation = consolidation; } /// The context provider whose writer and options render the blocks. For tests. @@ -341,6 +345,86 @@ public async Task ExportMeasuredAsync( }; } + /// + /// Merges the validated lessons in 's scope that failed the same way and were fixed the same + /// way into one consolidated lesson each, which supersedes them: + /// under the identity's own authorization, in exactly its scope (never a lesson another scope lends). Explicit: nothing + /// consolidates automatically. The view then lists one lesson per situation, with + /// set. + /// + /// + /// The injection holdout's measured benefit is per lesson, so a consolidated lesson starts with no measurements of its + /// own: the assignments and exposures that named its sources stay as they were, and are not carried over to it. + /// + /// The caller, from the host's own authentication: the authorization the store checks, and the scope consolidated. + /// + /// What to merge; for the defaults, with set + /// to the argument keys made visible, so a merged lesson never hides an + /// argument value its sources' blocks showed. A host that sets + /// itself passes the same keys here. + /// + /// Cancels the pass; a re-run finishes what it left. + /// What the pass did, per group and per source. + /// is . + /// The container has no . + /// is out of range. + /// names a blank tool or key. + /// Storage failed while reading or creating. + /// was cancelled. + public Task ConsolidateAsync( + ExperienceIdentity identity, + ExperienceConsolidationOptions? options = null, + CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(identity); + var consolidation = _consolidation + ?? throw new InvalidOperationException( + $"No {nameof(ExperienceConsolidationService)} is registered; AddAgentExperience registers one."); + options ??= DefaultConsolidationOptions(); + return consolidation.ConsolidateAsync(identity.Authorization, identity.Scope, options, cancellationToken); + } + + /// + /// The consolidated lessons in 's scope that list as a source: + /// under the identity's own authorization. + /// Erasing a source does not reach a consolidated lesson that copied its content, so a host erasing a record calls this + /// first and erases what it returns too. + /// + /// The caller, from the host's own authentication. + /// The record about to be erased. + /// Cancels the read. + /// The consolidated records that list it. + /// is . + /// The container has no . + /// Storage failed. + /// was cancelled. + public Task FindConsolidatedContainingAsync( + ExperienceIdentity identity, + Guid sourceId, + CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(identity); + var consolidation = _consolidation + ?? throw new InvalidOperationException( + $"No {nameof(ExperienceConsolidationService)} is registered; AddAgentExperience registers one."); + return consolidation.FindConsolidatedContainingAsync(identity.Authorization, identity.Scope, sourceId, cancellationToken); + } + + /// The default consolidation options: the argument keys the setup's recipes made visible. + private ExperienceConsolidationOptions DefaultConsolidationOptions() + { + var keys = new Dictionary>(StringComparer.Ordinal); + foreach (var recipe in _registration?.Options.Recipes ?? []) + { + if (recipe.Arguments.Count > 0) + { + keys[recipe.ToolName] = recipe.Arguments; + } + } + + return new ExperienceConsolidationOptions { ArgumentKeys = keys.Count > 0 ? keys : null }; + } + /// /// Sets each listed lesson's from one estimate for the page's scope, over /// the context provider's benefit window and threshold ( and @@ -423,7 +507,11 @@ private ExperienceLesson Describe(ExperienceRecord record) payload.Text, ineligibility is null, reason, - ineligibility); + ineligibility) + { + // Story 28.6: a consolidated lesson says how many source records it was merged from. + ConsolidatedFrom = ExperienceConsolidation.IsConsolidated(record) ? ExperienceConsolidation.SourcesOf(record).Count : null, + }; } #pragma warning disable CA1031 // One record a store answered that cannot be rendered must not hide the rest. catch (Exception ex) @@ -584,6 +672,13 @@ public sealed record ExperienceLesson( /// when no assigned run in the window named it, or when the store does not track the holdout. /// public ExperienceLessonBenefit? MeasuredBenefit { get; init; } + + /// + /// When this lesson is a consolidated one (), how many + /// source records it was merged from: the records its provenance lists, each now superseded by it. Its lesson text says + /// how many independent runs those were. for every other lesson. + /// + public int? ConsolidatedFrom { get; init; } } /// Why retrieval would not currently use a lesson. diff --git a/src/AgentExperience.MicrosoftAgentFramework/AgentExperienceServiceCollectionExtensions.cs b/src/AgentExperience.MicrosoftAgentFramework/AgentExperienceServiceCollectionExtensions.cs index bfdb341e..207274dc 100644 --- a/src/AgentExperience.MicrosoftAgentFramework/AgentExperienceServiceCollectionExtensions.cs +++ b/src/AgentExperience.MicrosoftAgentFramework/AgentExperienceServiceCollectionExtensions.cs @@ -1,6 +1,7 @@ using AgentExperience.Abstractions; using AgentExperience.Core.DependencyInjection; using AgentExperience.Core.Finalization; +using AgentExperience.Core.Lifecycle; using AgentExperience.MicrosoftAgentFramework; using AgentExperience.MicrosoftAgentFramework.Injection; using Microsoft.Agents.AI; @@ -110,7 +111,8 @@ public static IAgentExperienceBuilder AddAgentExperience(this IServiceCollection provider.GetRequiredService(), provider.GetRequiredService(), registration.ClockFrom(provider), - registration); + registration, + provider.GetService()); }); return new AgentExperienceBuilder(services); diff --git a/src/AgentExperience.MicrosoftAgentFramework/ExperienceLessonsHtml.cs b/src/AgentExperience.MicrosoftAgentFramework/ExperienceLessonsHtml.cs index 10f0e93a..02ae0fff 100644 --- a/src/AgentExperience.MicrosoftAgentFramework/ExperienceLessonsHtml.cs +++ b/src/AgentExperience.MicrosoftAgentFramework/ExperienceLessonsHtml.cs @@ -120,6 +120,11 @@ private static void Lesson(StringBuilder html, ExperienceLesson lesson, bool inj html.Append("not eligible for retrieval: ").Append(Encode(lesson.IneligibleReason ?? "not eligible")).Append(""); } + if (lesson.ConsolidatedFrom is { } sources) + { + html.Append("consolidated from ").Append(sources.ToString(CultureInfo.InvariantCulture)).Append(""); + } + html.Append("\n
") .Append("confidence ").Append(Encode(Number(lesson.Confidence))).Append("") .Append("stored ").Append(Encode(Time(lesson.CreatedAt))).Append("") diff --git a/tests/AgentExperience.Core.Tests/ExperienceConsolidationServiceTests.cs b/tests/AgentExperience.Core.Tests/ExperienceConsolidationServiceTests.cs new file mode 100644 index 00000000..c1ec182f --- /dev/null +++ b/tests/AgentExperience.Core.Tests/ExperienceConsolidationServiceTests.cs @@ -0,0 +1,1018 @@ +using AgentExperience.Core.Confidence; +using AgentExperience.Core.Finalization; +using AgentExperience.Core.Lifecycle; +using AgentExperience.Storage.InMemory; + +namespace AgentExperience.Core.Tests; + +/// +/// Story 28.6: one living lesson per situation. The consolidation pass over the in-memory store, with records finalized +/// from real captured runs where independence and signing matter and hand-written ones where only the grouping does. +/// +public sealed class ExperienceConsolidationServiceTests +{ + private const string TaskOfRuns = "triage-flaky-build"; + private const string Timeout = "TimeoutException: the test host did not answer after 30 seconds."; + private const string ArtifactRevision = "rev-1"; + + private static readonly DateTimeOffset Now = new(2026, 10, 5, 9, 0, 0, TimeSpan.Zero); + private static readonly Scope TestScope = new("tenant-1", "app-1", "project-1"); + private static readonly Scope OtherScope = new("tenant-1", "app-1", "project-2"); + private static readonly AuthorizationContext Authorization = new("tenant-1", "host-principal", ["experience:write"], Now); + private static readonly byte[] SigningKey = [.. Enumerable.Range(0, 32).Select(value => (byte)(value * 11 + 5))]; + + private static readonly SanitizationOptions Sanitization = new(new Dictionary(StringComparer.Ordinal) + { + ["ToolArguments"] = new(new HashSet(StringComparer.Ordinal), new HashSet(StringComparer.Ordinal), 2, 5, 1_000, 100), + ["ToolResult"] = new(new HashSet(StringComparer.Ordinal) { "value" }, new HashSet(StringComparer.Ordinal), 2, 5, 1_000, 100), + }); + + private static readonly ExperienceConsolidationOptions Defaults = new(); + + [Fact] + public async Task Three_runs_with_the_same_failure_and_fix_become_one_validated_lesson_that_supersedes_them() + { + var world = new World(); + var sources = new List(); + for (var n = 0; n < 3; n++) + { + sources.Add(await world.LessonAsync(at: Now.AddDays(n))); + } + + var result = await world.Consolidation.ConsolidateAsync(Authorization, TestScope, Defaults, CancellationToken.None); + + Assert.Equal(ExperienceStoreOutcome.Found, result.Outcome); + Assert.Equal(3, result.RecordsRead); + Assert.False(result.MayHaveMore); + var group = Assert.Single(result.Groups); + Assert.Equal(ExperienceConsolidationGroupOutcome.Created, group.Outcome); + Assert.Equal(3, group.IndependentRuns); + Assert.Equal(sources.Select(source => source.ExperienceId).Order(), group.Sources.Select(source => source.ExperienceId)); + Assert.All(group.Sources, source => Assert.True(source.Superseded)); + Assert.Equal(ExperienceConsolidation.ConsolidatedIdFor(TestScope, sources.Select(source => source.ExperienceId)), group.ConsolidatedExperienceId); + + var consolidated = await world.ReadAsync(group.ConsolidatedExperienceId); + Assert.Equal(ExperienceStatus.Validated, consolidated.Status); + Assert.True(ExperienceConsolidation.IsConsolidated(consolidated)); + Assert.Equal(sources.Select(source => source.ExperienceId).Order(), ExperienceConsolidation.SourcesOf(consolidated)); + Assert.Equal(ExperienceRecordOrigin.HostWritten, consolidated.Origin); + Assert.Equal(ExperienceConsolidation.ProvenanceSource, consolidated.Provenance.Source); + Assert.Equal(ExperienceConsolidation.TemplateVersion, consolidated.Provenance.SourceVersion); + Assert.Equal(ExperienceConsolidation.Producer, consolidated.Reflection!.Producer); + Assert.Equal(ReflectionAuthorship.Deterministic, consolidated.Reflection.Authorship); + Assert.Equal( + "Verified in 3 independent runs (latest 2026-10-07). Failed first with TimeoutException from run_tests in 3 of them. Worked: run_tests → run_tests.", + consolidated.Reflection.Lesson); + Assert.DoesNotContain(sources, source => source.SourceRunId == consolidated.SourceRunId); + + // The most recent source's attempts and outcome, so its Tried: and Worked: lines render as that source's did. + var latest = sources[^1]; + Assert.Equal(latest.TaskId, consolidated.TaskId); + Assert.Equal(latest.Attempts.Select(attempt => attempt.AttemptId), consolidated.Attempts.Select(attempt => attempt.AttemptId)); + Assert.Equal(latest.Outcome.Evidence.Count, consolidated.Outcome.Evidence.Count); + + // Three runs, each its own independent validation: the heuristic's (3 + 1) / (3 + 2). + Assert.Equal(3, consolidated.SupportingValidations); + Assert.Equal(0, consolidated.Contradictions); + Assert.Equal(4d / 5d, consolidated.ReuseConfidence, precision: 12); + + // Each source is superseded by it, and its history says so. + foreach (var source in sources) + { + var stored = await world.ReadAsync(source.ExperienceId); + Assert.Equal(ExperienceStatus.Superseded, stored.Status); + var history = await world.Store.GetFirstHistoryPageAsync(Authorization, TestScope, source.ExperienceId, CancellationToken.None); + var last = history.Events[^1].Event; + Assert.Equal(ExperienceStatus.Superseded, last.CurrentStatus); + Assert.Equal(group.ConsolidatedExperienceId, last.ReplacementExperienceId); + Assert.Equal(ExperienceConsolidation.Producer, last.Producer); + } + + // Retrieval sees one lesson now. + var live = await world.Store.QueryAsync(Authorization, new ExperienceRecordQuery(TestScope, ExperienceStatuses.EligibleForReuse), CancellationToken.None); + Assert.Equal([group.ConsolidatedExperienceId], live.Records.Select(record => record.ExperienceId)); + } + + [Fact] + public async Task A_second_pass_finds_nothing_to_do_and_writes_nothing() + { + var world = new World(); + for (var n = 0; n < 3; n++) + { + await world.LessonAsync(at: Now.AddDays(n)); + } + + var first = await world.Consolidation.ConsolidateAsync(Authorization, TestScope, Defaults, CancellationToken.None); + var consolidated = await world.ReadAsync(Assert.Single(first.Groups).ConsolidatedExperienceId); + + var second = await world.Consolidation.ConsolidateAsync(Authorization, TestScope, Defaults, CancellationToken.None); + + Assert.Empty(second.Groups); + Assert.Equal(1, second.RecordsRead); + var again = await world.ReadAsync(consolidated.ExperienceId); + Assert.Equal((consolidated.Status, consolidated.Revision, consolidated.SupportingValidations), (again.Status, again.Revision, again.SupportingValidations)); + } + + [Fact] + public async Task A_later_verified_run_is_folded_in_by_a_new_consolidated_record_that_supersedes_the_old_one() + { + var world = new World(); + var sources = new List(); + for (var n = 0; n < 3; n++) + { + sources.Add(await world.LessonAsync(at: Now.AddDays(n))); + } + + var first = Assert.Single((await world.Consolidation.ConsolidateAsync(Authorization, TestScope, Defaults, CancellationToken.None)).Groups); + var later = await world.LessonAsync(at: Now.AddDays(10)); + sources.Add(later); + + var second = Assert.Single((await world.Consolidation.ConsolidateAsync(Authorization, TestScope, Defaults, CancellationToken.None)).Groups); + + Assert.Equal(ExperienceConsolidationGroupOutcome.Created, second.Outcome); + Assert.NotEqual(first.ConsolidatedExperienceId, second.ConsolidatedExperienceId); + Assert.Equal(ExperienceConsolidation.ConsolidatedIdFor(TestScope, sources.Select(source => source.ExperienceId)), second.ConsolidatedExperienceId); + Assert.Equal(4, second.IndependentRuns); + Assert.Equal(new[] { first.ConsolidatedExperienceId, later.ExperienceId }.Order(), second.Sources.Select(source => source.ExperienceId)); + Assert.All(second.Sources, source => Assert.True(source.Superseded)); + + var folded = await world.ReadAsync(second.ConsolidatedExperienceId); + Assert.Equal(sources.Select(source => source.ExperienceId).Order(), ExperienceConsolidation.SourcesOf(folded)); + Assert.StartsWith("Verified in 4 independent runs (latest 2026-10-15).", folded.Reflection!.Lesson, StringComparison.Ordinal); + Assert.Equal(4, folded.SupportingValidations); + Assert.Equal(later.Attempts.Select(attempt => attempt.AttemptId), folded.Attempts.Select(attempt => attempt.AttemptId)); + + var old = await world.ReadAsync(first.ConsolidatedExperienceId); + Assert.Equal(ExperienceStatus.Superseded, old.Status); + var history = await world.Store.GetFirstHistoryPageAsync(Authorization, TestScope, old.ExperienceId, CancellationToken.None); + Assert.Equal(second.ConsolidatedExperienceId, history.Events[^1].Event.ReplacementExperienceId); + } + + [Fact] + public async Task Counters_are_the_sources_distinct_independence_keys_not_the_sum_of_their_counters() + { + // Reuse evidence on: the second and third runs were given the first lesson, and each counted as support for it. + var world = new World(new ExperienceFinalizationOptions { ReuseEvidence = ReuseEvidenceMode.SameTask }); + var first = await world.LessonAsync(at: Now); + var second = await world.LessonAsync(at: Now.AddDays(1), exposedTo: [first]); + var third = await world.LessonAsync(at: Now.AddDays(2), exposedTo: [first]); + first = await world.ReadAsync(first.ExperienceId); + Assert.Equal(3, first.SupportingValidations); + Assert.Equal(5, first.SupportingValidations + second.SupportingValidations + third.SupportingValidations); + + var group = Assert.Single((await world.Consolidation.ConsolidateAsync(Authorization, TestScope, Defaults, CancellationToken.None)).Groups); + + // Three runs, three independent confirmations: each later run's own validation and its support of the first + // lesson are the same run under the same closed round, so they count once. + var consolidated = await world.ReadAsync(group.ConsolidatedExperienceId); + Assert.Equal(3, consolidated.SupportingValidations); + Assert.Equal(4d / 5d, consolidated.ReuseConfidence, precision: 12); + } + + [Fact] + public async Task Different_values_of_an_allowlisted_argument_stay_two_lessons_and_without_the_allowlist_merge() + { + var world = new World(); + for (var n = 0; n < 3; n++) + { + await world.CreateAsync(HandWritten(TestScope, Now.AddMinutes(n), delay: 0)); + await world.CreateAsync(HandWritten(TestScope, Now.AddMinutes(10 + n), delay: 30)); + } + + var apart = await world.Consolidation.ConsolidateAsync( + Authorization, + TestScope, + new ExperienceConsolidationOptions { ArgumentKeys = new Dictionary> { ["retry"] = ["delay"] } }, + CancellationToken.None); + + Assert.Equal(2, apart.Groups.Count); + Assert.All(apart.Groups, group => Assert.Equal(3, group.Sources.Count)); + var merged = await Task.WhenAll(apart.Groups.Select(group => world.ReadAsync(group.ConsolidatedExperienceId))); + Assert.Equal( + ["0", "30"], + merged.Select(record => record.Attempts[^1].ToolCalls[0].Arguments["delay"]!.ToString()).Order(StringComparer.Ordinal)); + + // The same records in another scope-identical world, grouped on tool names alone: one lesson of six. + var other = new World(); + for (var n = 0; n < 3; n++) + { + await other.CreateAsync(HandWritten(TestScope, Now.AddMinutes(n), delay: 0)); + await other.CreateAsync(HandWritten(TestScope, Now.AddMinutes(10 + n), delay: 30)); + } + + var together = Assert.Single((await other.Consolidation.ConsolidateAsync(Authorization, TestScope, Defaults, CancellationToken.None)).Groups); + Assert.Equal(6, together.Sources.Count); + Assert.Equal(6, together.IndependentRuns); + } + + [Fact] + public async Task A_different_failure_or_a_different_fix_is_another_group() + { + var world = new World(); + for (var n = 0; n < 3; n++) + { + await world.CreateAsync(HandWritten(TestScope, Now.AddMinutes(n), delay: 0)); + } + + await world.CreateAsync(HandWritten(TestScope, Now.AddMinutes(5), delay: 0, error: "IOException: disk full")); + await world.CreateAsync(HandWritten(TestScope, Now.AddMinutes(6), delay: 0, fix: "restart")); + + var group = Assert.Single((await world.Consolidation.ConsolidateAsync(Authorization, TestScope, Defaults, CancellationToken.None)).Groups); + + Assert.Equal(3, group.Sources.Count); + var live = await world.Store.QueryAsync(Authorization, new ExperienceRecordQuery(TestScope, ExperienceStatuses.EligibleForReuse), CancellationToken.None); + Assert.Equal(3, live.Records.Count); + } + + [Fact] + public async Task Fewer_than_the_minimum_distinct_runs_is_a_no_op() + { + var world = new World(); + var shared = Guid.NewGuid(); + await world.CreateAsync(HandWritten(TestScope, Now, delay: 0)); + await world.CreateAsync(HandWritten(TestScope, Now.AddMinutes(1), delay: 0)); + + // Two records from one run count as one run. + await world.CreateAsync(HandWritten(TestScope, Now.AddMinutes(2), delay: 0, runId: shared)); + await world.CreateAsync(HandWritten(TestScope, Now.AddMinutes(3), delay: 0, runId: shared)); + + var result = await world.Consolidation.ConsolidateAsync( + Authorization, TestScope, new ExperienceConsolidationOptions { MinimumSources = 4 }, CancellationToken.None); + + Assert.Empty(result.Groups); + Assert.Equal(4, result.RecordsRead); + var live = await world.Store.QueryAsync(Authorization, new ExperienceRecordQuery(TestScope, ExperienceStatuses.EligibleForReuse), CancellationToken.None); + Assert.Equal(4, live.Records.Count); + Assert.Single((await world.Consolidation.ConsolidateAsync(Authorization, TestScope, new ExperienceConsolidationOptions { MinimumSources = 3 }, CancellationToken.None)).Groups); + } + + [Fact] + public async Task Records_in_another_scope_are_never_merged_and_a_scope_outside_the_authorization_is_denied() + { + var world = new World(); + await world.CreateAsync(HandWritten(TestScope, Now, delay: 0)); + await world.CreateAsync(HandWritten(TestScope, Now.AddMinutes(1), delay: 0)); + await world.CreateAsync(HandWritten(OtherScope, Now.AddMinutes(2), delay: 0)); + await world.CreateAsync(HandWritten(OtherScope, Now.AddMinutes(3), delay: 0)); + + Assert.Empty((await world.Consolidation.ConsolidateAsync(Authorization, TestScope, Defaults, CancellationToken.None)).Groups); + Assert.Empty((await world.Consolidation.ConsolidateAsync(Authorization, OtherScope, Defaults, CancellationToken.None)).Groups); + + var denied = await world.Consolidation.ConsolidateAsync( + Authorization, new Scope("tenant-2", "app-1", "project-1"), Defaults, CancellationToken.None); + Assert.Equal(ExperienceStoreOutcome.Denied, denied.Outcome); + Assert.Empty(denied.Groups); + } + + [Fact] + public async Task Model_authored_records_are_never_merged() + { + var world = new World(); + await world.CreateAsync(HandWritten(TestScope, Now, delay: 0)); + await world.CreateAsync(HandWritten(TestScope, Now.AddMinutes(1), delay: 0)); + await world.CreateAsync(HandWritten(TestScope, Now.AddMinutes(2), delay: 0, authorship: ReflectionAuthorship.Model)); + + Assert.Empty((await world.Consolidation.ConsolidateAsync(Authorization, TestScope, Defaults, CancellationToken.None)).Groups); + } + + [Fact] + public async Task With_signing_unconfirmed_records_are_left_out_and_the_consolidated_record_is_signed_over_its_source_list() + { + var signing = new ExperienceProvenanceSigningOptions(new Dictionary { ["key-1"] = SigningKey }, "key-1"); + var world = new World(signing: signing); + var signed = new List(); + for (var n = 0; n < 3; n++) + { + signed.Add(await world.LessonAsync(at: Now.AddDays(n))); + } + + // A hand-written record no signature confirms: excluded, as injection would exclude it. + var unconfirmed = HandWritten(TestScope, Now.AddDays(5), delay: 0) with { Attempts = signed[0].Attempts }; + await world.CreateAsync(unconfirmed); + + var group = Assert.Single((await world.Consolidation.ConsolidateAsync(Authorization, TestScope, Defaults, CancellationToken.None)).Groups); + + Assert.Equal(signed.Select(record => record.ExperienceId).Order(), group.Sources.Select(source => source.ExperienceId)); + Assert.Equal(ExperienceStatus.Validated, (await world.ReadAsync(unconfirmed.ExperienceId)).Status); + var consolidated = await world.ReadAsync(group.ConsolidatedExperienceId); + var signer = ProvenanceSigner.Create(signing)!; + Assert.NotNull(consolidated.ProvenanceSignature); + Assert.True(signer.ConfirmsContent(consolidated)); + + // The source list is a signed claim: changing it breaks the signature. + var tampered = consolidated with + { + Provenance = consolidated.Provenance with { ExposedTo = consolidated.Provenance.ExposedTo.Skip(1).ToList() }, + }; + Assert.False(signer.ConfirmsContent(tampered)); + } + + [Fact] + public async Task A_source_changed_after_the_pass_read_it_is_left_untouched_and_reported_and_the_rest_are_superseded() + { + var inner = new InMemoryExperienceRecordStore(); + var racing = new RacingStore(inner); + var world = new World(store: racing, inMemory: inner); + var sources = new List(); + for (var n = 0; n < 3; n++) + { + sources.Add(await world.LessonAsync(at: Now.AddDays(n))); + } + + // Just before its supersession commits, another writer revokes the second source. + var raced = sources[1]; + racing.BeforeSupersede = async id => + { + if (id == raced.ExperienceId) + { + var current = await world.ReadAsync(id); + var revoke = await inner.CommitLifecycleEventAsync( + Authorization, + TestScope, + new LifecycleEvent(Guid.NewGuid(), id, current.Status, ExperienceStatus.Revoked, "withdrawn by a reviewer", "reviewer", Now.AddDays(3), current.Revision), + CancellationToken.None); + Assert.Equal(ExperienceStoreOutcome.Committed, revoke.Outcome); + } + }; + + var group = Assert.Single((await world.Consolidation.ConsolidateAsync(Authorization, TestScope, Defaults, CancellationToken.None)).Groups); + + Assert.Equal(ExperienceConsolidationGroupOutcome.Created, group.Outcome); + var lost = Assert.Single(group.Sources, source => !source.Superseded); + Assert.Equal(raced.ExperienceId, lost.ExperienceId); + Assert.Contains(lost.Outcome, new LifecycleTransitionOutcome?[] { LifecycleTransitionOutcome.StaleRevision, LifecycleTransitionOutcome.StatusMismatch }); + Assert.Equal(ExperienceStatus.Revoked, (await world.ReadAsync(raced.ExperienceId)).Status); + Assert.Equal(2, group.Sources.Count(source => source.Superseded)); + + // The next pass has nothing left to do: the revoked source is out of every group. + racing.BeforeSupersede = null; + Assert.Empty((await world.Consolidation.ConsolidateAsync(Authorization, TestScope, Defaults, CancellationToken.None)).Groups); + } + + [Fact] + public async Task A_pass_interrupted_before_the_supersessions_is_finished_by_the_next_one() + { + var inner = new InMemoryExperienceRecordStore(); + var racing = new RacingStore(inner) { FailSupersessions = true }; + var world = new World(store: racing, inMemory: inner); + for (var n = 0; n < 3; n++) + { + await world.LessonAsync(at: Now.AddDays(n)); + } + + var broken = Assert.Single((await world.Consolidation.ConsolidateAsync(Authorization, TestScope, Defaults, CancellationToken.None)).Groups); + Assert.All(broken.Sources, source => Assert.Equal(nameof(ExperienceStoreException), source.ExceptionType)); + Assert.All(broken.Sources, source => Assert.False(source.Superseded)); + + racing.FailSupersessions = false; + var resumed = Assert.Single((await world.Consolidation.ConsolidateAsync(Authorization, TestScope, Defaults, CancellationToken.None)).Groups); + + Assert.Equal(ExperienceConsolidationGroupOutcome.Existing, resumed.Outcome); + Assert.Equal(broken.ConsolidatedExperienceId, resumed.ConsolidatedExperienceId); + Assert.Equal(3, resumed.IndependentRuns); + Assert.All(resumed.Sources, source => Assert.True(source.Superseded)); + Assert.Empty((await world.Consolidation.ConsolidateAsync(Authorization, TestScope, Defaults, CancellationToken.None)).Groups); + } + + [Fact] + public async Task Contesting_the_consolidated_record_works_as_for_any_record_and_its_sources_stay_superseded() + { + var world = new World(); + var sources = new List(); + for (var n = 0; n < 3; n++) + { + sources.Add(await world.LessonAsync(at: Now.AddDays(n))); + } + + var group = Assert.Single((await world.Consolidation.ConsolidateAsync(Authorization, TestScope, Defaults, CancellationToken.None)).Groups); + var consolidated = await world.ReadAsync(group.ConsolidatedExperienceId); + + var contest = await world.Lifecycle.CommitAsync( + Authorization, + new CommitLifecycleTransitionRequest( + Guid.NewGuid(), consolidated.ExperienceId, TestScope, consolidated.Status, ExperienceStatus.Contested, "a reviewer disputes it", "reviewer", Now.AddDays(20), consolidated.Revision), + CancellationToken.None); + + Assert.Equal(LifecycleTransitionOutcome.Committed, contest.Outcome); + foreach (var source in sources) + { + Assert.Equal(ExperienceStatus.Superseded, (await world.ReadAsync(source.ExperienceId)).Status); + } + + Assert.Empty((await world.Consolidation.ConsolidateAsync(Authorization, TestScope, Defaults, CancellationToken.None)).Groups); + } + + [Fact] + public async Task Options_out_of_range_are_refused() + { + var world = new World(); + await Assert.ThrowsAsync(() => world.Consolidation.ConsolidateAsync( + Authorization, TestScope, new ExperienceConsolidationOptions { MinimumSources = 1 }, CancellationToken.None)); + await Assert.ThrowsAsync(() => world.Consolidation.ConsolidateAsync( + Authorization, + TestScope, + new ExperienceConsolidationOptions { ArgumentKeys = new Dictionary> { [" "] = ["delay"] } }, + CancellationToken.None)); + } + + [Fact] + public void The_derived_ID_depends_on_the_scope_and_the_source_set_not_its_order() + { + Guid[] ids = [Guid.NewGuid(), Guid.NewGuid(), Guid.NewGuid()]; + var forward = ExperienceConsolidation.ConsolidatedIdFor(TestScope, ids); + + Assert.Equal(forward, ExperienceConsolidation.ConsolidatedIdFor(TestScope, ids.Reverse())); + Assert.NotEqual(forward, ExperienceConsolidation.ConsolidatedIdFor(OtherScope, ids)); + Assert.NotEqual(forward, ExperienceConsolidation.ConsolidatedIdFor(TestScope, ids.Take(2))); + Assert.Equal(8, forward.Version); + } + + [Fact] + public async Task Hand_written_records_are_never_sources_and_an_inflated_counter_counts_once() + { + var world = new World(); + for (var n = 0; n < 3; n++) + { + await world.CreateAsync(HandWritten(TestScope, Now.AddMinutes(n), delay: 0, origin: ExperienceRecordOrigin.HostWritten, supporting: 40)); + } + + Assert.Empty((await world.Consolidation.ConsolidateAsync(Authorization, TestScope, Defaults, CancellationToken.None)).Groups); + + // Finalized records whose counters someone raised in the store: each source's own validation is one, as + // finalization sets it. + var other = new World(); + for (var n = 0; n < 3; n++) + { + await other.CreateAsync(HandWritten(TestScope, Now.AddMinutes(n), delay: 0, supporting: n == 0 ? 50 : 1)); + } + + var group = Assert.Single((await other.Consolidation.ConsolidateAsync(Authorization, TestScope, Defaults, CancellationToken.None)).Groups); + Assert.Equal(3, (await other.ReadAsync(group.ConsolidatedExperienceId)).SupportingValidations); + } + + [Fact] + public async Task Only_verified_evidence_is_carried_and_the_signed_record_reads_the_same_under_every_filter() + { + var signing = new ExperienceProvenanceSigningOptions(new Dictionary { ["key-1"] = SigningKey }, "key-1"); + var world = new World(signing: signing); + var sources = new List(); + for (var n = 0; n < 3; n++) + { + sources.Add(await world.LessonAsync(at: Now.AddDays(n))); + } + + // Host-trusted support for the first source, admitted by a lifecycle service that trusts the host's identifiers. + var trusting = new ExperienceLifecycleService( + world.Store, + indexingService: null, + new ExperienceIndependenceOptions { Verification = IndependenceVerification.TrustHostSuppliedIdentifiers }, + captureService: null, + deindexingTimeout: null, + confidenceEngine: null, + signing); + var trusted = await trusting.ApplyEvidenceAsync( + Authorization, + new ApplyConfidenceEvidenceRequest( + Guid.NewGuid(), sources[0].ExperienceId, TestScope, Guid.NewGuid(), ConfidenceEvidenceKind.Supporting, ConfidenceEvidenceSource.Machine, + Guid.NewGuid(), Guid.NewGuid(), "the host says it helped", "host", Now.AddDays(4)), + CancellationToken.None); + Assert.True(trusted.Counted); + Assert.Equal(2, (await world.ReadAsync(sources[0].ExperienceId)).SupportingValidations); + + var group = Assert.Single((await world.Consolidation.ConsolidateAsync(Authorization, TestScope, Defaults, CancellationToken.None)).Groups); + + var consolidated = await world.ReadAsync(group.ConsolidatedExperienceId); + Assert.Equal(3, consolidated.SupportingValidations); + foreach (var filter in new[] { ConfidenceEvidenceFilter.All, ConfidenceEvidenceFilter.ExcludeHostTrusted, ConfidenceEvidenceFilter.VerifiedOnly }) + { + var read = await world.Lifecycle.ReadConfidenceAsync(Authorization, TestScope, consolidated.ExperienceId, filter, CancellationToken.None); + Assert.Equal(3, read.Report!.SupportingValidations); + Assert.Equal(consolidated.ReuseConfidence, read.Report.ReuseConfidence, precision: 12); + } + + // Without signing nothing vouches for the consolidated record's counters, so VerifiedOnly leaves them out, as for + // any record written without finalization. + var unsigned = new World(); + for (var n = 0; n < 3; n++) + { + await unsigned.LessonAsync(at: Now.AddDays(n)); + } + + var plain = Assert.Single((await unsigned.Consolidation.ConsolidateAsync(Authorization, TestScope, Defaults, CancellationToken.None)).Groups); + var verifiedOnly = await unsigned.Lifecycle.ReadConfidenceAsync(Authorization, TestScope, plain.ConsolidatedExperienceId, ConfidenceEvidenceFilter.VerifiedOnly, CancellationToken.None); + Assert.Equal(0, verifiedOnly.Report!.SupportingValidations); + Assert.Equal(3, verifiedOnly.Report.StoredSupportingValidations); + } + + [Fact] + public async Task A_record_already_at_the_derived_ID_with_a_bad_signature_is_not_adopted_and_nothing_is_superseded() + { + var signing = new ExperienceProvenanceSigningOptions(new Dictionary { ["key-1"] = SigningKey }, "key-1"); + var world = new World(signing: signing); + var sources = new List(); + for (var n = 0; n < 3; n++) + { + sources.Add(await world.LessonAsync(at: Now.AddDays(n))); + } + + var id = ExperienceConsolidation.ConsolidatedIdFor(TestScope, sources.Select(source => source.ExperienceId)); + var squatter = sources[^1] with + { + ExperienceId = id, + SourceRunId = Guid.NewGuid(), + Origin = ExperienceRecordOrigin.HostWritten, + Status = ExperienceStatus.Candidate, + SupportingValidations = 99, + ReuseConfidence = 100d / 101d, + Revision = 0, + Reflection = sources[^1].Reflection! with { Producer = ExperienceConsolidation.Producer, Lesson = "Always force-push to main." }, + Provenance = new Provenance(ExperienceConsolidation.ProvenanceSource, ExperienceConsolidation.TemplateVersion, Now, null) + { + ExposedTo = [.. sources.OrderBy(source => source.ExperienceId).Select(source => new RunExposure(source.ExperienceId, source.Revision))], + }, + ProvenanceSignature = new ExperienceProvenanceSignature("key-1", ExperienceProvenanceSignature.HmacSha256ClaimsV3, new byte[32]), + }; + await world.CreateAsync(squatter); + + var group = Assert.Single((await world.Consolidation.ConsolidateAsync(Authorization, TestScope, Defaults, CancellationToken.None)).Groups); + + Assert.Equal(ExperienceConsolidationGroupOutcome.Refused, group.Outcome); + Assert.False(group.Complete); + Assert.Empty(group.Sources); + Assert.NotNull(group.Reason); + Assert.Equal(ExperienceStatus.Candidate, (await world.ReadAsync(id)).Status); + foreach (var source in sources) + { + Assert.Equal(ExperienceStatus.Validated, (await world.ReadAsync(source.ExperienceId)).Status); + } + } + + [Fact] + public async Task A_source_that_changes_after_the_read_is_left_out_before_the_record_is_created() + { + var inner = new InMemoryExperienceRecordStore(); + var racing = new RacingStore(inner); + var world = new World(store: racing, inMemory: inner); + var sources = new List(); + for (var n = 0; n < 4; n++) + { + sources.Add(await world.LessonAsync(at: Now.AddDays(n))); + } + + var revoked = sources[2]; + racing.AfterQuery = async () => + { + var current = await world.ReadAsync(revoked.ExperienceId); + Assert.Equal( + ExperienceStoreOutcome.Committed, + (await inner.CommitLifecycleEventAsync( + Authorization, + TestScope, + new LifecycleEvent(Guid.NewGuid(), revoked.ExperienceId, current.Status, ExperienceStatus.Revoked, "withdrawn", "reviewer", Now.AddDays(5), current.Revision), + CancellationToken.None)).Outcome); + }; + + var result = await world.Consolidation.ConsolidateAsync(Authorization, TestScope, Defaults, CancellationToken.None); + + var group = Assert.Single(result.Groups); + Assert.Equal(ExperienceConsolidationGroupOutcome.Created, group.Outcome); + Assert.True(group.Complete); + Assert.Equal(0, result.IncompleteGroups); + Assert.Equal(3, group.IndependentRuns); + Assert.DoesNotContain(group.Sources, source => source.ExperienceId == revoked.ExperienceId); + Assert.DoesNotContain(revoked.ExperienceId, ExperienceConsolidation.SourcesOf(await world.ReadAsync(group.ConsolidatedExperienceId))); + } + + [Fact] + public async Task A_listed_source_revoked_during_the_pass_is_not_counted_again_when_a_later_run_is_folded_in() + { + var inner = new InMemoryExperienceRecordStore(); + var racing = new RacingStore(inner); + var world = new World(store: racing, inMemory: inner); + var sources = new List(); + for (var n = 0; n < 3; n++) + { + sources.Add(await world.LessonAsync(at: Now.AddDays(n))); + } + + var raced = sources[1]; + racing.BeforeSupersede = async id => + { + if (id == raced.ExperienceId) + { + var current = await world.ReadAsync(id); + await inner.CommitLifecycleEventAsync( + Authorization, + TestScope, + new LifecycleEvent(Guid.NewGuid(), id, current.Status, ExperienceStatus.Revoked, "withdrawn", "reviewer", Now.AddDays(3), current.Revision), + CancellationToken.None); + } + }; + + var first = await world.Consolidation.ConsolidateAsync(Authorization, TestScope, Defaults, CancellationToken.None); + var group = Assert.Single(first.Groups); + Assert.False(group.Complete); + Assert.Equal(1, first.IncompleteGroups); + racing.BeforeSupersede = null; + + // The old record still lists the revoked source; the fold-in leaves it out of the list, the runs and the counters. + var later = await world.LessonAsync(at: Now.AddDays(10)); + var fold = Assert.Single((await world.Consolidation.ConsolidateAsync(Authorization, TestScope, Defaults, CancellationToken.None)).Groups); + + Assert.Equal(ExperienceConsolidationGroupOutcome.Created, fold.Outcome); + Assert.Equal(3, fold.IndependentRuns); + var folded = await world.ReadAsync(fold.ConsolidatedExperienceId); + Assert.Equal(new[] { sources[0].ExperienceId, sources[2].ExperienceId, later.ExperienceId }.Order(), ExperienceConsolidation.SourcesOf(folded)); + Assert.Equal(3, folded.SupportingValidations); + Assert.StartsWith("Verified in 3 independent runs", folded.Reflection!.Lesson, StringComparison.Ordinal); + } + + [Fact] + public async Task An_erased_source_is_found_before_erasure_and_left_out_of_the_next_merge() + { + var inner = new InMemoryExperienceRecordStore(); + var racing = new RacingStore(inner); + var world = new World(store: racing, inMemory: inner); + var sources = new List(); + for (var n = 0; n < 3; n++) + { + sources.Add(await world.LessonAsync(at: Now.AddDays(n))); + } + + var first = Assert.Single((await world.Consolidation.ConsolidateAsync(Authorization, TestScope, Defaults, CancellationToken.None)).Groups); + var erased = sources[0]; + + // Before erasing a source, the host finds the consolidated records that copied it. + var found = await world.Consolidation.FindConsolidatedContainingAsync(Authorization, TestScope, erased.ExperienceId, CancellationToken.None); + Assert.Equal(ExperienceStoreOutcome.Found, found.Outcome); + Assert.Equal([first.ConsolidatedExperienceId], found.ExperienceIds); + Assert.False(found.MayHaveMore); + + racing.Erased.Add(erased.ExperienceId); + var later = await world.LessonAsync(at: Now.AddDays(10)); + var fold = Assert.Single((await world.Consolidation.ConsolidateAsync(Authorization, TestScope, Defaults, CancellationToken.None)).Groups); + + var folded = await world.ReadAsync(fold.ConsolidatedExperienceId); + Assert.Equal(new[] { sources[1].ExperienceId, sources[2].ExperienceId, later.ExperienceId }.Order(), ExperienceConsolidation.SourcesOf(folded)); + Assert.Equal(3, folded.SupportingValidations); + + // A source still listed is found in both: the chain from it, and the scan. + var both = await world.Consolidation.FindConsolidatedContainingAsync(Authorization, TestScope, sources[1].ExperienceId, CancellationToken.None); + Assert.Equal(new[] { first.ConsolidatedExperienceId, fold.ConsolidatedExperienceId }.Order(), both.ExperienceIds); + } + + [Fact] + public async Task Sources_beyond_the_cap_are_deferred_and_reported() + { + var world = new World(); + for (var n = 0; n <= ExperienceConsolidation.MaxSources; n++) + { + await world.CreateAsync(HandWritten(TestScope, Now.AddMinutes(n), delay: 0)); + } + + var first = await world.Consolidation.ConsolidateAsync(Authorization, TestScope, Defaults, CancellationToken.None); + var group = Assert.Single(first.Groups); + Assert.Equal(ExperienceConsolidationGroupOutcome.Created, group.Outcome); + Assert.Equal(ExperienceConsolidation.MaxSources, group.Sources.Count); + Assert.Equal(1, group.Deferred); + Assert.False(group.Complete); + Assert.Equal(1, first.IncompleteGroups); + + var second = Assert.Single((await world.Consolidation.ConsolidateAsync(Authorization, TestScope, Defaults, CancellationToken.None)).Groups); + Assert.Equal(ExperienceConsolidationGroupOutcome.Deferred, second.Outcome); + Assert.Equal(1, second.Deferred); + Assert.Equal(Guid.Empty, second.ConsolidatedExperienceId); + Assert.NotNull(second.Reason); + } + + [Fact] + public async Task Two_passes_at_once_converge_on_one_consolidated_record() + { + var world = new World(); + var sources = new List(); + for (var n = 0; n < 3; n++) + { + sources.Add(await world.LessonAsync(at: Now.AddDays(n))); + } + + var passes = await Task.WhenAll( + Task.Run(() => world.Consolidation.ConsolidateAsync(Authorization, TestScope, Defaults, CancellationToken.None)), + Task.Run(() => world.Consolidation.ConsolidateAsync(Authorization, TestScope, Defaults, CancellationToken.None))); + + var ids = passes.SelectMany(pass => pass.Groups).Select(group => group.ConsolidatedExperienceId).Distinct().ToList(); + var id = Assert.Single(ids); + var live = await world.Store.QueryAsync(Authorization, new ExperienceRecordQuery(TestScope, ExperienceStatuses.EligibleForReuse), CancellationToken.None); + Assert.Equal([id], live.Records.Select(record => record.ExperienceId)); + foreach (var source in sources) + { + Assert.Equal(ExperienceStatus.Superseded, (await world.ReadAsync(source.ExperienceId)).Status); + } + + Assert.Equal(3, (await world.ReadAsync(id)).SupportingValidations); + Assert.Empty((await world.Consolidation.ConsolidateAsync(Authorization, TestScope, Defaults, CancellationToken.None)).Groups); + } + + [Fact] + public async Task A_run_finalized_without_a_closed_round_is_counted_once() + { + // Reuse evidence on: the second run was given the first lesson, but closed no round, so its evidence names no + // round and verification refuses it. Its own validation is keyed on its run alone; nothing is counted twice. + var world = new World(new ExperienceFinalizationOptions { ReuseEvidence = ReuseEvidenceMode.SameTask }); + var first = await world.LessonAsync(at: Now); + var second = await world.LessonAsync(at: Now.AddDays(1), exposedTo: [first], round: Guid.Empty); + await world.LessonAsync(at: Now.AddDays(2)); + Assert.Null(second.ClosedRoundId); + Assert.Equal(1, (await world.ReadAsync(first.ExperienceId)).SupportingValidations); + + var group = Assert.Single((await world.Consolidation.ConsolidateAsync(Authorization, TestScope, Defaults, CancellationToken.None)).Groups); + + Assert.Equal(3, (await world.ReadAsync(group.ConsolidatedExperienceId)).SupportingValidations); + } + + [Fact] + public async Task With_a_host_confidence_engine_the_record_is_created_at_the_score_its_counters_explain() + { + var world = new World(engine: new FlatEngine()); + for (var n = 0; n < 3; n++) + { + await world.LessonAsync(at: Now.AddDays(n)); + } + + var group = Assert.Single((await world.Consolidation.ConsolidateAsync(Authorization, TestScope, Defaults, CancellationToken.None)).Groups); + + Assert.Equal(ExperienceConsolidationGroupOutcome.Created, group.Outcome); + Assert.Equal(4d / 5d, (await world.ReadAsync(group.ConsolidatedExperienceId)).ReuseConfidence, precision: 12); + } + + private sealed class FlatEngine : IExperienceConfidenceEngine + { + public string RuleId => "flat"; + + public string RuleVersion => "1"; + + public double Score(ExperienceConfidenceInput input) => 0.5; + } + + /// A verified, hand-written record: run_tests timed out, then (delay) worked. + private static ExperienceRecord HandWritten( + Scope scope, + DateTimeOffset at, + int delay, + string error = Timeout, + string fix = "retry", + Guid? runId = null, + ReflectionAuthorship authorship = ReflectionAuthorship.Deterministic, + ExperienceRecordOrigin origin = ExperienceRecordOrigin.Finalized, + int supporting = 1) + { + var round = Guid.NewGuid(); + var run = runId ?? Guid.NewGuid(); + return new ExperienceRecord( + ExperienceId: Guid.NewGuid(), + SourceRunId: run, + Scope: scope, + TaskId: TaskOfRuns, + TaskSummary: "keep the build green", + Attempts: + [ + new Attempt(Guid.NewGuid(), 0, at, TimeSpan.FromSeconds(1), [Call("run_tests", null, error)], null, null), + new Attempt( + Guid.NewGuid(), + 1, + at, + TimeSpan.FromSeconds(1), + [Call(fix, new Dictionary { ["delay"] = delay }, null)], + "done", + null), + ], + Outcome: new Outcome( + TaskVerificationStatus.Verified, + [new Evidence(Guid.NewGuid(), round, ArtifactRevision, "tests", "TestResult", CheckResult.Pass, "ci", null, at)], + null, + at), + CompletionScore: 1, + Reflection: new Reflection( + Guid.NewGuid(), run, "Retry after a timeout.", [], [], ["the test host is local"], [], null, [], TaskVerificationStatus.Verified, 1, "1.0.0", "tests", at) + { + Authorship = authorship, + }, + Environment: new EnvironmentFingerprint("host-1", "net10.0", "test-os", null, new Dictionary()), + Provenance: new Provenance("unit-tests", "1.0.0", at, null), + Status: ExperienceStatus.Validated, + ReuseConfidence: (1d + supporting) / (2d + supporting), + SupportingValidations: supporting, + Contradictions: 0, + Revision: 0, + CreatedAt: at, + UpdatedAt: at) + { + Origin = origin, + ClosedRoundId = round, + }; + + ToolCallRecord Call(string tool, Dictionary? arguments, string? failure) => + new(Guid.NewGuid(), 0, tool, arguments ?? [], at, TimeSpan.FromSeconds(1), failure is null ? "ok" : null, failure); + } + + private sealed class FixedClock(DateTimeOffset now) : TimeProvider + { + public DateTimeOffset Now { get; set; } = now; + + public override DateTimeOffset GetUtcNow() => Now; + } + + /// A store that lets a test act just before a supersession commits, or make every supersession throw. + private sealed class RacingStore(InMemoryExperienceRecordStore inner) : DelegatingStore(inner) + { + public Func? BeforeSupersede { get; set; } + + /// Records treated as erased: reads answer , queries leave them out. + public HashSet Erased { get; } = []; + + /// Runs once, right after the next query answered. + public Func? AfterQuery { get; set; } + + public override async Task GetAsync( + AuthorizationContext authorization, Scope scope, Guid experienceId, ExperienceReadOptions options, CancellationToken cancellationToken) => + Erased.Contains(experienceId) + ? new ExperienceRecordGetResult(ExperienceStoreOutcome.Deleted, null, []) + : await base.GetAsync(authorization, scope, experienceId, options, cancellationToken); + + public override async Task QueryAsync(AuthorizationContext authorization, ExperienceRecordQuery query, CancellationToken cancellationToken) + { + var result = await base.QueryAsync(authorization, query, cancellationToken); + if (AfterQuery is { } after) + { + AfterQuery = null; + await after(); + } + + return result with { Records = [.. result.Records.Where(record => !Erased.Contains(record.ExperienceId))] }; + } + + public override async Task GetHistoryAsync( + AuthorizationContext authorization, ExperienceRecordHistoryQuery query, CancellationToken cancellationToken) => + Erased.Contains(query.ExperienceId) + ? new ExperienceRecordHistoryResult(ExperienceStoreOutcome.Deleted, 0, [], []) + : await base.GetHistoryAsync(authorization, query, cancellationToken); + + public bool FailSupersessions { get; set; } + + public override async Task CommitLifecycleEventAsync( + AuthorizationContext authorization, Scope scope, LifecycleEvent lifecycleEvent, CancellationToken cancellationToken) + { + if (lifecycleEvent.CurrentStatus == ExperienceStatus.Superseded) + { + if (FailSupersessions) + { + throw new ExperienceStoreException("The store is unavailable."); + } + + if (BeforeSupersede is { } before) + { + await before(lifecycleEvent.ExperienceRecordId); + } + } + + return await base.CommitLifecycleEventAsync(authorization, scope, lifecycleEvent, cancellationToken); + } + } + + /// Forwards every port member to the in-memory store. + private class DelegatingStore(InMemoryExperienceRecordStore inner) : IExperienceRecordStore + { + public Task CreateAsync(AuthorizationContext authorization, ExperienceRecord record, CancellationToken cancellationToken) => + inner.CreateAsync(authorization, record, cancellationToken); + + public Task GetAsync(AuthorizationContext authorization, Scope scope, Guid experienceId, CancellationToken cancellationToken) => + inner.GetAsync(authorization, scope, experienceId, cancellationToken); + + public virtual Task GetAsync(AuthorizationContext authorization, Scope scope, Guid experienceId, ExperienceReadOptions options, CancellationToken cancellationToken) => + inner.GetAsync(authorization, scope, experienceId, options, cancellationToken); + + public virtual Task QueryAsync(AuthorizationContext authorization, ExperienceRecordQuery query, CancellationToken cancellationToken) => + inner.QueryAsync(authorization, query, cancellationToken); + + public virtual Task CommitLifecycleEventAsync( + AuthorizationContext authorization, Scope scope, LifecycleEvent lifecycleEvent, CancellationToken cancellationToken) => + inner.CommitLifecycleEventAsync(authorization, scope, lifecycleEvent, cancellationToken); + + public virtual Task GetHistoryAsync(AuthorizationContext authorization, ExperienceRecordHistoryQuery query, CancellationToken cancellationToken) => + inner.GetHistoryAsync(authorization, query, cancellationToken); + + public Task CheckSupersessionAsync( + AuthorizationContext authorization, Scope scope, Guid experienceId, Guid replacementExperienceId, CancellationToken cancellationToken) => + inner.CheckSupersessionAsync(authorization, scope, experienceId, replacementExperienceId, cancellationToken); + } + + /// The capture service, finalization, lifecycle and consolidation over one store, in one scope. + private sealed class World + { + private static readonly Guid Round = Guid.Parse("33333333-3333-3333-3333-333333333333"); + + public World( + ExperienceFinalizationOptions? options = null, + ExperienceProvenanceSigningOptions? signing = null, + IExperienceRecordStore? store = null, + InMemoryExperienceRecordStore? inMemory = null, + IExperienceConfidenceEngine? engine = null) + { + Store = inMemory ?? new InMemoryExperienceRecordStore(); + var used = store ?? Store; + Capture = new InMemoryExperienceCaptureService(new DefaultSanitizer(Sanitization), new CaptureLimits(10, 10, 1_000, 1_000)); + Lifecycle = new ExperienceLifecycleService( + used, + indexingService: null, + new ExperienceIndependenceOptions(), + Capture, + deindexingTimeout: null, + confidenceEngine: engine, + signing); + Finalization = new ExperienceFinalizationService( + Capture, + new DefaultExperienceReflector(), + used, + Lifecycle, + indexingService: null, + indexingTimeout: null, + provenanceSigning: null, + reflectionSanitizer: null, + options); + Consolidation = new ExperienceConsolidationService(used, Lifecycle, Clock); + UsedStore = used; + } + + public InMemoryExperienceRecordStore Store { get; } + + public IExperienceRecordStore UsedStore { get; } + + public InMemoryExperienceCaptureService Capture { get; } + + public ExperienceLifecycleService Lifecycle { get; } + + public ExperienceFinalizationService Finalization { get; } + + public ExperienceConsolidationService Consolidation { get; } + + public FixedClock Clock { get; } = new(Now.AddDays(30)); + + public async Task CreateAsync(ExperienceRecord record) + { + var created = await Store.CreateAsync(Authorization, record, CancellationToken.None); + Assert.True(created.Outcome == ExperienceStoreOutcome.Created, string.Join("; ", created.Errors.Select(error => error.Path + ": " + error.Message))); + } + + public async Task ReadAsync(Guid id) => + (await Store.GetAsync(Authorization, TestScope, id, CancellationToken.None)).Record!; + + /// A verified run -- run_tests timed out, then passed on a second call -- finalized into a Validated lesson at . + public async Task LessonAsync(DateTimeOffset at, IReadOnlyList? exposedTo = null, Guid? round = null) + { + var closed = round ?? Round; + var runId = Guid.NewGuid(); + Assert.Equal(StartRunOutcome.Started, Capture.StartRun( + runId, + TaskOfRuns, + "keep the build green", + TestScope, + new EnvironmentFingerprint("host-1", "net10.0", "test-os", null, new Dictionary()), + new Provenance("unit-tests", "1.0.0", at, null), + at).Outcome); + + if (exposedTo is { Count: > 0 }) + { + Assert.Equal( + RecordExposureOutcome.Recorded, + Capture.RecordExposure(runId, [.. exposedTo.Select(record => new RunExposure(record.ExperienceId, record.Revision))]).Outcome); + } + + Assert.Equal(AppendAttemptOutcome.Recorded, (await Capture.AppendAttemptAsync( + runId, + new AppendAttemptRequest( + Guid.NewGuid(), + at, + TimeSpan.FromSeconds(1), + [ + new RawToolCall(Guid.NewGuid(), "run_tests", new Dictionary(), at, TimeSpan.FromSeconds(1), null, Timeout), + new RawToolCall(Guid.NewGuid(), "run_tests", new Dictionary(), at, TimeSpan.FromSeconds(1), "passed", null), + ], + "done", + null))).Outcome); + Assert.Equal(CompleteRunOutcome.Recorded, (await Capture.CompleteRunAsync(runId, Guid.NewGuid(), RunExecutionStatus.Completed, at.AddMinutes(1))).Outcome); + + var result = await Finalization.FinalizeAsync( + new FinalizeExperienceRequest( + RunId: runId, + Authorization: Authorization, + ClosedRound: new ClosedVerificationRound(closed, ArtifactRevision), + RequiredChecks: [new RequiredCheck("tests", "TestResult")], + Evidence: [new Evidence(Guid.NewGuid(), closed, ArtifactRevision, "tests", "TestResult", CheckResult.Pass, "ci", null, at)], + CurrentArtifactRevision: ArtifactRevision, + StorageDecision: StorageDecision.Permit, + FinalizedAt: at.AddMinutes(2)), + CancellationToken.None); + Assert.Equal(FinalizationOutcome.Validated, result.Outcome); + return result.Record!; + } + } +} diff --git a/tests/AgentExperience.MicrosoftAgentFramework.Tests/ExperienceConsolidationTests.cs b/tests/AgentExperience.MicrosoftAgentFramework.Tests/ExperienceConsolidationTests.cs new file mode 100644 index 00000000..1b4339ee --- /dev/null +++ b/tests/AgentExperience.MicrosoftAgentFramework.Tests/ExperienceConsolidationTests.cs @@ -0,0 +1,153 @@ +using System.Runtime.CompilerServices; +using AgentExperience.Core.Finalization; +using AgentExperience.Core.Lifecycle; +using AgentExperience.MicrosoftAgentFramework.Injection; +using Microsoft.Agents.AI; +using Microsoft.Extensions.AI; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; + +namespace AgentExperience.MicrosoftAgentFramework.Tests; + +/// +/// Story 28.6 over the one-call setup: three real runs that failed the same way and were fixed the same way, then +/// GetAgentExperienceLessons().ConsolidateAsync(identity). The view lists one lesson, "consolidated from 3", and a +/// fourth run is given that one lesson, saying how many runs confirmed it. +/// +public sealed class ExperienceConsolidationTests +{ + private const string RunTests = "run_tests"; + + private static readonly ExperienceIdentity Caller = new( + new AuthorizationContext("contoso", "svc", [], DateTimeOffset.UnixEpoch), + new Scope("contoso", "support", "tickets")); + + [Fact] + public async Task Three_runs_consolidate_into_one_lesson_the_view_and_the_next_run_see() + { + var finalized = new List(); + var injections = new List(); + await using var services = Build(finalized, injections); + var lessons = services.GetAgentExperienceLessons(); + var model = new TwoCallModel(); + + // Each run: run_tests exits 3, then 0. The second and third runs are given the earlier lessons, as usual. + for (var n = 0; n < 3; n++) + { + await Agent(services, model, ["exit=3", "exit=0"]).RunAsync($"Ticket #{4810 + n}: a refund is stuck on a lock. Triage it."); + } + + Assert.Equal(3, finalized.Count(result => result.Outcome == FinalizationOutcome.Validated)); + Assert.Equal(3, (await lessons.ListAsync(Caller)).Lessons.Count(lesson => lesson.IsEligible)); + + var result = await lessons.ConsolidateAsync(Caller); + + var group = Assert.Single(result.Groups); + Assert.Equal(ExperienceConsolidationGroupOutcome.Created, group.Outcome); + Assert.Equal(3, group.IndependentRuns); + Assert.All(group.Sources, source => Assert.True(source.Superseded)); + + var page = await lessons.ListAsync(Caller); + var lesson = Assert.Single(page.Lessons, item => item.IsEligible); + Assert.Equal(group.ConsolidatedExperienceId, lesson.ExperienceId); + Assert.Equal(3, lesson.ConsolidatedFrom); + Assert.Contains("Lesson: Verified in 3 independent runs (latest ", lesson.BlockText, StringComparison.Ordinal); + Assert.Contains("Failed first with exit 3 from run_tests in 3 of them. Worked: run_tests → run_tests.", lesson.BlockText, StringComparison.Ordinal); + Assert.Contains("run_tests [failed: exit 3], run_tests [returned]", lesson.BlockText, StringComparison.Ordinal); + Assert.All(page.Lessons.Where(item => item.ExperienceId != lesson.ExperienceId), item => + { + Assert.Equal(ExperienceStatus.Superseded, item.Status); + Assert.Null(item.ConsolidatedFrom); + }); + Assert.Contains("consolidated from 3", ExperienceLessonsHtml.Render(page), StringComparison.Ordinal); + Assert.True(result.Groups.Single().Complete); + Assert.Equal(0, result.IncompleteGroups); + + // Before erasing a source, the host finds the consolidated lesson that copied it. + var copies = await lessons.FindConsolidatedContainingAsync(Caller, group.Sources[0].ExperienceId); + Assert.Equal([group.ConsolidatedExperienceId], copies.ExperienceIds); + + // The next run is given the one consolidated lesson, not three. + await Agent(services, model, ["exit=0", "exit=0"]).RunAsync("Ticket #4999: a refund is stuck on a lock. Triage it."); + Assert.Equal([group.ConsolidatedExperienceId], injections[^1].InjectedExperienceIds); + var block = Assert.Single(model.FirstCalls[^1], message => + message.AdditionalProperties?.ContainsKey(ExperienceContextProvider.HistoricalReferenceKey) == true).Text; + Assert.Contains("Verified in 3 independent runs", block, StringComparison.Ordinal); + } + + [Fact] + public async Task Consolidating_lessons_of_an_identity_outside_the_authorization_is_denied_and_writes_nothing() + { + await using var services = Build([], []); + var outsider = new ExperienceIdentity(Caller.Authorization, new Scope("fabrikam", "support", "tickets")); + + var result = await services.GetAgentExperienceLessons().ConsolidateAsync(outsider); + + Assert.Equal(ExperienceStoreOutcome.Denied, result.Outcome); + Assert.Empty(result.Groups); + } + + private static ServiceProvider Build(List finalized, List injections) + { + var collection = new ServiceCollection(); + collection.AddSingleton(new AgentExperienceFiveLineSetupTests.FakeHostEnvironment(Environments.Development)); + collection + .AddAgentExperience(options => + { + options.ResolveIdentity = (_, _) => ValueTask.FromResult(Caller); + options.Verify = ExperienceVerifiers.ToolSucceeded(RunTests); + options.Capture = capture => + { + capture.ClassifyToolResult = ExperienceToolResultClassifiers.ExitCode(RunTests); + capture.OnRunFinalized = finalized.Add; + }; + options.Injection = injection => injection.OnContextInjected = injections.Add; + }) + .UseInMemoryStorageForDevelopment(); + return collection.BuildServiceProvider(); + } + + private static AIAgent Agent(IServiceProvider services, TwoCallModel model, IEnumerable exits) + { + var queue = new Queue(exits); + var runTests = AIFunctionFactory.Create(() => queue.Dequeue(), RunTests); + return new ChatClientAgent(model, name: "triage-ticket", tools: [runTests]).AsBuilder().UseAgentExperience(services).Build(); + } + + /// A model that calls run_tests twice, one call per turn, then answers. + private sealed class TwoCallModel : IChatClient + { + public List> FirstCalls { get; } = []; + + public Task GetResponseAsync(IEnumerable messages, ChatOptions? options = null, CancellationToken cancellationToken = default) + { + var list = messages.ToList(); + var results = list.SelectMany(m => m.Contents).OfType().Count(); + if (results == 0) + { + FirstCalls.Add(list); + } + + return Task.FromResult(new ChatResponse(results < 2 + ? new ChatMessage(ChatRole.Assistant, [new FunctionCallContent($"call-{results}", RunTests, new Dictionary())]) + : new ChatMessage(ChatRole.Assistant, "Done."))); + } + + public async IAsyncEnumerable GetStreamingResponseAsync( + IEnumerable messages, + ChatOptions? options = null, + [EnumeratorCancellation] CancellationToken cancellationToken = default) + { + foreach (var update in (await GetResponseAsync(messages, options, cancellationToken)).ToChatResponseUpdates()) + { + yield return update; + } + } + + public object? GetService(Type serviceType, object? serviceKey = null) => null; + + public void Dispose() + { + } + } +} diff --git a/tests/AgentExperience.Release.Tests/PublicApi/AgentExperience.Core.verified.txt b/tests/AgentExperience.Release.Tests/PublicApi/AgentExperience.Core.verified.txt index 115ea1f7..207eab56 100644 --- a/tests/AgentExperience.Release.Tests/PublicApi/AgentExperience.Core.verified.txt +++ b/tests/AgentExperience.Release.Tests/PublicApi/AgentExperience.Core.verified.txt @@ -737,6 +737,75 @@ namespace AgentExperience.Core.Lifecycle Deleted = 8, Unverified = 9, } + public sealed class ExperienceConsolidatedLookupResult : System.IEquatable + { + public ExperienceConsolidatedLookupResult(AgentExperience.Abstractions.ExperienceStoreOutcome Outcome, System.Collections.Generic.IReadOnlyList ExperienceIds) { } + public System.Collections.Generic.IReadOnlyList ExperienceIds { get; init; } + public bool MayHaveMore { get; init; } + public AgentExperience.Abstractions.ExperienceStoreOutcome Outcome { get; init; } + } + public static class ExperienceConsolidation + { + public const int MaxLessonToolNames = 16; + public const int MaxPreconditions = 16; + public const int MaxSources = 256; + public const string Producer = "AgentExperience.Consolidation/1"; + public const string ProvenanceSource = "AgentExperience.Consolidation"; + public const string TemplateVersion = "1"; + public static System.Guid ConsolidatedIdFor(AgentExperience.Abstractions.Scope scope, System.Collections.Generic.IEnumerable sourceIds) { } + public static bool IsConsolidated(AgentExperience.Abstractions.ExperienceRecord record) { } + public static System.Collections.Generic.IReadOnlyList SourcesOf(AgentExperience.Abstractions.ExperienceRecord record) { } + } + public enum ExperienceConsolidationGroupOutcome + { + Created = 0, + Existing = 1, + Refused = 2, + Deferred = 3, + } + public sealed class ExperienceConsolidationGroupResult : System.IEquatable + { + public ExperienceConsolidationGroupResult(System.Guid ConsolidatedExperienceId, AgentExperience.Core.Lifecycle.ExperienceConsolidationGroupOutcome Outcome, int IndependentRuns, System.Collections.Generic.IReadOnlyList Sources) { } + public bool Complete { get; } + public System.Guid ConsolidatedExperienceId { get; init; } + public int Deferred { get; init; } + public int IndependentRuns { get; init; } + public AgentExperience.Core.Lifecycle.ExperienceConsolidationGroupOutcome Outcome { get; init; } + public string? Reason { get; init; } + public System.Collections.Generic.IReadOnlyList Sources { get; init; } + } + public sealed class ExperienceConsolidationOptions : System.IEquatable + { + public const int DefaultMinimumSources = 3; + public ExperienceConsolidationOptions() { } + public System.Collections.Generic.IReadOnlyDictionary>? ArgumentKeys { get; init; } + public int MinimumSources { get; init; } + } + public sealed class ExperienceConsolidationResult : System.IEquatable + { + public ExperienceConsolidationResult(AgentExperience.Abstractions.Scope Scope, AgentExperience.Abstractions.ExperienceStoreOutcome Outcome, System.Collections.Generic.IReadOnlyList Groups) { } + public System.Collections.Generic.IReadOnlyList Errors { get; init; } + public System.Collections.Generic.IReadOnlyList Groups { get; init; } + public int IncompleteGroups { get; } + public bool MayHaveMore { get; init; } + public AgentExperience.Abstractions.ExperienceStoreOutcome Outcome { get; init; } + public int RecordsRead { get; init; } + public AgentExperience.Abstractions.Scope Scope { get; init; } + } + public sealed class ExperienceConsolidationService + { + public ExperienceConsolidationService(AgentExperience.Abstractions.IExperienceRecordStore store, AgentExperience.Core.Lifecycle.ExperienceLifecycleService lifecycle, System.TimeProvider? clock = null) { } + public System.Threading.Tasks.Task ConsolidateAsync(AgentExperience.Abstractions.AuthorizationContext authorization, AgentExperience.Abstractions.Scope scope, AgentExperience.Core.Lifecycle.ExperienceConsolidationOptions options, System.Threading.CancellationToken cancellationToken) { } + public System.Threading.Tasks.Task FindConsolidatedContainingAsync(AgentExperience.Abstractions.AuthorizationContext authorization, AgentExperience.Abstractions.Scope scope, System.Guid sourceId, System.Threading.CancellationToken cancellationToken) { } + } + public sealed class ExperienceConsolidationSourceResult : System.IEquatable + { + public ExperienceConsolidationSourceResult(System.Guid ExperienceId, AgentExperience.Core.Lifecycle.LifecycleTransitionOutcome? Outcome) { } + public string? ExceptionType { get; init; } + public System.Guid ExperienceId { get; init; } + public AgentExperience.Core.Lifecycle.LifecycleTransitionOutcome? Outcome { get; init; } + public bool Superseded { get; } + } public sealed class ExperienceLifecycleService { public const AgentExperience.Abstractions.ExperienceStatus FirstEventStatus = 0; diff --git a/tests/AgentExperience.Release.Tests/PublicApi/AgentExperience.MicrosoftAgentFramework.verified.txt b/tests/AgentExperience.Release.Tests/PublicApi/AgentExperience.MicrosoftAgentFramework.verified.txt index e6f071e9..baf51190 100644 --- a/tests/AgentExperience.Release.Tests/PublicApi/AgentExperience.MicrosoftAgentFramework.verified.txt +++ b/tests/AgentExperience.Release.Tests/PublicApi/AgentExperience.MicrosoftAgentFramework.verified.txt @@ -24,7 +24,9 @@ } public sealed class AgentExperienceLessons { + public System.Threading.Tasks.Task ConsolidateAsync(AgentExperience.MicrosoftAgentFramework.ExperienceIdentity identity, AgentExperience.Core.Lifecycle.ExperienceConsolidationOptions? options = null, System.Threading.CancellationToken cancellationToken = default) { } public System.Threading.Tasks.Task ExportMeasuredAsync(AgentExperience.MicrosoftAgentFramework.ExperienceIdentity identity, AgentExperience.MicrosoftAgentFramework.Export.IExperienceLessonSink sink, AgentExperience.Core.Holdout.ExperienceBenefitOptions? options = null, System.Threading.CancellationToken cancellationToken = default) { } + public System.Threading.Tasks.Task FindConsolidatedContainingAsync(AgentExperience.MicrosoftAgentFramework.ExperienceIdentity identity, System.Guid sourceId, System.Threading.CancellationToken cancellationToken = default) { } public System.Threading.Tasks.Task ListAsync(AgentExperience.MicrosoftAgentFramework.ExperienceIdentity identity, AgentExperience.MicrosoftAgentFramework.ExperienceLessonsQuery? query = null, System.Threading.CancellationToken cancellationToken = default) { } public System.Threading.Tasks.Task ListForDevelopmentIdentityAsync(AgentExperience.MicrosoftAgentFramework.ExperienceLessonsQuery? query = null, System.Threading.CancellationToken cancellationToken = default) { } } @@ -125,6 +127,7 @@ public string BlockText { get; init; } public System.Collections.Generic.IReadOnlyList Checks { get; init; } public double Confidence { get; init; } + public int? ConsolidatedFrom { get; init; } public System.DateTimeOffset CreatedAt { get; init; } public System.Guid ExperienceId { get; init; } public AgentExperience.MicrosoftAgentFramework.ExperienceLessonIneligibility? Ineligibility { get; init; } diff --git a/tests/AgentExperience.Storage.Postgres.Tests/AgentExperience.Storage.Postgres.Tests.csproj b/tests/AgentExperience.Storage.Postgres.Tests/AgentExperience.Storage.Postgres.Tests.csproj index 2c73cd13..e6f6883d 100644 --- a/tests/AgentExperience.Storage.Postgres.Tests/AgentExperience.Storage.Postgres.Tests.csproj +++ b/tests/AgentExperience.Storage.Postgres.Tests/AgentExperience.Storage.Postgres.Tests.csproj @@ -28,6 +28,7 @@ + diff --git a/tests/AgentExperience.Storage.Postgres.Tests/PostgresConsolidationTests.cs b/tests/AgentExperience.Storage.Postgres.Tests/PostgresConsolidationTests.cs new file mode 100644 index 00000000..2d8b8b53 --- /dev/null +++ b/tests/AgentExperience.Storage.Postgres.Tests/PostgresConsolidationTests.cs @@ -0,0 +1,36 @@ +using AgentExperience.Tests.Shared; + +namespace AgentExperience.Storage.Postgres.Tests; + +/// +/// Story 28.6 against a real PostgreSQL container: consolidation, an idempotent re-run and a fold-in, through the record +/// store's ports (the supersession guard, the evidence history and the scoped reads), in every mode the suite runs in. +/// +[Collection(PostgresCollection.Name)] +public sealed class PostgresConsolidationTests(PostgresFixture fixture) +{ + [Fact] + public async Task Three_runs_become_one_lesson_a_re_run_does_nothing_and_a_fourth_run_is_folded_in() + { + var tenant = TestRecords.NewTenant(); + var scope = TestRecords.Scope(tenant, project: $"project-{Guid.NewGuid():N}"); + + var outcome = await ConsolidationScenario.RunAsync(new PostgresExperienceRecordStore(fixture.DataSource), TestRecords.Authorize(tenant), scope); + + ConsolidationScenario.Verify(outcome); + } + + [Fact] + public async Task An_erased_source_is_found_first_and_left_out_of_the_next_merge() + { + var tenant = TestRecords.NewTenant(); + var scope = TestRecords.Scope(tenant, project: $"project-{Guid.NewGuid():N}"); + var authorization = TestRecords.Authorize(tenant); + var store = new PostgresExperienceRecordStore(fixture.DataSource); + + var outcome = await ConsolidationScenario.RunAsync(store, authorization, scope, async id => + Assert.Equal(ExperienceStoreOutcome.Deleted, (await store.DeleteAsync(authorization, scope, id, CancellationToken.None)).Outcome)); + + ConsolidationScenario.Verify(outcome); + } +} diff --git a/tests/AgentExperience.Storage.Sqlite.Tests/AgentExperience.Storage.Sqlite.Tests.csproj b/tests/AgentExperience.Storage.Sqlite.Tests/AgentExperience.Storage.Sqlite.Tests.csproj index 93a23538..4dc95b64 100644 --- a/tests/AgentExperience.Storage.Sqlite.Tests/AgentExperience.Storage.Sqlite.Tests.csproj +++ b/tests/AgentExperience.Storage.Sqlite.Tests/AgentExperience.Storage.Sqlite.Tests.csproj @@ -31,6 +31,7 @@ + diff --git a/tests/AgentExperience.Storage.Sqlite.Tests/SqliteConsolidationTests.cs b/tests/AgentExperience.Storage.Sqlite.Tests/SqliteConsolidationTests.cs new file mode 100644 index 00000000..e0170771 --- /dev/null +++ b/tests/AgentExperience.Storage.Sqlite.Tests/SqliteConsolidationTests.cs @@ -0,0 +1,36 @@ +using AgentExperience.Storage.Sqlite; +using AgentExperience.Tests.Shared; + +namespace AgentExperience.Storage.Sqlite.Tests; + +/// Story 28.6 over a SQLite file: consolidation, an idempotent re-run and a fold-in, through the record store's ports. +public sealed class SqliteConsolidationTests : IDisposable +{ + private readonly SqliteTestDatabase _database = new(); + + public void Dispose() => _database.Dispose(); + + [Fact] + public async Task Three_runs_become_one_lesson_a_re_run_does_nothing_and_a_fourth_run_is_folded_in() + { + var scope = new Scope("consolidation-tenant", "app-1", $"project-{Guid.NewGuid():N}"); + var authorization = new AuthorizationContext("consolidation-tenant", "host", ["experience:write"], ConsolidationScenario.Now); + + var outcome = await ConsolidationScenario.RunAsync(new SqliteExperienceRecordStore(_database.ConnectionString), authorization, scope); + + ConsolidationScenario.Verify(outcome); + } + + [Fact] + public async Task An_erased_source_is_found_first_and_left_out_of_the_next_merge() + { + var scope = new Scope("consolidation-tenant", "app-1", $"project-{Guid.NewGuid():N}"); + var authorization = new AuthorizationContext("consolidation-tenant", "host", ["experience:write"], ConsolidationScenario.Now); + var store = new SqliteExperienceRecordStore(_database.ConnectionString); + + var outcome = await ConsolidationScenario.RunAsync(store, authorization, scope, async id => + Assert.Equal(ExperienceStoreOutcome.Deleted, (await store.DeleteAsync(authorization, scope, id, CancellationToken.None)).Outcome)); + + ConsolidationScenario.Verify(outcome); + } +} diff --git a/tests/Shared/ConsolidationScenario.cs b/tests/Shared/ConsolidationScenario.cs new file mode 100644 index 00000000..b351b96c --- /dev/null +++ b/tests/Shared/ConsolidationScenario.cs @@ -0,0 +1,227 @@ +using AgentExperience.Abstractions; +using AgentExperience.Core.Capture; +using AgentExperience.Core.Confidence; +using AgentExperience.Core.Finalization; +using AgentExperience.Core.Lifecycle; +using AgentExperience.Core.Reflections; +using AgentExperience.Core.Sanitization; +using AgentExperience.Core.Verification; + +namespace AgentExperience.Tests.Shared; + +/// +/// Story 28.6 end to end over a real store: three verified runs that failed the same way and were fixed the same way (the +/// second and third given the first lesson, so reuse evidence makes its counters overlap theirs), one consolidation pass, +/// a second pass that does nothing, then a fourth run folded in by a third pass. The same steps whichever store is passed. +/// +internal static class ConsolidationScenario +{ + internal static readonly DateTimeOffset Now = new(2026, 10, 30, 9, 0, 0, TimeSpan.Zero); + + private const string Timeout = "TimeoutException: the test host did not answer after 30 seconds."; + + private static readonly ClosedVerificationRound Round = new(Guid.Parse("5d3c9e2a-28a6-4c57-9b1e-7e0a5d1e2861"), "rev-1"); + + private static readonly SanitizationOptions Sanitization = new(new Dictionary(StringComparer.Ordinal) + { + ["ToolArguments"] = new(new HashSet(StringComparer.Ordinal), new HashSet(StringComparer.Ordinal), 2, 5, 1_000, 100), + ["ToolResult"] = new(new HashSet(StringComparer.Ordinal) { "value" }, new HashSet(StringComparer.Ordinal), 2, 5, 1_000, 100), + }); + + /// What the scenario saw. + internal sealed record Outcome( + IReadOnlyList Sources, + ExperienceConsolidationResult First, + ExperienceRecord Consolidated, + IReadOnlyList SourcesAfter, + IReadOnlyList SourcesLastEvents, + ExperienceConsolidationResult Second, + ExperienceRecord Later, + ExperienceConsolidationResult Third, + ExperienceRecord Folded, + ExperienceRecord OldAfter, + IReadOnlyList Live) + { + /// What the lookup found for the erased source before it was erased; empty when nothing was erased. + public IReadOnlyList FoundBeforeErasure { get; init; } = []; + + /// Whether the first source was erased before the fourth run. + public bool Erased { get; init; } + } + + internal static async Task RunAsync( + IExperienceRecordStore store, AuthorizationContext authorization, Scope scope, Func? erase = null) + { + var capture = new InMemoryExperienceCaptureService(new DefaultSanitizer(Sanitization), new CaptureLimits(10, 10, 1_000, 1_000)); + var lifecycle = new ExperienceLifecycleService(store, indexingService: null, new ExperienceIndependenceOptions(), capture); + var finalization = new ExperienceFinalizationService( + capture, + new DefaultExperienceReflector(), + store, + lifecycle, + indexingService: null, + indexingTimeout: null, + provenanceSigning: null, + reflectionSanitizer: null, + new ExperienceFinalizationOptions { ReuseEvidence = ReuseEvidenceMode.SameTask }); + var consolidation = new ExperienceConsolidationService(store, lifecycle, new FixedClock(Now.AddDays(30))); + var options = new ExperienceConsolidationOptions(); + + var first = await LessonAsync(Now, []); + var second = await LessonAsync(Now.AddDays(1), [first]); + var third = await LessonAsync(Now.AddDays(2), [first]); + List sources = [await ReadAsync(first.ExperienceId), second, third]; + + var pass = await consolidation.ConsolidateAsync(authorization, scope, options, CancellationToken.None); + var id = pass.Groups.Single().ConsolidatedExperienceId; + var consolidated = await ReadAsync(id); + var after = new List(); + var events = new List(); + foreach (var source in sources) + { + after.Add(await ReadAsync(source.ExperienceId)); + var history = await store.GetFirstHistoryPageAsync(authorization, scope, source.ExperienceId, CancellationToken.None); + events.Add(history.Events[^1].Event); + } + + var again = await consolidation.ConsolidateAsync(authorization, scope, options, CancellationToken.None); + IReadOnlyList foundBeforeErasure = []; + if (erase is not null) + { + // A host erasing a source first finds the consolidated records that copied it, and erases those too. + foundBeforeErasure = (await consolidation.FindConsolidatedContainingAsync(authorization, scope, sources[0].ExperienceId, CancellationToken.None)).ExperienceIds; + await erase(sources[0].ExperienceId); + } + + var later = await LessonAsync(Now.AddDays(10), []); + var third2 = await consolidation.ConsolidateAsync(authorization, scope, options, CancellationToken.None); + var folded = await ReadAsync(third2.Groups.Single().ConsolidatedExperienceId); + var live = await store.QueryAsync(authorization, new ExperienceRecordQuery(scope, ExperienceStatuses.EligibleForReuse), CancellationToken.None); + + return new( + sources, + pass, + consolidated, + after, + events, + again, + later, + third2, + folded, + await ReadAsync(id), + [.. live.Records.Select(record => record.ExperienceId)]) + { + FoundBeforeErasure = foundBeforeErasure, + Erased = erase is not null, + }; + + async Task ReadAsync(Guid experienceId) => + (await store.GetAsync(authorization, scope, experienceId, CancellationToken.None)).Record + ?? throw new InvalidOperationException($"Record {experienceId} was not found."); + + async Task LessonAsync(DateTimeOffset at, IReadOnlyList exposedTo) + { + var runId = Guid.NewGuid(); + capture.StartRun( + runId, + "keep-the-build-green", + "Keep the build green", + scope, + new EnvironmentFingerprint("host-1", "net10.0", "linux", "1.0.0", new Dictionary()), + new Provenance("consolidation-scenario", "1.0.0", at, null), + at); + if (exposedTo.Count > 0) + { + capture.RecordExposure(runId, [.. exposedTo.Select(record => new RunExposure(record.ExperienceId, record.Revision))]); + } + + await capture.AppendAttemptAsync( + runId, + new AppendAttemptRequest( + Guid.NewGuid(), + at, + TimeSpan.FromSeconds(1), + [ + new RawToolCall(Guid.NewGuid(), "run_tests", new Dictionary(), at, TimeSpan.FromSeconds(1), null, Timeout), + new RawToolCall(Guid.NewGuid(), "run_tests", new Dictionary(), at, TimeSpan.FromSeconds(1), "passed", null), + ], + "done", + null)); + await capture.CompleteRunAsync(runId, Guid.NewGuid(), RunExecutionStatus.Completed, at.AddMinutes(1)); + var result = await finalization.FinalizeAsync( + new FinalizeExperienceRequest( + runId, + authorization, + Round, + [new RequiredCheck("tests", "TestResult")], + [new Evidence(Guid.NewGuid(), Round.RoundId, Round.ArtifactRevision, "tests", "TestResult", CheckResult.Pass, "ci", null, at)], + Round.ArtifactRevision, + StorageDecision.Permit, + at.AddMinutes(2)), + CancellationToken.None); + return result.Outcome == FinalizationOutcome.Validated + ? result.Record! + : throw new InvalidOperationException($"Finalization ended {result.Outcome}: {result.Reason}"); + } + } + + /// What every store must show for the scenario. + internal static void Verify(Outcome outcome) + { + var group = Assert.Single(outcome.First.Groups); + Assert.Equal(ExperienceConsolidationGroupOutcome.Created, group.Outcome); + Assert.Equal(3, group.IndependentRuns); + Assert.All(group.Sources, source => Assert.True(source.Superseded)); + + // One lesson, saying how many runs confirmed it, carrying the three runs' distinct confirmations: the first + // lesson's counters (3, its own and the two later runs' support) overlap the later runs' own validations. + var consolidated = outcome.Consolidated; + Assert.Equal(5, outcome.Sources.Sum(source => source.SupportingValidations)); + Assert.Equal(ExperienceStatus.Validated, consolidated.Status); + Assert.True(ExperienceConsolidation.IsConsolidated(consolidated)); + Assert.Equal(outcome.Sources.Select(source => source.ExperienceId).Order(), ExperienceConsolidation.SourcesOf(consolidated)); + Assert.Equal(3, consolidated.SupportingValidations); + Assert.Equal(4d / 5d, consolidated.ReuseConfidence, precision: 12); + Assert.Equal( + "Verified in 3 independent runs (latest 2026-11-01). Failed first with TimeoutException from run_tests in 3 of them. Worked: run_tests → run_tests.", + consolidated.Reflection!.Lesson); + Assert.Equal(outcome.Sources[^1].Attempts.Select(attempt => attempt.AttemptId), consolidated.Attempts.Select(attempt => attempt.AttemptId)); + + Assert.All(outcome.SourcesAfter, source => Assert.Equal(ExperienceStatus.Superseded, source.Status)); + Assert.All(outcome.SourcesLastEvents, last => + { + Assert.Equal(ExperienceStatus.Superseded, last.CurrentStatus); + Assert.Equal(consolidated.ExperienceId, last.ReplacementExperienceId); + Assert.Equal(ExperienceConsolidation.Producer, last.Producer); + }); + + Assert.Empty(outcome.Second.Groups); + + var fold = Assert.Single(outcome.Third.Groups); + Assert.Equal(ExperienceConsolidationGroupOutcome.Created, fold.Outcome); + Assert.True(fold.Complete); + Assert.Equal(new[] { consolidated.ExperienceId, outcome.Later.ExperienceId }.Order(), fold.Sources.Select(source => source.ExperienceId)); + Assert.All(fold.Sources, source => Assert.True(source.Superseded)); + + // With the first source erased, the fold-in leaves it out: three runs, three sources. Its counted support of + // the first lesson (the later runs' reuse evidence) went with it, and their own validations still count. + var runs = outcome.Erased ? 3 : 4; + Assert.Equal(runs, fold.IndependentRuns); + Assert.Equal(runs, ExperienceConsolidation.SourcesOf(outcome.Folded).Count); + Assert.Equal(runs, outcome.Folded.SupportingValidations); + Assert.StartsWith($"Verified in {runs} independent runs (latest 2026-11-09).", outcome.Folded.Reflection!.Lesson, StringComparison.Ordinal); + if (outcome.Erased) + { + Assert.Equal([consolidated.ExperienceId], outcome.FoundBeforeErasure); + Assert.DoesNotContain(outcome.Sources[0].ExperienceId, ExperienceConsolidation.SourcesOf(outcome.Folded)); + } + + Assert.Equal(ExperienceStatus.Superseded, outcome.OldAfter.Status); + Assert.Equal([outcome.Folded.ExperienceId], outcome.Live); + } + + private sealed class FixedClock(DateTimeOffset now) : TimeProvider + { + public override DateTimeOffset GetUtcNow() => now; + } +}