diff --git a/CHANGELOG.md b/CHANGELOG.md index 2142be34..5c210015 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -316,17 +316,6 @@ AgentExperience.NET is a **preview**. It is not production ready, and public API others `AlreadyRecorded`, a changed resubmission `Conflict` from all three), all three read the same history and confidence, and each agent alone leaves the same ledger, evidence, event and record rows in its own database. -## 0.1.0-preview.17 - -This preview measures whether injected lessons help. An opt-in holdout withholds the block on a deterministic share of -runs and records both arms; `AgentExperienceBenefit.EstimateAsync` compares the arms' verified outcomes, overall, per -lesson and optionally per model; the injection provider can act on the measured benefit; the same holdout measures any -other MAF memory provider; and lessons measured as helping can be exported to other memory stores. PostgreSQL gains -migrations `0027` to `0029` (re-run `ApplyApplicationRolePrivilegesAsync` after migrating). The Python and Go ports -take part in the holdout through contract 1.8 (add-only). Every new option is off by default, and the public API -changes are additions. The same tag releases the six NuGet packages, the Python distribution `agenticexperience` and -the six Go modules at one version. - ### A scoped supersession lock and ranking to the bit: contract 1.18 (story 28.1) - **A supersession locks only rows in the caller's scope.** The PostgreSQL store locked the record and its proposed @@ -487,6 +476,40 @@ the six Go modules at one version. roles, vectors, retention sweeps. Internally, the PostgreSQL payload's attempt, outcome, environment and provenance mappings are each one method now, shared with the SQLite store; nothing stored changes. +### Precision before recall: an injection-time match floor (story 28.5) + +- **`ExperienceInjectionOptions.MinimumMatch`** (type `ExperienceMatchFloor`, off by default): a record is injected only + when it has an exact failure-signature match on the current run (a recall hint), text coverage at least + `MinimumTextCoverage` (the share of the request's distinct terms it contains, relative to the store's terms; required, + no default), or semantic similarity at least `MinimumSemanticSimilarity` (by default the retrieval policy's + `MinimumVectorRelevance`; with neither set the semantic leg is off). A weaker record is omitted as the new + `InjectionOmissionReason.WeakMatch`, with a content-free detail, before the capability gate and the record limit, so + it takes no slot and a stronger one backfills; a host-limited request is widened like the capability gate's. A block + that injects nothing with a `WeakMatch` omission ends `NothingToInject`, and its `inject` span carries + `agentexperience.nothing_reason = weak_match`. The holdout splits after the floor. Under a floor, recall never asks + for a coarse signature match. Which candidates retrieval returns is unchanged, and with the floor off everything is + byte for byte what it was. +- **Each channel's strength travels with the record.** `ExperienceCandidate.TermCoverage` (the in-memory, PostgreSQL and + SQLite text sources set it), `RankedExperience.TextCoverage` and `RankedExperience.VectorRelevance` (set by + retrieval). Ranking is unchanged. +- **Chosen by a rule fixed before measuring, and the rule said off.** The retrieval benchmark's goldens gain an + `INJECTION FLOOR` sweep (0 to 0.6). No floor kept unexpected records at or below 0.15 in the first 8 on both stores, and + none removed the distractor-overlap noise (distractors containing every word of a short query), while every floor + cost long, realistic requests their lesson; so the floor stays off and has no recommended value + ([the table](docs/benchmarks.md#the-injection-floor-story-285)). +- The Python and Go ports and the contract are unchanged; the ports do not apply a floor yet. + +## 0.1.0-preview.17 + +This preview measures whether injected lessons help. An opt-in holdout withholds the block on a deterministic share of +runs and records both arms; `AgentExperienceBenefit.EstimateAsync` compares the arms' verified outcomes, overall, per +lesson and optionally per model; the injection provider can act on the measured benefit; the same holdout measures any +other MAF memory provider; and lessons measured as helping can be exported to other memory stores. PostgreSQL gains +migrations `0027` to `0029` (re-run `ApplyApplicationRolePrivilegesAsync` after migrating). The Python and Go ports +take part in the holdout through contract 1.8 (add-only). Every new option is off by default, and the public API +changes are additions. The same tag releases the six NuGet packages, the Python distribution `agenticexperience` and +the six Go modules at one version. + ### The holdout in Python and Go: contract 1.8 (story 25.6) - **Contract 1.8** ([what it adds](contract/README-1.8.md)) brings the injection holdout into the language-neutral diff --git a/docs/benchmarks.md b/docs/benchmarks.md index 04c304f6..5d7fb425 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -287,3 +287,35 @@ compound cannot meet the minimum alone. **Today's numbers** (recorded 2026-10-10 - **This is the source's order, not Core's final ranking.** The benchmark calls the candidate source directly; Core then re-scores candidates with confidence, recency, status and environment, and on PostgreSQL sees every full match at the same text relevance, so what an agent is finally shown can be ordered differently. + +### The injection floor (story 28.5) + +Story 28.5 added an injection-time match floor (`ExperienceInjectionOptions.MinimumMatch`, see +[Keeping weak matches out](guide/injection.md#keeping-weak-matches-out)) and extended the goldens with an +`INJECTION FLOOR` section: what would reach an 8-record block if only candidates covering at least a share `f` of the +query's distinct terms were injected. It takes the candidates in the source's relevance order (not Core's composite +score, which also weighs confidence, recency, status and environment), cut to 8, text only; a test checks that its +`f = 0` row equals each golden's overall R@8 and FP@8. The rule was fixed before any run: sweep `f` over 0, 0.3, 0.4, +0.5 and 0.6, keep the floors whose FP@8 is at most 0.15 on both adapters (in-memory and PostgreSQL) and whose +`distractor-overlap` queries inject nothing unexpected, and take the one with the highest PostgreSQL R@8 (ties to the +higher floor); if one is chosen it becomes the default, otherwise the floor stays off. + +| Floor | In-memory R@8 | In-memory FP@8 | In-memory distractor FP@8 | PostgreSQL R@8 | PostgreSQL FP@8 | PostgreSQL distractor FP@8 | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 0 (off) | 0.700 | 0.417 | 0.667 | 0.844 | 0.521 | 0.667 | +| 0.3 | 0.611 | 0.250 | 0.667 | 0.733 | 0.313 | 0.667 | +| 0.4 | 0.533 | 0.250 | 0.667 | 0.656 | 0.313 | 0.667 | +| 0.5 | 0.533 | 0.250 | 0.667 | 0.622 | 0.313 | 0.667 | +| 0.6 | 0.467 | 0.250 | 0.667 | 0.578 | 0.313 | 0.667 | + +**No floor qualifies, so the floor stays off and has no default value.** Above 0.3 the noise does not move: what is left +are distractors that contain *every* word of a short query (`distractor-03`'s "reduce image size" and `dis-07`), which no +coverage floor can tell from the right lesson. What a floor removes first is the right lesson for long, realistic +requests, where three matched words are a small share of the request: at 0.3, 5 realistic queries lose their record on +PostgreSQL and 4 in memory. The SQLite golden (added after the rule was fixed, so not part of it) has the same shape: +FP@8 0.625 at 0, 0.354 at 0.3 and 0.313 at 0.6, with distractor FP@8 0.667 throughout. Coverage is relative to each +store's terms (unstemmed words in memory, stemmed lexemes on PostgreSQL, Porter stems on SQLite). + +The hybrid column needs a real embedding provider, so the sweep was not run with vectors; its harness counts a record +the vector channel found above the policy's floor as passing (the semantic leg). The real-task question, whether +injecting fewer, stronger lessons helps an agent, is the holdout's to answer per deployment. diff --git a/docs/guide/injection.md b/docs/guide/injection.md index 7e4cdf27..1c49db54 100644 --- a/docs/guide/injection.md +++ b/docs/guide/injection.md @@ -165,6 +165,7 @@ it through its own redaction first. It recognises no language: no stemming, no s | `MessageRole` | `User` | The chat role the block is sent in: `User` or `System`. Some chat APIs reject, move or merge a system message that is not first, so test `System` with your provider. See [The payload](#the-payload). | | `DecideInjection` | none (permit) | Per-candidate host risk decision, asked after the final eligibility check. Fail-closed: a callback that throws or returns `null` denies. Synchronous on purpose: with session tracking on it runs while the session's lock is held. Per-candidate I/O has no async hook: prefetch what it needs, keyed by scope, in `ResolveRequestAsync`, or decide offline. | | `ReceivingAgent` | none (off) | The receiving agent's tools (declared, or the run's own with `UseRunTools`) and maximum risk class. A record whose verified approach it cannot, or must not, carry out is not injected and takes no record slot. See [Gating on the receiving agent's capabilities](#gating-on-the-receiving-agents-capabilities). | +| `MinimumMatch` | `null` (off) | The match-strength floor: a record is injected only when its match is strong enough (an exact failure-signature match, enough of the request's terms, or enough semantic similarity); weaker ones are omitted as `WeakMatch` and take no slot. See [Keeping weak matches out](#keeping-weak-matches-out). | | `ModelAuthoredLessons` (since `0.1.0-preview.5`) | `Include` | Whether records whose free text a model wrote are injected (labelled) or omitted. `Exclude` asks retrieval to leave them out before its limit (since `0.1.0-preview.6`); the provider still drops any that arrive. See [Model-authored lessons](#model-authored-lessons). | | `OnContextInjected` | none | Receives the content-free account of every attempt, including every omission and its reason. Exceptions it throws are swallowed. No result is emitted when the caller cancels the invocation: the cancellation propagates instead. | | `TimeProvider` | `TimeProvider.System` | The clock the final eligibility check measures record expiry and its own timeout with. | @@ -195,7 +196,7 @@ agent-framework does not hand them to a context provider, so only the agent's ow added count. In Go a zero window multiplier means 4, and `Deny("")` reports the default detail. Still .NET-only: session tracking, withdrawal and the final re-read (`SessionLimits`, `SessionStateKey`, `Limits.EligibilityCheckTimeout`, `Limits.MaxAbandonedReads`), unconfirmed content, the decision context's grant fields and required environment -attributes. +attributes, and the match floor (`MinimumMatch`): the ports do not apply a floor yet. ## What the agent sees @@ -737,6 +738,61 @@ new ExperienceInjectionOptions collection, or an undefined `ToolRiskClass`, with `ArgumentException`. With `ReceivingAgent` unset, the block, the omissions and the outcomes are byte for byte what they are without it. `CapabilityGateTests` pins each case. +## Keeping weak matches out + +Retrieval's channel floors decide which records are *candidates*: by default, a text candidate needs any three of the +request's terms (see [How the text channel matches](retrieval.md#how-the-text-channel-matches)), which finds long, +paraphrased requests at the price of some records that share only a few words. Without a floor, every candidate that +survives the checks below is injected however weakly it matched. The real-task experiment found that a wrong or +empty lesson costs the agent: its mismatched-cluster control did worse than no memory. + +`MinimumMatch` is a second, separate gate, applied at injection time. A record is injected only if **at least one** of +these holds: + +| Leg | Passes when | Read from | +| --- | --- | --- | +| Failure signature | an exact signature match on the current run, which only a [recall hint](#recall-at-the-moment-of-failure) has; the before-run block has none | the recall lookup | +| Text | the record contains at least `MinimumTextCoverage` (required, no default) of the request's distinct terms | `RankedExperience.TextCoverage` | +| Semantic | its vector relevance is at least `MinimumSemanticSimilarity`, or, when that is `null`, the retrieval policy's `MinimumVectorRelevance`; with neither set this leg is **off** | `RankedExperience.VectorRelevance` | + +```csharp +options.MinimumMatch = new ExperienceMatchFloor { MinimumTextCoverage = 0.5 }; +``` + +A record below it is omitted as `WeakMatch`, with a content-free detail naming its strengths and the floor. It is +checked on the ranked candidate after `AlreadyDelivered` and before the capability gate and the record limit, so a weak +match takes no slot and a stronger record ranked below it fills it; like the capability gate, a request that names +its own `Limit` asks retrieval for a window four times `MaxRecords` to backfill from (a weak match fetched only as +backfill, past the host's own limit, is not reported). When nothing is injected and at least one record was omitted as +`WeakMatch`, the outcome is `NothingToInject` and the `inject` span carries `agentexperience.nothing_reason = +weak_match`, even if other reasons (a capability gate, an ineligible re-read) kept other records out too: the attribute +says the floor contributed, and the result's `Omitted` list says what kept each record out. The holdout splits its arms +after the floor, so both arms are judged on the same eligible set. A recall hint is a signature match only, so under a +floor only an exact match counts and recall never asks for a coarse one, whatever `AllowCoarseMatch` says. + +Things to know: + +- **It is off by default, on evidence, and has no recommended value.** On the + [retrieval benchmark](../benchmarks.md#the-injection-floor-story-285), no floor met the precision target fixed before + measuring (at most 0.15 unexpected records in the first 8 on both stores, and none on the distractor-overlap queries), + so `MinimumTextCoverage` has no default: choose it for your requests and measure. The noise a coverage floor cannot remove is distractors that contain + *every* word of a short request; what it removes first is the right lesson for a long, realistic one. Turn it on + where your requests are short and your lessons varied, and measure with the holdout. +- **Coverage is a share, not a count.** A twelve-term request matched on three terms covers 0.25, which a floor of 0.3 + keeps out. That is the trade: the channel floor is a count precisely so long requests find something. +- **Coverage is relative to the store.** A term is what the text source counts: whole, unstemmed words in memory + ("invoices" and "invoice" are two); stemmed lexemes on PostgreSQL, less stopwords, with a hyphenated compound counted + once; Porter-stemmed words on SQLite. The same request can have a different number of terms, and a record a + different coverage, on each store, so a floor tuned on one is not tuned on another. +- **The semantic leg fails closed.** With neither `MinimumSemanticSimilarity` nor the policy's `MinimumVectorRelevance` + set, it is off: a record only the vector channel found is a weak match however close. Setting + `MinimumSemanticSimilarity` below the policy's floor does nothing, because retrieval never returns a vector candidate + below the policy's floor. +- **Comparisons allow 1e-6**, so a coverage PostgreSQL computes in single precision (3 of 10 terms as 0.3000000119) is not + lost at a floor of 0.3. +- The lessons view sets the task aside and runs no retrieval, so it shows no floor. +- The Python and Go ports do not apply a floor yet. + ## Model-authored lessons A lesson written by a model-backed reflector (`Reflection.Authorship` is anything but diff --git a/docs/guide/retrieval.md b/docs/guide/retrieval.md index c45e3d0f..b6e877d7 100644 --- a/docs/guide/retrieval.md +++ b/docs/guide/retrieval.md @@ -151,6 +151,12 @@ finds anything. var policy = RetrievalPolicy.Default with { MinimumMatchedTerms = ExperienceCandidateQuery.AllTerms }; ``` +Each candidate also reports its plain coverage, the share of the request's distinct terms it contains whatever fields +hold them (`ExperienceCandidate.TermCoverage`), which retrieval carries to `RankedExperience.TextCoverage` beside the +vector channel's `VectorRelevance`. Neither changes ranking; the injection provider's optional match floor reads them +to keep weak matches out of the block without changing which candidates retrieval returns (see +[Keeping weak matches out](injection.md#keeping-weak-matches-out)). + ## Ranking is explainable Every returned record carries all five normalized components (each in 0–1) and the effective weight applied to it, diff --git a/docs/telemetry-names.json b/docs/telemetry-names.json index aab1183f..277697c5 100644 --- a/docs/telemetry-names.json +++ b/docs/telemetry-names.json @@ -45,6 +45,7 @@ "agentexperience.feedback_id", "agentexperience.history_read", "agentexperience.independence.refusal", + "agentexperience.nothing_reason", "agentexperience.omitted_count", "agentexperience.operation", "agentexperience.outcome", diff --git a/docs/telemetry.md b/docs/telemetry.md index 236e2850..d2cab0d3 100644 --- a/docs/telemetry.md +++ b/docs/telemetry.md @@ -245,6 +245,7 @@ These are the only span attributes the library writes. | `agentexperience.omitted_count` | integer: how many ranked records the injection left out. The omission reasons stay on the typed result | `inject` | | `agentexperience.retracted_count` | integer: how many withdrawal notices the injected block carried, for records delivered earlier in the session. The IDs stay on the typed result | `inject`, only when the count is not zero (session tracking) | | `agentexperience.history_read` | `failed` or `timed_out`: why the single-call wiring's read of the chat history a `ChatClientAgent` replays joined nothing (the injection went on with the caller's messages alone) | `inject`, only under `UseAgentExperience` without the provider on the agent's options, and only when that read failed or ran past its 500-millisecond bound | +| `agentexperience.nothing_reason` | `weak_match`: the match floor (`ExperienceInjectionOptions.MinimumMatch`) kept at least one record out of a block that injected nothing | `inject`, only on a `NothingToInject` outcome with a `WeakMatch` omission | | `agentexperience.erased_count` | integer: how many records the sweep erased, how many grants the purge removed, or how many access rows the access purge removed. Which ones stays in the database | `retention.sweep`, `grant.purge`, `grant.access.purge`, on every returned result (0 on a refusal); `retention.sweep` also when it threw `ExperienceRetentionSweepInterruptedException`, taken from its `Partial` | | `agentexperience.interrupted` | boolean: whether the sweep stopped before the end of its batch | `retention.sweep`, where `erased_count` is written | | `agentexperience.scope_match` | `Exact` or `Subtree`: how wide the call was asked to reach (the `ScopeMatch` member's name, never the scope) | `retention.sweep`, `grant.access.purge`, `record.seal`, `record.authorship.backfill`, `record.signature.backfill`, whenever the caller passed a defined `ScopeMatch`; the five-argument `SweepExpiredAsync` reports `Exact` | @@ -301,7 +302,7 @@ a value is a deliberate change that widens every instrument's cardinality. | `deindex` | `ExperienceIndexingService.RemoveAsync` | `ExperienceDeindexingOutcome`: `Removed`, `NotIndexed`, `Denied`, `Failed` | `experience_id` | — | | `reindex` | `ExperienceIndexingService.ReindexAsync` | `ExperienceReindexOutcome`: `Completed`, `Denied`, `Invalid`, `Failed` | none (a pass has no single record) | — | | `reuse_feedback` | `ExperienceReuseFeedbackService.RecordAsync` | `ExperienceReuseFeedbackOutcome`: `Recorded`, `AlreadyRecorded`, `Conflict`, `Denied`, `Invalid` | `feedback_id`, `run_id` | `confidence.apply` (once per attributed exposed record) | -| `inject` | `ExperienceContextProvider` (MAF `AIContextProvider`) | `InjectionOutcome`: `Injected`, `NothingToInject`, `Skipped`, `RetrievalTimedOut`, `RetrievalDenied`, `RetrievalFailed`, `Failed`, `Retracted` (a block of withdrawal notices only), `SessionBudgetExhausted`, `Withheld` (the injection holdout withheld a built block from a captured run; see `ExperienceInjectionOptions.Holdout`) | `correlation_id`, `omitted_count`, `retracted_count`, `history_read` | none on its own meter; its `retrieve` is emitted on the Core meter with `nested=false` | +| `inject` | `ExperienceContextProvider` (MAF `AIContextProvider`) | `InjectionOutcome`: `Injected`, `NothingToInject`, `Skipped`, `RetrievalTimedOut`, `RetrievalDenied`, `RetrievalFailed`, `Failed`, `Retracted` (a block of withdrawal notices only), `SessionBudgetExhausted`, `Withheld` (the injection holdout withheld a built block from a captured run; see `ExperienceInjectionOptions.Holdout`) | `correlation_id`, `omitted_count`, `retracted_count`, `history_read`, `nothing_reason` | none on its own meter; its `retrieve` is emitted on the Core meter with `nested=false` | | `recall` | `ExperienceContextProvider`, once per failed tool call it looks at when recall at the moment of failure is on (`ExperienceInjectionOptions.Recall`, off by default), inside the tool call, before the model's next call | `ExperienceRecallOutcome`: `Hinted`, `Withheld` (the run is in the injection holdout's holdout arm), `NoMatch`, `TimedOut`, `Failed`, `Bounded` (the signature was already looked up in the run, or a run or session bound was reached) | `correlation_id`, `signature.match`, `record_count` | none on its own meter; its `retrieve.signature` is emitted on the Core meter with `nested=false` | | `record_injection` | `ExperienceContextProvider`, in the background after an injected block is handed over: it records when the block's records were injected, for the lessons view. No span (the injection's has closed); counted, timed and failure-counted on the same instruments | `Recorded`; `NotTracked` (the store does not track injections, `IExperienceRecordStore.TracksInjections`); `NotRecorded` (the store refused or failed the write; `error.class` `Infrastructure` or `Unexpected` when it threw, `Timeout` when too many earlier writes are still running); `OutcomeUnknown` (the write overran its few-second bound, `Timeout`, or was cancelled, `Cancelled`: it may have landed) | — | — | | `record_holdout` | `ExperienceContextProvider`, in the background, when the injection holdout (`ExperienceInjectionOptions.Holdout`, off by default) assigned a captured run to an arm: it records the run's arm and the records its block named, delivered or withheld. No span; counted, timed and failure-counted on the same instruments | The same four as `record_injection`; `NotTracked` means the store does not record holdout assignments (`IExperienceRecordStore.TracksHoldout`) | — | — | @@ -420,7 +421,8 @@ them. Differences from .NET that a dashboard should know: `omitted_count` (on `inject`), and `event_id`, `confidence.admission` and `independence.refusal` (on `lifecycle.commit` and `confidence.apply`, contract 1.16), `feedback_id` (on `reuse_feedback`, contract 1.17), and `reflection.screening_refusal` (on `finalize`). They do not write - `correlation_id`, `retracted_count`, `history_read` or `reuse_evidence.submitted`, because they have no such thing. + `correlation_id`, `retracted_count`, `history_read`, `nothing_reason` or `reuse_evidence.submitted`, because they have + no such thing. - An `inject` is `Faulted` only when something escapes it (the invocation's own cancellation in Python); every other failure is a returned outcome, as in .NET. diff --git a/src/AgentExperience.Abstractions/ExperienceCandidateSource.cs b/src/AgentExperience.Abstractions/ExperienceCandidateSource.cs index 36080eaf..12d7e991 100644 --- a/src/AgentExperience.Abstractions/ExperienceCandidateSource.cs +++ b/src/AgentExperience.Abstractions/ExperienceCandidateSource.cs @@ -197,6 +197,20 @@ public sealed record ExperienceCandidate( /// /// public ExperienceGrantDisclosure? GrantDisclosure { get; init; } + + /// + /// The share of the query's distinct terms the record contains, in [0, 1], as the text search counted them, or + /// when the source does not report it (a vector match, or a source written before it existed). + /// Unlike , which a source may weight by field, it is a plain count: three of a twelve-term + /// query's terms is 0.25 whichever fields hold them. + /// + /// + /// Both shipped text sources report it: on PostgreSQL it equals (computed in single + /// precision), and in memory it is the share of whole-word terms, before the field weighting. Core's retrieval + /// carries it to RankedExperience.TextCoverage, falling back to when a text source + /// leaves it ; the injection provider's match floor reads it there. + /// + public double? TermCoverage { get; init; } } /// diff --git a/src/AgentExperience.Core/Retrieval/ExperienceRetrievalService.cs b/src/AgentExperience.Core/Retrieval/ExperienceRetrievalService.cs index 765e0788..d7e693a7 100644 --- a/src/AgentExperience.Core/Retrieval/ExperienceRetrievalService.cs +++ b/src/AgentExperience.Core/Retrieval/ExperienceRetrievalService.cs @@ -829,12 +829,16 @@ private ExperienceRetrievalResult Rank( var positions = new Dictionary(); var truncated = false; - if (Absorb(request, channels.Text.Candidates, "candidate source", merged, positions, ref truncated) is { } textFailure) + // Story 28.5: what each channel said about each record, kept apart from the merged relevance, for the injection floor. + var textCoverage = new Dictionary(); + var vectorRelevance = new Dictionary(); + + if (Absorb(request, channels.Text.Candidates, "candidate source", merged, positions, ref truncated, textCoverage, static candidate => candidate.TermCoverage ?? candidate.Relevance) is { } textFailure) { return Empty(RetrievalOutcome.Failed, request, unrestricted, startedAt, textFailure, channels.Vector.Fallback); } - if (Absorb(request, channels.Vector.Candidates, "embedding index", merged, positions, ref truncated) is { } vectorFailure) + if (Absorb(request, channels.Vector.Candidates, "embedding index", merged, positions, ref truncated, vectorRelevance, static candidate => candidate.Relevance) is { } vectorFailure) { // A vector channel that answered with something unverifiable is treated exactly like a // text one that did: fail-closed. Silently dropping it would mean returning a result @@ -903,7 +907,12 @@ private ExperienceRetrievalResult Rank( channels.Vector.Fallback); } - ranked.Add((Score(record, candidate, environment, now), record.ExperienceId.ToString("D"))); + var scored = Score(record, candidate, environment, now) with + { + TextCoverage = textCoverage.TryGetValue(record.ExperienceId, out var coverage) ? coverage : null, + VectorRelevance = vectorRelevance.TryGetValue(record.ExperienceId, out var vector) ? vector : null, + }; + ranked.Add((scored, record.ExperienceId.ToString("D"))); } // Ties sort by ExperienceId ascending and ordinal, so the order is total and stable rather than @@ -952,7 +961,9 @@ private ExperienceRetrievalResult Rank( string channel, List merged, Dictionary positions, - ref bool truncated) + ref bool truncated, + Dictionary strengths, + Func strength) { if (candidates.Count > _policy.CandidateLimit) { @@ -995,6 +1006,7 @@ private ExperienceRetrievalResult Rank( return new RetrievalFailure($"The {channel} returned the same record more than once.", Exception: null); } + strengths[record.ExperienceId] = Normalize(strength(candidate)); Fold(merged, positions, candidate); } diff --git a/src/AgentExperience.Core/Retrieval/RetrievalResults.cs b/src/AgentExperience.Core/Retrieval/RetrievalResults.cs index b6e0121c..779ba9ad 100644 --- a/src/AgentExperience.Core/Retrieval/RetrievalResults.cs +++ b/src/AgentExperience.Core/Retrieval/RetrievalResults.cs @@ -182,6 +182,22 @@ public sealed record RankedExperience( /// . /// public ExperienceGrantDisclosure? MatchedGrantDisclosure { get; init; } + + /// + /// How much of the request's text the record covers, when the text channel found it: the share of the + /// request's distinct terms it contains (), or the text channel's + /// relevance when its source reports no coverage, normalized to [0, 1]. when only the vector + /// channel found the record. Set by retrieval; it does not enter , whose relevance component is the + /// higher of the two channels' relevances. The injection provider's match floor reads it. + /// + public double? TextCoverage { get; init; } + + /// + /// The vector channel's relevance for the record, (1 + cosine) / 2 normalized to [0, 1], when that channel + /// found it; when only the text channel did, or with no vector channel. Set by retrieval; the + /// injection provider's match floor reads it. + /// + public double? VectorRelevance { get; init; } } /// Why a candidate the search returned was not ranked. diff --git a/src/AgentExperience.MicrosoftAgentFramework/Diagnostics/InjectionDiagnostics.cs b/src/AgentExperience.MicrosoftAgentFramework/Diagnostics/InjectionDiagnostics.cs index 9ef1f7dc..df4af4e1 100644 --- a/src/AgentExperience.MicrosoftAgentFramework/Diagnostics/InjectionDiagnostics.cs +++ b/src/AgentExperience.MicrosoftAgentFramework/Diagnostics/InjectionDiagnostics.cs @@ -89,6 +89,15 @@ internal static class InjectionDiagnostics /// internal const string HistoryReadAttribute = "agentexperience.history_read"; + /// + /// Why an injection that ran every step injected nothing, when the match floor (ExperienceInjectionOptions.MinimumMatch) + /// kept at least one record out of it: . Set only then, on a NothingToInject outcome. + /// + internal const string NothingReasonAttribute = "agentexperience.nothing_reason"; + + /// The value for a block the match floor emptied. + internal const string WeakMatchReason = "weak_match"; + /// /// The operation recall at the moment of failure emits, once per failed tool call it looks at /// (ExperienceRecallOptions): a span like 's, with one of the diff --git a/src/AgentExperience.MicrosoftAgentFramework/Injection/ExperienceContextProvider.Recall.cs b/src/AgentExperience.MicrosoftAgentFramework/Injection/ExperienceContextProvider.Recall.cs index 2196983e..a9a47d2c 100644 --- a/src/AgentExperience.MicrosoftAgentFramework/Injection/ExperienceContextProvider.Recall.cs +++ b/src/AgentExperience.MicrosoftAgentFramework/Injection/ExperienceContextProvider.Recall.cs @@ -235,7 +235,8 @@ ExperienceRecallResult Result(ExperienceRecallOutcome outcome) => var omitted = new List(); var candidates = new List(); var seen = new HashSet(); - foreach (var match in recall.AllowCoarseMatch ? ExactThenCoarse : ExactOnly) + // Under the match floor only an exact signature match is strong enough, so a coarse one is never asked for. + foreach (var match in recall.AllowCoarseMatch && _minimumMatch is null ? ExactThenCoarse : ExactOnly) { var read = await ReadWithinAsync( token => lookup.FindAsync( diff --git a/src/AgentExperience.MicrosoftAgentFramework/Injection/ExperienceContextProvider.cs b/src/AgentExperience.MicrosoftAgentFramework/Injection/ExperienceContextProvider.cs index cbf5ba0f..686e12e4 100644 --- a/src/AgentExperience.MicrosoftAgentFramework/Injection/ExperienceContextProvider.cs +++ b/src/AgentExperience.MicrosoftAgentFramework/Injection/ExperienceContextProvider.cs @@ -170,6 +170,9 @@ public sealed partial class ExperienceContextProvider : AIContextProvider private readonly RecipeRendering _recipe; private readonly CapabilityGate? _capabilityGate; private readonly bool _excludeModelAuthored; + + /// The match-strength floor, or when it is off. + private readonly ExperienceMatchFloor? _minimumMatch; private readonly string _stateKey; private readonly IReadOnlyList _stateKeys; @@ -217,6 +220,7 @@ public ExperienceContextProvider( _recipe = recipeExcerpts.IsEmpty ? RecipeRendering.None : new RecipeRendering(recipeExcerpts, retrieval.AreRecipeExcerptsConfirmed); _capabilityGate = capabilityGate; _excludeModelAuthored = options.ModelAuthoredLessons == ModelAuthoredLessonPolicy.Exclude; + _minimumMatch = options.MinimumMatch; // Snapshotted with the rest: the key this provider reads and writes never changes under it. _stateKey = options.SessionStateKey; @@ -587,7 +591,7 @@ private async ValueTask InjectResolvedAsync( { try { - retrieved = await _retrieval.RetrieveAsync(GatedWindow(request, capabilityGate, _options.Limits.MaxRecords, _retrieval.Policy.CandidateLimit), cancellationToken).ConfigureAwait(false); + retrieved = await _retrieval.RetrieveAsync(GatedWindow(request, capabilityGate, _options.Limits.MaxRecords, _retrieval.Policy.CandidateLimit, _minimumMatch is not null), cancellationToken).ConfigureAwait(false); } catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) { @@ -1316,11 +1320,13 @@ protected override async ValueTask InvokedCoreAsync(InvokedContext context, Canc /// by the retrieval policy's candidate limit, so records the gate keeps out can be backfilled from below. A request /// with no limit already gets the whole candidate window; one whose limit retrieval would refuse is left for /// retrieval to refuse; and a gate that can reject nothing widens nothing. What the host's own limit admits is still - /// all that can be injected (). + /// all that can be injected (). The match floor () widens the window + /// the same way, by , or by the gate's multiplier when that is larger. /// - internal static RetrieveExperienceRequest GatedWindow(RetrieveExperienceRequest request, CapabilityGate? gate, int maxRecords, int candidateLimit) + internal static RetrieveExperienceRequest GatedWindow(RetrieveExperienceRequest request, CapabilityGate? gate, int maxRecords, int candidateLimit, bool floorActive = false) { - if (gate is not { CanReject: true } || request.Limit is not { } limit) + var multiplier = Math.Max(gate is { CanReject: true } ? gate.WindowMultiplier : 0, floorActive ? FloorWindowMultiplier : 0); + if (multiplier == 0 || request.Limit is not { } limit) { return request; } @@ -1332,10 +1338,13 @@ internal static RetrieveExperienceRequest GatedWindow(RetrieveExperienceRequest var window = (int)Math.Min( candidateLimit, - Math.Max(limit, (long)maxRecords * gate.WindowMultiplier)); + Math.Max(limit, (long)maxRecords * multiplier)); return window == limit ? request : request with { Limit = window }; } + /// How much wider than the record limit the match floor asks retrieval to look, when the request names a limit: the capability gate's default. + internal const int FloorWindowMultiplier = 4; + /// The omission detail of a lesson measured as hurting. internal const string MeasuredHarmfulDetail = "The injection holdout measured this lesson as hurting: runs given it verified less often than runs it was withheld from."; @@ -1429,7 +1438,7 @@ private RetrieveExperienceRequest WithModelAuthoredExclusion(RetrieveExperienceR /// and under . For contract 1.13's inject-options fixtures. /// internal RetrieveExperienceRequest RetrievalRequest(RetrieveExperienceRequest request, IEnumerable? runTools) => - GatedWindow(WithModelAuthoredExclusion(request), GateFor(runTools), _options.Limits.MaxRecords, _retrieval.Policy.CandidateLimit); + GatedWindow(WithModelAuthoredExclusion(request), GateFor(runTools), _options.Limits.MaxRecords, _retrieval.Policy.CandidateLimit, _minimumMatch is not null); /// /// Takes the top in rank order and records the @@ -1497,6 +1506,17 @@ private List Select(IReadOnlyList ranked, Li continue; } + // The match floor (story 28.5), before the limit: a weak match takes no slot, so a stronger one ranked below it + // fills it. Never re-checked on the re-read record: how well a record matched the request does not change. + if (_minimumMatch is { } floor && !floor.Passes(candidate, _retrieval.Policy.MinimumVectorRelevance)) + { + omittedHere.Add(new OmittedExperience( + candidate.Record.ExperienceId, + InjectionOmissionReason.WeakMatch, + floor.Detail(candidate, _retrieval.Policy.MinimumVectorRelevance))); + continue; + } + // The capability gate (story 20.7), before the limit: a record whose approach this agent cannot, or must // not, carry out takes no slot. Exactly the check the re-read record gets below, on the ranked one. A borrowed // one is decided by the level its grant was matched under (story 20.8): under a level that shows the approach @@ -2394,6 +2414,12 @@ public ExperienceInjectionSessionUsage Usage() private void Report(InjectionTrace trace, ExperienceInjectionResult result) { InjectionDiagnostics.Tag(trace, InjectionDiagnostics.OmittedCountAttribute, result.Omitted.Count); + if (result.Outcome == InjectionOutcome.NothingToInject + && result.Omitted.Any(omission => omission.Reason == InjectionOmissionReason.WeakMatch)) + { + InjectionDiagnostics.Tag(trace, InjectionDiagnostics.NothingReasonAttribute, InjectionDiagnostics.WeakMatchReason); + } + if (result.RetractedExperienceIds.Count > 0) { InjectionDiagnostics.Tag(trace, InjectionDiagnostics.RetractedCountAttribute, result.RetractedExperienceIds.Count); diff --git a/src/AgentExperience.MicrosoftAgentFramework/Injection/ExperienceInjectionOptions.cs b/src/AgentExperience.MicrosoftAgentFramework/Injection/ExperienceInjectionOptions.cs index 23bf64d7..1512da8d 100644 --- a/src/AgentExperience.MicrosoftAgentFramework/Injection/ExperienceInjectionOptions.cs +++ b/src/AgentExperience.MicrosoftAgentFramework/Injection/ExperienceInjectionOptions.cs @@ -778,6 +778,35 @@ public sealed class ExperienceInjectionOptions /// A bound out of range is refused when the provider is constructed. public ExperienceRecallOptions? Recall { get; set; } + /// + /// The injection-time match-strength floor: with one set, a record is injected only when its match is strong enough + /// (an exact failure-signature match on the current run, enough of the request's terms, or enough semantic + /// similarity; see ), and a block whose every record falls below it injects nothing. + /// (the default) is off: the block, telemetry and storage are exactly what they are without it. + /// + /// + /// + /// It is an injection gate, not a retrieval one: which candidates retrieval returns is unchanged. Each ranked + /// candidate is checked after and before the capability gate + /// and the cut, so a weak match takes no slot and a lower-ranked, + /// stronger one fills it; it is reported as , with a content-free detail + /// naming the strengths and the floor. As with the capability gate, a request that names its own + /// asks retrieval for a wider window to backfill from; a weak match + /// fetched only as backfill, past the host's own limit, is not reported. A block that injects nothing with at least + /// one ends as and its + /// inject span carries agentexperience.nothing_reason = weak_match, even when other reasons kept other + /// records out too: the attribute says the floor contributed, the typed result says what each record was omitted for. + /// + /// + /// The injection holdout splits its arms after the floor, so both arms are judged on the same eligible set. Recall + /// hints are checked too: only an exact signature match passes, so with a floor set recall never asks for a coarse + /// match, whatever says. There is no recommended coverage: + /// no value met the retrieval benchmark's pre-registered precision target. The Python and Go ports do not apply a + /// floor yet. + /// + /// + public ExperienceMatchFloor? MinimumMatch { get; set; } + /// A shallow copy, taken before validation, so a later change to these options changes nothing. internal ExperienceInjectionOptions Snapshot() { diff --git a/src/AgentExperience.MicrosoftAgentFramework/Injection/ExperienceMatchFloor.cs b/src/AgentExperience.MicrosoftAgentFramework/Injection/ExperienceMatchFloor.cs new file mode 100644 index 00000000..983d6e78 --- /dev/null +++ b/src/AgentExperience.MicrosoftAgentFramework/Injection/ExperienceMatchFloor.cs @@ -0,0 +1,103 @@ +using AgentExperience.Core.Retrieval; + +namespace AgentExperience.MicrosoftAgentFramework.Injection; + +/// +/// The injection-time match-strength floor (): a record is injected +/// only when its match is strong enough to be worth the context. Separate from retrieval's candidate floors, which it +/// never changes: retrieval still returns what it returns, and a record below this floor is omitted from the block as +/// . +/// +/// +/// +/// A record passes when at least one of these holds: +/// +/// +/// a failure-signature match, exact, on the current run: a recall hint +/// () found by an exact signature lookup. The before-run block has no +/// signature, so this leg never applies to it; with a floor set, recall never asks for a coarse match; +/// its text coverage (: the share of the request's distinct +/// terms it contains) is at least ; +/// its semantic similarity () is at least +/// , or, when that is , the retrieval policy's +/// . With neither set, this leg is off: a record the vector channel +/// found passes only on its text. A below the policy's floor does nothing, since +/// retrieval never returns a vector candidate below the policy's floor. +/// +/// +/// Coverage is adapter-relative. What a term is depends on the text source: whole, unstemmed words in memory; +/// stemmed lexemes on PostgreSQL, with a hyphenated compound counted once; Porter-stemmed words on SQLite. The same +/// request can therefore have a different term count, and a record a different coverage, on each store. No value is +/// recommended: on the retrieval benchmark no floor met the precision target fixed before measuring (see +/// docs/benchmarks.md), so the floor is off by default and has no default coverage. +/// +/// +/// Values are compared with a tolerance of , so a coverage the database computed in single +/// precision (three of ten terms as 0.3000000119) is not lost to rounding at a floor of 0.3. +/// +/// +public sealed record ExperienceMatchFloor +{ + /// The tolerance every comparison against the floor allows. + public const double Tolerance = 1e-6; + + /// + /// Required: the smallest share of the request's distinct terms a record must contain to pass on its text match, in + /// [0, 1]. 0 lets every record the text channel found pass. Relative to the store's terms (see the remarks). + /// + public required double MinimumTextCoverage + { + get; + init => field = value is >= 0d and <= 1d + ? value + : throw new ArgumentOutOfRangeException(nameof(MinimumTextCoverage), value, "The minimum text coverage must be between 0 and 1 inclusive."); + } + + /// + /// The smallest vector relevance, (1 + cosine) / 2 in [0, 1], a record must have to pass on its semantic match, + /// or (the default) for the retrieval policy's ; + /// with neither set, the semantic leg is off. + /// + public double? MinimumSemanticSimilarity + { + get; + init => field = value is not { } floor || floor is >= 0d and <= 1d + ? value + : throw new ArgumentOutOfRangeException(nameof(MinimumSemanticSimilarity), value, "The minimum semantic similarity must be between 0 and 1 inclusive."); + } + + /// + /// Whether passes the floor, without a signature match, under a retrieval policy whose vector + /// floor is . Fails closed: with no semantic floor anywhere, the semantic leg is off. + /// + internal bool Passes(RankedExperience ranked, double? policyVectorFloor) + { + if (ranked.TextCoverage is { } coverage && AtLeast(coverage, MinimumTextCoverage)) + { + return true; + } + + return ranked.VectorRelevance is { } similarity + && (MinimumSemanticSimilarity ?? policyVectorFloor) is { } floor + && AtLeast(similarity, floor); + } + + /// The content-free omission detail for a record that did not pass. + internal string Detail(RankedExperience ranked, double? policyVectorFloor) + { + var text = ranked.TextCoverage is { } coverage + ? $"text coverage {Number(coverage)} is below {Number(MinimumTextCoverage)}" + : "no text match"; + var semantic = (ranked.VectorRelevance, MinimumSemanticSimilarity ?? policyVectorFloor) switch + { + (null, _) => "no semantic match", + ({ } similarity, { } floor) => $"semantic similarity {Number(similarity)} is below {Number(floor)}", + (_, null) => "no semantic floor is set, so a semantic match does not count", + }; + return $"Below the match floor: {text}, {semantic}, and no exact failure-signature match."; + } + + private static bool AtLeast(double value, double floor) => !double.IsNaN(value) && value >= floor - Tolerance; + + private static string Number(double value) => value.ToString("0.###", System.Globalization.CultureInfo.InvariantCulture); +} diff --git a/src/AgentExperience.MicrosoftAgentFramework/Injection/InjectionResults.cs b/src/AgentExperience.MicrosoftAgentFramework/Injection/InjectionResults.cs index 1850344f..86ee71c8 100644 --- a/src/AgentExperience.MicrosoftAgentFramework/Injection/InjectionResults.cs +++ b/src/AgentExperience.MicrosoftAgentFramework/Injection/InjectionResults.cs @@ -133,6 +133,14 @@ public enum InjectionOmissionReason /// record itself is unchanged; the verdict can reverse as new evidence arrives and old evidence leaves the window. /// MeasuredHarmful, + + /// + /// The match-strength floor (): the record's match was too weak to + /// be worth the context -- no exact failure-signature match on the current run, too small a share of the request's + /// terms, and too little semantic similarity. Decided on the ranked candidate before the capability gate and the record + /// limit, so it takes no slot. The detail names the strengths and the floor, never record content. + /// + WeakMatch, } /// diff --git a/src/AgentExperience.Storage.InMemory/InMemoryExperienceCandidateSource.cs b/src/AgentExperience.Storage.InMemory/InMemoryExperienceCandidateSource.cs index c6fef243..788c4aad 100644 --- a/src/AgentExperience.Storage.InMemory/InMemoryExperienceCandidateSource.cs +++ b/src/AgentExperience.Storage.InMemory/InMemoryExperienceCandidateSource.cs @@ -107,7 +107,11 @@ record => statuses.Contains(record.Status) && record.ReuseConfidence >= query.MinimumConfidence && !(query.ExcludeModelAuthored && IsModelAuthored(record))) .Where(indexed => terms.Count(indexed.Text.Contains) >= required) - .Select(indexed => new ExperienceCandidate(indexed.Record, Relevance(indexed.Text, terms))) + .Select(indexed => new ExperienceCandidate(indexed.Record, Relevance(indexed.Text, terms)) + { + // Story 28.5: the plain share of the query's terms, before the field weighting, for the injection floor. + TermCoverage = (double)terms.Count(indexed.Text.Contains) / terms.Count, + }) .OrderByDescending(candidate => candidate.Relevance) .ThenBy(candidate => candidate.Record.ExperienceId) .Take(query.Limit) diff --git a/src/AgentExperience.Storage.Postgres/PostgresExperienceCandidateSource.cs b/src/AgentExperience.Storage.Postgres/PostgresExperienceCandidateSource.cs index e2461231..cff2c646 100644 --- a/src/AgentExperience.Storage.Postgres/PostgresExperienceCandidateSource.cs +++ b/src/AgentExperience.Storage.Postgres/PostgresExperienceCandidateSource.cs @@ -505,13 +505,16 @@ private async Task RecordGrantAccessAsync( // The level the predicate matched the record under travels with the shared flag, and only with it. A // shared row whose level is missing or unknown has none here, which every consumer reads as LessonOnly. + // The relevance is the coverage (story 20.2), so it is reported as the term coverage too (story 28.5). + var relevance = ReadRelevance(row); candidates.Add(new ExperienceCandidate( record, - ReadRelevance(row), + relevance, shared, ExperienceRecordRows.ReadPermittingGrant(row)) { GrantDisclosure = shared ? disclosure : null, + TermCoverage = relevance, }); disclosures.Add(disclosure); } diff --git a/src/AgentExperience.Storage.Sqlite/SqliteExperienceCandidateSource.cs b/src/AgentExperience.Storage.Sqlite/SqliteExperienceCandidateSource.cs index cacd7d67..f7e578fb 100644 --- a/src/AgentExperience.Storage.Sqlite/SqliteExperienceCandidateSource.cs +++ b/src/AgentExperience.Storage.Sqlite/SqliteExperienceCandidateSource.cs @@ -25,6 +25,10 @@ namespace AgentExperience.Storage.Sqlite; /// first, then by ID, after the filters (exact scope, eligible status, minimum confidence, model authorship), and only /// then cut to the limit. Its values are not comparable with the PostgreSQL source's. /// +/// +/// Term coverage () is the plain share of the query's terms the record +/// holds in any field, as the in-memory source counts it, except that a term matches on its Porter stem. +/// /// There are no grants here: a search returns only the requester's own records. /// public sealed class SqliteExperienceCandidateSource : IExperienceCandidateSource @@ -143,14 +147,14 @@ private static async Task SearchCoreAsync( var ranked = hits .Where(hit => hit.Value.Count(fields => fields != 0) >= required) - .Select(hit => (Id: Guid.Parse(hit.Key), Relevance: Relevance(hit.Value))) + .Select(hit => (Id: Guid.Parse(hit.Key), Relevance: Relevance(hit.Value), Coverage: (double)hit.Value.Count(fields => fields != 0) / hit.Value.Length)) .OrderByDescending(hit => hit.Relevance) .ThenBy(hit => hit.Id) .Take(query.Limit) .ToList(); var candidates = new List(ranked.Count); - foreach (var (id, relevance) in ranked) + foreach (var (id, relevance, coverage) in ranked) { using var command = Command( connection, @@ -160,7 +164,8 @@ private static async Task SearchCoreAsync( using var reader = await command.ExecuteReaderAsync(cancellationToken).ConfigureAwait(false); if (await reader.ReadAsync(cancellationToken).ConfigureAwait(false) && SqliteExperienceRecordStore.ReadRecord(reader) is { } record) { - candidates.Add(new ExperienceCandidate(record, relevance)); + // The plain share of the query's terms, before the field weighting, as the in-memory source reports it. + candidates.Add(new ExperienceCandidate(record, relevance) { TermCoverage = coverage }); } } diff --git a/tests/AgentExperience.Core.Tests/HybridRetrievalTests.cs b/tests/AgentExperience.Core.Tests/HybridRetrievalTests.cs index c4171ca5..6573d991 100644 --- a/tests/AgentExperience.Core.Tests/HybridRetrievalTests.cs +++ b/tests/AgentExperience.Core.Tests/HybridRetrievalTests.cs @@ -81,6 +81,30 @@ public async Task The_higher_normalized_relevance_wins_whichever_channel_it_came Assert.Equal(0.95, Relevance(vectorWins)); } + // ---------------------------------------------------------------- story 28.5: each channel's own strength + + [Fact] + public async Task Each_ranked_record_carries_its_text_coverage_and_vector_relevance_apart_from_the_merged_relevance() + { + var both = Record(Id(1)); + var textOnly = Record(Id(2)); + var vectorOnly = Record(Id(3)); + var noCoverage = Record(Id(4)); + + var result = await Service( + text: [Candidate(both, 0.6) with { TermCoverage = 0.25 }, Candidate(textOnly, 0.9) with { TermCoverage = 0.5 }, Candidate(noCoverage, 0.4)], + vector: [Candidate(both, 0.9), Candidate(vectorOnly, 0.8)]).RetrieveAsync(Request()); + + RankedExperience Of(Guid id) => result.Records.Single(r => r.Record.ExperienceId == id); + Assert.Equal((0.25, 0.9), (Of(Id(1)).TextCoverage, Of(Id(1)).VectorRelevance)); + Assert.Equal(0.9, Of(Id(1)).Components.Single(c => c.Kind == RankingComponentKind.Relevance).Value); + Assert.Equal((0.5, (double?)null), (Of(Id(2)).TextCoverage, Of(Id(2)).VectorRelevance)); + Assert.Equal(((double?)null, 0.8), (Of(Id(3)).TextCoverage, Of(Id(3)).VectorRelevance)); + + // A text source that reports no coverage: its relevance stands in. + Assert.Equal(0.4, Of(Id(4)).TextCoverage); + } + // ---------------------------------------------------------------- the vector relevance floor [Fact] diff --git a/tests/AgentExperience.MicrosoftAgentFramework.Tests/MatchFloorTests.cs b/tests/AgentExperience.MicrosoftAgentFramework.Tests/MatchFloorTests.cs new file mode 100644 index 00000000..c5f4f727 --- /dev/null +++ b/tests/AgentExperience.MicrosoftAgentFramework.Tests/MatchFloorTests.cs @@ -0,0 +1,382 @@ +using AgentExperience.Core.Retrieval; +using AgentExperience.MicrosoftAgentFramework.Diagnostics; +using AgentExperience.MicrosoftAgentFramework.Injection; +using AgentExperience.MicrosoftAgentFramework.Tests.Diagnostics; +using Microsoft.Agents.AI; + +namespace AgentExperience.MicrosoftAgentFramework.Tests; + +/// +/// Story 28.5: the injection-time match-strength floor (). The fake +/// world's candidates report no term coverage, so retrieval's is their +/// relevance, which each test sets. +/// +public class MatchFloorTests +{ + private static readonly Scope TestScope = new("tenant-1", "app-1", "project-1"); + private static readonly AuthorizationContext Authorization = new("tenant-1", "host", ["experience:read"], DateTimeOffset.UnixEpoch); + + [Fact] + public async Task Off_by_default_a_weak_match_is_injected_as_before() + { + var harness = new Harness(); + harness.World.Publish(InjectionRecords.Record(InjectionRecords.Id(1), TestScope), relevance: 0.1d); + + await harness.Agent().RunAsync("refund ticket stuck on a lock"); + + Assert.Null(new ExperienceInjectionOptions().MinimumMatch); + Assert.Equal(InjectionOutcome.Injected, harness.Last.Outcome); + Assert.Equal([InjectionRecords.Id(1)], harness.Last.InjectedExperienceIds); + Assert.Empty(harness.Last.Omitted); + } + + [Fact] + public async Task A_weak_match_is_omitted_as_weak_match_with_a_content_free_detail_and_a_strong_one_injected() + { + var harness = new Harness { MinimumMatch = new ExperienceMatchFloor { MinimumTextCoverage = 0.5 } }; + harness.World.Publish(InjectionRecords.Record(InjectionRecords.Id(1), TestScope), relevance: 0.9d); + harness.World.Publish(InjectionRecords.Record(InjectionRecords.Id(2), TestScope), relevance: 0.25d); + + await harness.Agent().RunAsync("refund ticket stuck on a lock"); + + Assert.Equal([InjectionRecords.Id(1)], harness.Last.InjectedExperienceIds); + var omission = Assert.Single(harness.Last.Omitted); + Assert.Equal(InjectionRecords.Id(2), omission.ExperienceId); + Assert.Equal(InjectionOmissionReason.WeakMatch, omission.Reason); + Assert.Equal( + "Below the match floor: text coverage 0.25 is below 0.5, no semantic match, and no exact failure-signature match.", + omission.Detail); + Assert.DoesNotContain(InjectionRecords.Id(2).ToString("D"), harness.InjectedText(), StringComparison.Ordinal); + Assert.DoesNotContain(InjectionRecords.Id(2), harness.World.Reads); + } + + [Fact] + public async Task A_weak_match_takes_no_slot_so_a_stronger_one_ranked_below_it_fills_the_limit() + { + // The weak record outranks the strong one on confidence and status, so with one slot and no floor it would take it. + var harness = new Harness + { + MinimumMatch = new ExperienceMatchFloor { MinimumTextCoverage = 0.3 }, + Limits = ExperienceInjectionLimits.Default with { MaxRecords = 1 }, + }; + harness.World.Publish(InjectionRecords.Record(InjectionRecords.Id(1), TestScope, status: ExperienceStatus.Reinforced, confidence: 1d), relevance: 0.29d); + harness.World.Publish(InjectionRecords.Record(InjectionRecords.Id(2), TestScope, confidence: 0.5d), relevance: 0.31d); + + await harness.Agent().RunAsync("refund ticket stuck on a lock"); + + Assert.Equal([InjectionRecords.Id(2)], harness.Last.InjectedExperienceIds); + Assert.Equal(new[] { InjectionOmissionReason.WeakMatch }, harness.Last.Omitted.Select(o => o.Reason)); + + var unfloored = new Harness { Limits = ExperienceInjectionLimits.Default with { MaxRecords = 1 } }; + unfloored.World.Publish(InjectionRecords.Record(InjectionRecords.Id(1), TestScope, status: ExperienceStatus.Reinforced, confidence: 1d), relevance: 0.29d); + unfloored.World.Publish(InjectionRecords.Record(InjectionRecords.Id(2), TestScope, confidence: 0.5d), relevance: 0.31d); + await unfloored.Agent().RunAsync("refund ticket stuck on a lock"); + Assert.Equal([InjectionRecords.Id(1)], unfloored.Last.InjectedExperienceIds); + } + + [Fact] + public async Task A_block_whose_every_record_is_weak_injects_nothing() + { + var harness = new Harness { MinimumMatch = new ExperienceMatchFloor { MinimumTextCoverage = 0.5 } }; + harness.World.Publish(InjectionRecords.Record(InjectionRecords.Id(1), TestScope), relevance: 0.2d); + + await harness.Agent().RunAsync("refund ticket stuck on a lock"); + + Assert.Equal(InjectionOutcome.NothingToInject, harness.Last.Outcome); + Assert.Empty(harness.Last.InjectedExperienceIds); + Assert.Equal(InjectionOmissionReason.WeakMatch, Assert.Single(harness.Last.Omitted).Reason); + Assert.Null(harness.InjectedText()); + } + + [Fact] + public async Task A_coverage_a_hair_below_the_floor_from_single_precision_still_passes() + { + var harness = new Harness { MinimumMatch = new ExperienceMatchFloor { MinimumTextCoverage = 0.3 } }; + harness.World.Publish(InjectionRecords.Record(InjectionRecords.Id(1), TestScope), relevance: 0.3d - 5e-7); + + await harness.Agent().RunAsync("refund ticket stuck on a lock"); + + Assert.Equal([InjectionRecords.Id(1)], harness.Last.InjectedExperienceIds); + } + + [Fact] + public async Task The_floor_is_snapshotted_when_the_provider_is_built() + { + var options = new ExperienceInjectionOptions + { + ResolveRequest = _ => new RetrieveExperienceRequest(Authorization, TestScope, "refund ticket stuck on a lock"), + TimeProvider = new FrozenTimeProvider(InjectionRecords.Now), + }; + var results = new List(); + options.OnContextInjected = results.Add; + var world = new FakeExperienceWorld(); + world.Publish(InjectionRecords.Record(InjectionRecords.Id(1), TestScope), relevance: 0.1d); + var provider = new ExperienceContextProvider( + new ExperienceRetrievalService(world, RetrievalPolicy.Default, RankingWeights.Default, new FrozenTimeProvider(InjectionRecords.Now)), + world, + options); + + options.MinimumMatch = new ExperienceMatchFloor { MinimumTextCoverage = 1d }; + await new ChatClientAgent(new RecordingChatClient(), new ChatClientAgentOptions { AIContextProviders = [provider] }).RunAsync("x"); + + Assert.Equal([InjectionRecords.Id(1)], Assert.Single(results).InjectedExperienceIds); + } + + [Fact] + public void A_host_limited_request_is_widened_for_backfill_only_while_the_floor_is_on() + { + var request = new RetrieveExperienceRequest(Authorization, TestScope, "refund", Limit: 8); + + Assert.Same(request, ExperienceContextProvider.GatedWindow(request, gate: null, maxRecords: 8, candidateLimit: 50)); + Assert.Equal(32, ExperienceContextProvider.GatedWindow(request, gate: null, maxRecords: 8, candidateLimit: 50, floorActive: true).Limit); + Assert.Equal(20, ExperienceContextProvider.GatedWindow(request, gate: null, maxRecords: 8, candidateLimit: 20, floorActive: true).Limit); + + var unlimited = request with { Limit = null }; + Assert.Same(unlimited, ExperienceContextProvider.GatedWindow(unlimited, gate: null, maxRecords: 8, candidateLimit: 50, floorActive: true)); + } + + // ---- The floor itself ---------------------------------------------------------------------------- + + [Fact] + public async Task A_weak_match_fetched_only_as_backfill_past_the_host_s_limit_is_not_reported() + { + // The host asks for one record; the floor widens retrieval to 32. The weak record ranked first is reported, the + // strong one backfills, and the weak one ranked third was never in the host's window, so it is not reported. + var harness = new Harness { MinimumMatch = new ExperienceMatchFloor { MinimumTextCoverage = 0.5 }, RequestLimit = 1 }; + harness.World.Publish(InjectionRecords.Record(InjectionRecords.Id(1), TestScope, status: ExperienceStatus.Reinforced, confidence: 1d), relevance: 0.45d); + harness.World.Publish(InjectionRecords.Record(InjectionRecords.Id(2), TestScope, confidence: 0.6d), relevance: 0.5d); + harness.World.Publish(InjectionRecords.Record(InjectionRecords.Id(3), TestScope, confidence: 0.5d), relevance: 0.1d); + + await harness.Agent().RunAsync("refund ticket stuck on a lock"); + + Assert.Equal([InjectionRecords.Id(2)], harness.Last.InjectedExperienceIds); + var omission = Assert.Single(harness.Last.Omitted); + Assert.Equal((InjectionRecords.Id(1), InjectionOmissionReason.WeakMatch), (omission.ExperienceId, omission.Reason)); + } + + // ---- The semantic leg, through a hybrid retrieval --------------------------------------------------- + + [Fact] + public async Task A_vector_only_match_passes_on_a_semantic_floor_and_fails_closed_without_one() + { + var record = InjectionRecords.Record(InjectionRecords.Id(1), TestScope); + var withPolicyFloor = new Harness + { + MinimumMatch = new ExperienceMatchFloor { MinimumTextCoverage = 0.5 }, + Policy = RetrievalPolicy.Default with { MinimumVectorRelevance = 0.75 }, + Vectors = new ScriptedVectorIndex(new ExperienceCandidate(record, 0.8)), + }; + var withOwnFloor = new Harness + { + MinimumMatch = new ExperienceMatchFloor { MinimumTextCoverage = 0.5, MinimumSemanticSimilarity = 0.85 }, + Policy = RetrievalPolicy.Default with { MinimumVectorRelevance = 0.75 }, + Vectors = new ScriptedVectorIndex(new ExperienceCandidate(record, 0.8)), + }; + var withNone = new Harness + { + MinimumMatch = new ExperienceMatchFloor { MinimumTextCoverage = 0.5 }, + Vectors = new ScriptedVectorIndex(new ExperienceCandidate(record, 0.99)), + }; + foreach (var harness in new[] { withPolicyFloor, withOwnFloor, withNone }) + { + harness.World.Store(record); + await harness.Agent().RunAsync("refund ticket stuck on a lock"); + } + + Assert.Equal([InjectionRecords.Id(1)], withPolicyFloor.Last.InjectedExperienceIds); + Assert.Null(withPolicyFloor.Last.VectorFallback); + + Assert.Equal(InjectionOutcome.NothingToInject, withOwnFloor.Last.Outcome); + Assert.Equal( + "Below the match floor: no text match, semantic similarity 0.8 is below 0.85, and no exact failure-signature match.", + Assert.Single(withOwnFloor.Last.Omitted).Detail); + + // No semantic floor anywhere: the semantic leg is off, so a vector-only record fails however close it is. + Assert.Equal(InjectionOutcome.NothingToInject, withNone.Last.Outcome); + var omission = Assert.Single(withNone.Last.Omitted); + Assert.Equal(InjectionOmissionReason.WeakMatch, omission.Reason); + Assert.Equal( + "Below the match floor: no text match, no semantic floor is set, so a semantic match does not count, and no exact failure-signature match.", + omission.Detail); + } + + // ---- The floor itself ---------------------------------------------------------------------------- + + [Theory] + [InlineData(-0.01)] + [InlineData(1.01)] + [InlineData(double.NaN)] + public void A_floor_outside_zero_to_one_is_refused(double value) + { + Assert.Throws(() => new ExperienceMatchFloor { MinimumTextCoverage = value }); + Assert.Throws(() => new ExperienceMatchFloor { MinimumTextCoverage = 0.5, MinimumSemanticSimilarity = value }); + } + + [Fact] + public void The_coverage_has_no_default_and_the_semantic_floor_defaults_to_the_policy_s() + { + Assert.True(typeof(ExperienceMatchFloor).GetProperty(nameof(ExperienceMatchFloor.MinimumTextCoverage))! + .IsDefined(typeof(System.Runtime.CompilerServices.RequiredMemberAttribute), inherit: false)); + Assert.Null(new ExperienceMatchFloor { MinimumTextCoverage = 0.5 }.MinimumSemanticSimilarity); + } + + [Fact] + public void Any_one_leg_passes_a_record_and_a_missing_leg_never_does() + { + var floor = new ExperienceMatchFloor { MinimumTextCoverage = 0.5, MinimumSemanticSimilarity = 0.8 }; + + Assert.True(floor.Passes(Ranked(text: 0.5, vector: null), policyVectorFloor: null)); + Assert.True(floor.Passes(Ranked(text: 0.1, vector: 0.8), policyVectorFloor: null)); + Assert.True(floor.Passes(Ranked(text: null, vector: 0.9), policyVectorFloor: null)); + Assert.False(floor.Passes(Ranked(text: 0.1, vector: 0.7), policyVectorFloor: null)); + Assert.False(floor.Passes(Ranked(text: null, vector: null), policyVectorFloor: null)); + Assert.False(floor.Passes(Ranked(text: double.NaN, vector: null), policyVectorFloor: null)); + } + + [Fact] + public void The_semantic_leg_defaults_to_the_retrieval_policy_s_vector_floor_and_is_off_without_one() + { + var floor = new ExperienceMatchFloor { MinimumTextCoverage = 1d }; + var hit = Ranked(text: null, vector: 0.7); + + Assert.False(floor.Passes(hit, policyVectorFloor: 0.75)); + Assert.True(floor.Passes(hit, policyVectorFloor: 0.7)); + Assert.False(floor.Passes(Ranked(text: null, vector: 1d), policyVectorFloor: null)); + Assert.False((floor with { MinimumSemanticSimilarity = 0.71 }).Passes(hit, policyVectorFloor: 0.5)); + Assert.True((floor with { MinimumSemanticSimilarity = 0.7 }).Passes(hit, policyVectorFloor: null)); + Assert.Equal( + "Below the match floor: no text match, semantic similarity 0.7 is below 0.75, and no exact failure-signature match.", + floor.Detail(hit, policyVectorFloor: 0.75)); + } + + private static RankedExperience Ranked(double? text, double? vector) => + new(InjectionRecords.Record(InjectionRecords.Id(1), TestScope), 0.5, []) + { + TextCoverage = text, + VectorRelevance = vector, + }; + + internal sealed class Harness + { + private readonly List _results = []; + + public FakeExperienceWorld World { get; } = new(); + + public RecordingChatClient Client { get; } = new(); + + public ExperienceMatchFloor? MinimumMatch { get; init; } + + public ExperienceInjectionLimits Limits { get; init; } = ExperienceInjectionLimits.Default; + + public string CorrelationId { get; init; } = "corr-floor"; + + public int? RequestLimit { get; init; } + + public RetrievalPolicy Policy { get; init; } = RetrievalPolicy.Default; + + public ScriptedVectorIndex? Vectors { get; init; } + + public ExperienceInjectionResult Last + { + get + { + lock (_results) + { + return _results[^1]; + } + } + } + + public ChatClientAgent Agent() => new(Client, new ChatClientAgentOptions { AIContextProviders = [Provider()] }); + + public string? InjectedText() => Client.LastMessages + ?.FirstOrDefault(m => m.AdditionalProperties?.ContainsKey(ExperienceContextProvider.HistoricalReferenceKey) == true) + ?.Text; + + public ExperienceContextProvider Provider() + { + var clock = new FrozenTimeProvider(InjectionRecords.Now); + return new( + new ExperienceRetrievalService(World, Policy, RankingWeights.Default, clock, Vectors, Vectors is null ? null : new ConstantGenerator()), + World, + new ExperienceInjectionOptions + { + ResolveRequest = _ => new RetrieveExperienceRequest(Authorization, TestScope, "refund ticket stuck on a lock", CorrelationId: CorrelationId, Limit: RequestLimit), + Limits = Limits, + TimeProvider = clock, + MinimumMatch = MinimumMatch, + OnContextInjected = result => + { + lock (_results) + { + _results.Add(result); + } + }, + }); + } + } + /// An embedding index that answers every search with the same candidates and nothing else. + internal sealed class ScriptedVectorIndex(params ExperienceCandidate[] candidates) : IExperienceEmbeddingIndex + { + public Task WriteAsync(AuthorizationContext authorization, ExperienceIndexWrite write, CancellationToken cancellationToken) => + throw new NotSupportedException(); + + public Task ScanAsync(AuthorizationContext authorization, ExperienceIndexScan scan, CancellationToken cancellationToken) => + throw new NotSupportedException(); + + public Task RemoveAsync(AuthorizationContext authorization, Scope scope, Guid experienceId, CancellationToken cancellationToken) => + throw new NotSupportedException(); + + public Task SearchAsync(AuthorizationContext authorization, ExperienceVectorQuery query, CancellationToken cancellationToken) => + Task.FromResult(new ExperienceVectorSearchResult(ExperienceVectorSearchOutcome.Found, candidates, [])); + } + + /// A two-dimensional generator whose vector never matters: the scripted index ignores it. + private sealed class ConstantGenerator : IExperienceEmbeddingGenerator + { + public string ModelId => "test-model"; + + public int Dimension => 2; + + public Task> GenerateAsync(string text, CancellationToken cancellationToken) => + Task.FromResult>(new float[] { 1f, 0f }); + } +} + +/// Story 28.5: the telemetry of a block the match floor emptied. Process-wide listeners, so its own collection. +[Collection(TelemetryCollection.Name)] +public class MatchFloorTelemetryTests +{ + private static readonly Scope TestScope = new("tenant-1", "app-1", "project-1"); + + [Fact] + public async Task A_block_the_floor_emptied_reports_nothing_to_inject_with_a_weak_match_reason_and_others_carry_none() + { + using var probe = TelemetryProbe.Start(); + + var emptied = new MatchFloorTests.Harness { MinimumMatch = new ExperienceMatchFloor { MinimumTextCoverage = 0.5 }, CorrelationId = "corr-floor-emptied" }; + emptied.World.Publish(InjectionRecords.Record(InjectionRecords.Id(1), TestScope), relevance: 0.1d); + await emptied.Agent().RunAsync("refund ticket stuck on a lock"); + + var injected = new MatchFloorTests.Harness { MinimumMatch = new ExperienceMatchFloor { MinimumTextCoverage = 0.5 }, CorrelationId = "corr-floor-injected" }; + injected.World.Publish(InjectionRecords.Record(InjectionRecords.Id(1), TestScope), relevance: 1d); + injected.World.Publish(InjectionRecords.Record(InjectionRecords.Id(2), TestScope), relevance: 0.1d); + await injected.Agent().RunAsync("refund ticket stuck on a lock"); + + var empty = new MatchFloorTests.Harness { MinimumMatch = new ExperienceMatchFloor { MinimumTextCoverage = 0.5 }, CorrelationId = "corr-floor-nothing" }; + await empty.Agent().RunAsync("refund ticket stuck on a lock"); + + var span = Assert.Single(InjectSpans(probe), activity => Equals(activity.GetTagItem(InjectionDiagnostics.CorrelationIdAttribute), "corr-floor-emptied")); + Assert.Equal("NothingToInject", span.GetTagItem(InjectionDiagnostics.OutcomeAttribute)); + Assert.Equal(InjectionDiagnostics.WeakMatchReason, span.GetTagItem(InjectionDiagnostics.NothingReasonAttribute)); + + foreach (var correlation in new[] { "corr-floor-injected", "corr-floor-nothing" }) + { + var other = Assert.Single(InjectSpans(probe), activity => Equals(activity.GetTagItem(InjectionDiagnostics.CorrelationIdAttribute), correlation)); + Assert.Null(other.GetTagItem(InjectionDiagnostics.NothingReasonAttribute)); + } + } + + private static IEnumerable InjectSpans(TelemetryProbe probe) => + probe.LibraryActivities.Where(activity => Equals(activity.GetTagItem(InjectionDiagnostics.OperationAttribute), InjectionDiagnostics.Inject)); +} diff --git a/tests/AgentExperience.MicrosoftAgentFramework.Tests/RecallProviderTests.cs b/tests/AgentExperience.MicrosoftAgentFramework.Tests/RecallProviderTests.cs index 68eda7df..5d573e0d 100644 --- a/tests/AgentExperience.MicrosoftAgentFramework.Tests/RecallProviderTests.cs +++ b/tests/AgentExperience.MicrosoftAgentFramework.Tests/RecallProviderTests.cs @@ -335,6 +335,70 @@ public async Task A_run_whose_block_was_held_out_is_held_out_of_the_hint_too_and Assert.Single(assignments.Select(a => a.RunId).Distinct()); } + [Fact] + public async Task Under_the_match_floor_an_exact_signature_match_still_gives_a_hint() + { + var h = new Harness { MinimumMatch = new ExperienceMatchFloor { MinimumTextCoverage = 1d } }; + h.Publish(Lesson(1)); + + await h.RunAsync("run_tests"); + + var recall = Assert.Single(h.Recalls); + Assert.Equal(ExperienceRecallOutcome.Hinted, recall.Outcome); + Assert.Equal([InjectionRecords.Id(1)], recall.ExperienceIds); + Assert.NotNull(h.Hint); + } + + [Fact] + public async Task Under_the_match_floor_recall_never_asks_for_a_coarse_match() + { + var h = new Harness + { + Recall = new ExperienceRecallOptions { AllowCoarseMatch = true }, + MinimumMatch = new ExperienceMatchFloor { MinimumTextCoverage = 0.5 }, + }; + var lesson = Lesson(1); + h.World.Store(lesson); + h.Source.Add(lesson, match: ExperienceSignatureMatch.Coarse); + + await h.RunAsync("run_tests"); + + var recall = Assert.Single(h.Recalls); + Assert.Equal(ExperienceRecallOutcome.NoMatch, recall.Outcome); + Assert.Equal([ExperienceSignatureMatch.Exact], h.Source.Queries.Select(q => q.Match)); + Assert.Null(h.Hint); + } + + [Fact] + public async Task The_match_floor_runs_before_the_holdout_split_so_a_held_out_block_withholds_only_what_passed() + { + var assignments = new List(); + var h = new Harness { Holdout = new ExperienceHoldoutOptions { Rate = 1 }, MinimumMatch = new ExperienceMatchFloor { MinimumTextCoverage = 0.5 } }; + h.World.Publish(Lesson(1, "deploy"), relevance: 1d); + h.World.Publish(Lesson(2, "deploy"), relevance: 0.1d); + h.World.TracksHoldout = true; + h.World.OnRecordHoldout = (_, _, assignment, _) => + { + lock (assignments) + { + assignments.Add(assignment); + } + + return Task.FromResult(new ExperienceHoldoutRecordResult(ExperienceStoreOutcome.Committed, [])); + }; + + await h.RunAsync(); + await h.Provider.InjectionRecordsSettledAsync(); + + var injection = Assert.Single(h.Injections); + Assert.Equal(InjectionOutcome.Withheld, injection.Outcome); + Assert.Equal([InjectionRecords.Id(1)], injection.WithheldExperienceIds); + Assert.Equal(new OmittedExperience(InjectionRecords.Id(2), InjectionOmissionReason.WeakMatch, injection.Omitted[0].Detail), Assert.Single(injection.Omitted)); + var assignment = Assert.Single(assignments); + Assert.Equal(HoldoutArm.Holdout, assignment.Arm); + Assert.Equal([InjectionRecords.Id(1)], assignment.Records.Select(r => r.ExperienceId)); + } + /// A scripted signature source: candidates by tool and match kind; it records every query. private sealed class ScriptedSignatureSource : IExperienceSignatureSource { @@ -453,6 +517,8 @@ private sealed class Harness public ExperienceHoldoutOptions Holdout { get; init; } = ExperienceHoldoutOptions.Off; + public ExperienceMatchFloor? MinimumMatch { get; init; } + public string? Instructions { get; init; } = "Base instructions."; public bool Concurrent { get; init; } @@ -474,6 +540,7 @@ private sealed class Harness Benefit = Benefit, SessionLimits = SessionLimits, Holdout = Holdout, + MinimumMatch = MinimumMatch, OnContextInjected = Injections.Add, Recall = Recall with { diff --git a/tests/AgentExperience.Release.Tests/PublicApi/AgentExperience.Abstractions.verified.txt b/tests/AgentExperience.Release.Tests/PublicApi/AgentExperience.Abstractions.verified.txt index b144f761..1209dd2e 100644 --- a/tests/AgentExperience.Release.Tests/PublicApi/AgentExperience.Abstractions.verified.txt +++ b/tests/AgentExperience.Release.Tests/PublicApi/AgentExperience.Abstractions.verified.txt @@ -111,6 +111,7 @@ public AgentExperience.Abstractions.ExperienceRecord Record { get; init; } public double Relevance { get; init; } public bool SharedByGrant { get; init; } + public double? TermCoverage { get; init; } } public sealed class ExperienceCandidateQuery : System.IEquatable { diff --git a/tests/AgentExperience.Release.Tests/PublicApi/AgentExperience.Core.verified.txt b/tests/AgentExperience.Release.Tests/PublicApi/AgentExperience.Core.verified.txt index 9e72a67d..115ea1f7 100644 --- a/tests/AgentExperience.Release.Tests/PublicApi/AgentExperience.Core.verified.txt +++ b/tests/AgentExperience.Release.Tests/PublicApi/AgentExperience.Core.verified.txt @@ -918,6 +918,8 @@ namespace AgentExperience.Core.Retrieval public AgentExperience.Abstractions.ExperienceRecord Record { get; init; } public double Score { get; init; } public bool SharedByGrant { get; init; } + public double? TextCoverage { get; init; } + public double? VectorRelevance { get; init; } } public sealed class RankingComponent : System.IEquatable { diff --git a/tests/AgentExperience.Release.Tests/PublicApi/AgentExperience.MicrosoftAgentFramework.verified.txt b/tests/AgentExperience.Release.Tests/PublicApi/AgentExperience.MicrosoftAgentFramework.verified.txt index 9c8bd9e6..e6f071e9 100644 --- a/tests/AgentExperience.Release.Tests/PublicApi/AgentExperience.MicrosoftAgentFramework.verified.txt +++ b/tests/AgentExperience.Release.Tests/PublicApi/AgentExperience.MicrosoftAgentFramework.verified.txt @@ -400,6 +400,7 @@ namespace AgentExperience.MicrosoftAgentFramework.Injection public AgentExperience.MicrosoftAgentFramework.Injection.ExperienceHoldoutOptions Holdout { get; set; } public AgentExperience.MicrosoftAgentFramework.Injection.ExperienceInjectionLimits Limits { get; set; } public AgentExperience.MicrosoftAgentFramework.Injection.HistoricalReferenceMessageRole MessageRole { get; set; } + public AgentExperience.MicrosoftAgentFramework.Injection.ExperienceMatchFloor? MinimumMatch { get; set; } public AgentExperience.MicrosoftAgentFramework.Injection.ModelAuthoredLessonPolicy ModelAuthoredLessons { get; set; } public System.Action? OnContextInjected { get; set; } public AgentExperience.MicrosoftAgentFramework.Injection.ExperienceRecallOptions? Recall { get; set; } @@ -454,6 +455,13 @@ namespace AgentExperience.MicrosoftAgentFramework.Injection public int RecordsUsed { get; init; } public int TrackedRecords { get; init; } } + public sealed class ExperienceMatchFloor : System.IEquatable + { + public const double Tolerance = 1E-06D; + public ExperienceMatchFloor() { } + public double? MinimumSemanticSimilarity { get; init; } + public required double MinimumTextCoverage { get; init; } + } public sealed class ExperienceRecallOptions : System.IEquatable { public const int DefaultMaxHintBytes = 600; @@ -602,6 +610,7 @@ namespace AgentExperience.MicrosoftAgentFramework.Injection ModelAuthored = 9, UnconfirmedContent = 10, MeasuredHarmful = 11, + WeakMatch = 12, } public enum InjectionOutcome { diff --git a/tests/AgentExperience.RetrievalQuality/GoldenInMemoryReport.txt b/tests/AgentExperience.RetrievalQuality/GoldenInMemoryReport.txt index a24903e2..4746e82e 100644 --- a/tests/AgentExperience.RetrievalQuality/GoldenInMemoryReport.txt +++ b/tests/AgentExperience.RetrievalQuality/GoldenInMemoryReport.txt @@ -419,3 +419,101 @@ sig-18 [call_api, HTTP 403] expected: none exact: none coarse: none + +INJECTION FLOOR + +What it measures: what would reach the injected block at MaxRecords = 8 under ExperienceInjectionOptions.MinimumMatch +with MinimumTextCoverage f and no vector channel: the text candidates above, in order, whose share of the query's +distinct terms is at least f, cut to 8. f = 0 is the block with no floor, so its row repeats the overall metrics. +dFP@8: FP@8 over the distractor-overlap queries only. R@8, FP@8 and zero as above. The order is the source's +relevance, not Core's composite score, which also weighs confidence, recency, status and environment. + +floor n lab R@8 FP@8 dFP@8 zero +0.0 48 45 0.700 0.417 0.667 0.292 +0.3 48 45 0.611 0.250 0.667 0.396 +0.4 48 45 0.533 0.250 0.667 0.479 +0.5 48 45 0.533 0.250 0.667 0.479 +0.6 48 45 0.467 0.250 0.667 0.542 + +at 0.3: + expected records it keeps out: + realistic-01 (mig-01) + realistic-03 (dock-01) + realistic-04 (auth-02) + realistic-06 (auth-03) + unexpected records still injected: + exact-01 (mig-02, mig-06, mig-03) + exact-03 (dis-07) + distractor-01 (dis-02) + distractor-02 (dis-03) + distractor-03 (dis-07) + distractor-06 (dis-12, dis-11) + distractor-08 (dis-05) + stopword-02 (mig-04) + stopword-05 (dis-10) + +at 0.4: + expected records it keeps out: + paraphrase-08 (pkg-06) + realistic-01 (mig-01) + realistic-03 (dock-01) + realistic-04 (auth-02) + realistic-05 (pkg-01) + realistic-06 (auth-03) + realistic-07 (mig-05) + realistic-08 (mig-03) + unexpected records still injected: + exact-01 (mig-02, mig-06, mig-03) + exact-03 (dis-07) + distractor-01 (dis-02) + distractor-02 (dis-03) + distractor-03 (dis-07) + distractor-06 (dis-12, dis-11) + distractor-08 (dis-05) + stopword-02 (mig-04) + stopword-05 (dis-10) + +at 0.5: + expected records it keeps out: + paraphrase-08 (pkg-06) + realistic-01 (mig-01) + realistic-03 (dock-01) + realistic-04 (auth-02) + realistic-05 (pkg-01) + realistic-06 (auth-03) + realistic-07 (mig-05) + realistic-08 (mig-03) + unexpected records still injected: + exact-01 (mig-02, mig-06, mig-03) + exact-03 (dis-07) + distractor-01 (dis-02) + distractor-02 (dis-03) + distractor-03 (dis-07) + distractor-06 (dis-12, dis-11) + distractor-08 (dis-05) + stopword-02 (mig-04) + stopword-05 (dis-10) + +at 0.6: + expected records it keeps out: + paraphrase-01 (mig-04) + paraphrase-05 (dock-06) + paraphrase-06 (ci-03) + paraphrase-08 (pkg-06) + realistic-01 (mig-01) + realistic-03 (dock-01) + realistic-04 (auth-02) + realistic-05 (pkg-01) + realistic-06 (auth-03) + realistic-07 (mig-05) + realistic-08 (mig-03) + unexpected records still injected: + exact-01 (mig-02, mig-06, mig-03) + exact-03 (dis-07) + distractor-01 (dis-02) + distractor-02 (dis-03) + distractor-03 (dis-07) + distractor-06 (dis-12, dis-11) + distractor-08 (dis-05) + stopword-02 (mig-04) + stopword-05 (dis-10) diff --git a/tests/AgentExperience.RetrievalQuality/GoldenPostgresReport.txt b/tests/AgentExperience.RetrievalQuality/GoldenPostgresReport.txt index de0e6a5d..5d2279da 100644 --- a/tests/AgentExperience.RetrievalQuality/GoldenPostgresReport.txt +++ b/tests/AgentExperience.RetrievalQuality/GoldenPostgresReport.txt @@ -414,3 +414,110 @@ sig-18 [call_api, HTTP 403] expected: none exact: none coarse: none + +INJECTION FLOOR + +What it measures: what would reach the injected block at MaxRecords = 8 under ExperienceInjectionOptions.MinimumMatch +with MinimumTextCoverage f and no vector channel: the text candidates above, in order, whose share of the query's +distinct terms is at least f, cut to 8. f = 0 is the block with no floor, so its row repeats the overall metrics. +dFP@8: FP@8 over the distractor-overlap queries only. R@8, FP@8 and zero as above. The order is the source's +relevance, not Core's composite score, which also weighs confidence, recency, status and environment. + +floor n lab R@8 FP@8 dFP@8 zero +0.0 48 45 0.844 0.521 0.667 0.188 +0.3 48 45 0.733 0.313 0.667 0.292 +0.4 48 45 0.656 0.313 0.667 0.354 +0.5 48 45 0.622 0.313 0.667 0.396 +0.6 48 45 0.578 0.313 0.667 0.438 + +at 0.3: + expected records it keeps out: + realistic-02 (ci-01) + realistic-03 (dock-01) + realistic-04 (auth-02) + realistic-06 (auth-03) + realistic-10 (ci-05) + unexpected records still injected: + exact-01 (mig-02, mig-03, mig-06) + exact-03 (dis-07) + morphology-01 (dis-02) + distractor-01 (dis-02) + distractor-02 (dis-03) + distractor-03 (dis-07) + distractor-06 (dis-12, dis-11) + distractor-08 (dis-05) + stopword-02 (mig-04, mig-02) + stopword-05 (ci-03, dis-10) + +at 0.4: + expected records it keeps out: + paraphrase-08 (pkg-01) + realistic-01 (mig-01) + realistic-02 (ci-01) + realistic-03 (dock-01) + realistic-04 (auth-02) + realistic-05 (pkg-01) + realistic-06 (auth-03) + realistic-07 (mig-05) + realistic-10 (ci-05) + unexpected records still injected: + exact-01 (mig-02, mig-03, mig-06) + exact-03 (dis-07) + morphology-01 (dis-02) + distractor-01 (dis-02) + distractor-02 (dis-03) + distractor-03 (dis-07) + distractor-06 (dis-12, dis-11) + distractor-08 (dis-05) + stopword-02 (mig-04, mig-02) + stopword-05 (ci-03, dis-10) + +at 0.5: + expected records it keeps out: + paraphrase-08 (pkg-06, pkg-01) + realistic-01 (mig-01) + realistic-02 (ci-01) + realistic-03 (dock-01) + realistic-04 (auth-02) + realistic-05 (pkg-01) + realistic-06 (auth-03) + realistic-07 (mig-05) + realistic-08 (mig-03) + realistic-10 (ci-05) + unexpected records still injected: + exact-01 (mig-02, mig-03, mig-06) + exact-03 (dis-07) + morphology-01 (dis-02) + distractor-01 (dis-02) + distractor-02 (dis-03) + distractor-03 (dis-07) + distractor-06 (dis-12, dis-11) + distractor-08 (dis-05) + stopword-02 (mig-04, mig-02) + stopword-05 (ci-03, dis-10) + +at 0.6: + expected records it keeps out: + paraphrase-01 (mig-04) + paraphrase-06 (ci-03) + paraphrase-08 (pkg-06, pkg-01) + realistic-01 (mig-01) + realistic-02 (ci-01) + realistic-03 (dock-01) + realistic-04 (auth-02) + realistic-05 (pkg-01) + realistic-06 (auth-03) + realistic-07 (mig-05) + realistic-08 (mig-03) + realistic-10 (ci-05) + unexpected records still injected: + exact-01 (mig-02, mig-03, mig-06) + exact-03 (dis-07) + morphology-01 (dis-02) + distractor-01 (dis-02) + distractor-02 (dis-03) + distractor-03 (dis-07) + distractor-06 (dis-12, dis-11) + distractor-08 (dis-05) + stopword-02 (mig-04, mig-02) + stopword-05 (ci-03, dis-10) diff --git a/tests/AgentExperience.RetrievalQuality/GoldenSqliteReport.txt b/tests/AgentExperience.RetrievalQuality/GoldenSqliteReport.txt index 28160d30..82602d16 100644 --- a/tests/AgentExperience.RetrievalQuality/GoldenSqliteReport.txt +++ b/tests/AgentExperience.RetrievalQuality/GoldenSqliteReport.txt @@ -416,3 +416,113 @@ sig-18 [call_api, HTTP 403] expected: none exact: none coarse: none + +INJECTION FLOOR + +What it measures: what would reach the injected block at MaxRecords = 8 under ExperienceInjectionOptions.MinimumMatch +with MinimumTextCoverage f and no vector channel: the text candidates above, in order, whose share of the query's +distinct terms is at least f, cut to 8. f = 0 is the block with no floor, so its row repeats the overall metrics. +dFP@8: FP@8 over the distractor-overlap queries only. R@8, FP@8 and zero as above. The order is the source's +relevance, not Core's composite score, which also weighs confidence, recency, status and environment. + +floor n lab R@8 FP@8 dFP@8 zero +0.0 48 45 0.867 0.625 0.667 0.167 +0.3 48 45 0.800 0.354 0.667 0.229 +0.4 48 45 0.678 0.333 0.667 0.333 +0.5 48 45 0.644 0.333 0.667 0.375 +0.6 48 45 0.578 0.313 0.667 0.438 + +at 0.3: + expected records it keeps out: + realistic-03 (dock-01) + realistic-06 (auth-03) + realistic-10 (ci-05) + unexpected records still injected: + exact-01 (mig-02, mig-06, mig-03) + exact-03 (dis-07) + exact-04 (pkg-03) + realistic-02 (ci-03) + morphology-01 (dis-02) + distractor-01 (dis-02) + distractor-02 (dis-03) + distractor-03 (dis-07) + distractor-06 (dis-12, dis-11) + distractor-08 (dis-05) + stopword-02 (mig-04, mig-02) + stopword-05 (ci-03, dis-10) + +at 0.4: + expected records it keeps out: + paraphrase-08 (pkg-01) + realistic-01 (mig-01) + realistic-02 (ci-01) + realistic-03 (dock-01) + realistic-04 (auth-02) + realistic-05 (pkg-01) + realistic-06 (auth-03) + realistic-07 (mig-05) + realistic-10 (ci-05) + unexpected records still injected: + exact-01 (mig-02, mig-06, mig-03) + exact-03 (dis-07) + exact-04 (pkg-03) + morphology-01 (dis-02) + distractor-01 (dis-02) + distractor-02 (dis-03) + distractor-03 (dis-07) + distractor-06 (dis-12, dis-11) + distractor-08 (dis-05) + stopword-02 (mig-04, mig-02) + stopword-05 (ci-03, dis-10) + +at 0.5: + expected records it keeps out: + paraphrase-08 (pkg-01, pkg-06) + realistic-01 (mig-01) + realistic-02 (ci-01) + realistic-03 (dock-01) + realistic-04 (auth-02) + realistic-05 (pkg-01) + realistic-06 (auth-03) + realistic-07 (mig-05) + realistic-08 (mig-03) + realistic-10 (ci-05) + unexpected records still injected: + exact-01 (mig-02, mig-06, mig-03) + exact-03 (dis-07) + exact-04 (pkg-03) + morphology-01 (dis-02) + distractor-01 (dis-02) + distractor-02 (dis-03) + distractor-03 (dis-07) + distractor-06 (dis-12, dis-11) + distractor-08 (dis-05) + stopword-02 (mig-04, mig-02) + stopword-05 (ci-03, dis-10) + +at 0.6: + expected records it keeps out: + paraphrase-01 (mig-04) + paraphrase-05 (dock-06) + paraphrase-06 (ci-03) + paraphrase-08 (pkg-01, pkg-06) + realistic-01 (mig-01) + realistic-02 (ci-01) + realistic-03 (dock-01) + realistic-04 (auth-02) + realistic-05 (pkg-01) + realistic-06 (auth-03) + realistic-07 (mig-05) + realistic-08 (mig-03) + realistic-10 (ci-05) + unexpected records still injected: + exact-01 (mig-02, mig-06, mig-03) + exact-03 (dis-07) + morphology-01 (dis-02) + distractor-01 (dis-02) + distractor-02 (dis-03) + distractor-03 (dis-07) + distractor-06 (dis-12, dis-11) + distractor-08 (dis-05) + stopword-02 (mig-04, mig-02) + stopword-05 (ci-03, dis-10) diff --git a/tests/AgentExperience.RetrievalQuality/Harness/HybridRetrievalRun.cs b/tests/AgentExperience.RetrievalQuality/Harness/HybridRetrievalRun.cs index ec82ba87..c5fb0f32 100644 --- a/tests/AgentExperience.RetrievalQuality/Harness/HybridRetrievalRun.cs +++ b/tests/AgentExperience.RetrievalQuality/Harness/HybridRetrievalRun.cs @@ -79,9 +79,19 @@ await generator.GenerateAsync(query.Text, cancellationToken), cancellationToken); Ensure(vector.Outcome == ExperienceVectorSearchOutcome.Found, query.Id, vector.Outcome.ToString()); - text.Add(new QueryOutcome(query, [.. lexical.Candidates.Select(c => Id(c, query.Id))])); + text.Add(new QueryOutcome(query, [.. lexical.Candidates.Select(c => Id(c, query.Id))], [.. lexical.Candidates.Select(c => c.TermCoverage ?? 0d)])); var aboveFloor = vector.Candidates.Where(c => c.Relevance >= MinimumVectorRelevance).ToList(); - hybrid.Add(new QueryOutcome(query, [.. Merge(lexical.Candidates, aboveFloor).Select(c => Id(c, query.Id))])); + var merged = Merge(lexical.Candidates, aboveFloor); + + // Story 28.5: a record the vector channel found above the policy's floor passes the injection floor's semantic + // leg (MinimumSemanticSimilarity defaults to that floor), so it counts as full strength; the rest by their text + // coverage. + var semantic = aboveFloor.Select(c => c.Record.ExperienceId).ToHashSet(); + var coverage = lexical.Candidates.ToDictionary(c => c.Record.ExperienceId, c => c.TermCoverage ?? 0d); + hybrid.Add(new QueryOutcome( + query, + [.. merged.Select(c => Id(c, query.Id))], + [.. merged.Select(c => semantic.Contains(c.Record.ExperienceId) ? 1d : coverage.GetValueOrDefault(c.Record.ExperienceId))])); } // Story 28.2: the signature channel does not depend on vectors, so both columns carry the same lookups. diff --git a/tests/AgentExperience.RetrievalQuality/Harness/InjectionFloorSweep.cs b/tests/AgentExperience.RetrievalQuality/Harness/InjectionFloorSweep.cs new file mode 100644 index 00000000..4d5fdd93 --- /dev/null +++ b/tests/AgentExperience.RetrievalQuality/Harness/InjectionFloorSweep.cs @@ -0,0 +1,31 @@ +namespace AgentExperience.RetrievalQuality.Harness; + +/// +/// Story 28.5: the injection decision under a match floor. A query's injected set at text-coverage floor f is its +/// candidates, in the source's order, whose term coverage (the share of the query's distinct terms they contain) is at +/// least f, cut to the first 8: what ExperienceInjectionOptions.MinimumMatch with that +/// MinimumTextCoverage would let reach the block at MaxRecords = 8, with no vector channel. A floor of 0 is +/// the block without one, so its row repeats the overall metrics. +/// +internal static class InjectionFloorSweep +{ + /// The floors swept, fixed in the story's rule before any run. + public static IReadOnlyList Floors { get; } = [0d, 0.3, 0.4, 0.5, 0.6]; + + /// The comparison tolerance, the injection floor's own (ExperienceMatchFloor.Tolerance). + public const double Tolerance = 1e-6; + + /// What reaches the block for at . + public static QueryOutcome Apply(QueryOutcome outcome, double floor) + { + var coverage = outcome.Coverage ?? throw new InvalidOperationException($"{outcome.Query.Id} carries no coverage."); + return new QueryOutcome( + outcome.Query, + [.. outcome.Returned.Where((_, index) => coverage[index] >= floor - Tolerance).Take(8)], + null); + } + + /// Every query of at , in corpus order. + public static IReadOnlyList Apply(RetrievalQualityResult result, double floor) => + [.. result.Outcomes.Select(outcome => Apply(outcome, floor))]; +} diff --git a/tests/AgentExperience.RetrievalQuality/Harness/RetrievalQualityReport.cs b/tests/AgentExperience.RetrievalQuality/Harness/RetrievalQualityReport.cs index 5c82efb6..d3896ee5 100644 --- a/tests/AgentExperience.RetrievalQuality/Harness/RetrievalQualityReport.cs +++ b/tests/AgentExperience.RetrievalQuality/Harness/RetrievalQualityReport.cs @@ -102,6 +102,7 @@ public static string Render(RetrievalQualityResult result) } RenderSignatures(result, Line); + RenderInjectionFloor(result, Line); return builder.ToString(); } @@ -146,6 +147,60 @@ static string Returned(QueryOutcome outcome) => outcome.Returned.Count == 0 : string.Join(", ", outcome.Returned.Take(Shown)) + (outcome.Returned.Count > Shown ? Invariant($", ... {outcome.Returned.Count - Shown} more") : string.Empty); } + /// + /// Story 28.5: the injection floor's sweep: per text-coverage floor, recall and noise of what would reach the block, the + /// distractor-overlap queries' noise, and which expected records and which unexpected ones each floor changes. + /// + private static void RenderInjectionFloor(RetrievalQualityResult result, Action line) + { + line(string.Empty); + line("INJECTION FLOOR"); + line(string.Empty); + line("What it measures: what would reach the injected block at MaxRecords = 8 under ExperienceInjectionOptions.MinimumMatch"); + line("with MinimumTextCoverage f and no vector channel: the text candidates above, in order, whose share of the query's"); + line("distinct terms is at least f, cut to 8. f = 0 is the block with no floor, so its row repeats the overall metrics."); + line("dFP@8: FP@8 over the distractor-overlap queries only. R@8, FP@8 and zero as above. The order is the source's"); + line("relevance, not Core's composite score, which also weighs confidence, recency, status and environment."); + line(string.Empty); + line(Row("floor", "n", "lab", "R@8", "FP@8", "dFP@8", "zero")); + var unfloored = InjectionFloorSweep.Apply(result, 0d); + foreach (var floor in InjectionFloorSweep.Floors) + { + var floored = InjectionFloorSweep.Apply(result, floor); + var metrics = RetrievalMetrics.Of(floored); + var distractor = RetrievalMetrics.Of([.. floored.Where(o => o.Query.Category == "distractor-overlap")]); + line(Row( + floor.ToString("0.0", CultureInfo.InvariantCulture), + Invariant($"{metrics.Queries}"), + Invariant($"{metrics.Labelled}"), + Number(metrics.RecallAt8), + Number(metrics.FalsePositivesAt8), + Number(distractor.FalsePositivesAt8), + Number(metrics.ZeroCandidateShare))); + } + + foreach (var floor in InjectionFloorSweep.Floors.Where(f => f > 0d)) + { + var floored = InjectionFloorSweep.Apply(result, floor); + line(string.Empty); + line(Invariant($"at {floor:0.0}:")); + var lost = unfloored.Zip(floored) + .Select(pair => (pair.First.Query.Id, Ids: pair.First.Returned.Except(pair.Second.Returned).Where(pair.First.Query.Expected.Contains).ToList())) + .Where(entry => entry.Ids.Count > 0) + .Select(entry => $"{entry.Id} ({string.Join(", ", entry.Ids)})") + .ToList(); + line(" expected records it keeps out:" + (lost.Count == 0 ? " none" : string.Empty)); + lost.ForEach(entry => line(" " + entry)); + var noise = floored + .Select(o => (o.Query.Id, Ids: o.Returned.Where(id => !o.Query.Expected.Contains(id)).ToList())) + .Where(entry => entry.Ids.Count > 0) + .Select(entry => $"{entry.Id} ({string.Join(", ", entry.Ids)})") + .ToList(); + line(" unexpected records still injected:" + (noise.Count == 0 ? " none" : string.Empty)); + noise.ForEach(entry => line(" " + entry)); + } + } + private static string Ineligibility(RetrievalCorpus corpus) { var ineligible = corpus.Records.Where(r => r.Group == RetrievalCorpus.Ineligible).ToList(); diff --git a/tests/AgentExperience.RetrievalQuality/Harness/RetrievalQualityRun.cs b/tests/AgentExperience.RetrievalQuality/Harness/RetrievalQualityRun.cs index c8f257ee..6d456b18 100644 --- a/tests/AgentExperience.RetrievalQuality/Harness/RetrievalQualityRun.cs +++ b/tests/AgentExperience.RetrievalQuality/Harness/RetrievalQualityRun.cs @@ -5,7 +5,11 @@ namespace AgentExperience.RetrievalQuality.Harness; /// What one query's search returned, by corpus record ID, strongest match first. /// The query. /// The corpus IDs of the candidates, in the order the source returned them. -internal sealed record QueryOutcome(CorpusQuery Query, IReadOnlyList Returned); +/// +/// Story 28.5: each candidate's , parallel to , for a +/// text query; for a signature lookup. +/// +internal sealed record QueryOutcome(CorpusQuery Query, IReadOnlyList Returned, IReadOnlyList? Coverage = null); /// Story 28.2: what one signature query's two lookups returned, by corpus record ID, in the source's order. /// The signature query. @@ -69,7 +73,9 @@ public static async Task RunAsync( query, [.. result.Candidates.Select(c => byExperienceId.TryGetValue(c.Record.ExperienceId, out var id) ? id - : throw new InvalidOperationException($"Query {query.Id} returned {c.Record.ExperienceId}, which is not a corpus record."))])); + : throw new InvalidOperationException($"Query {query.Id} returned {c.Record.ExperienceId}, which is not a corpus record."))], + [.. result.Candidates.Select(c => c.TermCoverage + ?? throw new InvalidOperationException($"Query {query.Id}'s candidate {c.Record.ExperienceId} carries no term coverage."))])); } return new RetrievalQualityResult( diff --git a/tests/AgentExperience.RetrievalQuality/RetrievalQualityTests.cs b/tests/AgentExperience.RetrievalQuality/RetrievalQualityTests.cs index 7877a698..ca882a47 100644 --- a/tests/AgentExperience.RetrievalQuality/RetrievalQualityTests.cs +++ b/tests/AgentExperience.RetrievalQuality/RetrievalQualityTests.cs @@ -123,6 +123,27 @@ public async Task The_signature_channel_s_exact_match_is_precise_and_its_coarse_ Assert.Contains(result.Signatures, o => o.Exact.Returned.Count > 0); } + /// + /// Story 28.5: the injection floor's sweep at 0 is the block with no floor, so its row is the overall metrics' R@8 + /// and FP@8, and the golden's sweep table says so. + /// + [Fact] + public async Task The_floor_sweep_at_zero_repeats_the_overall_recall_and_noise() + { + var result = await RunAsync(RetrievalCorpus.Load()); + + var overall = RetrievalMetrics.Of(result.Outcomes); + var unfloored = RetrievalMetrics.Of(InjectionFloorSweep.Apply(result, 0d)); + Assert.Equal(overall.RecallAt8, unfloored.RecallAt8); + Assert.Equal(overall.FalsePositivesAt8, unfloored.FalsePositivesAt8); + + var golden = GoldenFile.Read(GoldenFileName).Split('\n'); + string Cells(string line) => string.Join(' ', line.Split(' ', StringSplitOptions.RemoveEmptyEntries)); + var overallRow = Cells(golden.First(line => line.StartsWith("overall ", StringComparison.Ordinal))).Split(' '); + var zeroRow = Cells(golden.First(line => line.StartsWith("0.0 ", StringComparison.Ordinal))).Split(' '); + Assert.Equal((overallRow[5], overallRow[8]), (zeroRow[3], zeroRow[4])); + } + private static string FirstDifference(string golden, string rendered) { var expected = golden.Split('\n'); diff --git a/tests/AgentExperience.Storage.Conformance/CandidateSourceConformanceTests.cs b/tests/AgentExperience.Storage.Conformance/CandidateSourceConformanceTests.cs index d059321f..1ac0bbe7 100644 --- a/tests/AgentExperience.Storage.Conformance/CandidateSourceConformanceTests.cs +++ b/tests/AgentExperience.Storage.Conformance/CandidateSourceConformanceTests.cs @@ -244,6 +244,27 @@ public async Task A_record_covering_more_of_the_query_s_terms_ranks_first_with_a Assert.True(result.Candidates[2].Relevance > 0d); } + /// Story 28.5: every candidate reports the plain share of the query's distinct terms it contains, whatever fields hold them. + [Fact] + public async Task Each_candidate_reports_the_share_of_the_query_s_terms_it_contains_whatever_field_holds_them() + { + var tenant = NewTenant(); + var scope = Scope(tenant); + var all = await SeedAsync(tenant, scope, taskId: "t1", summary: "Refund policy for a disputed invoice", lesson: "Escalate"); + var inSummary = await SeedAsync(tenant, scope, taskId: "t2", summary: "Refund policy questions", lesson: "Escalate"); + var inLesson = await SeedAsync(tenant, scope, taskId: "t3", summary: "Questions from a customer", lesson: "Check the refund policy first"); + var one = await SeedAsync(tenant, scope, taskId: "t4", summary: "Refund requested by the customer", lesson: "Escalate"); + + var result = await SearchAsync(tenant, scope, "refund policy invoice", minimumMatchedTerms: 1); + + double Coverage(Guid id) => result.Candidates.Single(c => c.Record.ExperienceId == id).TermCoverage + ?? throw new Xunit.Sdk.XunitException("A shipped text source must report the term coverage."); + Assert.Equal(1d, Coverage(all), 6); + Assert.Equal(2d / 3d, Coverage(inSummary), 6); + Assert.Equal(2d / 3d, Coverage(inLesson), 6); + Assert.Equal(1d / 3d, Coverage(one), 6); + } + [Fact] public async Task A_record_below_the_minimum_of_matched_terms_is_not_returned_and_the_minimum_is_capped_at_the_query_s_terms() { diff --git a/tests/AgentExperience.Storage.Postgres.Tests/PostgresGrantTests.cs b/tests/AgentExperience.Storage.Postgres.Tests/PostgresGrantTests.cs index 221dd679..c201b20c 100644 --- a/tests/AgentExperience.Storage.Postgres.Tests/PostgresGrantTests.cs +++ b/tests/AgentExperience.Storage.Postgres.Tests/PostgresGrantTests.cs @@ -602,6 +602,9 @@ public async Task The_text_channel_returns_a_granted_record_under_the_same_eligi // in the same owner scope is not reachable at all. Assert.Equal([shared], result.Candidates.Select(candidate => candidate.Record.ExperienceId)); Assert.True(Assert.Single(result.Candidates).SharedByGrant); + + // Story 28.5: a borrowed candidate reports its term coverage like an owned one. + Assert.Equal(1d, Assert.Single(result.Candidates).TermCoverage); Assert.DoesNotContain(ineligible, result.Candidates.Select(candidate => candidate.Record.ExperienceId)); Assert.DoesNotContain(ungranted, result.Candidates.Select(candidate => candidate.Record.ExperienceId)); diff --git a/tests/AgentExperience.Storage.Postgres.Tests/PostgresTextSearchRowLevelSecurityTests.cs b/tests/AgentExperience.Storage.Postgres.Tests/PostgresTextSearchRowLevelSecurityTests.cs index a77dcb12..a060b01b 100644 --- a/tests/AgentExperience.Storage.Postgres.Tests/PostgresTextSearchRowLevelSecurityTests.cs +++ b/tests/AgentExperience.Storage.Postgres.Tests/PostgresTextSearchRowLevelSecurityTests.cs @@ -163,6 +163,9 @@ public async Task Any_term_matching_answers_exactly_the_same_with_row_level_secu Assert.Contains(startup.ToString(), off[12], StringComparison.Ordinal); Assert.Contains(docker.ToString(), off[16], StringComparison.Ordinal); + // Story 28.5: every candidate carries its term coverage, equal to its relevance, through either route. + Assert.All(off[0].Split('\n'), line => Assert.Equal(line.Split(' ')[1], line.Split(' ')[2])); + await world.ApplyAsync(enable: true); Assert.True(await world.RoutesThroughFunctionAsync(tenant)); for (var i = 0; i < requests.Count; i++) @@ -698,7 +701,7 @@ public async Task AnswerAsync(AuthorizationContext authorization, Experi Assert.Equal(ExperienceStoreOutcome.Found, result.Outcome); var lines = result.Candidates - .Select(c => $"{c.Record.ExperienceId} {c.Relevance:R} {c.SharedByGrant} {c.PermittingGrantId} {c.GrantDisclosure}") + .Select(c => $"{c.Record.ExperienceId} {c.Relevance:R} {c.TermCoverage:R} {c.SharedByGrant} {c.PermittingGrantId} {c.GrantDisclosure}") .ToList(); await using var command = Owner.CreateCommand( diff --git a/tests/AgentExperience.Upgrade.Tests/README.md b/tests/AgentExperience.Upgrade.Tests/README.md index 7142b905..b3a7dbda 100644 --- a/tests/AgentExperience.Upgrade.Tests/README.md +++ b/tests/AgentExperience.Upgrade.Tests/README.md @@ -129,7 +129,7 @@ the documented behaviour instead: The comparison enforces the same list from the other side: a member today's API returns that the preview's did not fails unless it is one of these, new by design, each listed with its reason in `UpgradeReport.NewByDesign`: `closedRoundId`, `origin`, `exposedTo`, `provenanceSignature`, `authorship`, `assessmentId`, `admission`, -`disclosure`, `approachArguments`, `grantDisclosure`, `grantApproachArguments`, `recipeExcerpts`, `grantRecipeExcerpts` and `injections`. Everything else must read back +`disclosure`, `approachArguments`, `grantDisclosure`, `grantApproachArguments`, `recipeExcerpts`, `grantRecipeExcerpts`, `injections` and `termCoverage` (computed by each search, so only its presence is checked). Everything else must read back identical. ## Running it diff --git a/tests/AgentExperience.Upgrade.Tests/UpgradeReport.cs b/tests/AgentExperience.Upgrade.Tests/UpgradeReport.cs index df95eb3a..0aafe212 100644 --- a/tests/AgentExperience.Upgrade.Tests/UpgradeReport.cs +++ b/tests/AgentExperience.Upgrade.Tests/UpgradeReport.cs @@ -45,8 +45,15 @@ internal sealed class UpgradeReport(string subject, int major) ["recipeExcerpts"] = "ExperienceGrant, unreleased (story 28.4, 0031): null when absent", ["grantRecipeExcerpts"] = "ExperienceRecordGetResult, unreleased (story 28.4, 0031): null when absent", ["injections"] = "ExperienceRecordQueryResult, unreleased (story 21.8, 0026): QueryWithInjectionsAsync's summaries; null from QueryAsync", + ["termCoverage"] = "ExperienceCandidate, unreleased (story 28.5): computed by each search, never stored", }; + /// + /// The members of that a search computes rather than reads from a row: a preview never + /// returned them, so the value it would have had is unknowable, and only their presence is checked, not their value. + /// + public static readonly IReadOnlySet ComputedByDesign = new HashSet(StringComparer.Ordinal) { "termCoverage" }; + private const int MaxDifferencesPerItem = 5; private const int MaxShownLength = 160; @@ -113,6 +120,11 @@ private static void Diff(string path, JsonNode? expected, JsonNode? actual, List continue; } + if (ev is null && ComputedByDesign.Contains(name)) + { + continue; + } + Diff($"{path}.{name}", ev, av, differences); }