Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 34 additions & 11 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -316,17 +316,6 @@ AgentExperience.NET is a **preview**. It is not production ready, and public API
others `AlreadyRecorded`, a changed resubmission `Conflict` from all three), all three read the same history and
confidence, and each agent alone leaves the same ledger, evidence, event and record rows in its own database.

## 0.1.0-preview.17

This preview measures whether injected lessons help. An opt-in holdout withholds the block on a deterministic share of
runs and records both arms; `AgentExperienceBenefit.EstimateAsync` compares the arms' verified outcomes, overall, per
lesson and optionally per model; the injection provider can act on the measured benefit; the same holdout measures any
other MAF memory provider; and lessons measured as helping can be exported to other memory stores. PostgreSQL gains
migrations `0027` to `0029` (re-run `ApplyApplicationRolePrivilegesAsync` after migrating). The Python and Go ports
take part in the holdout through contract 1.8 (add-only). Every new option is off by default, and the public API
changes are additions. The same tag releases the six NuGet packages, the Python distribution `agenticexperience` and
the six Go modules at one version.

### A scoped supersession lock and ranking to the bit: contract 1.18 (story 28.1)

- **A supersession locks only rows in the caller's scope.** The PostgreSQL store locked the record and its proposed
Expand Down Expand Up @@ -487,6 +476,40 @@ the six Go modules at one version.
roles, vectors, retention sweeps. Internally, the PostgreSQL payload's attempt, outcome, environment and provenance
mappings are each one method now, shared with the SQLite store; nothing stored changes.

### Precision before recall: an injection-time match floor (story 28.5)

- **`ExperienceInjectionOptions.MinimumMatch`** (type `ExperienceMatchFloor`, off by default): a record is injected only
when it has an exact failure-signature match on the current run (a recall hint), text coverage at least
`MinimumTextCoverage` (the share of the request's distinct terms it contains, relative to the store's terms; required,
no default), or semantic similarity at least `MinimumSemanticSimilarity` (by default the retrieval policy's
`MinimumVectorRelevance`; with neither set the semantic leg is off). A weaker record is omitted as the new
`InjectionOmissionReason.WeakMatch`, with a content-free detail, before the capability gate and the record limit, so
it takes no slot and a stronger one backfills; a host-limited request is widened like the capability gate's. A block
that injects nothing with a `WeakMatch` omission ends `NothingToInject`, and its `inject` span carries
`agentexperience.nothing_reason = weak_match`. The holdout splits after the floor. Under a floor, recall never asks
for a coarse signature match. Which candidates retrieval returns is unchanged, and with the floor off everything is
byte for byte what it was.
- **Each channel's strength travels with the record.** `ExperienceCandidate.TermCoverage` (the in-memory, PostgreSQL and
SQLite text sources set it), `RankedExperience.TextCoverage` and `RankedExperience.VectorRelevance` (set by
retrieval). Ranking is unchanged.
- **Chosen by a rule fixed before measuring, and the rule said off.** The retrieval benchmark's goldens gain an
`INJECTION FLOOR` sweep (0 to 0.6). No floor kept unexpected records at or below 0.15 in the first 8 on both stores, and
none removed the distractor-overlap noise (distractors containing every word of a short query), while every floor
cost long, realistic requests their lesson; so the floor stays off and has no recommended value
([the table](docs/benchmarks.md#the-injection-floor-story-285)).
- The Python and Go ports and the contract are unchanged; the ports do not apply a floor yet.

## 0.1.0-preview.17

This preview measures whether injected lessons help. An opt-in holdout withholds the block on a deterministic share of
runs and records both arms; `AgentExperienceBenefit.EstimateAsync` compares the arms' verified outcomes, overall, per
lesson and optionally per model; the injection provider can act on the measured benefit; the same holdout measures any
other MAF memory provider; and lessons measured as helping can be exported to other memory stores. PostgreSQL gains
migrations `0027` to `0029` (re-run `ApplyApplicationRolePrivilegesAsync` after migrating). The Python and Go ports
take part in the holdout through contract 1.8 (add-only). Every new option is off by default, and the public API
changes are additions. The same tag releases the six NuGet packages, the Python distribution `agenticexperience` and
the six Go modules at one version.

### The holdout in Python and Go: contract 1.8 (story 25.6)

- **Contract 1.8** ([what it adds](contract/README-1.8.md)) brings the injection holdout into the language-neutral
Expand Down
32 changes: 32 additions & 0 deletions docs/benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -287,3 +287,35 @@ compound cannot meet the minimum alone. **Today's numbers** (recorded 2026-10-10
- **This is the source's order, not Core's final ranking.** The benchmark calls the candidate source directly; Core
then re-scores candidates with confidence, recency, status and environment, and on PostgreSQL sees every full match
at the same text relevance, so what an agent is finally shown can be ordered differently.

### The injection floor (story 28.5)

Story 28.5 added an injection-time match floor (`ExperienceInjectionOptions.MinimumMatch`, see
[Keeping weak matches out](guide/injection.md#keeping-weak-matches-out)) and extended the goldens with an
`INJECTION FLOOR` section: what would reach an 8-record block if only candidates covering at least a share `f` of the
query's distinct terms were injected. It takes the candidates in the source's relevance order (not Core's composite
score, which also weighs confidence, recency, status and environment), cut to 8, text only; a test checks that its
`f = 0` row equals each golden's overall R@8 and FP@8. The rule was fixed before any run: sweep `f` over 0, 0.3, 0.4,
0.5 and 0.6, keep the floors whose FP@8 is at most 0.15 on both adapters (in-memory and PostgreSQL) and whose
`distractor-overlap` queries inject nothing unexpected, and take the one with the highest PostgreSQL R@8 (ties to the
higher floor); if one is chosen it becomes the default, otherwise the floor stays off.

| Floor | In-memory R@8 | In-memory FP@8 | In-memory distractor FP@8 | PostgreSQL R@8 | PostgreSQL FP@8 | PostgreSQL distractor FP@8 |
| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 0 (off) | 0.700 | 0.417 | 0.667 | 0.844 | 0.521 | 0.667 |
| 0.3 | 0.611 | 0.250 | 0.667 | 0.733 | 0.313 | 0.667 |
| 0.4 | 0.533 | 0.250 | 0.667 | 0.656 | 0.313 | 0.667 |
| 0.5 | 0.533 | 0.250 | 0.667 | 0.622 | 0.313 | 0.667 |
| 0.6 | 0.467 | 0.250 | 0.667 | 0.578 | 0.313 | 0.667 |

**No floor qualifies, so the floor stays off and has no default value.** Above 0.3 the noise does not move: what is left
are distractors that contain *every* word of a short query (`distractor-03`'s "reduce image size" and `dis-07`), which no
coverage floor can tell from the right lesson. What a floor removes first is the right lesson for long, realistic
requests, where three matched words are a small share of the request: at 0.3, 5 realistic queries lose their record on
PostgreSQL and 4 in memory. The SQLite golden (added after the rule was fixed, so not part of it) has the same shape:
FP@8 0.625 at 0, 0.354 at 0.3 and 0.313 at 0.6, with distractor FP@8 0.667 throughout. Coverage is relative to each
store's terms (unstemmed words in memory, stemmed lexemes on PostgreSQL, Porter stems on SQLite).

The hybrid column needs a real embedding provider, so the sweep was not run with vectors; its harness counts a record
the vector channel found above the policy's floor as passing (the semantic leg). The real-task question, whether
injecting fewer, stronger lessons helps an agent, is the holdout's to answer per deployment.
58 changes: 57 additions & 1 deletion docs/guide/injection.md
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,7 @@ it through its own redaction first. It recognises no language: no stemming, no s
| `MessageRole` | `User` | The chat role the block is sent in: `User` or `System`. Some chat APIs reject, move or merge a system message that is not first, so test `System` with your provider. See [The payload](#the-payload). |
| `DecideInjection` | none (permit) | Per-candidate host risk decision, asked after the final eligibility check. Fail-closed: a callback that throws or returns `null` denies. Synchronous on purpose: with session tracking on it runs while the session's lock is held. Per-candidate I/O has no async hook: prefetch what it needs, keyed by scope, in `ResolveRequestAsync`, or decide offline. |
| `ReceivingAgent` | none (off) | The receiving agent's tools (declared, or the run's own with `UseRunTools`) and maximum risk class. A record whose verified approach it cannot, or must not, carry out is not injected and takes no record slot. See [Gating on the receiving agent's capabilities](#gating-on-the-receiving-agents-capabilities). |
| `MinimumMatch` | `null` (off) | The match-strength floor: a record is injected only when its match is strong enough (an exact failure-signature match, enough of the request's terms, or enough semantic similarity); weaker ones are omitted as `WeakMatch` and take no slot. See [Keeping weak matches out](#keeping-weak-matches-out). |
| `ModelAuthoredLessons` (since `0.1.0-preview.5`) | `Include` | Whether records whose free text a model wrote are injected (labelled) or omitted. `Exclude` asks retrieval to leave them out before its limit (since `0.1.0-preview.6`); the provider still drops any that arrive. See [Model-authored lessons](#model-authored-lessons). |
| `OnContextInjected` | none | Receives the content-free account of every attempt, including every omission and its reason. Exceptions it throws are swallowed. No result is emitted when the caller cancels the invocation: the cancellation propagates instead. |
| `TimeProvider` | `TimeProvider.System` | The clock the final eligibility check measures record expiry and its own timeout with. |
Expand Down Expand Up @@ -195,7 +196,7 @@ agent-framework does not hand them to a context provider, so only the agent's ow
added count. In Go a zero window multiplier means 4, and `Deny("")` reports the default detail. Still .NET-only: session
tracking, withdrawal and the final re-read (`SessionLimits`, `SessionStateKey`, `Limits.EligibilityCheckTimeout`,
`Limits.MaxAbandonedReads`), unconfirmed content, the decision context's grant fields and required environment
attributes.
attributes, and the match floor (`MinimumMatch`): the ports do not apply a floor yet.

## What the agent sees

Expand Down Expand Up @@ -737,6 +738,61 @@ new ExperienceInjectionOptions
collection, or an undefined `ToolRiskClass`, with `ArgumentException`. With `ReceivingAgent` unset, the block, the omissions and the outcomes are byte for byte what
they are without it. `CapabilityGateTests` pins each case.

## Keeping weak matches out

Retrieval's channel floors decide which records are *candidates*: by default, a text candidate needs any three of the
request's terms (see [How the text channel matches](retrieval.md#how-the-text-channel-matches)), which finds long,
paraphrased requests at the price of some records that share only a few words. Without a floor, every candidate that
survives the checks below is injected however weakly it matched. The real-task experiment found that a wrong or
empty lesson costs the agent: its mismatched-cluster control did worse than no memory.

`MinimumMatch` is a second, separate gate, applied at injection time. A record is injected only if **at least one** of
these holds:

| Leg | Passes when | Read from |
| --- | --- | --- |
| Failure signature | an exact signature match on the current run, which only a [recall hint](#recall-at-the-moment-of-failure) has; the before-run block has none | the recall lookup |
| Text | the record contains at least `MinimumTextCoverage` (required, no default) of the request's distinct terms | `RankedExperience.TextCoverage` |
| Semantic | its vector relevance is at least `MinimumSemanticSimilarity`, or, when that is `null`, the retrieval policy's `MinimumVectorRelevance`; with neither set this leg is **off** | `RankedExperience.VectorRelevance` |

```csharp
options.MinimumMatch = new ExperienceMatchFloor { MinimumTextCoverage = 0.5 };
```

A record below it is omitted as `WeakMatch`, with a content-free detail naming its strengths and the floor. It is
checked on the ranked candidate after `AlreadyDelivered` and before the capability gate and the record limit, so a weak
match takes no slot and a stronger record ranked below it fills it; like the capability gate, a request that names
its own `Limit` asks retrieval for a window four times `MaxRecords` to backfill from (a weak match fetched only as
backfill, past the host's own limit, is not reported). When nothing is injected and at least one record was omitted as
`WeakMatch`, the outcome is `NothingToInject` and the `inject` span carries `agentexperience.nothing_reason =
weak_match`, even if other reasons (a capability gate, an ineligible re-read) kept other records out too: the attribute
says the floor contributed, and the result's `Omitted` list says what kept each record out. The holdout splits its arms
after the floor, so both arms are judged on the same eligible set. A recall hint is a signature match only, so under a
floor only an exact match counts and recall never asks for a coarse one, whatever `AllowCoarseMatch` says.

Things to know:

- **It is off by default, on evidence, and has no recommended value.** On the
[retrieval benchmark](../benchmarks.md#the-injection-floor-story-285), no floor met the precision target fixed before
measuring (at most 0.15 unexpected records in the first 8 on both stores, and none on the distractor-overlap queries),
so `MinimumTextCoverage` has no default: choose it for your requests and measure. The noise a coverage floor cannot remove is distractors that contain
*every* word of a short request; what it removes first is the right lesson for a long, realistic one. Turn it on
where your requests are short and your lessons varied, and measure with the holdout.
- **Coverage is a share, not a count.** A twelve-term request matched on three terms covers 0.25, which a floor of 0.3
keeps out. That is the trade: the channel floor is a count precisely so long requests find something.
- **Coverage is relative to the store.** A term is what the text source counts: whole, unstemmed words in memory
("invoices" and "invoice" are two); stemmed lexemes on PostgreSQL, less stopwords, with a hyphenated compound counted
once; Porter-stemmed words on SQLite. The same request can have a different number of terms, and a record a
different coverage, on each store, so a floor tuned on one is not tuned on another.
- **The semantic leg fails closed.** With neither `MinimumSemanticSimilarity` nor the policy's `MinimumVectorRelevance`
set, it is off: a record only the vector channel found is a weak match however close. Setting
`MinimumSemanticSimilarity` below the policy's floor does nothing, because retrieval never returns a vector candidate
below the policy's floor.
- **Comparisons allow 1e-6**, so a coverage PostgreSQL computes in single precision (3 of 10 terms as 0.3000000119) is not
lost at a floor of 0.3.
- The lessons view sets the task aside and runs no retrieval, so it shows no floor.
- The Python and Go ports do not apply a floor yet.

## Model-authored lessons

A lesson written by a model-backed reflector (`Reflection.Authorship` is anything but
Expand Down
6 changes: 6 additions & 0 deletions docs/guide/retrieval.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,12 @@ finds anything.
var policy = RetrievalPolicy.Default with { MinimumMatchedTerms = ExperienceCandidateQuery.AllTerms };
```

Each candidate also reports its plain coverage, the share of the request's distinct terms it contains whatever fields
hold them (`ExperienceCandidate.TermCoverage`), which retrieval carries to `RankedExperience.TextCoverage` beside the
vector channel's `VectorRelevance`. Neither changes ranking; the injection provider's optional match floor reads them
to keep weak matches out of the block without changing which candidates retrieval returns (see
[Keeping weak matches out](injection.md#keeping-weak-matches-out)).

## Ranking is explainable

Every returned record carries all five normalized components (each in 0–1) and the effective weight applied to it,
Expand Down
1 change: 1 addition & 0 deletions docs/telemetry-names.json
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@
"agentexperience.feedback_id",
"agentexperience.history_read",
"agentexperience.independence.refusal",
"agentexperience.nothing_reason",
"agentexperience.omitted_count",
"agentexperience.operation",
"agentexperience.outcome",
Expand Down
Loading
Loading