diff --git a/CHANGELOG.md b/CHANGELOG.md index dca3f17e..34894470 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -327,6 +327,28 @@ take part in the holdout through contract 1.8 (add-only). Every new option is of changes are additions. The same tag releases the six NuGet packages, the Python distribution `agenticexperience` and the six Go modules at one version. +### A scoped supersession lock and ranking to the bit: contract 1.18 (story 28.1) + +- **A supersession locks only rows in the caller's scope.** The PostgreSQL store locked the record and its proposed + replacement by ID alone, so with row-level security off a caller naming another tenant's record ID as the + replacement held a row lock on that tenant's record for its transaction: no read or write, but a way to make another + tenant's writes wait and to time them. The lock now carries the same exact-scope predicate as every other lifecycle + statement. The outcome is unchanged (a replacement in another scope is `ReplacementNotAllowed`, as one that does not + exist), and a test proves a supersession naming a row another transaction holds no longer waits for it. +- **Contract 1.18** ([what it adds](contract/README-1.18.md)), add-only: `sql/lifecycle-1.18.sql`, the scoped lock + (`lock-supersession-rows-scoped`), which ports run in place of 1.16's `lock-supersession-rows`; and + `retrieval-policy/ranking-near-tie`, two pairs of records whose scores differ by one ulp, ranked against their + experience-ID tie-break, so a port whose arithmetic differs from .NET's in the last bit orders them differently + and fails. Every 1.0-1.17 file is unchanged. +- **The Python and Go ports run the scoped lock**, with PostgreSQL tests. **Go's decay now agrees with .NET's on the + fixture**: its `math.Pow` was 2 ulps off on the new fixture's input (about 30% of inputs are off by an ulp or more), + which tied two records .NET orders, so the Go port computes `2^x` with its own correctly rounded double-double + `exp2`, which agrees with .NET's C-library `pow` except on rare inputs where that `pow` misrounds (about 1 in 1,500 + measured on glibc); and it rounds each ranking component's product before adding it, so the compiler cannot fuse the + two (an FMA, which Go does on arm64). Python, which calls the platform's `pow` as .NET does, already matched. Only + the fixture's inputs are pinned: near-tied records whose recency or decay factor falls on a misrounded input may + still be ordered differently across runtimes. + ### The failure signature (story 28.2) - **A failed tool call now has a failure signature**: `ExperienceFailureSignature` (Abstractions), its tool name, its diff --git a/contract/README-1.18.md b/contract/README-1.18.md new file mode 100644 index 00000000..63a4469e --- /dev/null +++ b/contract/README-1.18.md @@ -0,0 +1,73 @@ +# Contract 1.18: additions + +This page lists what contract 1.18 adds to [contract 1.0](README.md) and the pages of 1.1 to 1.17 +([1.1](README-1.1.md), [1.2](README-1.2.md), [1.3](README-1.3.md), [1.4](README-1.4.md), [1.5](README-1.5.md), +[1.6](README-1.6.md), [1.7](README-1.7.md), [1.8](README-1.8.md), [1.9](README-1.9.md), [1.10](README-1.10.md), +[1.11](README-1.11.md), [1.12](README-1.12.md), [1.13](README-1.13.md), [1.14](README-1.14.md), [1.15](README-1.15.md), +[1.16](README-1.16.md) and [1.17](README-1.17.md)). Those pages are still normative and unchanged, except where this +page names the one statement a port now runs in place of another: a minor version only adds files, this page among +them. The contract's version is the one `lock.json` records (`contractVersion`), which the .NET suite holds in +`ContractFiles.ContractVersion`. + +A port that passes contract 1.18 passes every 1.0 to 1.17 fixture and the fixtures below. + +## The supersession lock, scoped to the request + +Contract 1.16's `sql/lifecycle.sql` locks the record being superseded and its replacement with +`lock-supersession-rows`, which names no scope: + +```sql +SELECT experience_id FROM agent_experience.experience_records WHERE experience_id = ANY(@lock_ids) ORDER BY experience_id FOR UPDATE +``` + +Without row-level security, a caller naming another tenant's record ID as the replacement therefore took a row lock on +that tenant's record for the rest of its transaction. It read and wrote nothing there (`check-supersession` reads the +replacement only in the request scope, so the outcome was `ReplacementNotAllowed` as for a record that does not exist), +but one tenant could make another's writes wait, and time them. + +**From 1.18, the .NET store and every port run [`sql/lifecycle-1.18.sql`](sql/lifecycle-1.18.sql)'s +`lock-supersession-rows-scoped` in its place**, in lifecycle.sql's COMMIT step 3, with the same `@lock_ids` and the +request scope bound as every other lifecycle statement binds it (`@tenant_id` ... `@user_id`, NULL for an unset +optional field). It locks only the rows of the request scope, still in experience-ID order, so two supersessions naming +each other still cannot deadlock. Nothing else changes: `check-supersession` follows it, and a replacement in another +scope reads as not found exactly as before (`ReplacementNotAllowed`, with no current status). No outcome, result or +table a fixture pins moves, so the 1.16 `lifecycle-sql` fixtures are unchanged and still pass. + +1.16's `lock-supersession-rows` stays in `lifecycle.sql` because a published file never changes; a 1.18 port does not +run it. The statement is generated from the .NET store's source, like the rest of `lifecycle.sql`, and checked against +it on every build. The application role needs no new privilege. + +## Ranking near ties, pinned to the bit + +Ranking orders records by score, highest first, and breaks an exact tie by experience ID as text, ascending. Fixtures +compare scores within 1e-12, so before 1.18 a port whose arithmetic differed from .NET's in the last bit could pass every +fixture and still order two nearly tied records the other way. + +**retrieval-policy/ranking-near-tie** is a new case in 1.12's `retrieval-policy` family (same files, same comparison): +two pairs of records whose scores differ by one ulp, the higher score on the larger experience ID. The order (lesson 2, +lesson 1, lesson 4, lesson 3) is compared exactly, as every `ranked` list is. A port reproduces it only with .NET's +arithmetic: + +- **Recency and confidence decay are `2^(-age / halfLife)` as .NET's `Math.Pow` gives it**, `age` and `halfLife` each + .NET's `TimeSpan.TotalSeconds` (whole ticks as a double, divided by 10,000,000), the quotient negated and rounded to a + double before the power. .NET's `Math.Pow` is the platform C library's `pow` (glibc on the Linux the fixtures are + checked on), which is not correctly rounded on every input. Lesson 1's recency, `2^(-803580 / 2592000)`, is + `0.8066288166995434`; an implementation 2 ulps off (Go's `math.Pow` gives `0.8066288166995436`) ties lesson 1 with + lesson 2, and the tie-break then puts lesson 1 first. The case's inputs are far from rounding midpoints, so glibc's + and macOS's `pow` and a correctly rounded `2^x` all give them exactly. +- **Each component's contribution is its value times its weight, rounded to a double, then added** to the running + total, which starts at 0 and takes the five in component order (relevance, confidence, recency, status, + environment). A compiler that fuses the product into the sum (a fused multiply-add) ties lessons 3 and 4. + +**What 1.18 does not pin.** Only the fixture's inputs. C libraries' `pow` is not correctly rounded everywhere: glibc's +`pow(2, x)` was measured 1 ulp from the correctly rounded value on about 0.07% of `x` in (-60, 0) (about 1 input in +1,500), macOS's on about 0.12%. A port that computes `2^x` correctly rounded (the Go port does) agrees with .NET except +on those rare inputs, and a port that calls its platform's `pow` (the Python port does) agrees with .NET on the same +platform. Two near-tied records whose recency or decay factor falls on such an input may still be ordered differently +by different runtimes. + +## What 1.18 adds + +- **sql:** [`lifecycle-1.18.sql`](sql/lifecycle-1.18.sql), one statement, `lock-supersession-rows-scoped`. +- **retrieval-policy/ranking-near-tie**: one new case in an existing family, generated through .NET's + `ExperienceRetrievalService` like the others. diff --git a/contract/fixtures/retrieval-policy/ranking-near-tie/candidates.json b/contract/fixtures/retrieval-policy/ranking-near-tie/candidates.json new file mode 100644 index 00000000..ba0c442c --- /dev/null +++ b/contract/fixtures/retrieval-policy/ranking-near-tie/candidates.json @@ -0,0 +1,330 @@ +[ + { + "record": { + "columns": { + "experience_id": "000000c0-0000-4000-8000-000000000001", + "source_run_id": "000000c0-0000-4000-8000-000000000101", + "tenant_id": "contract-tenant", + "application_id": "contract-app", + "project_id": "contract-project", + "team_id": null, + "agent_id": "refund-agent", + "user_id": null, + "task_id": "rotate-keys", + "status": "Validated", + "reuse_confidence": 0.7660227099802732, + "supporting_validations": 1, + "contradictions": 0, + "revision": 1, + "created_at": "2026-01-01T09:00:00+00:00", + "updated_at": "2026-02-21T01:47:00+00:00", + "payload_version": 1 + }, + "payload": { + "taskSummary": "Rotate the signing keys", + "attempts": [], + "outcome": { + "status": "Verified", + "evidence": [ + { + "evidenceId": "000000c0-0000-4000-8000-000000000e01", + "verificationRoundId": "000000c0-0000-4000-8000-000000000d00", + "artifactRevision": "rev-1", + "checkId": "check", + "kind": "integration-test", + "result": "Pass", + "producer": "contract-ci", + "detail": null, + "capturedAt": "2026-02-20T09:00:00+00:00" + } + ], + "reason": null, + "evaluatedAt": "2026-02-20T09:00:00+00:00" + }, + "completionScore": 1, + "reflection": { + "reflectionId": "000000c0-0000-4000-8000-000000000201", + "experienceRunId": "000000c0-0000-4000-8000-000000000101", + "lesson": "Lesson 1: verified after 1 attempt.", + "successfulApproaches": [], + "failedApproaches": [], + "preconditions": [ + "Runtime version: runtime-1" + ], + "warnings": [], + "reuseGuidance": "Reuse only where the listed preconditions match.", + "evidenceIds": [ + "000000c0-0000-4000-8000-000000000e01" + ], + "verificationStatus": "Verified", + "completionScore": 1, + "verificationRuleVersion": "1.0.0", + "producer": "host-reflector/1.0", + "createdAt": "2026-02-20T09:00:00+00:00" + }, + "environment": { + "hostName": "contract-host", + "runtimeVersion": "runtime-1", + "operatingSystem": "linux", + "applicationVersion": "1.0.0", + "metadata": { + "region": "eu-west" + } + }, + "provenance": { + "source": "contract-fixture", + "sourceVersion": "1", + "recordedAt": "2026-02-20T09:00:00+00:00", + "correlationId": null + } + } + }, + "relevance": 1 + }, + { + "record": { + "columns": { + "experience_id": "000000c0-0000-4000-8000-000000000002", + "source_run_id": "000000c0-0000-4000-8000-000000000102", + "tenant_id": "contract-tenant", + "application_id": "contract-app", + "project_id": "contract-project", + "team_id": null, + "agent_id": "refund-agent", + "user_id": null, + "task_id": "rotate-keys", + "status": "Validated", + "reuse_confidence": 0.95, + "supporting_validations": 1, + "contradictions": 0, + "revision": 1, + "created_at": "2026-01-01T09:00:00+00:00", + "updated_at": "2026-01-31T09:00:00+00:00", + "payload_version": 1 + }, + "payload": { + "taskSummary": "Rotate the signing keys", + "attempts": [], + "outcome": { + "status": "Verified", + "evidence": [ + { + "evidenceId": "000000c0-0000-4000-8000-000000000e02", + "verificationRoundId": "000000c0-0000-4000-8000-000000000d00", + "artifactRevision": "rev-1", + "checkId": "check", + "kind": "integration-test", + "result": "Pass", + "producer": "contract-ci", + "detail": null, + "capturedAt": "2026-02-20T09:00:00+00:00" + } + ], + "reason": null, + "evaluatedAt": "2026-02-20T09:00:00+00:00" + }, + "completionScore": 1, + "reflection": { + "reflectionId": "000000c0-0000-4000-8000-000000000202", + "experienceRunId": "000000c0-0000-4000-8000-000000000102", + "lesson": "Lesson 2: verified after 1 attempt.", + "successfulApproaches": [], + "failedApproaches": [], + "preconditions": [ + "Runtime version: runtime-1" + ], + "warnings": [], + "reuseGuidance": "Reuse only where the listed preconditions match.", + "evidenceIds": [ + "000000c0-0000-4000-8000-000000000e02" + ], + "verificationStatus": "Verified", + "completionScore": 1, + "verificationRuleVersion": "1.0.0", + "producer": "host-reflector/1.0", + "createdAt": "2026-02-20T09:00:00+00:00" + }, + "environment": { + "hostName": "contract-host", + "runtimeVersion": "runtime-1", + "operatingSystem": "linux", + "applicationVersion": "1.0.0", + "metadata": { + "region": "eu-west" + } + }, + "provenance": { + "source": "contract-fixture", + "sourceVersion": "1", + "recordedAt": "2026-02-20T09:00:00+00:00", + "correlationId": null + } + } + }, + "relevance": 1 + }, + { + "record": { + "columns": { + "experience_id": "000000c0-0000-4000-8000-000000000003", + "source_run_id": "000000c0-0000-4000-8000-000000000103", + "tenant_id": "contract-tenant", + "application_id": "contract-app", + "project_id": "contract-project", + "team_id": null, + "agent_id": "refund-agent", + "user_id": null, + "task_id": "rotate-keys", + "status": "Validated", + "reuse_confidence": 0.5220950305647717, + "supporting_validations": 1, + "contradictions": 0, + "revision": 1, + "created_at": "2026-01-01T09:00:00+00:00", + "updated_at": "2026-02-19T03:19:00+00:00", + "payload_version": 1 + }, + "payload": { + "taskSummary": "Rotate the signing keys", + "attempts": [], + "outcome": { + "status": "Verified", + "evidence": [ + { + "evidenceId": "000000c0-0000-4000-8000-000000000e03", + "verificationRoundId": "000000c0-0000-4000-8000-000000000d00", + "artifactRevision": "rev-1", + "checkId": "check", + "kind": "integration-test", + "result": "Pass", + "producer": "contract-ci", + "detail": null, + "capturedAt": "2026-02-20T09:00:00+00:00" + } + ], + "reason": null, + "evaluatedAt": "2026-02-20T09:00:00+00:00" + }, + "completionScore": 1, + "reflection": { + "reflectionId": "000000c0-0000-4000-8000-000000000203", + "experienceRunId": "000000c0-0000-4000-8000-000000000103", + "lesson": "Lesson 3: verified after 1 attempt.", + "successfulApproaches": [], + "failedApproaches": [], + "preconditions": [ + "Runtime version: runtime-1" + ], + "warnings": [], + "reuseGuidance": "Reuse only where the listed preconditions match.", + "evidenceIds": [ + "000000c0-0000-4000-8000-000000000e03" + ], + "verificationStatus": "Verified", + "completionScore": 1, + "verificationRuleVersion": "1.0.0", + "producer": "host-reflector/1.0", + "createdAt": "2026-02-20T09:00:00+00:00" + }, + "environment": { + "hostName": "contract-host", + "runtimeVersion": "runtime-1", + "operatingSystem": "linux", + "applicationVersion": "1.0.0", + "metadata": { + "region": "eu-west" + } + }, + "provenance": { + "source": "contract-fixture", + "sourceVersion": "1", + "recordedAt": "2026-02-20T09:00:00+00:00", + "correlationId": null + } + } + }, + "relevance": 1 + }, + { + "record": { + "columns": { + "experience_id": "000000c0-0000-4000-8000-000000000004", + "source_run_id": "000000c0-0000-4000-8000-000000000104", + "tenant_id": "contract-tenant", + "application_id": "contract-app", + "project_id": "contract-project", + "team_id": null, + "agent_id": "refund-agent", + "user_id": null, + "task_id": "rotate-keys", + "status": "Validated", + "reuse_confidence": 0.6848993455402582, + "supporting_validations": 1, + "contradictions": 0, + "revision": 1, + "created_at": "2026-01-01T09:00:00+00:00", + "updated_at": "2026-01-31T09:00:00+00:00", + "payload_version": 1 + }, + "payload": { + "taskSummary": "Rotate the signing keys", + "attempts": [], + "outcome": { + "status": "Verified", + "evidence": [ + { + "evidenceId": "000000c0-0000-4000-8000-000000000e04", + "verificationRoundId": "000000c0-0000-4000-8000-000000000d00", + "artifactRevision": "rev-1", + "checkId": "check", + "kind": "integration-test", + "result": "Pass", + "producer": "contract-ci", + "detail": null, + "capturedAt": "2026-02-20T09:00:00+00:00" + } + ], + "reason": null, + "evaluatedAt": "2026-02-20T09:00:00+00:00" + }, + "completionScore": 1, + "reflection": { + "reflectionId": "000000c0-0000-4000-8000-000000000204", + "experienceRunId": "000000c0-0000-4000-8000-000000000104", + "lesson": "Lesson 4: verified after 1 attempt.", + "successfulApproaches": [], + "failedApproaches": [], + "preconditions": [ + "Runtime version: runtime-1" + ], + "warnings": [], + "reuseGuidance": "Reuse only where the listed preconditions match.", + "evidenceIds": [ + "000000c0-0000-4000-8000-000000000e04" + ], + "verificationStatus": "Verified", + "completionScore": 1, + "verificationRuleVersion": "1.0.0", + "producer": "host-reflector/1.0", + "createdAt": "2026-02-20T09:00:00+00:00" + }, + "environment": { + "hostName": "contract-host", + "runtimeVersion": "runtime-1", + "operatingSystem": "linux", + "applicationVersion": "1.0.0", + "metadata": { + "region": "eu-west" + } + }, + "provenance": { + "source": "contract-fixture", + "sourceVersion": "1", + "recordedAt": "2026-02-20T09:00:00+00:00", + "correlationId": null + } + } + }, + "relevance": 1 + } +] diff --git a/contract/fixtures/retrieval-policy/ranking-near-tie/expected.json b/contract/fixtures/retrieval-policy/ranking-near-tie/expected.json new file mode 100644 index 00000000..9127d6f2 --- /dev/null +++ b/contract/fixtures/retrieval-policy/ranking-near-tie/expected.json @@ -0,0 +1,157 @@ +{ + "outcome": "Completed", + "query": { + "minConfidence": 0.5, + "limit": 51, + "minimumMatchedTerms": 3, + "excludeModelAuthored": false + }, + "truncated": false, + "ranked": [ + { + "experienceId": "000000c0-0000-4000-8000-000000000002", + "score": 0.8374999999999998, + "components": [ + { + "kind": "Relevance", + "value": 1, + "weight": 0.35, + "undecayedValue": null + }, + { + "kind": "Confidence", + "value": 0.95, + "weight": 0.25, + "undecayedValue": null + }, + { + "kind": "Recency", + "value": 0.5, + "weight": 0.15, + "undecayedValue": null + }, + { + "kind": "Status", + "value": 0.5, + "weight": 0.15, + "undecayedValue": null + }, + { + "kind": "EnvironmentCompatibility", + "value": 1, + "weight": 0.1, + "undecayedValue": null + } + ] + }, + { + "experienceId": "000000c0-0000-4000-8000-000000000001", + "score": 0.8374999999999997, + "components": [ + { + "kind": "Relevance", + "value": 1, + "weight": 0.35, + "undecayedValue": null + }, + { + "kind": "Confidence", + "value": 0.7660227099802732, + "weight": 0.25, + "undecayedValue": null + }, + { + "kind": "Recency", + "value": 0.8066288166995434, + "weight": 0.15, + "undecayedValue": null + }, + { + "kind": "Status", + "value": 0.5, + "weight": 0.15, + "undecayedValue": null + }, + { + "kind": "EnvironmentCompatibility", + "value": 1, + "weight": 0.1, + "undecayedValue": null + } + ] + }, + { + "experienceId": "000000c0-0000-4000-8000-000000000004", + "score": 0.7712248363850644, + "components": [ + { + "kind": "Relevance", + "value": 1, + "weight": 0.35, + "undecayedValue": null + }, + { + "kind": "Confidence", + "value": 0.6848993455402582, + "weight": 0.25, + "undecayedValue": null + }, + { + "kind": "Recency", + "value": 0.5, + "weight": 0.15, + "undecayedValue": null + }, + { + "kind": "Status", + "value": 0.5, + "weight": 0.15, + "undecayedValue": null + }, + { + "kind": "EnvironmentCompatibility", + "value": 1, + "weight": 0.1, + "undecayedValue": null + } + ] + }, + { + "experienceId": "000000c0-0000-4000-8000-000000000003", + "score": 0.7712248363850643, + "components": [ + { + "kind": "Relevance", + "value": 1, + "weight": 0.35, + "undecayedValue": null + }, + { + "kind": "Confidence", + "value": 0.5220950305647717, + "weight": 0.25, + "undecayedValue": null + }, + { + "kind": "Recency", + "value": 0.7713405249591436, + "weight": 0.15, + "undecayedValue": null + }, + { + "kind": "Status", + "value": 0.5, + "weight": 0.15, + "undecayedValue": null + }, + { + "kind": "EnvironmentCompatibility", + "value": 1, + "weight": 0.1, + "undecayedValue": null + } + ] + } + ], + "excluded": [] +} diff --git a/contract/fixtures/retrieval-policy/ranking-near-tie/policy.json b/contract/fixtures/retrieval-policy/ranking-near-tie/policy.json new file mode 100644 index 00000000..7db71b66 --- /dev/null +++ b/contract/fixtures/retrieval-policy/ranking-near-tie/policy.json @@ -0,0 +1,26 @@ +{ + "description": "Near ties pinned to the bit: lesson 2 outranks lesson 1, and lesson 4 lesson 3, by one ulp of the score, against the experience-ID tie-break. A port reproduces the order only with .NET's arithmetic: recency 2^(-age / halfLife) as .NET's Math.Pow gives it for these inputs, each component's value times its weight rounded before it is added, and the five added in component order (relevance, confidence, recency, status, environment) starting from 0.", + "scope": { + "tenantId": "contract-tenant", + "applicationId": "contract-app", + "projectId": "contract-project", + "teamId": null, + "agentId": "refund-agent", + "userId": null + }, + "taskText": "rotate the signing keys", + "now": "2026-03-02T09:00:00+00:00", + "preferredEnvironment": {}, + "excludeModelAuthored": false, + "policy": { + "timeoutTicks": 5000000, + "minimumConfidence": 0.5, + "maxAgeTicks": null, + "recencyHalfLifeTicks": 25920000000000, + "candidateLimit": 50, + "maxAbandonedSearches": 16, + "minimumMatchedTerms": 3, + "minimumVectorRelevance": null + }, + "confidenceDecay": null +} diff --git a/contract/lock.json b/contract/lock.json index 830e1e0c..65ba6dbf 100644 --- a/contract/lock.json +++ b/contract/lock.json @@ -1,5 +1,5 @@ { - "contractVersion": "1.17", + "contractVersion": "1.18", "files": { "README-1.1.md": "eb7c3a0ec370ec5370aae110a57701d1818d70348f9eef1d88e73442b05398aa", "README-1.10.md": "6192567b6e5a519a35efe1e3308b1f2a08415e14a12926bff80ef75e46dabced", @@ -10,6 +10,7 @@ "README-1.15.md": "2d304f3ef277dd43d9c3a6dd34e499c0c91cce06ddfc275c92d81a9089f2da56", "README-1.16.md": "1c1853010e66c72b6fcaeacdb42d86b188d0eb673b34a371508e5c6ff03224d9", "README-1.17.md": "9182fd79a719c3d24490cd41b462d5fcdd65ca83538e0d2b2b26c21f34535863", + "README-1.18.md": "a42e6c9a9b0e2c715ccceec0d4b86773be68aae5c1dbaa420dc1529ce9ad1902", "README-1.2.md": "71948ff812426e85ebe3cc849025db6b8e6a959eb34fd8e10bd4757ea4065d94", "README-1.3.md": "b63d0342726eca79d2c321b1dda1115c3f8d7de5bf94228f459dbc8c3372b8e3", "README-1.4.md": "6e0281d621ea903718126c6d6f6b00c565a459063e7f70d902fb054375200a75", @@ -1302,6 +1303,9 @@ "fixtures/retrieval-policy/model-authored-excluded/expected.json": "c2fecfc926e5dc4331c63923618ea057f5b7dde98133762a5704c39be1e233f9", "fixtures/retrieval-policy/model-authored-excluded/policy.json": "24c8f3779950d08d4ec269ddce28ce27aa9a541f0930529048e0f3a26ed1a043", "fixtures/retrieval-policy/policy-validation.json": "6e406d2e7f9bfbbf9071d1b51e9725a5509f495bf62f55e3d7bca03938768db8", + "fixtures/retrieval-policy/ranking-near-tie/candidates.json": "4c6caf8a95126644dac4b460ba239521bac48f99c40abfa2c2229d1dffe39f03", + "fixtures/retrieval-policy/ranking-near-tie/expected.json": "73521589c35e8c20568b418dfe6e3f5a30c233496f075e5b17fdeaf48d009b22", + "fixtures/retrieval-policy/ranking-near-tie/policy.json": "7e34752ff44de6792c449f7ca9e070d527a00713c8bb5c61d47a2b8bc57ba0a7", "fixtures/retrieval-policy/recency-half-life/candidates.json": "ba38c7922b2c370a7ecdfe84a246ef3069e0eb98deb8f3de3b10f60256bd11fc", "fixtures/retrieval-policy/recency-half-life/expected.json": "9ed3416496b9db37ded0baecaca9b6962420f0ec8a93b14d6da8cfd17ad746e1", "fixtures/retrieval-policy/recency-half-life/policy.json": "7c6b189597f182e67ea4704bafa7edb9d478dfbc1b2cd393b53c7378da3455c5", @@ -1434,6 +1438,7 @@ "sql/application-role-privileges-1.8.sql": "5ffd2a33b2e87d580e523d79ba36bb007446930e8fbd6124b6fca08dde092508", "sql/application-role-privileges.sql": "d6ba9aa62c4eae99f07be06afb1909002ba961796c53bfc94906352745eae789", "sql/declare-bounds.sql": "dfcbfa701cdbf18e2da9ab04bf8cd91cb143116bafd8d4f9c3f534ef90e1a2b7", + "sql/lifecycle-1.18.sql": "d89d962bb7c44f84fb3861ce5c6799053faef9f2b5819abe89d472d190d10296", "sql/lifecycle.sql": "a480263981ad46a3c98154420e728f1e994b345ea947df816dad8f841e9fdafe", "sql/list-experience-lessons.sql": "8bd01c5c24262c257a71c08374730a803a831cca6090acd0cebb06554633fdc7", "sql/query-holdout-outcomes.sql": "7c881fbe1a1f302de79e6786ef18289396ddeeaa372c615af401030f41cca270", diff --git a/contract/sql/lifecycle-1.18.sql b/contract/sql/lifecycle-1.18.sql new file mode 100644 index 00000000..0d11d494 --- /dev/null +++ b/contract/sql/lifecycle-1.18.sql @@ -0,0 +1,9 @@ +-- contract 1.18: the supersession lock scoped to the request, exactly as the .NET PostgreSQL store +-- (PostgresExperienceRecordStore) runs it, generated from its source and checked against it on every build. In +-- lifecycle.sql's COMMIT step 3, run "lock-supersession-rows-scoped" in place of 1.16's "lock-supersession-rows", +-- with the same @lock_ids (the record and the replacement, a uuid[]) and the request scope (@tenant_id ... +-- @user_id, NULL for an unset optional field). It locks only the rows of the request scope: a replacement in +-- another scope is not locked, and "check-supersession" then reads it as not found, exactly as before +-- (ReplacementNotAllowed). Every other statement of lifecycle.sql is unchanged. +-- statement: lock-supersession-rows-scoped +SELECT experience_id FROM agent_experience.experience_records WHERE experience_id = ANY(@lock_ids) AND tenant_id = @tenant_id AND application_id = @application_id AND project_id = @project_id AND team_id IS NOT DISTINCT FROM @team_id AND agent_id IS NOT DISTINCT FROM @agent_id AND user_id IS NOT DISTINCT FROM @user_id ORDER BY experience_id FOR UPDATE diff --git a/docs/guide/lifecycle.md b/docs/guide/lifecycle.md index 40fcf7df..bd19904a 100644 --- a/docs/guide/lifecycle.md +++ b/docs/guide/lifecycle.md @@ -119,7 +119,10 @@ exactly as for the store's other operations, and the scope predicate is applied unstorable however the write arrives. Which replacements are *acceptable* stays Core's decision; the store answers the parts only a scoped query can (see below) and persists what Core decided. - **The supersession gate is inside the commit.** An event carrying `ReplacementExperienceId` is checked *within the - commit transaction*, after both record rows are locked `FOR UPDATE` in a deterministic order: the replacement must + commit transaction*, after both record rows are locked `FOR UPDATE` in a deterministic order (only rows in the + caller's exact scope: a replacement ID naming another tenant's or another scope's record takes no lock on it, so a + supersession can neither hold nor wait on a row outside its scope, with or without row-level security; contract + 1.18's `sql/lifecycle-1.18.sql`, which the Python and Go ports run too): the replacement must exist in exactly this scope, be eligible for reuse, and not already sit on a chain of `replacement_experience_id` links leading back to the record. Otherwise the commit is `ReplacementNotAllowed`, carrying the replacement's stored status, and nothing is written. Checking it anywhere else would not hold: two supersessions naming each diff --git a/go/README.md b/go/README.md index dd091353..2c853fca 100644 --- a/go/README.md +++ b/go/README.md @@ -4,8 +4,8 @@ A Go core of AgentExperience that implements the [language-neutral contract](../ learned by a Go program is stored in the format the .NET and Python libraries read, and the other way round, with a Microsoft Agent Framework for Go integration. -**Status: preview, contract 1.17, with a Microsoft Agent Framework integration (the `maf` module).** It passes every -fixture of contract 1.17 (and so every 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 1.10, 1.11, 1.12, 1.13, 1.14, 1.15 and 1.16 fixture). Every release of this repository tags each module at the +**Status: preview, contract 1.18, with a Microsoft Agent Framework integration (the `maf` module).** It passes every +fixture of contract 1.18 (and so every 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 1.10, 1.11, 1.12, 1.13, 1.14, 1.15, 1.16 and 1.17 fixture). Every release of this repository tags each module at the same version as the .NET packages (`v0.1.0-preview.N`), and the Go module proxy serves it from there. Like the .NET packages, it claims no production readiness, and its API may change between previews. It is Apache-2.0, as the [repository](../LICENSE) is. @@ -727,6 +727,12 @@ err := maf.UseAgentExperience(&config, maf.Options{ refuses (a timeout over a day, a candidate limit over 199, a confidence outside 0 to 1 or NaN, a blank domain key or domain, a zero or negative half-life in `HalfLives` or `DefaultHalfLife`), is refused, naming the .NET option. The zero `RetrievalPolicy` is .NET's `RetrievalPolicy.Default`, and ranks exactly as before 1.12. +- **Scores to the bit.** Recency and decay are `2^(-age / halfLife)` computed by the port's own correctly rounded + double-double `exp2` (Go's `math.Pow` is off by an ulp or more on about 30% of inputs), which agrees with .NET's + C-library `Math.Pow` except on rare inputs where that `pow` misrounds (about 1 in 1,500 measured on glibc). Each + component's value times its weight is rounded before it is added, never fused into the sum. Contract 1.18's + `retrieval-policy/ranking-near-tie` pins two near ties; two records whose scores differ in the last bit through a + misrounded `pow` input may still rank differently from .NET. - **Decay never excludes.** It only reorders: the confidence floor is the search's, on stored confidence. In the verbose block the `Confidence:` line shows the stored value and the applicability line the decayed one, as .NET's. - **Timing out.** Past `Timeout` the invocation gets no block, its inject outcome is `RetrievalTimedOut` (retrieve: @@ -902,7 +908,8 @@ Python agents on the same database. role, reads records back, runs the contract's text search, records injections (`RecordInjection`, contract 1.4) and holdout assignments (`RecordHoldoutAssignment`, contract 1.8), writes and searches embeddings (`WriteEmbedding` and `SearchVectors`, contract 1.6) and reads the lessons view's page (`ListLessonRecords`, contract 1.7) and the holdout's outcome rows (`QueryHoldoutOutcomes`, contract 1.9), - commits lifecycle events and reads their history (contract 1.16), and records reuse feedback (contract 1.17), every transaction first declaring its bounds. The contract's statement files are embedded and run as + commits lifecycle events and reads their history (contract 1.16; a supersession locks only rows of the request's + scope, `sql/lifecycle-1.18.sql`), and records reuse feedback (contract 1.17), every transaction first declaring its bounds. The contract's statement files are embedded and run as written (a test keeps the copies in `postgres/sql/` identical to `contract/sql/`); only their `@name` placeholders are numbered for the driver. The store needs a schema the .NET migrator applied, with row-level security on; it never creates or alters a schema object. The application role's privileges are the .NET privileges call's, or the @@ -991,7 +998,7 @@ Every rule is .NET's, and Go's standard library differs from .NET in places ## What the Go port does not do yet -The port covers contract 1.0 to 1.17: capture, finalize, store, text and semantic retrieval under .NET's retrieval +The port covers contract 1.0 to 1.18: capture, finalize, store, text and semantic retrieval under .NET's retrieval policy (expiry, confidence decay by domain, the timeout and abandoned-search cap), the injected block and the injection options (the receiving agent, the model-authored policy, the request resolver and the injection decision), injection recording, the lessons view and its HTML page, exporting measured lessons, the model-backed reflector and its @@ -1031,7 +1038,7 @@ In the core, `go test ./...` runs the unit tests and the conformance runner ([`c every fixture family from [`../contract/fixtures`](../contract/fixtures) and compares this package's output exactly: blocks and task text byte for byte, JSON by value, with a number written as an integer never equal to one written with a fraction or an exponent. It checks every contract file against `contract/lock.json`, and fails with a clear -message if the lock names a contract version this package does not support (1.0 to 1.17 are). The verifier and +message if the lock names a contract version this package does not support (1.0 to 1.18 are). The verifier and classifier tests port the cases of .NET's and Python's. In `postgres/`, `go test ./...` validates every record of the fixtures, and every record the core writes for a finalize fixture, against the contract's JSON Schema (there, because it needs a validator and the core module requires none). The diff --git a/go/block.go b/go/block.go index f012e283..1f8a2899 100644 --- a/go/block.go +++ b/go/block.go @@ -160,8 +160,10 @@ type RankingComponent struct { UndecayedValue *float64 } -// Contribution is the component's value times its weight. -func (c RankingComponent) Contribution() float64 { return c.Value * c.Weight } +// Contribution is the component's value times its weight, rounded to a float64. The explicit conversion keeps the +// compiler from fusing the product into a sum it is added to (an FMA, which Go allows on arm64 and others): ranking +// adds each contribution as .NET does, product rounded first (contract 1.18's ranking-near-tie). +func (c RankingComponent) Contribution() float64 { return float64(c.Value * c.Weight) } // RankedRecord is a record as retrieval ranked it: its score and the components it was computed from. type RankedRecord struct { diff --git a/go/contract_test.go b/go/contract_test.go index eb62f267..6276d026 100644 --- a/go/contract_test.go +++ b/go/contract_test.go @@ -47,15 +47,15 @@ func TestTheContractVersionIsOneThisPackageSupports(t *testing.T) { if version != ae.ContractVersion { t.Fatalf("the lock is contract %s, this package implements %s", version, ae.ContractVersion) } - if strings.Join(ae.SupportedContractVersions, ", ") != "1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 1.10, 1.11, 1.12, 1.13, 1.14, 1.15, 1.16, 1.17" { + if strings.Join(ae.SupportedContractVersions, ", ") != "1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 1.10, 1.11, 1.12, 1.13, 1.14, 1.15, 1.16, 1.17, 1.18" { t.Fatalf("supported versions: %v", ae.SupportedContractVersions) } } func TestAnUnsupportedContractVersionFailsWithAClearMessage(t *testing.T) { - for _, version := range []any{"2.0", "1.18", nil} { + for _, version := range []any{"2.0", "1.19", nil} { _, err := ct.CheckContractVersion(ae.NewObject(ae.Member{Key: "contractVersion", Value: version})) - if !errors.Is(err, ct.ErrUnsupportedContractVersion) || !strings.Contains(err.Error(), "supports only 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 1.10, 1.11, 1.12, 1.13, 1.14, 1.15, 1.16, 1.17") { + if !errors.Is(err, ct.ErrUnsupportedContractVersion) || !strings.Contains(err.Error(), "supports only 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 1.10, 1.11, 1.12, 1.13, 1.14, 1.15, 1.16, 1.17, 1.18") { t.Fatalf("version %v: %v", version, err) } } diff --git a/go/doc.go b/go/doc.go index 503cf7d9..6d5a05df 100644 --- a/go/doc.go +++ b/go/doc.go @@ -43,7 +43,7 @@ package agentexperience // ContractVersion is the contract version this package implements. -const ContractVersion = "1.17" +const ContractVersion = "1.18" // SupportedContractVersions are the contract versions this package passes: every fixture of each. -var SupportedContractVersions = []string{"1.0", "1.1", "1.2", "1.3", "1.4", "1.5", "1.6", "1.7", "1.8", "1.9", "1.10", "1.11", "1.12", "1.13", "1.14", "1.15", "1.16", "1.17"} +var SupportedContractVersions = []string{"1.0", "1.1", "1.2", "1.3", "1.4", "1.5", "1.6", "1.7", "1.8", "1.9", "1.10", "1.11", "1.12", "1.13", "1.14", "1.15", "1.16", "1.17", "1.18"} diff --git a/go/exp2.go b/go/exp2.go new file mode 100644 index 00000000..34358d63 --- /dev/null +++ b/go/exp2.go @@ -0,0 +1,107 @@ +package agentexperience + +import "math" + +// exp2 is 2^x correctly rounded (to nearest, ties to even), for ranking's recency and confidence decay. +// +// .NET's Math.Pow(2, x) is the platform C library's pow, which is not correctly rounded on every input: glibc's was +// measured 1 ulp off on about 0.07% of x in (-60, 0) (about 1 in 1,500), macOS's on about 0.12%. Go's math.Pow is +// further off (about 30% of inputs by an ulp or more; math.Pow(2, -803580.0/2592000) is 2 ulps above), and ranking +// orders records on the last bit of a score. A correctly rounded 2^x agrees with .NET's pow except on those rare +// inputs; contract 1.18's retrieval-policy/ranking-near-tie pins only its own inputs, which every one of them gives +// alike. Near-tied records whose recency or decay falls on a misrounded input may still rank differently from .NET. +// +// It splits x into an integer k and a fraction f in [-1/2, 1/2] (exactly), evaluates 2^f = e^(f ln 2) in double-double +// arithmetic (about 104 bits, so the one rounding at the end is correct unless 2^f lies within about 2^-100 of a +// midpoint between two doubles), and scales by 2^k. The scaling is exact, so the result is rounded once: a normal +// result is the double-double's rounding scaled by 2^k; a subnormal one is rounded once, from the double-double, to +// a whole multiple of 2^-1074. +func exp2(x float64) float64 { + switch { + case math.IsNaN(x): + return x + case x == 0: + return 1 + case x > 1024: + return math.Inf(1) + case x < -1080: + return 0 + } + k := math.Round(x) + f := x - k // exact: k is the integer nearest x + // t = f * ln 2, in double-double. + t := ddMulDouble(dd{hi: ln2Hi, lo: ln2Lo}, f) + // e^t by Horner's rule on its Taylor series, |t| <= 0.347: the 25th term is below 2^-120 of the sum. + sum := dd{hi: 1} + for n := 24; n >= 1; n-- { + sum = ddAddOne(ddDivInt(ddMul(t, sum), float64(n))) + } + if k > -1022 { + // 2^f is in [0.707, 1.415), so the result is at least 2^-1021.5: normal, and the scaling exact. + return math.Ldexp(sum.hi+sum.lo, int(k)) + } + // Possibly subnormal: count in units of 2^-1074. Scaling both halves by 2^(k+1074) >= 2^-6 is exact, and the + // count is below 2^53, so rounding it to a whole number is the result's one rounding. + units := dd{hi: math.Ldexp(sum.hi, int(k)+1074), lo: math.Ldexp(sum.lo, int(k)+1074)} + n := math.RoundToEven(units.hi) + // units.hi - n is exact. Only when units.hi is itself a midpoint can the low half move the count, and then its + // sign alone decides: |lo| is at most half an ulp of units.hi, never enough to reach another midpoint. + switch remainder := units.hi - n; { + case remainder == 0.5 && units.lo > 0: + n++ + case remainder == -0.5 && units.lo < 0: + n-- + } + return math.Ldexp(n, -1074) +} + +// ln 2 to 107 bits, as the sum of two doubles. +const ( + ln2Hi = 0x1.62e42fefa39efp-01 + ln2Lo = 0x1.abc9e3b39803fp-56 +) + +// dd is an unevaluated sum hi + lo with |lo| <= ulp(hi)/2. +type dd struct{ hi, lo float64 } + +// twoSum is a + b as a double and its exact rounding error. +func twoSum(a, b float64) dd { + s := a + b + bb := s - a + return dd{hi: s, lo: (a - (s - bb)) + (b - bb)} +} + +// quickTwoSum is twoSum for |a| >= |b|. +func quickTwoSum(a, b float64) dd { + s := a + b + return dd{hi: s, lo: b - (s - a)} +} + +// twoProd is a * b as a double and its exact rounding error. The conversion rounds the product where it is +// computed: without it, Go may fuse the product into an expression that uses p (ddDivInt's a.hi - p.hi), which then +// sees the exact product and counts its error twice. +func twoProd(a, b float64) dd { + p := float64(a * b) + return dd{hi: p, lo: math.FMA(a, b, -p)} +} + +func ddMul(a, b dd) dd { + p := twoProd(a.hi, b.hi) + return quickTwoSum(p.hi, p.lo+(a.hi*b.lo+a.lo*b.hi)) +} + +func ddMulDouble(a dd, b float64) dd { + p := twoProd(a.hi, b) + return quickTwoSum(p.hi, p.lo+a.lo*b) +} + +func ddDivInt(a dd, n float64) dd { + q := a.hi / n + p := twoProd(q, n) + return quickTwoSum(q, (((a.hi-p.hi)-p.lo)+a.lo)/n) +} + +func ddAddOne(a dd) dd { + s := twoSum(1, a.hi) + return quickTwoSum(s.hi, s.lo+a.lo) +} diff --git a/go/exp2_test.go b/go/exp2_test.go new file mode 100644 index 00000000..4cb6b5f8 --- /dev/null +++ b/go/exp2_test.go @@ -0,0 +1,174 @@ +package agentexperience + +// exp2 against 2^x correctly rounded, two ways: a fixed table, and a reference computed here with math/big at 320 bits +// over deterministic samples (reproducible: go test -run TestExp2AgainstABigFloatReference -exp2.samples=80000). + +import ( + "flag" + "math" + "math/big" + "math/rand/v2" + "sync" + "testing" +) + +var exp2Samples = flag.Int("exp2.samples", 8000, "how many inputs TestExp2AgainstABigFloatReference checks") + +// exp2Golden is 2^x correctly rounded, computed independently in Python (decimal, 70 to 120 significant digits, then +// float(Decimal), which rounds once, subnormals included): float((Decimal(x) * Decimal(2).ln()).exp()). +var exp2Golden = []struct{ x, want float64 }{ + {-0x1.3d76b54932710p-2, 0x1.9cfe73c778b00p-1}, // 2^-0.31002314814814813 = 0.8066288166995434 + {-0x1.7f8cb487042c0p-2, 0x1.8aed25318ed84p-1}, // 2^-0.3745601851851852 = 0.7713405249591436 + {-0x1.4740da740da74p-3, 0x1.ca51aafe3c769p-1}, // 2^-0.15979166666666667 = 0.8951543269488244 + {-0x1.0000000000000p-1, 0x1.6a09e667f3bcdp-1}, // 2^-0.5 = 0.7071067811865476 + {-0x1.0000000000000p+0, 0x1.0000000000000p-1}, // 2^-1.0 = 0.5 + {-0x1.19799812dea11p-40, 0x1.fffffffffe79dp-1}, // 2^-1e-12 = 0.9999999999993069 + {-0x1.501a7462c60e7p-1, 0x1.44d4e39abfeecp-1}, // 2^-0.6564518328259766 = 0.6344367147830474 + {-0x1.63775e53c31a1p+5, 0x1.7b2c6f71375d3p-45}, // 2^-44.433285383597884 = 4.209672338204587e-14 + {0x1.e000000000000p+1, 0x1.ae89f995ad3adp+3}, // 2^3.75 = 13.454342644059432 + {-0x1.f420000000000p+9, 0x1.ae89f995ad3adp-1001}, // 2^-1000.25 = 7.847780312860669e-302 + {-0x1.ec491682e977cp-2, 0x1.6ee70230b62e1p-1}, // 2^-0.48074755834601723 = 0.7166062054121448 + {-0x1.1d232bde4bfdbp-1, 0x1.5c091c85147c7p-1}, // 2^-0.5569089611872146 = 0.6797570145083619 + {-0x1.455eaf57abd5fp+1, 0x1.5fa99db06b3a6p-3}, // 2^-2.5419520547945207 = 0.1717102355445636 + {-0x1.3c188f77e66d5p+5, 0x1.670aaceadbad6p-40}, // 2^-39.51199239417989 = 1.2755723624392815e-12 + {-0x1.20b8935b332cdp-1, 0x1.5a59e7437c0eap-1}, // 2^-0.563908200152207 = 0.676467158305994 + {-0x1.8081fb4c14541p+4, 0x1.f4dc6475e43f9p-25}, // 2^-24.031733796296297 = 5.830788378438364e-08 + {-0x1.767696f1f7a80p+5, 0x1.2475ba9021135p-47}, // 2^-46.80790509259259 = 8.11739225768758e-15 + {-0x1.de4401ebecaeep+2, 0x1.70e74323f1895p-8}, // 2^-7.472900848765432 = 0.005629018680286664 + {-0x1.212937a48b59cp+5, 0x1.cf007d4151a2dp-37}, // 2^-36.14512566137566 = 1.3159305787040185e-11 + {-0x1.0d258a6dfc352p+7, 0x1.58193e4c0fe86p-135}, // 2^-134.57332175925927 = 3.0859830356812564e-41 + {-0x1.6540ddebbc99ap+5, 0x1.44c8203647e56p-45}, // 2^-44.65667328042328 = 3.605801674096303e-14 + {-0x1.694962fc962fdp+6, 0x1.99ac9a4fef420p-91}, // 2^-90.32166666666667 = 6.463519672443351e-28 + {-0x1.eddd14c03af2ap+3, 0x1.7b2fa95c3409ap-16}, // 2^-15.433237433862434 = 2.2601257442301204e-05 + {-0x1.5c4c30a5754adp+4, 0x1.2c897a62313fdp-22}, // 2^-21.768601080246913 = 2.7989691465626243e-07 + {-0x1.3525e0af1c02dp+7, 0x1.57f1125543866p-155}, // 2^-154.57397982804233 = 2.941680642062522e-47 + {-0x1.805eaad208e9cp-1, 0x1.3048decf3bdb3p-1}, // 2^-0.7507222539320142 = 0.5943059566470111 + {-0x1.1b21a40c81969p+1, 0x1.ba0a74a74c4fap-3}, // 2^-2.211964136225266 = 0.21584025512269261 + {-0x1.89ce5c5561d5ep+1, 0x1.e5855fec8c600p-4}, // 2^-3.0766101281075597 = 0.11853539915389177 + {-0x1.26ecc8d7de3e3p+0, 0x1.ccc8eb8917a21p-2}, // 2^-1.1520505454084222 = 0.44998519920582863 + {-0x1.fac0ca5c3ac83p-1, 0x1.01d32da0d48a1p-1}, // 2^-0.9897521245560629 = 0.5035642870737044 + {-0x1.8e85ed787120bp+6, 0x1.4aa958b270d07p-100}, // 2^-99.6307886904762 = 1.0189294446547113e-30 + {-0x1.632b4180e4db2p+5, 0x1.8511706efedb8p-45}, // 2^-44.396121031746034 = 4.3195238702578525e-14 + // Subnormal results, each rounded once, and large positive x. + {-0x1.ff40000000000p+9, 0x0.b504f333f9de6p-1022}, // 2^-1022.5 = 1.5733648139913585e-308 + {-0x1.0193333333333p+10, 0x0.00cfefc5e672ap-1022}, // 2^-1030.3 = 7.0598498970396e-311 + {-0x1.06b0000000000p+10, 0x0.00000009837f0p-1022}, // 2^-1050.75 = 4.928696e-317 + {-0x1.0b86666666666p+10, 0x0.000000000000fp-1022}, // 2^-1070.1 = 7.4e-323 + {-0x1.0c7999999999ap+10, 0x0.0000000000001p-1022}, // 2^-1073.9 = 5e-324 + {-0x1.0ca0000000000p+10, 0x0.0000000000001p-1022}, // 2^-1074.5 = 5e-324 + {-0x1.0c90000000000p+10, 0x0.0000000000001p-1022}, // 2^-1074.25 = 5e-324 + {-0x1.0cb0000000000p+10, 0x0.0000000000001p-1022}, // 2^-1074.75 = 5e-324 + {-0x1.0ce0000000000p+10, 0x0.0p+0}, // 2^-1075.5 = 0.0 + {0x1.f440000000000p+9, 0x1.6a09e667f3bcdp+1000}, // 2^1000.5 = 1.5153420044823246e+301 + {0x1.ffa0000000000p+9, 0x1.306fe0a31b715p+1023}, // 2^1023.25 = 1.068914733284906e+308 + {0x1.ffffdf3b645a2p+9, 0x1.ffa52de61c1b3p+1023}, // 2^1023.999 = 1.796447500688184e+308 + {0x1.0010000000000p+9, 0x1.172b83c7d517bp+512}, // 2^512.125 = 1.4621318225692964e+154 + {0x1.5e26666666666p+9, 0x1.3b2c47bff827ap+700}, // 2^700.3 = 6.4759869286261e+210 +} + +func TestExp2IsCorrectlyRounded(t *testing.T) { + for _, c := range exp2Golden { + if got := exp2(c.x); got != c.want { + t.Errorf("exp2(%v) = %v, want %v", c.x, got, c.want) + } + } + // Why ranking does not use math.Pow: on the near tie's input it is off (2 ulps above, with Go 1.26). Logged, not + // asserted, so a toolchain whose math.Pow improves does not fail the build. + if x := -803580.0 / 2592000.0; math.Pow(2, x) != exp2(x) { + t.Logf("math.Pow(2, %v) = %v, exp2 = %v", x, math.Pow(2, x), exp2(x)) + } +} + +func TestExp2Edges(t *testing.T) { + for _, c := range []struct{ x, want float64 }{ + {0, 1}, + {math.Copysign(0, -1), 1}, + {1, 2}, + {-1022, 0x1p-1022}, + {-1074, math.SmallestNonzeroFloat64}, + {-1075, 0}, // exactly halfway between 0 and 2^-1074: ties to even + {math.Nextafter(-1075, 0), math.SmallestNonzeroFloat64}, + {math.Nextafter(-1075, math.Inf(-1)), 0}, + {-1080, 0}, + {-1100, 0}, + {math.Inf(-1), 0}, + {1023, 0x1p1023}, + {1024, math.Inf(1)}, + {1025, math.Inf(1)}, + {math.Inf(1), math.Inf(1)}, + } { + if got := exp2(c.x); got != c.want { + t.Errorf("exp2(%v) = %v, want %v", c.x, got, c.want) + } + } + if !math.IsNaN(exp2(math.NaN())) { + t.Error("exp2(NaN) is not NaN") + } +} + +func TestExp2AgainstABigFloatReference(t *testing.T) { + random := rand.New(rand.NewPCG(28, 1)) + halfLives := []float64{604800, 1209600, 2592000, 7776000, 31536000} + for i := range *exp2Samples { + var x float64 + switch i % 6 { + case 0: // decay within a few half-lives + x = -random.Float64() + case 1: + x = -60 * random.Float64() + case 2: // an age in whole seconds over a half-life, as ranking divides them + x = -float64(random.IntN(1_000_000_000)+1) / halfLives[random.IntN(len(halfLives))] + case 3: // subnormal results + x = -1022 - 58*random.Float64() + case 4: // normal results down to the smallest + x = -1022 * random.Float64() + default: // large results + x = 1024 * random.Float64() + } + if got, want := exp2(x), exp2Reference(x); got != want { + t.Fatalf("exp2(%v) = %v, want %v", x, got, want) + } + } +} + +// exp2Reference is 2^x computed with math/big at 320 bits, by a different route from exp2's: e^(x ln 2) with the +// argument halved until it is tiny, a Taylor series, then squared back; big.Float's Float64 rounds the result once to +// the nearest double (ties to even), subnormals included. +func exp2Reference(x float64) float64 { + const prec = 320 + y := new(big.Float).SetPrec(prec).SetFloat64(x) + y.Mul(y, bigLn2Once()) + halvings := max(0, y.MantExp(nil)+8) + z := new(big.Float).SetPrec(prec).SetMantExp(y, -halvings) + sum := new(big.Float).SetPrec(prec).SetInt64(1) + term := new(big.Float).SetPrec(prec).SetInt64(1) + for n := int64(1); n < 200; n++ { + term.Mul(term, z) + term.Quo(term, new(big.Float).SetPrec(prec).SetInt64(n)) + sum.Add(sum, term) + if term.Sign() == 0 || term.MantExp(nil) < -prec-8 { + break + } + } + for range halvings { + sum.Mul(sum, sum) + } + result, _ := sum.Float64() + return result +} + +// bigLn2Once is bigLn2(320), computed once. +var bigLn2Once = sync.OnceValue(func() *big.Float { return bigLn2(320) }) + +// bigLn2 is ln 2 to prec bits: the sum of 1/(k 2^k) for k >= 1. +func bigLn2(prec uint) *big.Float { + work := prec + 16 + one := new(big.Float).SetPrec(work).SetInt64(1) + sum := new(big.Float).SetPrec(work) + for k := 1; k <= int(work); k++ { + denominator := new(big.Int).Lsh(big.NewInt(int64(k)), uint(k)) + term := new(big.Float).SetPrec(work).SetInt(denominator) + sum.Add(sum, term.Quo(one, term)) + } + return sum +} diff --git a/go/postgres/lifecycle.go b/go/postgres/lifecycle.go index 9409d25f..e417347b 100644 --- a/go/postgres/lifecycle.go +++ b/go/postgres/lifecycle.go @@ -24,10 +24,14 @@ import ( // LifecycleSQL is contract 1.16's: the lifecycle statements, each starting at its "-- statement: " line. const LifecycleSQL = "lifecycle.sql" +// LifecycleScopedLockSQL is contract 1.18's: the supersession lock scoped to the request, which a commit runs in place +// of LifecycleSQL's unscoped "lock-supersession-rows". +const LifecycleScopedLockSQL = "lifecycle-1.18.sql" + // The lifecycle statements a commit, a history, a supersession check and a read run. var lifecycleStatementNames = []string{"read-record", "sealed-record", "savepoint", "rollback-to-savepoint", "release-savepoint", "insert-evidence", "recorded-only-key-taken", "evidence-id-taken", "select-evidence", - "insert-event", "select-event", "lock-supersession-rows", "check-supersession", "update-projection", + "insert-event", "select-event", "check-supersession", "update-projection", "update-projection-with-confidence", "update-projection-recorded-only", "select-revision-and-status", "lock-revision-and-status", "read-history"} @@ -44,16 +48,24 @@ const ( ) func loadLifecycleStatements() (map[string]Statement, error) { - statements, err := ContractStatements(LifecycleSQL) - if err != nil { - return nil, err - } - loaded := make(map[string]Statement, len(lifecycleStatementNames)) - for _, name := range lifecycleStatementNames { - if statements[name] == "" { - return nil, fmt.Errorf("%s has no %q statement", LifecycleSQL, name) + loaded := make(map[string]Statement, len(lifecycleStatementNames)+1) + for _, file := range []struct { + name string + statements []string + }{ + {LifecycleSQL, lifecycleStatementNames}, + {LifecycleScopedLockSQL, []string{"lock-supersession-rows-scoped"}}, + } { + statements, err := ContractStatements(file.name) + if err != nil { + return nil, err + } + for _, name := range file.statements { + if statements[name] == "" { + return nil, fmt.Errorf("%s has no %q statement", file.name, name) + } + loaded[name] = ToDriverPlaceholders(statements[name]) } - loaded[name] = ToDriverPlaceholders(statements[name]) } return loaded, nil } @@ -796,10 +808,12 @@ func (c *commitState) eventValues(confidence *ae.ConfidenceUpdate) (map[string]a } // supersessionGate locks both rows and re-decides the replacement inside the commit: a refusal, or nil to proceed. +// Only rows of the request scope are locked (contract 1.18), so a replacement naming another tenant's record takes no +// lock on it; the check then reads it as not found. func (c *commitState) supersessionGate(tx pgx.Tx, replacementID ae.UUID) (*ae.LifecycleCommitResult, error) { s, ctx := c.store, c.ctx - if _, err := s.run(ctx, tx, "lock-supersession-rows", map[string]any{ - "lock_ids": []pgtype.UUID{uuidValue(c.event.ExperienceRecordID), uuidValue(replacementID)}}); err != nil { + if _, err := s.run(ctx, tx, "lock-supersession-rows-scoped", scopeValues(c.scope, map[string]any{ + "lock_ids": []pgtype.UUID{uuidValue(c.event.ExperienceRecordID), uuidValue(replacementID)}})); err != nil { return nil, err } check, err := s.readSupersession(ctx, tx, c.scope, c.event.ExperienceRecordID, replacementID) diff --git a/go/postgres/pgtest/lifecycle_review_postgres_test.go b/go/postgres/pgtest/lifecycle_review_postgres_test.go index 2b2f332a..0d31c15e 100644 --- a/go/postgres/pgtest/lifecycle_review_postgres_test.go +++ b/go/postgres/pgtest/lifecycle_review_postgres_test.go @@ -12,6 +12,8 @@ import ( "time" "github.com/jackc/pgx/v5" + "github.com/jackc/pgx/v5/pgconn" + "github.com/jackc/pgx/v5/pgxpool" ae "github.com/fabbrik/AgentExperience.NET/go" ct "github.com/fabbrik/AgentExperience.NET/go/internal/contracttest" @@ -272,3 +274,145 @@ func TestACommitCancelledWhileItWaitsWritesNothing(t *testing.T) { t.Fatalf("revision %d supporting %d", revision, supporting) } } + +// Contract 1.18's scoped supersession lock. Tenant B's store runs as the table owner, whom row-level security does not +// filter, so only the statement's own scope predicate keeps it off tenant A's row. Another transaction holds A's row +// FOR UPDATE throughout, and B's connections give up on any lock they would wait for (lock_timeout): a lock on A's row +// would fail with 55P03 instead of answering. No clock is read and nothing sleeps. +func TestASupersessionNamingAnotherTenantsRecordTakesNoLockOnIt(t *testing.T) { + admin := requireServer(t) + ctx := context.Background() + database := createDatabase(t, admin, "lifecycle_scoped_lock") + victim := seededRecord(t, newStore(t, admin, database)) + + storeB := impatientStore(t, admin.Owner(database), "10000") + recordB := victim + recordB.ExperienceID = ae.UUID{0: 0x72, 6: 0x40, 8: 0x80, 15: 1} + recordB.SourceRunID = ae.UUID{0: 0x72, 6: 0x40, 8: 0x80, 15: 2} + recordB.Scope.TenantID = "other-tenant" + if err := storeB.InsertRecord(ctx, ae.BoundsOf(recordB.Scope), &recordB); err != nil { + t.Fatal(err) + } + + owner := ownerConn(t, admin, database) + holder, err := owner.Begin(ctx) + if err != nil { + t.Fatal(err) + } + defer func() { _ = holder.Rollback(context.Background()) }() + var locked [16]byte + if err := holder.QueryRow(ctx, "SELECT experience_id FROM agent_experience.experience_records "+ + "WHERE experience_id = $1 FOR UPDATE", [16]byte(victim.ExperienceID)).Scan(&locked); err != nil || + ae.UUID(locked) != victim.ExperienceID { + t.Fatalf("%v %v", locked, err) + } + + prior := recordB.Status + replacement := victim.ExperienceID + result, err := storeB.CommitLifecycleEvent(ctx, ae.AuthorizationContext{PrincipalID: "tenant-b", + Bounds: ae.BoundsOf(recordB.Scope)}, recordB.Scope, ae.LifecycleEvent{ + EventID: ae.UUID{0: 0x72, 6: 0x40, 8: 0x80, 15: 3}, ExperienceRecordID: recordB.ExperienceID, + PriorStatus: &prior, CurrentStatus: ae.StatusSuperseded, Reason: "superseded by another tenant's record", + Producer: "scoped-lock/1.0", OccurredAt: recordB.CreatedAt, ExpectedRevision: recordB.Revision, + ReplacementExperienceID: &replacement}) + // The outcome is unchanged: another tenant's record reads as not found, and nothing is written. + if err != nil || result.Outcome != ae.StoreReplacementNotAllowed || result.CurrentStatus != nil { + t.Fatalf("%+v %v", result, err) + } + if err := holder.Rollback(ctx); err != nil { + t.Fatal(err) + } + if countRows(t, owner, "agent_experience.lifecycle_events", recordB.ExperienceID) != 0 { + t.Fatal("an event was written") + } +} + +// impatientStore is a store on connString whose connections give up on any lock they wait for longer than timeout +// milliseconds (lock_timeout, SQLSTATE 55P03). +func impatientStore(t *testing.T, connString string, timeout string) *postgres.Store { + t.Helper() + config, err := pgxpool.ParseConfig(connString) + if err != nil { + t.Fatal(err) + } + config.ConnConfig.RuntimeParams["lock_timeout"] = timeout + pool, err := pgxpool.NewWithConfig(context.Background(), config) + if err != nil { + t.Fatal(err) + } + t.Cleanup(pool.Close) + store, err := postgres.NewStore(pool) + if err != nil { + t.Fatal(err) + } + return store +} + +// The other side of contract 1.18's scoped lock: it still locks the caller's own replacement, whether the scope's +// optional fields are set or NULL (IS NOT DISTINCT FROM). The holder takes FOR NO KEY UPDATE, which conflicts with the +// supersession lock's FOR UPDATE but not with a foreign key's key-share lock, so only that lock can wait on it; +// lock_timeout ends the wait with 55P03. Nothing sleeps and no duration is asserted. +func TestHoldingTheInScopeReplacementsRowMakesTheSupersessionWait(t *testing.T) { + admin := requireServer(t) + for _, optionalFieldsSet := range []bool{false, true} { + name := "null-optional-scope" + if optionalFieldsSet { + name = "full-scope" + } + t.Run(name, func(t *testing.T) { + ctx := context.Background() + database := createDatabase(t, admin, "lifecycle_scoped_lock_waits") + store := newStore(t, admin, database) + seeded := seededRecord(t, store) + scope := seeded.Scope + scope.TenantID = "scoped-lock-tenant" + scope.TeamID, scope.AgentID, scope.UserID = nil, nil, nil + if optionalFieldsSet { + team, agent, user := "team-1", "agent-1", "user-1" + scope.TeamID, scope.AgentID, scope.UserID = &team, &agent, &user + } + records := make([]ae.ExperienceRecord, 2) + for i := range records { + records[i] = seeded + records[i].ExperienceID = ae.UUID{0: 0x73, 6: 0x40, 8: 0x80, 15: byte(1 + i)} + records[i].SourceRunID = ae.UUID{0: 0x73, 6: 0x40, 8: 0x80, 15: byte(0x11 + i)} + records[i].Scope = scope + if err := store.InsertRecord(ctx, ae.BoundsOf(scope), &records[i]); err != nil { + t.Fatal(err) + } + } + record, replacement := records[0], records[1] + + owner := ownerConn(t, admin, database) + holder, err := owner.Begin(ctx) + if err != nil { + t.Fatal(err) + } + defer func() { _ = holder.Rollback(context.Background()) }() + if _, err := holder.Exec(ctx, "SELECT experience_id FROM agent_experience.experience_records "+ + "WHERE experience_id = $1 FOR NO KEY UPDATE", [16]byte(replacement.ExperienceID)); err != nil { + t.Fatal(err) + } + + impatient := impatientStore(t, admin.ConnString(database, applicationRole, rolePassword), "500") + prior := record.Status + replacementID := replacement.ExperienceID + result, err := impatient.CommitLifecycleEvent(ctx, ae.AuthorizationContext{PrincipalID: "scoped", + Bounds: ae.BoundsOf(scope)}, scope, ae.LifecycleEvent{ + EventID: ae.UUID{0: 0x73, 6: 0x40, 8: 0x80, 15: 0x20}, ExperienceRecordID: record.ExperienceID, + PriorStatus: &prior, CurrentStatus: ae.StatusSuperseded, Reason: "superseded", + Producer: "scoped-lock/1.0", OccurredAt: record.CreatedAt, ExpectedRevision: record.Revision, + ReplacementExperienceID: &replacementID}) + var pgErr *pgconn.PgError + if !errors.As(err, &pgErr) || pgErr.Code != "55P03" { + t.Fatalf("the supersession did not wait on the held replacement: %+v %v", result, err) + } + if err := holder.Rollback(ctx); err != nil { + t.Fatal(err) + } + if countRows(t, owner, "agent_experience.lifecycle_events", record.ExperienceID) != 0 { + t.Fatal("an event was written") + } + }) + } +} diff --git a/go/postgres/sql/lifecycle-1.18.sql b/go/postgres/sql/lifecycle-1.18.sql new file mode 100644 index 00000000..0d11d494 --- /dev/null +++ b/go/postgres/sql/lifecycle-1.18.sql @@ -0,0 +1,9 @@ +-- contract 1.18: the supersession lock scoped to the request, exactly as the .NET PostgreSQL store +-- (PostgresExperienceRecordStore) runs it, generated from its source and checked against it on every build. In +-- lifecycle.sql's COMMIT step 3, run "lock-supersession-rows-scoped" in place of 1.16's "lock-supersession-rows", +-- with the same @lock_ids (the record and the replacement, a uuid[]) and the request scope (@tenant_id ... +-- @user_id, NULL for an unset optional field). It locks only the rows of the request scope: a replacement in +-- another scope is not locked, and "check-supersession" then reads it as not found, exactly as before +-- (ReplacementNotAllowed). Every other statement of lifecycle.sql is unchanged. +-- statement: lock-supersession-rows-scoped +SELECT experience_id FROM agent_experience.experience_records WHERE experience_id = ANY(@lock_ids) AND tenant_id = @tenant_id AND application_id = @application_id AND project_id = @project_id AND team_id IS NOT DISTINCT FROM @team_id AND agent_id IS NOT DISTINCT FROM @agent_id AND user_id IS NOT DISTINCT FROM @user_id ORDER BY experience_id FOR UPDATE diff --git a/go/postgres/store.go b/go/postgres/store.go index 90d4173e..263952fe 100644 --- a/go/postgres/store.go +++ b/go/postgres/store.go @@ -63,7 +63,7 @@ const ( //go:embed sql/write-experience-embedding.sql sql/search-experience-vector.sql sql/list-experience-lessons.sql //go:embed sql/record-holdout-assignment.sql sql/query-holdout-outcomes.sql //go:embed sql/record-provider-holdout-assignment.sql sql/query-provider-holdout-outcomes.sql -//go:embed sql/lifecycle.sql sql/reuse-feedback.sql +//go:embed sql/lifecycle.sql sql/reuse-feedback.sql sql/lifecycle-1.18.sql var contractFiles embed.FS // ContractSQL is one of the contract's SQL files, as this package embeds it (DeclareBoundsSQL, SearchSQL, diff --git a/go/postgres/store_test.go b/go/postgres/store_test.go index 2888f94c..bbb70237 100644 --- a/go/postgres/store_test.go +++ b/go/postgres/store_test.go @@ -19,7 +19,7 @@ import ( func TestTheEmbeddedSQLIsTheContractSQL(t *testing.T) { for _, name := range []string{DeclareBoundsSQL, SearchSQL, RecordInjectionSQL, WriteEmbeddingSQL, SearchVectorSQL, ListLessonsSQL, RecordHoldoutSQL, QueryHoldoutOutcomesSQL, RecordProviderHoldoutSQL, - QueryProviderHoldoutOutcomesSQL, LifecycleSQL, ReuseFeedbackSQL} { + QueryProviderHoldoutOutcomesSQL, LifecycleSQL, ReuseFeedbackSQL, LifecycleScopedLockSQL} { embedded, err := ContractSQL(name) if err != nil { t.Fatal(err) diff --git a/go/retrieval_policy.go b/go/retrieval_policy.go index 1e8cd4d8..0563b216 100644 --- a/go/retrieval_policy.go +++ b/go/retrieval_policy.go @@ -288,13 +288,14 @@ func durationSeconds(d time.Duration) float64 { return DurationFromTicks(int64(d / 100)).TotalSeconds() } -// decayFactor is 2^(-age / halfLife), normalized; 1 when since is now or later. +// decayFactor is 2^(-age / halfLife), normalized; 1 when since is now or later. The power is exp2's (correctly +// rounded), not math.Pow's, which is further from .NET's Math.Pow (contract 1.18). func decayFactor(since, now Instant, halfLifeSeconds float64) float64 { age := now.Sub(since) if age.Ticks() <= 0 { return 1.0 } - return normalizeScore(math.Pow(2.0, -age.TotalSeconds()/halfLifeSeconds)) + return normalizeScore(exp2(-age.TotalSeconds() / halfLifeSeconds)) } // confidenceComponent is the stored confidence, decayed by the record's domain's half-life when one applies (then diff --git a/python/README.md b/python/README.md index edda09a1..4dec3a79 100644 --- a/python/README.md +++ b/python/README.md @@ -3,8 +3,8 @@ A pure-Python core of AgentExperience that implements the [language-neutral contract](https://github.com/fabbrik/AgentExperience.NET/blob/main/contract/README.md), so a lesson learned by a Python agent is stored in the format the .NET library reads, and the other way round. -**Status: alpha, contract 1.17, with a Microsoft Agent Framework integration (the `maf` extra).** It passes every -fixture of contract 1.17 (and so every 1.0 to 1.16 fixture). It is published to PyPI as `agenticexperience` (the import name stays `agentexperience`), with every +**Status: alpha, contract 1.18, with a Microsoft Agent Framework integration (the `maf` extra).** It passes every +fixture of contract 1.18 (and so every 1.0 to 1.17 fixture). It is published to PyPI as `agenticexperience` (the import name stays `agentexperience`), with every release of this repository and at the same version as the .NET packages: .NET `0.1.0-preview.N` is `agenticexperience` `0.1.0aN` (a PEP 440 alpha; while every release is an alpha, pip and uv install the newest one). Like the .NET packages, it claims no production readiness, and its API may change between alphas. @@ -844,7 +844,8 @@ which shares lessons with .NET agents on the same database. - **The error class.** `error_class` reduces a captured error to its short class. - **PostgreSQL** (the `postgres` extra, psycopg 3). `PostgresExperienceStore` writes a finalized record and its initial event as the application role, reads records back, runs the contract's text search, records injections - (`record_injection`, contract 1.4), and commits lifecycle events (`sql/lifecycle.sql`, contract 1.16); + (`record_injection`, contract 1.4), and commits lifecycle events (`sql/lifecycle.sql`, contract 1.16, with `sql/lifecycle-1.18.sql`'s supersession lock, + which locks only rows of the request's scope); `PostgresExperienceReuseFeedbackStore` is the reuse-feedback ledger (`sql/reuse-feedback.sql`, contract 1.17). It ships the contract's SQL files and runs them as written. `rank` applies the contract's safety checks, truncation, eligibility and ranking to the rows the search returns, under the retrieval policy (contract 1.12) when one is given. The store needs a schema the .NET @@ -973,7 +974,7 @@ print(render_block(ranked.ranked, BlockOptions(rendering="Compact")).text) ## What the Python port does not do yet -The port covers contract 1.0 to 1.17: capture, finalize, store, text and semantic retrieval under .NET's retrieval +The port covers contract 1.0 to 1.18: capture, finalize, store, text and semantic retrieval under .NET's retrieval policy (expiry, confidence decay by domain, the timeout and abandoned-search cap), the injected block and every injection option that needs no session (the receiving agent's gate, the model-authored policy, the request resolver and the host's decision), diff --git a/python/src/agentexperience/__init__.py b/python/src/agentexperience/__init__.py index 9c5b84c5..85c0fa83 100644 --- a/python/src/agentexperience/__init__.py +++ b/python/src/agentexperience/__init__.py @@ -1,5 +1,5 @@ """ -AgentExperience for Python: a pure-Python core that implements the AgentExperience.NET language-neutral contract 1.17. +AgentExperience for Python: a pure-Python core that implements the AgentExperience.NET language-neutral contract 1.18. Pre-release. It finalizes a captured run into a stored Experience Record, renders the injected Historical Reference block, derives the task text, classifies errors, and (with the ``postgres`` extra) writes records to and searches them @@ -227,9 +227,9 @@ except _metadata.PackageNotFoundError: # running from a source tree that was never installed __version__ = "0.0.0+unknown" -CONTRACT_VERSION = "1.17" -"""The contract version this package passes: every fixture of contract 1.17, and so every 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, -1.6, 1.7, 1.8, 1.9, 1.10, 1.11, 1.12, 1.13, 1.14, 1.15 and 1.16 fixture.""" +CONTRACT_VERSION = "1.18" +"""The contract version this package passes: every fixture of contract 1.18, and so every 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, +1.6, 1.7, 1.8, 1.9, 1.10, 1.11, 1.12, 1.13, 1.14, 1.15, 1.16 and 1.17 fixture.""" SUPPORTED_CONTRACT_VERSIONS: tuple[str, ...] = ( "1.0", @@ -250,6 +250,7 @@ "1.15", "1.16", "1.17", + "1.18", ) """Every contract version whose fixtures this package passes.""" diff --git a/python/src/agentexperience/postgres.py b/python/src/agentexperience/postgres.py index d117aec2..3e041cf2 100644 --- a/python/src/agentexperience/postgres.py +++ b/python/src/agentexperience/postgres.py @@ -203,6 +203,7 @@ def contract_statements(name: str) -> dict[str, str]: RECORD_PROVIDER_HOLDOUT_SQL = "record-provider-holdout-assignment.sql" QUERY_PROVIDER_HOLDOUT_OUTCOMES_SQL = "query-provider-holdout-outcomes.sql" LIFECYCLE_SQL = "lifecycle.sql" +LIFECYCLE_SCOPED_LOCK_SQL = "lifecycle-1.18.sql" REUSE_FEEDBACK_SQL = "reuse-feedback.sql" _UNIQUE_VIOLATION = "23505" @@ -294,9 +295,9 @@ def opener() -> psycopg.Connection[Any]: self._lessons_statements = { key: to_driver_placeholders(text) for key, text in contract_statements(LIST_LESSONS_SQL).items() } - self._lifecycle = { - key: to_driver_placeholders(text) for key, text in contract_statements(LIFECYCLE_SQL).items() - } + # Contract 1.18: lifecycle-1.18.sql's scoped supersession lock runs in place of lifecycle.sql's unscoped one. + lifecycle = contract_statements(LIFECYCLE_SQL) | contract_statements(LIFECYCLE_SCOPED_LOCK_SQL) + self._lifecycle = {key: to_driver_placeholders(text) for key, text in lifecycle.items()} self._clock = clock @contextmanager @@ -1124,7 +1125,12 @@ def run(self) -> LifecycleCommitResult: replacement = event.replacement_experience_id if replacement is not None: - self._execute("lock-supersession-rows", {"lock_ids": [event.experience_record_id, replacement]}) + # Contract 1.18: only the rows of the request scope are locked, so a replacement naming another + # tenant's record takes no lock on it (check-supersession then reads it as not found). + self._execute( + "lock-supersession-rows-scoped", + {"lock_ids": [event.experience_record_id, replacement], **self._scope_parameters}, + ) check = self._fetch( "check-supersession", { diff --git a/python/src/agentexperience/sql/lifecycle-1.18.sql b/python/src/agentexperience/sql/lifecycle-1.18.sql new file mode 100644 index 00000000..0d11d494 --- /dev/null +++ b/python/src/agentexperience/sql/lifecycle-1.18.sql @@ -0,0 +1,9 @@ +-- contract 1.18: the supersession lock scoped to the request, exactly as the .NET PostgreSQL store +-- (PostgresExperienceRecordStore) runs it, generated from its source and checked against it on every build. In +-- lifecycle.sql's COMMIT step 3, run "lock-supersession-rows-scoped" in place of 1.16's "lock-supersession-rows", +-- with the same @lock_ids (the record and the replacement, a uuid[]) and the request scope (@tenant_id ... +-- @user_id, NULL for an unset optional field). It locks only the rows of the request scope: a replacement in +-- another scope is not locked, and "check-supersession" then reads it as not found, exactly as before +-- (ReplacementNotAllowed). Every other statement of lifecycle.sql is unchanged. +-- statement: lock-supersession-rows-scoped +SELECT experience_id FROM agent_experience.experience_records WHERE experience_id = ANY(@lock_ids) AND tenant_id = @tenant_id AND application_id = @application_id AND project_id = @project_id AND team_id IS NOT DISTINCT FROM @team_id AND agent_id IS NOT DISTINCT FROM @agent_id AND user_id IS NOT DISTINCT FROM @user_id ORDER BY experience_id FOR UPDATE diff --git a/python/tests/test_contract.py b/python/tests/test_contract.py index fd6643ee..888e3ddd 100644 --- a/python/tests/test_contract.py +++ b/python/tests/test_contract.py @@ -113,12 +113,12 @@ def test_the_contract_version_is_one_this_package_supports() -> None: assert version == CONTRACT_VERSION -@pytest.mark.parametrize("version", ["2.0", "1.18", None]) +@pytest.mark.parametrize("version", ["2.0", "1.19", None]) def test_an_unsupported_contract_version_fails_with_a_clear_message(version: str | None) -> None: with pytest.raises( UnsupportedContractVersion, match=r"supports only 1\.0, 1\.1, 1\.2, 1\.3, 1\.4, 1\.5, 1\.6, 1\.7, 1\.8, 1\.9, 1\.10, 1\.11, 1\.12, 1\.13, " - r"1\.14, 1\.15, 1\.16, 1\.17", + r"1\.14, 1\.15, 1\.16, 1\.17, 1\.18", ): check_contract_version({"contractVersion": version}) @@ -153,6 +153,7 @@ def test_every_contract_file_matches_the_lock() -> None: "query-provider-holdout-outcomes.sql", "lifecycle.sql", "reuse-feedback.sql", + "lifecycle-1.18.sql", ], ) def test_the_packaged_sql_is_the_contract_sql(name: str) -> None: diff --git a/python/tests/test_lifecycle_postgres.py b/python/tests/test_lifecycle_postgres.py index f1da924b..8d19d5d2 100644 --- a/python/tests/test_lifecycle_postgres.py +++ b/python/tests/test_lifecycle_postgres.py @@ -241,6 +241,91 @@ def run(index: int) -> None: ] +def test_a_supersession_naming_another_tenants_record_takes_no_lock_on_it(pg_server: Server) -> None: + """ + Contract 1.18's scoped supersession lock. The store runs as the table owner, whom row-level security does not + filter, so only the statement's own scope predicate keeps it off tenant A's row. Tenant A's row is held FOR UPDATE + by another transaction throughout; tenant B's store gives up on any lock it would wait for (lock_timeout), so a lock + on A's row would fail with 55P03 instead of answering. No clock is read and nothing sleeps. + """ + _, victim, _, owner = _seeded(pg_server, "lifecycle_scoped_lock") + scope_b = replace(victim.scope, tenant_id="other-tenant") + record_b = replace(victim, experience_id=uuid.UUID(int=0x7200), source_run_id=uuid.UUID(int=0x7201), scope=scope_b) + PostgresExperienceStore(owner).insert_record(AuthorizationBounds.of(scope_b), record_b) + authorization_b = AuthorizationContext("tenant-b", AuthorizationBounds.of(scope_b)) + store_b = PostgresExperienceStore(owner + " options='-c lock_timeout=10000'") + event = LifecycleEvent( + event_id=uuid.UUID(int=0x7202), + experience_record_id=record_b.experience_id, + prior_status=record_b.status, + current_status="Superseded", + reason="superseded by another tenant's record", + producer="scoped-lock/1.0", + occurred_at=Instant.parse("2026-03-02T09:00:00+00:00"), + expected_revision=record_b.revision, + replacement_experience_id=victim.experience_id, + ) + with psycopg.connect(owner) as holder, holder.transaction(): + locked = holder.execute( + "SELECT experience_id FROM agent_experience.experience_records WHERE experience_id = %s FOR UPDATE", + (victim.experience_id,), + ).fetchone() + assert locked == (victim.experience_id,) + result = store_b.commit_lifecycle_event(authorization_b, scope_b, event) + # The outcome is unchanged: another tenant's record reads as not found, and nothing is written. + assert (result.outcome, result.current_status) == ("ReplacementNotAllowed", None) + assert _rows(owner, "lifecycle_events", ["event_id"], "event_id") == [] + + +@pytest.mark.parametrize("optional_fields_set", [False, True], ids=["null-optional-scope", "full-scope"]) +def test_holding_the_in_scope_replacements_row_makes_the_supersession_wait( + pg_server: Server, optional_fields_set: bool +) -> None: + """ + The other side of contract 1.18's scoped lock: it still locks the caller's own replacement, whether the scope's + optional fields are set or NULL (IS NOT DISTINCT FROM). The holder takes FOR NO KEY UPDATE, which conflicts with the + supersession lock's FOR UPDATE but not with a foreign key's key-share lock, so only that lock can wait on it; + lock_timeout ends the wait with 55P03. Nothing sleeps and no duration is asserted. + """ + conninfo, seeded, _, owner = _seeded(pg_server, "lifecycle_scoped_lock_waits") + scope = replace( + seeded.scope, + tenant_id="scoped-lock-tenant", + team_id="team-1" if optional_fields_set else None, + agent_id="agent-1" if optional_fields_set else None, + user_id="user-1" if optional_fields_set else None, + ) + record, replacement = ( + replace(seeded, experience_id=uuid.UUID(int=0x7300 + n), source_run_id=uuid.UUID(int=0x7310 + n), scope=scope) + for n in (1, 2) + ) + store = PostgresExperienceStore(conninfo) + for each in (record, replacement): + store.insert_record(AuthorizationBounds.of(scope), each) + event = LifecycleEvent( + event_id=uuid.UUID(int=0x7320), + experience_record_id=record.experience_id, + prior_status=record.status, + current_status="Superseded", + reason="superseded", + producer="scoped-lock/1.0", + occurred_at=Instant.parse("2026-03-02T09:00:00+00:00"), + expected_revision=record.revision, + replacement_experience_id=replacement.experience_id, + ) + impatient = PostgresExperienceStore(conninfo + " options='-c lock_timeout=500'") + with psycopg.connect(owner) as holder, holder.transaction(): + holder.execute( + "SELECT experience_id FROM agent_experience.experience_records WHERE experience_id = %s FOR NO KEY UPDATE", + (replacement.experience_id,), + ) + with pytest.raises(psycopg.errors.LockNotAvailable): + impatient.commit_lifecycle_event( + AuthorizationContext("scoped", AuthorizationBounds.of(scope)), scope, event + ) + assert _rows(owner, "lifecycle_events", ["event_id"], "event_id") == [] + + def test_a_sealed_record_is_not_found_for_commit_history_and_read(pg_server: Server) -> None: conninfo, record, authorization, owner = _seeded(pg_server, "lifecycle_sealed") # A sealed row as the .NET sealing writes one (payload_version 2, opaque payload), inserted as the owner. diff --git a/src/AgentExperience.Storage.Postgres/PostgresExperienceRecordStore.Lifecycle.cs b/src/AgentExperience.Storage.Postgres/PostgresExperienceRecordStore.Lifecycle.cs index 4b9d363c..6422b1c2 100644 --- a/src/AgentExperience.Storage.Postgres/PostgresExperienceRecordStore.Lifecycle.cs +++ b/src/AgentExperience.Storage.Postgres/PostgresExperienceRecordStore.Lifecycle.cs @@ -197,9 +197,12 @@ public sealed partial class PostgresExperienceRecordStore /// which is what makes the replacement checks below atomic with the write: a concurrent transition /// of the replacement either lands before this lock (and is therefore seen by the check) or blocks /// behind it (and is therefore decided against a record this commit has already moved). + /// Only rows in the caller's exact scope are locked: a replacement ID naming another scope's record + /// takes no lock on it (it reads as not found in the check that follows, exactly as before), so one + /// tenant cannot hold or wait on another tenant's row through a supersession. /// private const string LockSupersessionRowsSql = - $"SELECT experience_id FROM {Table} WHERE experience_id = ANY(@lock_ids) ORDER BY experience_id FOR UPDATE"; + $"SELECT experience_id FROM {Table} WHERE experience_id = ANY(@lock_ids) AND {ScopePredicate} ORDER BY experience_id FOR UPDATE"; /// public async Task CommitLifecycleEventAsync( @@ -452,6 +455,7 @@ public async Task CommitLifecycleEventAsync( { TypedValue = [experienceId, replacementId], }); + AddScopeParameters(locks.Parameters, scope); await locks.ExecuteNonQueryAsync(cancellationToken).ConfigureAwait(false); } diff --git a/tests/AgentExperience.Contract.Tests/ContractFiles.cs b/tests/AgentExperience.Contract.Tests/ContractFiles.cs index 17982faf..e6baebcb 100644 --- a/tests/AgentExperience.Contract.Tests/ContractFiles.cs +++ b/tests/AgentExperience.Contract.Tests/ContractFiles.cs @@ -19,7 +19,7 @@ internal static class ContractFiles /// what minor version N adds) bumps the minor version; a change to anything already published (a fixture, the /// schema, the SQL, a README) bumps the major version. /// - public const string ContractVersion = "1.17"; + public const string ContractVersion = "1.18"; /// /// The SHA-256 of contract/lock.json for every published version. This, not lock.json, is the authority: @@ -46,6 +46,7 @@ internal static class ContractFiles ["1.15"] = "641639cb6b332b592562f765dff8724b8f458ca67cada62e6c254c95aae6931d", ["1.16"] = "99ceae576bc00677df6623e4501f80a7cd5d5bf9dd25c2e88eb874c692c575d6", ["1.17"] = "4eb841ac7dbdc49cee467198ae3dcbadecb2d30a5a5751f2ce38a3d4b4d0efb8", + ["1.18"] = "32450bd033d750e7c51971bbf18b6fa45b8c28d9b762924b6460707f50013051", }; /// The environment variable that rewrites the fixtures instead of only comparing against them. diff --git a/tests/AgentExperience.Contract.Tests/LifecycleContractFixtureTests.cs b/tests/AgentExperience.Contract.Tests/LifecycleContractFixtureTests.cs index 6e52a398..88abf1a3 100644 --- a/tests/AgentExperience.Contract.Tests/LifecycleContractFixtureTests.cs +++ b/tests/AgentExperience.Contract.Tests/LifecycleContractFixtureTests.cs @@ -78,6 +78,7 @@ public void The_lifecycle_statements_are_the_dotnet_postgres_stores_own() { var drift = new List(); ContractFiles.Verify(LifecycleSqlContract.File, LifecycleSqlContract.Render(), drift); + ContractFiles.Verify(LifecycleSqlContract.ScopedFile, LifecycleSqlContract.RenderScoped(), drift); Relock(drift); ContractFiles.AssertNoDrift(drift); } diff --git a/tests/AgentExperience.Contract.Tests/LifecycleSqlContract.cs b/tests/AgentExperience.Contract.Tests/LifecycleSqlContract.cs index 267ccf09..c91e8b6c 100644 --- a/tests/AgentExperience.Contract.Tests/LifecycleSqlContract.cs +++ b/tests/AgentExperience.Contract.Tests/LifecycleSqlContract.cs @@ -53,7 +53,7 @@ internal static class LifecycleSqlContract ("select-evidence", Store("SelectEvidenceSql")), ("insert-event", Store("InsertEventSql")), ("select-event", Store("SelectEventSql")), - ("lock-supersession-rows", Store("LockSupersessionRowsSql")), + ("lock-supersession-rows", LockSupersessionRows116), ("check-supersession", Store("SupersessionCheckSql").Trim()), ("update-projection", Store("UpdateProjectionSql")), ("update-projection-with-confidence", Store("UpdateProjectionWithConfidenceSql")), @@ -63,6 +63,35 @@ internal static class LifecycleSqlContract ("read-history", Store("HistorySql")), ]; + /// + /// Contract 1.16's supersession lock, frozen as published: it names no scope, so it can lock another tenant's row. The + /// .NET store has since run 's lock-supersession-rows-scoped (contract 1.18), and a + /// port runs that one instead; this text stays only because a published file never changes. + /// + private const string LockSupersessionRows116 = + $"SELECT experience_id FROM {ExperienceRecordSql.Table} WHERE experience_id = ANY(@lock_ids) ORDER BY experience_id FOR UPDATE"; + + // ---- sql/lifecycle-1.18.sql ----------------------------------------------------------------------------------- + + /// Contract 1.18's file: the scoped supersession lock that replaces 1.16's lock-supersession-rows. + public const string ScopedFile = "sql/lifecycle-1.18.sql"; + + private const string ScopedHeader = """ + -- contract 1.18: the supersession lock scoped to the request, exactly as the .NET PostgreSQL store + -- (PostgresExperienceRecordStore) runs it, generated from its source and checked against it on every build. In + -- lifecycle.sql's COMMIT step 3, run "lock-supersession-rows-scoped" in place of 1.16's "lock-supersession-rows", + -- with the same @lock_ids (the record and the replacement, a uuid[]) and the request scope (@tenant_id ... + -- @user_id, NULL for an unset optional field). It locks only the rows of the request scope: a replacement in + -- another scope is not locked, and "check-supersession" then reads it as not found, exactly as before + -- (ReplacementNotAllowed). Every other statement of lifecycle.sql is unchanged. + """; + + /// The whole of sql/lifecycle-1.18.sql. + public static string RenderScoped() => + ScopedHeader.Replace("\r\n", "\n", StringComparison.Ordinal) + "\n" + + "-- statement: lock-supersession-rows-scoped\n" + + Store("LockSupersessionRowsSql").Replace("\r\n", "\n", StringComparison.Ordinal).TrimEnd() + "\n"; + /// /// The one statement that is not the .NET store's: a port holds no key store, so it cannot write a sealed record's /// events the way the .NET store in crypto-shredding mode does, and refuses to commit against one (contract 1.0: a diff --git a/tests/AgentExperience.Contract.Tests/RetrievalPolicyContract.cs b/tests/AgentExperience.Contract.Tests/RetrievalPolicyContract.cs index d21e4f77..b365e4a7 100644 --- a/tests/AgentExperience.Contract.Tests/RetrievalPolicyContract.cs +++ b/tests/AgentExperience.Contract.Tests/RetrievalPolicyContract.cs @@ -90,8 +90,35 @@ public static JsonObject CrossLanguageDecay() => WithDefault( ("hybrid-vector-floor-from-the-policy", HybridVectorFloor()), ("hybrid-per-channel-candidate-limit", HybridPerChannelCandidateLimit()), ("hybrid-floor-before-truncation", HybridFloorBeforeTruncation()), + ("ranking-near-tie", RankingNearTie()), ]; + /// + /// Contract 1.18: two pairs of records whose scores differ in the last bit of a double, each pair ranked against its + /// experience-ID tie-break. The inputs were chosen so the order catches the two ways a port's arithmetic drifts from + /// .NET's: lesson 1's recency is 2^(-803580 / 2592000), which .NET's + /// gives 2 ulps below Go's math.Pow, which would tie lesson 1 with lesson 2; and lessons 3 and 4 tie if a + /// compiler fuses each weighted term into the running sum (an FMA) instead of rounding the product first. The + /// recency values here are far from rounding midpoints, so glibc's and macOS's pow and a correctly rounded + /// 2^x all give them exactly. Only these inputs are pinned: pow implementations still disagree in the + /// last bit on rare other inputs. + /// + private static RetrievalPolicyCase RankingNearTie() => new( + "Near ties pinned to the bit: lesson 2 outranks lesson 1, and lesson 4 lesson 3, by one ulp of the score, against " + + "the experience-ID tie-break. A port reproduces the order only with .NET's arithmetic: recency 2^(-age / halfLife) " + + "as .NET's Math.Pow gives it for these inputs, each component's value times its weight rounded before it is added, and the five added in " + + "component order (relevance, confidence, recency, status, environment) starting from 0.", + [ + (Lesson(1, created: Now.AddDays(-60), updated: Now.AddSeconds(-803_580), confidence: 0.7660227099802732), 1), + (Lesson(2, created: Now.AddDays(-60), updated: Now.AddDays(-30), confidence: 0.95), 1), + (Lesson(3, created: Now.AddDays(-60), updated: Now.AddSeconds(-970_860), confidence: 0.5220950305647717), 1), + (Lesson(4, created: Now.AddDays(-60), updated: Now.AddDays(-30), confidence: 0.6848993455402582), 1), + ], + Now, + Default(), + ConfidenceDecay: null, + NoPreference); + private static RetrievalPolicyCase ModelAuthoredExcluded() => new( "Model-authored lessons excluded, under MaxAge 90 days. Each candidate is excluded by the first rule that applies, " + "in .NET's order: a model-authored Candidate is IneligibleStatus; a model-authored record, expired or not, is " + diff --git a/tests/AgentExperience.Contract.Tests/RetrievalPolicyContractFixtureTests.cs b/tests/AgentExperience.Contract.Tests/RetrievalPolicyContractFixtureTests.cs index fa7072d1..1f257d34 100644 --- a/tests/AgentExperience.Contract.Tests/RetrievalPolicyContractFixtureTests.cs +++ b/tests/AgentExperience.Contract.Tests/RetrievalPolicyContractFixtureTests.cs @@ -129,6 +129,12 @@ public void The_retrieval_policy_cases_cover_every_rule() Assert.False(runs["hybrid-floor-before-truncation"].Result.Truncated); Assert.Equal(2, runs["hybrid-floor-before-truncation"].Result.Records.Count); + // The near tie (contract 1.18): each pair one ulp apart, the higher score on the larger experience ID. + var nearTie = runs["ranking-near-tie"].Result.Records; + Assert.Equal([2, 1, 4, 3], nearTie.Select(ranked => (int)(ranked.Record.ExperienceId.ToByteArray(bigEndian: true)[15]))); + Assert.Equal(1, BitConverter.DoubleToInt64Bits(nearTie[0].Score) - BitConverter.DoubleToInt64Bits(nearTie[1].Score)); + Assert.Equal(1, BitConverter.DoubleToInt64Bits(nearTie[2].Score) - BitConverter.DoubleToInt64Bits(nearTie[3].Score)); + // The cross-language policy: an expiry and decay together. Assert.Contains(runs["max-age-and-decay"].Result.Excluded, entry => entry.Reason == RetrievalExclusionReason.Expired); Assert.Contains(runs["max-age-and-decay"].Result.Records, ranked => Confidence(ranked).UndecayedValue is not null); diff --git a/tests/AgentExperience.Storage.Postgres.Tests/PostgresSupersessionAndAppendOnlyTests.cs b/tests/AgentExperience.Storage.Postgres.Tests/PostgresSupersessionAndAppendOnlyTests.cs index 24b6a6f0..25bc9a0d 100644 --- a/tests/AgentExperience.Storage.Postgres.Tests/PostgresSupersessionAndAppendOnlyTests.cs +++ b/tests/AgentExperience.Storage.Postgres.Tests/PostgresSupersessionAndAppendOnlyTests.cs @@ -174,6 +174,104 @@ public async Task A_replacement_in_another_scope_is_refused_and_reveals_nothing( await AssertUnchangedAsync(auth, otherScope, foreign.ExperienceId, ExperienceStatus.Validated, 1, events: 1); } + [Fact] + public async Task A_supersession_naming_another_tenants_record_takes_no_lock_on_it() + { + var tenantA = NewTenant(); + var tenantB = NewTenant(); + var authA = Authorize(tenantA); + var authB = Authorize(tenantB); + var scopeA = Scope(tenantA); + var scopeB = Scope(tenantB); + var victim = await ValidatedAsync(authA, scopeA); + var record = await ValidatedAsync(authB, scopeB); + + // Tenant A's row is held FOR UPDATE by another transaction for the whole test. Tenant B's store gives up on + // any lock it would have to wait for (lock_timeout): a store that locked A's row would fail with 55P03 instead + // of answering, so the outcome alone says whether it waited. No clock is read and nothing sleeps. + await using var holder = await _fixture.RawDataSource.OpenConnectionAsync(); + await using var held = await HoldRowAsync(holder, victim.ExperienceId, "FOR UPDATE"); + await using var impatient = Impatient(lockTimeoutMilliseconds: 10_000); + var storeB = new PostgresExperienceRecordStore(impatient); + + // The store's own commit, which runs the lock, and the lifecycle service over it. + var committed = await storeB.CommitLifecycleEventAsync( + authB, + scopeB, + Event(record.ExperienceId, ExperienceStatus.Validated, ExperienceStatus.Superseded, 1, Guid.NewGuid(), replacement: victim.ExperienceId), + CancellationToken.None); + var result = await new ExperienceLifecycleService(storeB).CommitAsync( + authB, + Transition(scopeB, record.ExperienceId, ExperienceStatus.Validated, ExperienceStatus.Superseded, 1, victim.ExperienceId), + CancellationToken.None); + + // The outcome is unchanged: another tenant's record reads as not found. + Assert.Equal(ExperienceStoreOutcome.ReplacementNotAllowed, committed.Outcome); + Assert.Equal(LifecycleTransitionOutcome.ReplacementNotAllowed, result.Outcome); + await held.RollbackAsync(); + await AssertUnchangedAsync(authB, scopeB, record.ExperienceId, ExperienceStatus.Validated, 1, events: 1); + await AssertUnchangedAsync(authA, scopeA, victim.ExperienceId, ExperienceStatus.Validated, 1, events: 1); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task Holding_the_in_scope_replacements_row_makes_the_supersession_wait(bool optionalScopeFieldsSet) + { + // The other side of the scoped lock: it still locks the caller's own replacement, whether the scope's optional + // fields are set or NULL (IS NOT DISTINCT FROM). The holder takes FOR NO KEY UPDATE, which conflicts with the + // supersession lock's FOR UPDATE but not with the key-share lock a foreign key takes, so only that lock can wait + // on it. lock_timeout ends the wait with 55P03: nothing sleeps and no duration is asserted. + var tenant = NewTenant(); + var auth = Authorize(tenant); + var scope = optionalScopeFieldsSet ? new Scope(tenant, "app-1", "project-1", "team-1", "agent-1", "user-1") : Scope(tenant); + var record = await ValidatedAsync(auth, scope); + var replacement = await ValidatedAsync(auth, scope); + + await using var holder = await _fixture.RawDataSource.OpenConnectionAsync(); + await using var held = await HoldRowAsync(holder, replacement.ExperienceId, "FOR NO KEY UPDATE"); + await using var impatient = Impatient(lockTimeoutMilliseconds: 500); + var store = new PostgresExperienceRecordStore(impatient); + + var thrown = await Record.ExceptionAsync(() => store.CommitLifecycleEventAsync( + auth, + scope, + Event(record.ExperienceId, ExperienceStatus.Validated, ExperienceStatus.Superseded, 1, Guid.NewGuid(), replacement: replacement.ExperienceId), + CancellationToken.None)); + + Assert.NotNull(thrown); + var postgres = Assert.IsType(Flatten(thrown).OfType().FirstOrDefault(), exactMatch: false); + Assert.Equal(PostgresErrorCodes.LockNotAvailable, postgres.SqlState); + await held.RollbackAsync(); + await AssertUnchangedAsync(auth, scope, record.ExperienceId, ExperienceStatus.Validated, 1, events: 1); + } + + private static IEnumerable Flatten(Exception exception) + { + for (Exception? current = exception; current is not null; current = current.InnerException) + { + yield return current; + } + } + + /// Opens a transaction on that holds one record's row with . + private static async Task HoldRowAsync(NpgsqlConnection holder, Guid experienceId, string lockClause) + { + var held = await holder.BeginTransactionAsync(); + await using var lockRow = new NpgsqlCommand( + $"SELECT experience_id FROM agent_experience.experience_records WHERE experience_id = @id {lockClause}", holder, held); + lockRow.Parameters.Add(new NpgsqlParameter("id", experienceId)); + Assert.Equal(experienceId, (Guid)(await lockRow.ExecuteScalarAsync())!); + return held; + } + + /// The application role's data source, giving up on any lock it waits for longer than the timeout. + private NpgsqlDataSource Impatient(int lockTimeoutMilliseconds) => NpgsqlDataSource.Create( + new NpgsqlConnectionStringBuilder(_fixture.ConnectionString(PostgresFixture.StoreDatabase, PostgresFixture.ApplicationRoleName)) + { + Options = $"-c lock_timeout={lockTimeoutMilliseconds}", + }.ConnectionString); + [Theory] [InlineData(ExperienceStatus.Quarantined)] [InlineData(ExperienceStatus.Revoked)]