diff --git a/docs/api/intelligence.md b/docs/api/intelligence.md index 29f46248..e5e67d53 100644 --- a/docs/api/intelligence.md +++ b/docs/api/intelligence.md @@ -394,6 +394,151 @@ The product owns the private state lookup and atomic write. *** +### AgentProfileCandidateMeasurementExecutor + +Product-owned executor for exact baseline/candidate profile measurement. + +#### Properties + +##### executionRef + +> **executionRef**: `AgentProfileImprovementExecutionRef` + +#### Methods + +##### measure() + +> **measure**(`input`): `Promise`\<`AgentProfileImprovementRunReceipt`\> + +###### Parameters + +###### input + +`AgentProfileImprovementExperimentExecutionInput` & `object` + +###### Returns + +`Promise`\<`AgentProfileImprovementRunReceipt`\> + +*** + +### ProposeAuthoredAgentProfileImprovementOptions + +Measure a complete human-authored, imported, or compound profile candidate. +No optimizer runs and no optimizer receipt is fabricated. + +#### Properties + +##### runId + +> **runId**: `string` + +##### source + +> **source**: `object` + +##### profile + +> **profile**: `AgentProfile` + +##### stateDigest + +> **stateDigest**: [`AgentImprovementProfileStateDigest`](#agentimprovementprofilestatedigest) + +##### candidateProfile + +> **candidateProfile**: `AgentProfile` + +##### candidateLineage + +> **candidateLineage**: [`AuthoredAgentProfileCandidateLineage`](#authoredagentprofilecandidatelineage) + +##### diff? + +> `optional` **diff?**: [`AgentImprovementTargetProfileDiffOptions`](#agentimprovementtargetprofilediffoptions) + +Optional source/artifact metadata used on Runtime-derived profile diff steps. + +##### findings? + +> `optional` **findings?**: readonly `ProposalFinding`[] + +##### benchmark + +> **benchmark**: [`AgentProfileImprovementBenchmark`](#agentprofileimprovementbenchmark) + +##### executor + +> **executor**: [`AgentProfileCandidateMeasurementExecutor`](#agentprofilecandidatemeasurementexecutor) + +##### budgetUsd + +> **budgetUsd**: `number` + +One customer-approved maximum for the held-out paired measurement. + +##### developmentScenarios? + +> `optional` **developmentScenarios?**: readonly `CampaignScenarioIdentity`[] + +Optional identities used to prove authored/imported development work is held out. + +##### maxConcurrency? + +> `optional` **maxConcurrency?**: `number` + +##### signal? + +> `optional` **signal?**: `AbortSignal` + +##### candidate? + +> `optional` **candidate?**: `object` + +##### metadata? + +> `optional` **metadata?**: `object` + +###### Index Signature + +\[`key`: `string`\]: `AgentCandidateJsonValue` + +##### now? + +> `optional` **now?**: () => `Date` + +###### Returns + +`Date` + +*** + +### ProposeAuthoredAgentProfileImprovementResult + +#### Properties + +##### candidateProfile + +> **candidateProfile**: `AgentProfile` + +##### candidateLineage + +> **candidateLineage**: `AgentCandidateLineage` + +##### experiment + +> **experiment**: `AgentProfileImprovementExperiment` + +##### measurements + +> **measurements**: `AgentProfileImprovementMeasurement`[] + +##### proposal + +> **proposal**: `AgentImprovementProposal` + +*** + ### CredentialRef A named secret a binding requires — declared, never carried. @@ -1507,7 +1652,7 @@ Runtime's expired-attempt path reuses this port only to stop and dispose. ###### Inherited from -[`ExactProcessCandidateExperimentExecutor`](#exactprocesscandidateexperimentexecutor).[`executor`](#executor) +[`ExactProcessCandidateExperimentExecutor`](#exactprocesscandidateexperimentexecutor).[`executor`](#executor-1) ##### recoveryPorts @@ -3503,7 +3648,7 @@ Full canonical profile used for this agent. Exported redacted with a stable hash ###### Inherited from -[`IntelligenceConfig`](#intelligenceconfig).[`profile`](#profile-4) +[`IntelligenceConfig`](#intelligenceconfig).[`profile`](#profile-5) ##### commitSha? @@ -3669,6 +3814,34 @@ Return undefined only when no target write can have committed. *** +### AuthoredAgentProfileCandidateLineage + +> **AuthoredAgentProfileCandidateLineage** = `Omit`\<`AgentCandidateLineage`, `"source"` \| `"profileDiffIds"`\> & `object` + +Lineage accepted by the direct candidate path. Optimizer lineage belongs to `improve()`. + +#### Type Declaration + +##### source + +> **source**: `Exclude`\<`AgentCandidateLineage`\[`"source"`\], `"optimizer"`\> + +##### profileDiffIds? + +> `optional` **profileDiffIds?**: `never` + +Runtime derives these from the exact profile change it seals. + +*** + +### AuthoredAgentProfileDiffOptions + +> **AuthoredAgentProfileDiffOptions** = `NonNullable`\<`Parameters`\<*typeof* [`agentImprovementProfileDiffs`](#agentimprovementprofilediffs)\>\[`2`\]\> + +Provenance attached while Runtime derives the exact profile diff. + +*** + ### JsonSchema > **JsonSchema** = `Record`\<`string`, `unknown`\> @@ -4197,6 +4370,25 @@ Validate and execute one product-owned activation transition. *** +### proposeAuthoredAgentProfileImprovement() + +> **proposeAuthoredAgentProfileImprovement**(`options`): `Promise`\<[`ProposeAuthoredAgentProfileImprovementResult`](#proposeauthoredagentprofileimprovementresult)\> + +Put a complete authored/imported profile through the canonical profile +experiment and proposal path without invoking `improve()`. + +#### Parameters + +##### options + +[`ProposeAuthoredAgentProfileImprovementOptions`](#proposeauthoredagentprofileimprovementoptions) + +#### Returns + +`Promise`\<[`ProposeAuthoredAgentProfileImprovementResult`](#proposeauthoredagentprofileimprovementresult)\> + +*** + ### manifestFromProfile() > **manifestFromProfile**(`profile`): [`CapabilityManifest`](#capabilitymanifest) diff --git a/docs/api/primitive-catalog.md b/docs/api/primitive-catalog.md index d8776ba9..5bde9a0d 100644 --- a/docs/api/primitive-catalog.md +++ b/docs/api/primitive-catalog.md @@ -7,7 +7,7 @@ # Primitive catalog — the never-stale anti-reinvention inventory -> **GENERATED** from `@tangle-network/agent-runtime@0.138.1` and `@tangle-network/agent-eval@0.146.0` by `scripts/gen-primitive-catalog.mjs`. Do NOT hand-edit — run `pnpm run docs:api`. This is the mechanical companion to the JUDGMENT in `canonical-api.md` (§2 decision table + §1.5 AgentProfile law): that doc says WHICH primitive to reach for and what NOT to build; this catalog proves WHAT exists. Per-symbol signatures + `file:line` live in the per-module pages under `docs/api/`. +> **GENERATED** from `@tangle-network/agent-runtime@0.139.0` and `@tangle-network/agent-eval@0.146.0` by `scripts/gen-primitive-catalog.mjs`. Do NOT hand-edit — run `pnpm run docs:api`. This is the mechanical companion to the JUDGMENT in `canonical-api.md` (§2 decision table + §1.5 AgentProfile law): that doc says WHICH primitive to reach for and what NOT to build; this catalog proves WHAT exists. Per-symbol signatures + `file:line` live in the per-module pages under `docs/api/`. ## 1. agent-runtime — own public surface @@ -401,7 +401,7 @@ Import from `@tangle-network/agent-runtime/tool-loop` — 12 exports. ### Intelligence SDK — Observe + provable-OFF billing -Import from `@tangle-network/agent-runtime/intelligence` — 166 exports. +Import from `@tangle-network/agent-runtime/intelligence` — 172 exports. | Symbol | Kind | Summary | |---|---|---| @@ -436,6 +436,7 @@ Import from `@tangle-network/agent-runtime/intelligence` — 166 exports. | `prepareAgentImprovementProfileActivation` | function | Compare product-owned profiles with an exact measured transition and prepare | | `proposeAgentImprovement` | function | Analyze, search, then remeasure the resulting exact candidate before proposing it. | | `proposeAgentProfileImprovement` | function | Analyze a product-owned profile, search one profile surface, then run the | +| `proposeAuthoredAgentProfileImprovement` | function | Put a complete authored/imported profile through the canonical profile | | `pullCertified` | function | Pull the certified composed profile for a target. Fail-closed: a network | | `resolveEffort` | function | Compile a named tier (plus optional per-field overrides) into the flat | | `resolveIntelligenceBaseUrl` | function | Resolve the ONE Intelligence base URL — the single knob both the send and | @@ -459,6 +460,7 @@ Import from `@tangle-network/agent-runtime/intelligence` — 166 exports. | `AgentImprovementActivationResult` | interface | Immutable outcome of one idempotent, transaction-wide activation attempt. | | `AgentImprovementMeasuredComparison` | interface | Portable paired held-out comparison produced by a sealed candidate executor. | | `AgentImprovementReview` | interface | Human or tenant-policy decision bound to one exact proposal. | +| `AgentProfileCandidateMeasurementExecutor` | interface | Product-owned executor for exact baseline/candidate profile measurement. | | `AgentProfileImprovementBenchmark` | interface | Product-owned task material that Runtime freezes before either profile state runs. | | `AgentProfileImprovementExecutor` | interface | One product execution adapter shared by optimizer search and exact profile | | `AppliedIntelligence` | interface | What the hook hands the agent each run. Additive over the prompt-only | @@ -486,6 +488,7 @@ Import from `@tangle-network/agent-runtime/intelligence` — 166 exports. | `ModeReadiness` | interface | One mode's readiness verdict. | | `ProfileImprovementActivationTransitionInput` | interface | A measured profile change without raw profile bytes. | | `ProposeAgentProfileImprovementOptions` | interface | Complete profile-improvement path for a product-owned source. | +| `ProposeAuthoredAgentProfileImprovementOptions` | interface | Measure a complete human-authored, imported, or compound profile candidate. | | `ProposedProfileDiff` | interface | A gate-certified profile diff the plane has already promoted, plus the | | `ProtectedExactProcessCandidateExperimentExecutor` | interface | Exact-process executor plus the ports required for durable recovery. | | `ProvisionedHost` | interface | A live, provisioned host the resolver tore up for a `process-on-infra` arm. | @@ -512,6 +515,8 @@ Import from `@tangle-network/agent-runtime/intelligence` — 166 exports. | `AgentImprovementProfileStateResolver` | type | Product-owned retained-state lookup used only for an explicit restore. | | `AgentImprovementProposalSubmissionState` | type | What Runtime knows about a failed proposal submission. | | `AgentProfileImprovementMethodOptions` | type | The portable profile changes that the measured-profile contract permits. | +| `AuthoredAgentProfileCandidateLineage` | type | Lineage accepted by the direct candidate path. Optimizer lineage belongs to `improve()`. | +| `AuthoredAgentProfileDiffOptions` | type | Provenance attached while Runtime derives the exact profile diff. | | `CapabilityAuth` | type | How a binding authenticates at resolve time. Declared as a REQUIREMENT in the | | `CapabilityInterface` | type | What the agent consumes. CLOSED — a new runtime kind NEVER extends this. Each | | `CapabilitySurface` | type | Every interface surface tag — the closed set the resolver fans into slots. | @@ -529,7 +534,7 @@ Import from `@tangle-network/agent-runtime/intelligence` — 166 exports. | `SubmitAgentImprovementProposalOutcome` | type | Typed result for proposal submission. A successful result contains the | | `UsageClass` | type | Usage class for billing. Base-stream tokens bill `'inference'`; every | -**Undocumented supporting types** (add a TSDoc line at the declaration to earn a table row): `AgentCandidateExperimentCellPlacement`, `AgentImprovementActivationResultStore`, `AgentImprovementActivationTargetPlan`, `AgentImprovementProfileReplacement`, `AgentImprovementProfileStateDigestInput`, `AgentImprovementProfileStateResolverInput`, `AgentImprovementProposal`, `AgentImprovementTargetProfileDiffOptions`, `AgentProfileImprovementActivationTargetPlan`, `CreateAgentImprovementActivationOptions`, `CreateAgentImprovementActivationResultOptions`, `CreateAgentImprovementProposalOptions`, `CreateExactProcessCandidateExperimentExecutorOptions`, `ExactProcessCandidateExperimentExecution`, `ExactProcessCandidateExperimentExecutor`, `ExecuteAgentCandidateExperimentCellOptions`, `ExecuteAgentImprovementActivationInput`, `ExecuteAgentImprovementActivationOptions`, `OptimizationActivationReceipt`, `OptimizationReceiptCost`, `ProposeAgentImprovementOptions`, `ProposeAgentImprovementResult`, `ProposeAgentProfileImprovementResult`, `PullCertifiedOptions`, `ReviewAgentImprovementInput`, `RunAgentCandidateExperimentOptions`, `RunAgentCandidateExperimentResult`, `SealedCandidateActivationTransitionInput`, `VerifyCandidateExecutionEvidenceOptions`, `AgentImprovementActivationIntent`, `AgentImprovementActivationOutcome`, `AgentImprovementActivationTargetIdentity`, `AgentImprovementActivationTransitionInput`, `AgentImprovementAnalysisOptions`, `AgentImprovementProfileActivationInput`, `AgentImprovementProfileActivationPreparation`, `AgentImprovementProfileActivationTarget`, `AgentImprovementProfileSurface`, `AgentImprovementProfileTargetState`, `AgentImprovementProfileTargetTransition`, `AgentImprovementReviewDecision`, `AgentProfileImprovementActivationOperation`, `AgentProfileMeasuredSurface`. +**Undocumented supporting types** (add a TSDoc line at the declaration to earn a table row): `AgentCandidateExperimentCellPlacement`, `AgentImprovementActivationResultStore`, `AgentImprovementActivationTargetPlan`, `AgentImprovementProfileReplacement`, `AgentImprovementProfileStateDigestInput`, `AgentImprovementProfileStateResolverInput`, `AgentImprovementProposal`, `AgentImprovementTargetProfileDiffOptions`, `AgentProfileImprovementActivationTargetPlan`, `CreateAgentImprovementActivationOptions`, `CreateAgentImprovementActivationResultOptions`, `CreateAgentImprovementProposalOptions`, `CreateExactProcessCandidateExperimentExecutorOptions`, `ExactProcessCandidateExperimentExecution`, `ExactProcessCandidateExperimentExecutor`, `ExecuteAgentCandidateExperimentCellOptions`, `ExecuteAgentImprovementActivationInput`, `ExecuteAgentImprovementActivationOptions`, `OptimizationActivationReceipt`, `OptimizationReceiptCost`, `ProposeAgentImprovementOptions`, `ProposeAgentImprovementResult`, `ProposeAgentProfileImprovementResult`, `ProposeAuthoredAgentProfileImprovementResult`, `PullCertifiedOptions`, `ReviewAgentImprovementInput`, `RunAgentCandidateExperimentOptions`, `RunAgentCandidateExperimentResult`, `SealedCandidateActivationTransitionInput`, `VerifyCandidateExecutionEvidenceOptions`, `AgentImprovementActivationIntent`, `AgentImprovementActivationOutcome`, `AgentImprovementActivationTargetIdentity`, `AgentImprovementActivationTransitionInput`, `AgentImprovementAnalysisOptions`, `AgentImprovementProfileActivationInput`, `AgentImprovementProfileActivationPreparation`, `AgentImprovementProfileActivationTarget`, `AgentImprovementProfileSurface`, `AgentImprovementProfileTargetState`, `AgentImprovementProfileTargetTransition`, `AgentImprovementReviewDecision`, `AgentProfileImprovementActivationOperation`, `AgentProfileMeasuredSurface`. ### Execution kernel — recursive atom, supervision, executors, round-synchronous loop diff --git a/docs/canonical-api.md b/docs/canonical-api.md index 11f96d1f..e6fe769f 100644 --- a/docs/canonical-api.md +++ b/docs/canonical-api.md @@ -4,7 +4,7 @@ Generated signatures and the complete export list live in docs/api/. Run pnpm docs:freshness after editing this file. --> -> **Version 0.138.1.** +> **Version 0.139.0.** > [`docs/api/primitive-catalog.md`](./api/primitive-catalog.md) lists every export and import path. > `agent-eval` must satisfy `>=0.146.0 <0.147.0`. > `sandbox` must satisfy `>=0.27.1 <0.28.0`. diff --git a/package.json b/package.json index 499e6fe7..a31f431a 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@tangle-network/agent-runtime", - "version": "0.138.1", + "version": "0.139.0", "description": "Shared task-lifecycle skeleton for agents: a recursive loop kernel for chat turns, one-shot tasks, and multi-attempt loops, with trace capture and eval-gated self-improvement. Domain behavior lives in adapters; scoring and ship-gates in @tangle-network/agent-eval.", "homepage": "https://github.com/tangle-network/agent-runtime#readme", "repository": { diff --git a/src/intelligence/authored-profile-improvement.ts b/src/intelligence/authored-profile-improvement.ts new file mode 100644 index 00000000..a73b56bf --- /dev/null +++ b/src/intelligence/authored-profile-improvement.ts @@ -0,0 +1,249 @@ +import { assertProposalFindings, type ProposalFinding } from '@tangle-network/agent-eval/analyst' +import type { CampaignScenarioIdentity } from '@tangle-network/agent-eval/campaign' +import { + type AgentProfileImprovementExperimentExecutionInput, + measuredComparisonFromAgentProfileImprovementExperiment, + runAgentProfileImprovementExperiment, + sealAgentProfileImprovementExperiment, + verifyAgentProfileImprovementExperimentComparison, +} from '@tangle-network/agent-eval/contract' +import type { + AgentCandidateLineage, + AgentImprovementProposal, + AgentImprovementSource, + AgentProfile, + AgentProfileImprovementExecutionRef, + AgentProfileImprovementExperiment, + AgentProfileImprovementMeasuredComparison, + AgentProfileImprovementMeasurement, + AgentProfileImprovementRunReceipt, + AgentProfileImprovementSuiteInputs, + Sha256Digest, +} from '@tangle-network/agent-interface' +import { agentImprovementSourceSchema } from '@tangle-network/agent-interface' +import { canonicalCandidateDigest, immutableCandidateValue } from '../candidate-execution/digest' +import { parseExactAgentProfile } from '../candidate-execution/profile' +import { + type AgentProfileImprovementBenchmark, + createAgentImprovementProposal, +} from './improvement-cycle' +import { agentImprovementProfileDiffs } from './improvement-surfaces' +import type { AgentImprovementProfileStateDigest } from './profile-activation' +import { + createProfileImprovementCostLedger, + profileImprovementMetadata, + profilePolicyWithBudget, + profilePreparationAccounting, + profileStateDigest, + profileTaskScenarioIdentity, + sealProfileImprovementBenchmark, +} from './profile-improvement-experiment' + +/** Lineage accepted by the direct candidate path. Optimizer lineage belongs to `improve()`. */ +export type AuthoredAgentProfileCandidateLineage = Omit< + AgentCandidateLineage, + 'source' | 'profileDiffIds' +> & { + source: Exclude + /** Runtime derives these from the exact profile change it seals. */ + profileDiffIds?: never +} + +/** Provenance attached while Runtime derives the exact profile diff. */ +export type AuthoredAgentProfileDiffOptions = NonNullable< + Parameters[2] +> + +/** Product-owned executor for exact baseline/candidate profile measurement. */ +export interface AgentProfileCandidateMeasurementExecutor { + executionRef: AgentProfileImprovementExecutionRef + measure( + input: AgentProfileImprovementExperimentExecutionInput & { profile: AgentProfile }, + ): Promise +} + +/** + * Measure a complete human-authored, imported, or compound profile candidate. + * No optimizer runs and no optimizer receipt is fabricated. + */ +export interface ProposeAuthoredAgentProfileImprovementOptions { + runId: string + source: AgentImprovementSource + profile: AgentProfile + stateDigest: AgentImprovementProfileStateDigest + candidateProfile: AgentProfile + candidateLineage: AuthoredAgentProfileCandidateLineage + /** Optional source/artifact metadata used on Runtime-derived profile diff steps. */ + diff?: AuthoredAgentProfileDiffOptions + findings?: readonly ProposalFinding[] + benchmark: AgentProfileImprovementBenchmark + executor: AgentProfileCandidateMeasurementExecutor + /** One customer-approved maximum for the held-out paired measurement. */ + budgetUsd: number + /** Optional identities used to prove authored/imported development work is held out. */ + developmentScenarios?: readonly CampaignScenarioIdentity[] + maxConcurrency?: number + signal?: AbortSignal + candidate?: AgentProfileImprovementMeasuredComparison['candidate'] + metadata?: AgentProfileImprovementMeasuredComparison['metadata'] + now?: () => Date +} + +export interface ProposeAuthoredAgentProfileImprovementResult { + candidateProfile: AgentProfile + candidateLineage: AgentCandidateLineage + experiment: AgentProfileImprovementExperiment + measurements: AgentProfileImprovementMeasurement[] + proposal: AgentImprovementProposal +} + +/** + * Put a complete authored/imported profile through the canonical profile + * experiment and proposal path without invoking `improve()`. + */ +export async function proposeAuthoredAgentProfileImprovement( + options: ProposeAuthoredAgentProfileImprovementOptions, +): Promise { + const source = agentImprovementSourceSchema.parse(options.source) + // Validate and seal caller metadata before allocating a cost ledger or + // invoking the product-owned executor. Reserved provenance fields and forged + // optimizer receipts must fail closed without spending measurement budget. + const metadata = profileImprovementMetadata(options.metadata, source) + const inputLineage = options.candidateLineage as AgentCandidateLineage + if (inputLineage.source === 'optimizer') { + throw new Error('authored profile improvement refuses optimizer lineage; use improve()') + } + if (Object.hasOwn(inputLineage, 'profileDiffIds')) { + throw new Error('authored profile improvement derives candidateLineage.profileDiffIds') + } + const findings = immutableCandidateValue([ + ...assertProposalFindings(options.findings ?? [], 'authored profile improvement findings'), + ]) + const costLedger = createProfileImprovementCostLedger( + options.budgetUsd, + 'authored profile improvement', + ) + const preparationStartedAt = performance.now() + const baselineProfile = parseExactAgentProfile(options.profile, 'authored profile baseline') + const candidateProfile = parseExactAgentProfile( + options.candidateProfile, + 'authored profile candidate', + ) + const baselineStateDigest = profileStateDigest( + options.stateDigest, + source.sourceIdentity, + baselineProfile, + ) + if (baselineStateDigest !== source.sourceDigest) { + throw new Error('authored profile source digest does not match the measured profile state') + } + const candidateStateDigest = profileStateDigest( + options.stateDigest, + source.sourceIdentity, + candidateProfile, + ) + if (candidateStateDigest === baselineStateDigest) { + throw new Error('authored profile candidate state digest matches the baseline') + } + + const change = agentImprovementProfileDiffs(baselineProfile, candidateProfile, { + ...options.diff, + id: options.diff?.id ?? `profile-improvement:${candidateStateDigest}`, + metadata: { + ...options.diff?.metadata, + sourceIdentity: source.sourceIdentity, + sourceRevision: source.sourceRevision, + }, + }) + const profileDiffIds = change.map((step) => { + if (!step.id) throw new Error('authored profile change requires an exact diff id') + return step.id + }) + const candidateLineage = immutableCandidateValue({ + ...inputLineage, + profileDiffIds, + }) + const policy = profilePolicyWithBudget( + options.benchmark.policy, + options.budgetUsd, + 'authored profile', + ) + const benchmark = sealProfileImprovementBenchmark({ ...options.benchmark, policy }) + assertDirectCandidateReleaseWorkIsFresh(benchmark, candidateLineage, options.developmentScenarios) + const experiment = sealAgentProfileImprovementExperiment({ + kind: 'agent-profile-improvement-experiment', + digestAlgorithm: 'rfc8785-sha256', + source, + executionRef: options.executor.executionRef, + baseline: { stateDigest: baselineStateDigest }, + candidate: { stateDigest: candidateStateDigest }, + change, + candidateLineage, + benchmark, + policy, + }) + const profilesByStateDigest = new Map([ + [baselineStateDigest, baselineProfile], + [candidateStateDigest, candidateProfile], + ]) + const preparation = profilePreparationAccounting(costLedger, preparationStartedAt) + const run = await runAgentProfileImprovementExperiment({ + experiment, + ...(options.maxConcurrency === undefined ? {} : { maxConcurrency: options.maxConcurrency }), + costLedger, + ...(options.signal ? { signal: options.signal } : {}), + execute: async (input) => { + const measuredProfile = profilesByStateDigest.get(input.stateDigest) + if (!measuredProfile) { + throw new Error('authored profile execution requested an unknown profile state') + } + return options.executor.measure({ ...input, profile: measuredProfile }) + }, + }) + const evaluation = verifyAgentProfileImprovementExperimentComparison( + measuredComparisonFromAgentProfileImprovementExperiment({ + experiment, + measurements: run.measurements, + runId: options.runId, + ...(options.candidate ? { candidate: options.candidate } : {}), + generationsExplored: 0, + preparation, + measurement: run.measurement, + metadata, + }), + ) + const proposal = createAgentImprovementProposal({ + runId: options.runId, + findings, + evaluation, + ...(options.now ? { now: options.now } : {}), + }) + return { + candidateProfile, + candidateLineage, + experiment, + measurements: run.measurements, + proposal, + } +} + +function assertDirectCandidateReleaseWorkIsFresh( + benchmark: AgentProfileImprovementSuiteInputs, + lineage: AgentCandidateLineage, + developmentScenarios: readonly CampaignScenarioIdentity[] | undefined, +): void { + if (lineage.developmentSplitDigest === benchmark.suite.splitDigest) { + throw new Error('authored profile development and held-out splits must be disjoint') + } + if (!developmentScenarios || developmentScenarios.length === 0) return + const development = new Set(developmentScenarios.map(canonicalCandidateDigest)) + const reused = benchmark.tasks + .map(profileTaskScenarioIdentity) + .filter((scenario) => development.has(canonicalCandidateDigest(scenario))) + .map((scenario) => scenario.id) + if (reused.length > 0) { + throw new Error( + `authored profile release reuses development scenario(s): [${reused.join(', ')}]`, + ) + } +} diff --git a/src/intelligence/improvement-cycle.ts b/src/intelligence/improvement-cycle.ts index d36ce760..baa01d21 100644 --- a/src/intelligence/improvement-cycle.ts +++ b/src/intelligence/improvement-cycle.ts @@ -1,9 +1,6 @@ import { CostLedger, type CostLedgerHandle } from '@tangle-network/agent-eval' import { assertProposalFindings, type ProposalFinding } from '@tangle-network/agent-eval/analyst' -import { - type CampaignScenarioIdentity, - campaignSplitDigestFromIdentities, -} from '@tangle-network/agent-eval/campaign' +import type { CampaignScenarioIdentity } from '@tangle-network/agent-eval/campaign' import { type AgentProfileImprovementExperimentExecutionInput, type CandidateExperimentExecutionInput, @@ -14,8 +11,6 @@ import { runCandidateExperiment, type Scenario, sealAgentProfileImprovementExperiment, - sealAgentProfileImprovementSuite, - sealAgentProfileImprovementTask, sealCandidateExperiment, verifyAgentProfileImprovementExperimentComparison, verifyCandidateExperiment, @@ -32,7 +27,6 @@ import type { AgentCandidateRunCell, AgentImprovementActivation, AgentImprovementActivationIntent, - AgentImprovementCost, AgentImprovementEvaluation, AgentImprovementMeasuredComparison, AgentImprovementProposal, @@ -46,21 +40,17 @@ import type { AgentProfileImprovementMeasurement, AgentProfileImprovementRunReceipt, AgentProfileImprovementSuiteInputs, - AgentProfileImprovementTask, AgentProfileImprovementTaskMaterial, CandidateExecutionEvidence, Sha256Digest, } from '@tangle-network/agent-interface' import { - AGENT_IMPROVEMENT_SOURCE_METADATA_KEY, agentCandidateMaterializationReceiptSchema, agentCandidateRunReceiptSchema, agentImprovementActivationSchema, agentImprovementProposalSchema, agentImprovementReviewSchema, - agentImprovementSourceMetadata, agentImprovementSourceSchema, - agentProfileImprovementArmSchema, agentProfileImprovementExecutionRefSchema, candidateExecutionEvidenceSchema, numbersApproximatelyEqual, @@ -125,6 +115,15 @@ import { optimizationActivationReceiptFromMetadata, } from './optimization-receipt' import type { AgentImprovementProfileStateDigest } from './profile-activation' +import { + createProfileImprovementCostLedger, + profileImprovementMetadata, + profilePolicyWithBudget, + profilePreparationAccounting, + profileStateDigest, + profileTaskScenarioIdentity, + sealProfileImprovementBenchmark, +} from './profile-improvement-experiment' export type { AgentImprovementActivation, @@ -569,40 +568,6 @@ function assertAnalysisCostRecorded( } } -function createProfileImprovementCostLedger(budgetUsd: number): CostLedger { - if (!Number.isFinite(budgetUsd) || budgetUsd < 0) { - throw new Error('profile improvement budgetUsd must be a non-negative finite number') - } - return new CostLedger({ costCeilingUsd: budgetUsd }) -} - -function profilePolicyWithBudget( - policy: AgentCandidateEvaluationPolicy, - budgetUsd: number, -): AgentCandidateEvaluationPolicy { - if (policy.budgetUsd !== undefined && !numbersApproximatelyEqual(policy.budgetUsd, budgetUsd)) { - throw new Error('profile improvement policy budgetUsd must equal the run budgetUsd') - } - return { ...policy, budgetUsd } -} - -function profilePreparationAccounting( - costLedger: CostLedgerHandle, - startedAt: number, -): { wallDurationMs: number; cost: AgentImprovementCost } { - const summary = costLedger.summary() - if (!summary.accountingComplete || summary.costProvenance.kind === 'uncaptured') { - throw new Error('profile improvement preparation cost is incomplete') - } - return { - wallDurationMs: Math.max(0, performance.now() - startedAt), - cost: { - usd: summary.costProvenance.usd, - provenance: summary.costProvenance.kind, - }, - } -} - function completeImprovementSearchAccounting( analysis: ImprovementSearchAccounting, improvement: { @@ -631,42 +596,6 @@ function completeImprovementSearchAccounting( } } -function profileStateDigest( - stateDigest: AgentImprovementProfileStateDigest, - identity: string, - profile: AgentProfile, -): Sha256Digest { - return agentProfileImprovementArmSchema.parse({ - stateDigest: stateDigest({ identity, profile }), - }).stateDigest -} - -function sealProfileImprovementBenchmark( - input: AgentProfileImprovementBenchmark, -): AgentProfileImprovementSuiteInputs { - const tasks = input.tasks.map((task) => sealAgentProfileImprovementTask(task)) as [ - AgentProfileImprovementTask, - ...AgentProfileImprovementTask[], - ] - return sealAgentProfileImprovementSuite({ - splitDigest: campaignSplitDigestFromIdentities( - tasks.map(profileTaskScenarioIdentity), - input.reps, - ), - tasks, - reps: input.reps, - seeds: input.seeds, - }) -} - -function profileTaskScenarioIdentity(task: AgentProfileImprovementTask): CampaignScenarioIdentity { - return { - id: task.scenario.id, - kind: task.scenario.kind, - scenarioDigest: task.scenario.digest, - } -} - function assertReleaseSplitIsFresh( heldOutSplitDigest: Sha256Digest, improvement: ImproveResult, @@ -733,24 +662,6 @@ function assertProfileReleaseWorkIsFresh( assertReleaseScenariosAreFresh(improvement, benchmark.tasks.map(profileTaskScenarioIdentity)) } -function profileImprovementMetadata( - metadata: AgentProfileImprovementMeasuredComparison['metadata'], - source: AgentImprovementSource, - optimizationReceipt: ReturnType, -): NonNullable { - assertNoCallerOptimizationReceipt(metadata) - if (metadata && Object.hasOwn(metadata, AGENT_IMPROVEMENT_SOURCE_METADATA_KEY)) { - throw new Error( - `candidate metadata reserves '${AGENT_IMPROVEMENT_SOURCE_METADATA_KEY}' for Runtime`, - ) - } - const sourceMetadata = agentImprovementSourceMetadata(source) - const merged = { ...(metadata ?? {}), ...sourceMetadata } - return optimizationReceipt - ? attachOptimizationActivationReceipt(merged, optimizationReceipt) - : immutableCandidateValue(merged) -} - /** * Analyze a product-owned profile, search one profile surface, then run the * exact baseline and candidate through the product executor before proposing. diff --git a/src/intelligence/index.ts b/src/intelligence/index.ts index 187df128..0704481d 100644 --- a/src/intelligence/index.ts +++ b/src/intelligence/index.ts @@ -78,6 +78,14 @@ export { executeAgentImprovementActivation, verifyAgentImprovementActivationResult, } from './activation' +export type { + AgentProfileCandidateMeasurementExecutor, + AuthoredAgentProfileCandidateLineage, + AuthoredAgentProfileDiffOptions, + ProposeAuthoredAgentProfileImprovementOptions, + ProposeAuthoredAgentProfileImprovementResult, +} from './authored-profile-improvement' +export { proposeAuthoredAgentProfileImprovement } from './authored-profile-improvement' export type { CapabilityAuth, CapabilityInterface, diff --git a/src/intelligence/profile-improvement-experiment.ts b/src/intelligence/profile-improvement-experiment.ts new file mode 100644 index 00000000..278b18a1 --- /dev/null +++ b/src/intelligence/profile-improvement-experiment.ts @@ -0,0 +1,133 @@ +import { CostLedger, type CostLedgerHandle } from '@tangle-network/agent-eval' +import { + type CampaignScenarioIdentity, + campaignSplitDigestFromIdentities, +} from '@tangle-network/agent-eval/campaign' +import { + sealAgentProfileImprovementSuite, + sealAgentProfileImprovementTask, +} from '@tangle-network/agent-eval/contract' +import type { + AgentCandidateEvaluationPolicy, + AgentImprovementCost, + AgentImprovementSource, + AgentProfile, + AgentProfileImprovementMeasuredComparison, + AgentProfileImprovementSuiteInputs, + AgentProfileImprovementTask, + AgentProfileImprovementTaskMaterial, + Sha256Digest, +} from '@tangle-network/agent-interface' +import { + AGENT_IMPROVEMENT_SOURCE_METADATA_KEY, + agentImprovementSourceMetadata, + agentProfileImprovementArmSchema, + numbersApproximatelyEqual, +} from '@tangle-network/agent-interface' +import { immutableCandidateValue } from '../candidate-execution/digest' +import { + assertNoCallerOptimizationReceipt, + attachOptimizationActivationReceipt, + type createOptimizationActivationReceipt, +} from './optimization-receipt' +import type { AgentImprovementProfileStateDigest } from './profile-activation' + +export interface ProfileImprovementBenchmarkInput { + tasks: [AgentProfileImprovementTaskMaterial, ...AgentProfileImprovementTaskMaterial[]] + reps: number + seeds: [number, ...number[]] + policy: AgentCandidateEvaluationPolicy +} + +export function createProfileImprovementCostLedger( + budgetUsd: number, + context = 'profile improvement', +): CostLedger { + if (!Number.isFinite(budgetUsd) || budgetUsd < 0) { + throw new Error(`${context} budgetUsd must be a non-negative finite number`) + } + return new CostLedger({ costCeilingUsd: budgetUsd }) +} + +export function profilePolicyWithBudget( + policy: AgentCandidateEvaluationPolicy, + budgetUsd: number, + context = 'profile improvement', +): AgentCandidateEvaluationPolicy { + if (policy.budgetUsd !== undefined && !numbersApproximatelyEqual(policy.budgetUsd, budgetUsd)) { + throw new Error(`${context} policy budgetUsd must equal the run budgetUsd`) + } + return { ...policy, budgetUsd } +} + +export function profilePreparationAccounting( + costLedger: CostLedgerHandle, + startedAt: number, +): { wallDurationMs: number; cost: AgentImprovementCost } { + const summary = costLedger.summary() + if (!summary.accountingComplete || summary.costProvenance.kind === 'uncaptured') { + throw new Error('profile improvement preparation cost is incomplete') + } + return { + wallDurationMs: Math.max(0, performance.now() - startedAt), + cost: { + usd: summary.costProvenance.usd, + provenance: summary.costProvenance.kind, + }, + } +} + +export function profileStateDigest( + stateDigest: AgentImprovementProfileStateDigest, + identity: string, + profile: AgentProfile, +): Sha256Digest { + return agentProfileImprovementArmSchema.parse({ + stateDigest: stateDigest({ identity, profile }), + }).stateDigest +} + +export function sealProfileImprovementBenchmark( + input: ProfileImprovementBenchmarkInput, +): AgentProfileImprovementSuiteInputs { + const tasks = input.tasks.map((task) => sealAgentProfileImprovementTask(task)) as [ + AgentProfileImprovementTask, + ...AgentProfileImprovementTask[], + ] + return sealAgentProfileImprovementSuite({ + splitDigest: campaignSplitDigestFromIdentities( + tasks.map(profileTaskScenarioIdentity), + input.reps, + ), + tasks, + reps: input.reps, + seeds: input.seeds, + }) +} + +export function profileTaskScenarioIdentity( + task: AgentProfileImprovementTask, +): CampaignScenarioIdentity { + return { + id: task.scenario.id, + kind: task.scenario.kind, + scenarioDigest: task.scenario.digest, + } +} + +export function profileImprovementMetadata( + metadata: AgentProfileImprovementMeasuredComparison['metadata'], + source: AgentImprovementSource, + optimizationReceipt?: ReturnType, +): NonNullable { + assertNoCallerOptimizationReceipt(metadata) + if (metadata && Object.hasOwn(metadata, AGENT_IMPROVEMENT_SOURCE_METADATA_KEY)) { + throw new Error( + `candidate metadata reserves '${AGENT_IMPROVEMENT_SOURCE_METADATA_KEY}' for Runtime`, + ) + } + const merged = { ...(metadata ?? {}), ...agentImprovementSourceMetadata(source) } + return optimizationReceipt + ? attachOptimizationActivationReceipt(merged, optimizationReceipt) + : immutableCandidateValue(merged) +} diff --git a/src/testing/fixtures/agent-improvement-proposal.json b/src/testing/fixtures/agent-improvement-proposal.json index 8c091ec8..3b430b8e 100644 --- a/src/testing/fixtures/agent-improvement-proposal.json +++ b/src/testing/fixtures/agent-improvement-proposal.json @@ -1,6 +1,6 @@ { "changedSurfaces": ["prompt"], - "digest": "sha256:96aa602a3f9d198264af652f5bba5cd51fb54519aaae91c28894eef2b6e694e5", + "digest": "sha256:8681ef365ff523163b585e6bec22b7e6b0de652647675763d97da8ecb0c423b4", "evaluation": { "decision": { "contributingChecks": [ @@ -4882,7 +4882,7 @@ ], "metadata": { "fixture": "agent-improvement-proposal", - "runtimeVersion": "0.138.1" + "runtimeVersion": "0.139.0" }, "objectives": [ { @@ -4993,8 +4993,8 @@ "baselineContentHash": "sha256:5c21ee53e513fc604cb09754e21c392b24a424da0ef37dbf8f1ee4a8a0b08f09", "candidateContentHash": "sha256:60fcbb1c728194bd51d7d19cb732d1c3f1881dce7e0a6266b41c8b98cfd65693", "kind": "agent-eval-loop", - "recordDigest": "sha256:5b1be97ac62edbe65f899224398159b1fbfc9a768d25a5d79044de6e53f76600", - "runId": "agent-runtime-0.138.1-proposal-fixture", + "recordDigest": "sha256:b5fc7a2d1c5ed0a168afdb6393ab9aa7090da6db00f11278f2ab39766963aa97", + "runId": "agent-runtime-0.139.0-proposal-fixture", "schema": "agent-candidate-experiment" } }, @@ -5021,5 +5021,5 @@ ], "kind": "agent-improvement-proposal", "proposedAt": "2026-07-10T01:00:00.000Z", - "runId": "agent-runtime-0.138.1-proposal-fixture" + "runId": "agent-runtime-0.139.0-proposal-fixture" } diff --git a/src/testing/fixtures/agent-profile-improvement-proposal.json b/src/testing/fixtures/agent-profile-improvement-proposal.json index edbe8cbd..88722142 100644 --- a/src/testing/fixtures/agent-profile-improvement-proposal.json +++ b/src/testing/fixtures/agent-profile-improvement-proposal.json @@ -1,6 +1,6 @@ { "changedSurfaces": ["prompt", "skills"], - "digest": "sha256:f98a80755978d5cfbca957967b1a97f29af8a974bf920746fda8b3deddf38304", + "digest": "sha256:35fd03f0e77f5f251219ee6c4e552590d50ba92a222da20b75d2a98cdbb29854", "evaluation": { "decision": { "contributingChecks": [ @@ -1715,7 +1715,7 @@ ], "metadata": { "fixture": "agent-profile-improvement-proposal", - "runtimeVersion": "0.138.1" + "runtimeVersion": "0.139.0" }, "objectives": [ { @@ -1826,7 +1826,7 @@ "baselineContentHash": "sha256:21c495a37c418c10bde64fbaa188beddeed31f1f051ea60a6a6582a9ee0db704", "candidateContentHash": "sha256:103f77bc8481601eef1ad5fe6ba84a40dffabc3a44f421f8c8559121edab84e9", "kind": "agent-eval-loop", - "recordDigest": "sha256:16e8733160306657bd773d677c478519634bb4360f0573484e6c5e85e4dcc6d4", + "recordDigest": "sha256:07205e57acf8fbe5927cb53da608ce553093efa116deebbcef1ca117e0bc53c1", "runId": "profile-improvement-1", "schema": "agent-profile-improvement-experiment" } diff --git a/tests/authored-profile-improvement.test.ts b/tests/authored-profile-improvement.test.ts new file mode 100644 index 00000000..bdd684d0 --- /dev/null +++ b/tests/authored-profile-improvement.test.ts @@ -0,0 +1,291 @@ +import { minimumPairsForPairedDeltaTest, type ProposalFinding } from '@tangle-network/agent-eval' +import type { CampaignScenarioIdentity } from '@tangle-network/agent-eval/campaign' +import { + AGENT_IMPROVEMENT_SOURCE_METADATA_KEY, + type AgentProfile, +} from '@tangle-network/agent-interface' +import { describe, expect, it } from 'vitest' + +import { canonicalCandidateDigest } from '../src/candidate-execution/digest' +import { + type ProposeAuthoredAgentProfileImprovementOptions, + proposeAuthoredAgentProfileImprovement, +} from '../src/intelligence/authored-profile-improvement' +import { optimizationActivationReceiptFromMetadata } from '../src/intelligence/optimization-receipt' +import { improvementFinding as fixtureFinding } from './helpers/improvement-method-fixture' +import { + createProfileImprovementFixture, + createProfileImprovementRunReceipt, +} from './helpers/profile-improvement-fixture' + +const minimumPairedRuns = minimumPairsForPairedDeltaTest(0.95) +const { proposal_origin: _fixtureOrigin, ...fixtureAnalystFinding } = + fixtureFinding as ProposalFinding +const productionFinding: ProposalFinding = { + ...fixtureAnalystFinding, + proposal_origin: 'production', + evidence_refs: [{ kind: 'span', uri: 'span-authored-profile' }], +} + +function setup() { + const template = createProfileImprovementFixture() + const task = template.evaluation.experiment.benchmark.tasks[0] + if (!task) throw new Error('expected a profile improvement task') + const { digest: _taskDigest, ...taskMaterial } = task + const identity = 'tenant/research/profile' + const baselineProfile: AgentProfile = { + name: 'researcher', + prompt: { systemPrompt: 'Investigate the question.' }, + model: { default: 'provider/old-model' }, + metadata: { lineage: 'baseline' }, + } + const candidateProfile: AgentProfile = { + ...baselineProfile, + prompt: { systemPrompt: 'Investigate, verify, and cite the earliest causal failure.' }, + model: { default: 'provider/new-model', reasoningEffort: 'high' }, + harness: 'codex', + tools: { Read: true, Bash: true }, + mcp: { literature: { command: 'literature-server' } }, + hooks: { Stop: [{ command: 'node verify-result.mjs', blocking: true }] }, + metadata: { lineage: 'human-reflection', reflectionRun: 'reflection-1' }, + } + const stateDigest = ({ + identity: profileIdentity, + profile, + }: { + identity: string + profile: AgentProfile + }) => canonicalCandidateDigest({ identity: profileIdentity, profile }) + const baselineStateDigest = stateDigest({ identity, profile: baselineProfile }) + const heldOutScenario: CampaignScenarioIdentity = { + id: task.scenario.id, + kind: task.scenario.kind, + scenarioDigest: task.scenario.digest, + } + const observed: Array<{ arm: 'baseline' | 'candidate'; profile: AgentProfile }> = [] + const options: ProposeAuthoredAgentProfileImprovementOptions = { + runId: 'authored-profile-run-1', + budgetUsd: minimumPairedRuns * 2, + source: { + kind: 'platform-agent-profile', + sourceIdentity: identity, + sourceDigest: baselineStateDigest, + sourceRevision: 1, + }, + profile: baselineProfile, + candidateProfile, + candidateLineage: { + source: 'human', + parentDigests: [baselineStateDigest], + runIds: ['reflection-1'], + modelSnapshots: ['provider/new-model'], + }, + diff: { + id: 'human-reflection-1', + source: { + kind: 'frontier-author', + artifacts: ['traces://reflection-1'], + notes: ['A human approved the profile authored from a trace autopsy.'], + }, + }, + findings: [productionFinding], + stateDigest, + benchmark: { + tasks: [taskMaterial], + reps: minimumPairedRuns, + seeds: Array.from({ length: minimumPairedRuns }, (_, index) => 101 + index) as [ + number, + ...number[], + ], + policy: template.evaluation.experiment.policy, + }, + executor: { + executionRef: { + kind: 'agent-profile-improvement-execution-ref', + identity: 'authored-profile-runner', + digest: canonicalCandidateDigest({ runner: 'authored-profile-runner', revision: 1 }), + }, + measure: async (input) => { + observed.push({ arm: input.arm, profile: input.profile }) + const variation = ((input.runCell.repetition % 3) - 1) * 0.02 + return createProfileImprovementRunReceipt( + input, + input.arm === 'baseline' ? 0.2 : 0.8 + variation, + ) + }, + }, + candidate: { + label: 'human trace reflection', + rationale: 'A complete profile authored from a cited trace autopsy.', + }, + now: () => new Date('2026-08-16T20:00:00.000Z'), + } + return { + options, + observed, + baselineProfile, + candidateProfile, + baselineStateDigest, + heldOutScenario, + } +} + +describe('authored profile improvement', { timeout: 30_000 }, () => { + it('measures a human-authored complete profile through the canonical proposal path', async () => { + const fixture = setup() + const inputLineage = structuredClone(fixture.options.candidateLineage) + + const result = await proposeAuthoredAgentProfileImprovement(fixture.options) + + expect(fixture.observed).toHaveLength(minimumPairedRuns * 2) + expect( + fixture.observed + .filter((entry) => entry.arm === 'baseline') + .every((entry) => entry.profile === fixture.baselineProfile), + ).toBe(false) + expect( + fixture.observed + .filter((entry) => entry.arm === 'baseline') + .every( + (entry) => + entry.profile.prompt?.systemPrompt === fixture.baselineProfile.prompt?.systemPrompt, + ), + ).toBe(true) + expect( + fixture.observed + .filter((entry) => entry.arm === 'candidate') + .every( + (entry) => + entry.profile.prompt?.systemPrompt === fixture.candidateProfile.prompt?.systemPrompt, + ), + ).toBe(true) + expect(result.candidateLineage).toMatchObject({ + source: 'human', + parentDigests: [fixture.baselineStateDigest], + runIds: ['reflection-1'], + }) + expect(result.candidateLineage.profileDiffIds).toEqual([ + 'human-reflection-1:agent-profile:reset', + 'human-reflection-1:agent-profile:set', + ]) + expect(result.experiment.change.map((step) => step.id)).toEqual( + result.candidateLineage.profileDiffIds, + ) + expect( + result.experiment.change.every( + (step) => + step.source?.kind === 'frontier-author' && + step.metadata?.sourceIdentity === fixture.options.source.sourceIdentity && + step.metadata?.sourceRevision === fixture.options.source.sourceRevision, + ), + ).toBe(true) + expect(fixture.options.candidateLineage).toEqual(inputLineage) + expect(Object.hasOwn(fixture.options.candidateLineage, 'profileDiffIds')).toBe(false) + expect(result.experiment.candidateLineage).toEqual(result.candidateLineage) + expect(result.proposal.evaluation.decision.outcome).toBe('ship') + expect(result.proposal.changedSurfaces).toEqual([ + 'prompt', + 'tools', + 'mcp', + 'hooks', + 'agent-profile', + ]) + expect(result.proposal.findings).toEqual([productionFinding]) + expect(result.proposal.evaluation.generationsExplored).toBeUndefined() + expect(optimizationActivationReceiptFromMetadata(result.proposal.evaluation.metadata)).toBe( + undefined, + ) + }) + + it('preserves import lineage without fabricating optimizer evidence', async () => { + const fixture = setup() + fixture.options.candidateLineage = { + source: 'import', + parentDigests: [fixture.baselineStateDigest], + runIds: ['external-profile-build-1'], + } + + const result = await proposeAuthoredAgentProfileImprovement(fixture.options) + + expect(result.candidateLineage.source).toBe('import') + expect(optimizationActivationReceiptFromMetadata(result.proposal.evaluation.metadata)).toBe( + undefined, + ) + }) + + it('refuses optimizer lineage and caller-supplied profile diff identities', async () => { + const optimizer = setup() + optimizer.options.candidateLineage = { + source: 'optimizer', + parentDigests: [optimizer.baselineStateDigest], + runIds: ['optimizer-run'], + developmentSplitDigest: canonicalCandidateDigest({ split: 'development' }), + } as never + await expect(proposeAuthoredAgentProfileImprovement(optimizer.options)).rejects.toThrow( + /refuses optimizer lineage/, + ) + + const suppliedIds = setup() + suppliedIds.options.candidateLineage = { + source: 'human', + profileDiffIds: ['caller-controlled-id'], + } as never + await expect(proposeAuthoredAgentProfileImprovement(suppliedIds.options)).rejects.toThrow( + /derives candidateLineage\.profileDiffIds/, + ) + }) + + it('refuses forged metadata and invalid or mismatched budgets before execution', async () => { + const forgedMetadata = setup() + forgedMetadata.options.metadata = { + [AGENT_IMPROVEMENT_SOURCE_METADATA_KEY]: 'caller-controlled-source', + } + await expect(proposeAuthoredAgentProfileImprovement(forgedMetadata.options)).rejects.toThrow( + /reserves/, + ) + expect(forgedMetadata.observed).toHaveLength(0) + + const invalidBudget = setup() + invalidBudget.options.budgetUsd = Number.NaN + await expect(proposeAuthoredAgentProfileImprovement(invalidBudget.options)).rejects.toThrow( + /non-negative finite number/, + ) + expect(invalidBudget.observed).toHaveLength(0) + + const mismatchedBudget = setup() + mismatchedBudget.options.benchmark = { + ...mismatchedBudget.options.benchmark, + policy: { + ...mismatchedBudget.options.benchmark.policy, + budgetUsd: mismatchedBudget.options.budgetUsd + 1, + }, + } + await expect(proposeAuthoredAgentProfileImprovement(mismatchedBudget.options)).rejects.toThrow( + /policy budgetUsd must equal/, + ) + expect(mismatchedBudget.observed).toHaveLength(0) + }) + + it('refuses unchanged candidates, source drift, and reused held-out scenarios', async () => { + const unchanged = setup() + unchanged.options.candidateProfile = unchanged.baselineProfile + await expect(proposeAuthoredAgentProfileImprovement(unchanged.options)).rejects.toThrow( + /matches the baseline/, + ) + + const drifted = setup() + drifted.options.source = { + ...drifted.options.source, + sourceDigest: canonicalCandidateDigest({ wrong: true }), + } + await expect(proposeAuthoredAgentProfileImprovement(drifted.options)).rejects.toThrow( + /source digest does not match/, + ) + + const leaked = setup() + leaked.options.developmentScenarios = [leaked.heldOutScenario] + await expect(proposeAuthoredAgentProfileImprovement(leaked.options)).rejects.toThrow( + /reuses development scenario/, + ) + }) +})