test: hermetic e2e suite for Altimate Base (registration, catalog, inference, rate-limit/budget, error surfacing) - #1248
Merged
anandgupta42 merged 2 commits intoSep 7, 2026
Conversation
Foundation for a 6-suite parallel Altimate Base e2e test partition (see the design doc). Adds `test/altimate/_fixtures/fake-gateway.ts` (a `FakeGateway` that intercepts `fetch` via `spyOn(globalThis, "fetch")` — the repo's existing proven pattern, not a real HTTP server — implementing `/register` and `/v1/chat/completions` with controllable knobs for every failure mode the suites need: per-minute token rate-limit, both `budget_exceeded` variants, request-too-large, 401, 5xx, timeout, malformed JSON, and success) and `test/altimate/_fixtures/altimate-base-harness.ts` (isolated XDG/home bootstrap + gateway-env reset helpers, extracted from `altimate-base.test.ts`'s existing pattern so every suite shares one implementation). Adds `altimate-base-harness-smoke.test.ts` proving the harness works in both directions: a register -> `authorizedFetch` happy-path round trip, and one scripted failure knob (per-minute token rate-limit -> `describeRateLimit`'s non-retryable message). Does not add any of the 6 planned suite files themselves — those are a separate, parallel follow-up. Copies the design doc (`docs/internal/2026-09-04-altimate-base-e2e-harness-plan.md`) into the branch so it travels with the PR. Stacked on `codex/altimate-base-release-final` (#1199) since the harness targets that branch's 131072/65536 limits and Altimate Base code.
…rming Brings together 5 independently-written hermetic e2e suites for Altimate Base onto the shared harness branch (53 tests): - `altimate-base-registration-gaps.test.ts` (11) — HTTP/network/malformed register failure mapping, payload shape, retry idempotency - `altimate-base-catalog.test.ts` (9) — model catalog / provider isolation - `altimate-base-inference-e2e.test.ts` (5) — register -> list -> fetch round trip, placeholder-vs-real-key isolation - `altimate-base-rate-limit-messages.test.ts` (21) — throttle/budget/ request-too-large message mapping - `altimate-base-error-surfacing.test.ts` (7) — 5xx/timeout/abort/ malformed-body/401 pass-through at the inference layer All 5 (plus the two pre-existing files, `altimate-base.test.ts` and `altimate-base-harness-smoke.test.ts`) independently called `FreeTierCapability.issueArmer()` at module scope. That capability is process-global and throws on a second call, so running the directory in one `bun test` invocation — as CI does — threw "Altimate Base consent armer already issued for this process" once a second armer-calling file loaded into the same worker process (reproducible with just the two pre-existing files, before any of these suites existed). Fix: centralize arming in the shared harness (`_fixtures/altimate-base-harness.ts`) behind a new `consented()` helper that lazily calls `issueArmer()` exactly once per process and caches the returned armer in a module-level singleton. Because bun caches modules per process, every suite file that imports `consented()` shares that one cached armer regardless of load order or file count. This adds no way to reset, re-claim, or otherwise weaken the one-shot guarantee `issueArmer()` already enforces — it is a cache in front of the single legitimate call, not a new capability. All 7 armer-calling files now import and use the shared helper instead of claiming their own. Verified with `bun test --timeout 90000 test/altimate/` (the directory CI covers, at CI's timeout) from `packages/opencode`: 5103 pass, 0 fail, zero armer-collision errors, in one process invocation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VqnuBDGkh1ZT65Ti7e6DHZ
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.
Tip: disable this comment in your organization's Code Review settings.
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_e9e0cb77-cc9e-4751-b90e-c28b01e5b412) |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Team Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
|
Thanks for your contribution! This PR doesn't have a linked issue. All PRs must reference an existing issue. Please:
See CONTRIBUTING.md for details. |
full receipts (3 sessions)
builder ·
|
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
2 similar comments
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d2973f4ec9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // kicking off the call could fire before the fake gateway's `timeout` branch has attached its | ||
| // `abort` listener. `AbortSignal.timeout` schedules the abort on a real timer instead, so it | ||
| // always fires after the listener is attached. | ||
| const promise = FreeTier.authorizedFetch(url, { ...init, signal: AbortSignal.timeout(50) }) |
There was a problem hiding this comment.
On a slow CI filesystem, credentialsForLoad() can take longer than 50 ms, so this signal may abort before FakeGateway.handleChat() installs its abort listener. The fake does not check signal.aborted when entering timeout mode, leaving its promise pending until Bun's 30-second test timeout. Publish gateway readiness and abort afterward, or make the fake immediately reject an already-aborted signal.
AGENTS.md reference: packages/opencode/test/AGENTS.md:L165-L169
Useful? React with 👍 / 👎.
Comment on lines
+81
to
+83
| function armer(): (token: string) => void { | ||
| if (!cachedArmer) cachedArmer = FreeTierCapability.issueArmer() | ||
| return cachedArmer |
There was a problem hiding this comment.
Because the shared armer is now claimed lazily, running only the existing unforgeable consent: no in-process caller can mint an independent authority test means no earlier call to consented() has occurred. Its direct FreeTierCapability.issueArmer() call therefore succeeds even though the test expects it to throw, so common bun test -t ... workflows fail. Claim the armer during module setup or explicitly initialize it in that test before checking the second-claim behavior.
Useful? React with 👍 / 👎.
Comment on lines
+84
to
+87
| const response = await FreeTier.authorizedFetch(`${GATEWAY_URL}/v1/chat/completions`, chatRequestInit()) | ||
|
|
||
| expect(response.status).toBe(200) | ||
| const body = (await response.json()) as { choices: [{ message: { content: string } }] } |
There was a problem hiding this comment.
This purported inference E2E path calls FreeTier.authorizedFetch directly and manually parses the response, so it never constructs the @ai-sdk/openai-compatible model or exercises Provider.getModel/Provider.getLanguage, request serialization, model selection, and SDK response decoding. A regression that leaves the fetch function present in provider options but makes the configured model unusable would therefore pass the entire new suite; drive a generation through the provider model, as other provider E2E tests do, instead of invoking the transport seam directly.
Useful? React with 👍 / 👎.
Comment on lines
+81
to
+84
| const url = typeof input === "string" ? input : input instanceof URL ? input.href : input.url | ||
| if (url.endsWith("/register")) return this.handleRegister(url, init) | ||
| if (url.includes("/v1/chat/completions")) return this.handleChat(url, init) | ||
| throw new Error(`FakeGateway: unhandled URL ${url}`) |
There was a problem hiding this comment.
The fake dispatches only by URL suffix/substring and never checks init.method, so it accepts requests that the real gateway rejects—for example, a registration accidentally changed from POST to GET, or a chat request sent to /v1/chat/completions-invalid. Because replacing global fetch also bypasses native rejection of a GET request with a body, the registration contract tests can remain green while the shipped client fails before reaching the gateway. Match the exact pathname and require POST for both routes.
Useful? React with 👍 / 👎.
![kilo-code-bot[bot]](https://avatars.githubusercontent.com/in/2193792?s=60&v=4){kind=small}
| import fs from "node:fs" | ||
| import os from "node:os" | ||
| import path from "node:path" | ||
| import { consented } from "./_fixtures/altimate-base-harness" |
There was a problem hiding this comment.
SUGGESTION: Redundant environment-isolation code — consolidate onto the shared helper.
This file still hand-rolls the XDG/home isolation (isolatedEnvironment, originalEnvironment, temporaryHome, and the afterAll cleanup at lines 8-21 and 68-75), duplicating what isolateAltimateBaseHome() in _fixtures/altimate-base-harness.ts now provides. The harness plan (docs/internal/2026-09-04-altimate-base-e2e-harness-plan.md) explicitly intended altimate-base.test.ts to import the shared bootstrap so there is "exactly one isolated-environment implementation", but only consented() was migrated. Replace the inline block with isolateAltimateBaseHome("altimate-base").
Reply with @kilocode-bot fix it to have Kilo Code address this issue.
| claimable exactly once **per process**, not per file. `bun test` runs each test file in its own | ||
| worker process by default (confirmed by the existing suite's comment at `altimate-base.test.ts:76-82` | ||
| treating this as safe), so each suite file gets its own fresh module instances and can safely call | ||
| `FreeTierCapability.issueArmer()` at module scope, exactly like the existing file does. **Do not** |
There was a problem hiding this comment.
SUGGESTION: This guidance contradicts the shipped fix and would reintroduce the crash.
This section instructs each suite to call FreeTierCapability.issueArmer() at module scope on the premise that "bun test runs each test file in its own worker process." The actual fix in this PR (consented() in _fixtures/altimate-base-harness.ts) exists precisely because multiple suite files load into one worker, where a second module-scope issueArmer() throws. A future implementer following this section (or the example test later in this file that calls issueArmer() directly) would reintroduce the "consent armer already issued" crash. Update this section to direct suites to use consented() instead.
Reply with @kilocode-bot fix it to have Kilo Code address this issue.
Code Review SummaryStatus: 2 Issues Found | Recommendation: Address before merge Overview
Issue Details (click to expand)SUGGESTION
Files Reviewed (10 files)
Fix these issues in Kilo Cloud Reviewed by deepseek-v4-pro · Input: 96.3K · Output: 32.8K · Cached: 1.3M Review guidance: REVIEW.md from base branch |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
8 issues found across 10 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/opencode/test/altimate/_fixtures/altimate-base-harness.ts">
<violation number="1" location="packages/opencode/test/altimate/_fixtures/altimate-base-harness.ts:59">
P3: resetGatewayEnv() permanently mutates process env: it deletes ALTIMATE_FREE_GATEWAY_URL and sets ALTIMATE_BASE_GATEWAY_URL to the fake gateway URL, but no teardown restores the prior values. isolateAltimateBaseHome()'s afterAll only restores the ISOLATED_ENV keys, so in a shared `bun test` worker the fake gateway URL stays set and the legacy var stays deleted after the suite. Follow the file's own restoration pattern: capture the original values once (e.g., alongside the ISOLATED_ENV snapshot in isolateAltimateBaseHome) and restore them in the afterAll.</violation>
<violation number="2" location="packages/opencode/test/altimate/_fixtures/altimate-base-harness.ts:79">
P2: When the unforgeable-consent test runs alone with `bun test -t`, `cachedArmer` is still unset, so its direct `issueArmer()` call succeeds instead of throwing. Claim the shared armer during module setup so filtered tests still exercise a second claim.</violation>
</file>
<file name="docs/internal/2026-09-04-altimate-base-e2e-harness-plan.md">
<violation number="1" location="docs/internal/2026-09-04-altimate-base-e2e-harness-plan.md:3">
P3: The doc's status header says "Not yet implemented" and describes itself as the contract for implementer agents, but every suite and fixture it specifies is implemented and shipped in this same PR. Its Deliverable 2 example and "Cross-file consent isolation" section also instruct each file to claim `issueArmer()` once at module scope, whereas the shipped harness uses a shared `consented()` singleton that claims the process's single armer — the exact collision this PR was built to fix. Update the status to reflect that the design is implemented and that the consent helper is a per-process `consented()` shared across files, not a per-file `issueArmer()`, so a future reader doesn't follow the stale design.</violation>
<violation number="2" location="docs/internal/2026-09-04-altimate-base-e2e-harness-plan.md:360">
P2: The fake gateway accepts invalid methods and paths, allowing contract tests to pass for requests the real gateway rejects. Dispatch only when the exact pathname is paired with `POST` for both routes.</violation>
</file>
<file name="packages/opencode/test/altimate/_fixtures/fake-gateway.ts">
<violation number="1" location="packages/opencode/test/altimate/_fixtures/fake-gateway.ts:151">
P3: The `timeout` chat mode never settles when the request has no AbortSignal, and also hangs if the signal is already aborted before the listener attaches. Check the signal and reject immediately in those cases so a mis-scripted test degrades to a clear rejection instead of a hang until Bun's global timeout.</violation>
</file>
<file name="packages/opencode/test/altimate/altimate-base-registration-gaps.test.ts">
<violation number="1" location="packages/opencode/test/altimate/altimate-base-registration-gaps.test.ts:143">
P3: The primary assertion in the cli_version test is a tautology: `sentVersion` is computed by the client as exactly `sanitizeCliVersion(Installation.VERSION)`, so comparing it to the same re-called function can never fail independently and gives false confidence that the sanitizer contract is being exercised. The only meaningful check is the following regex. Make the assertion self-contained by hardcoding an expected value, or rely on the regex alone and drop the self-comparison.</violation>
</file>
<file name="packages/opencode/test/altimate/altimate-base.test.ts">
<violation number="1" location="packages/opencode/test/altimate/altimate-base.test.ts:6">
P3: Use `isolateAltimateBaseHome("altimate-base")` here and remove the inline environment setup so this file has the same single isolation implementation as the new suites.</violation>
</file>
<file name="packages/opencode/test/altimate/altimate-base-inference-e2e.test.ts">
<violation number="1" location="packages/opencode/test/altimate/altimate-base-inference-e2e.test.ts:84">
P2: This test bypasses the configured `@ai-sdk/openai-compatible` model, so it cannot catch broken model construction, provider model selection, request serialization, or SDK response decoding. Drive a generation through the provider model instead.</violation>
</file>
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.
Re-trigger cubic
| // otherwise bypass that one-shot guarantee. It is purely a shared cache in front of the single | ||
| // legitimate call, so the underlying security property (only one in-process caller can ever obtain | ||
| // the ability to arm the production consent authority) is unchanged. | ||
| let cachedArmer: ((token: string) => void) | undefined |
There was a problem hiding this comment.
P2: When the unforgeable-consent test runs alone with bun test -t, cachedArmer is still unset, so its direct issueArmer() call succeeds instead of throwing. Claim the shared armer during module setup so filtered tests still exercise a second claim.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/opencode/test/altimate/_fixtures/altimate-base-harness.ts, line 79:
<comment>When the unforgeable-consent test runs alone with `bun test -t`, `cachedArmer` is still unset, so its direct `issueArmer()` call succeeds instead of throwing. Claim the shared armer during module setup so filtered tests still exercise a second claim.</comment>
<file context>
@@ -0,0 +1,95 @@
+// otherwise bypass that one-shot guarantee. It is purely a shared cache in front of the single
+// legitimate call, so the underlying security property (only one in-process caller can ever obtain
+// the ability to arm the production consent authority) is unchanged.
+let cachedArmer: ((token: string) => void) | undefined
+
+function armer(): (token: string) => void {
</file context>
Comment on lines
+360
to
+361
| if (url.endsWith("/register")) return this.handleRegister(url, init) | ||
| if (url.includes("/v1/chat/completions")) return this.handleChat(url, init) |
There was a problem hiding this comment.
P2: The fake gateway accepts invalid methods and paths, allowing contract tests to pass for requests the real gateway rejects. Dispatch only when the exact pathname is paired with POST for both routes.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At docs/internal/2026-09-04-altimate-base-e2e-harness-plan.md, line 360:
<comment>The fake gateway accepts invalid methods and paths, allowing contract tests to pass for requests the real gateway rejects. Dispatch only when the exact pathname is paired with `POST` for both routes.</comment>
<file context>
@@ -0,0 +1,672 @@
+ this.spy = spyOn(globalThis, "fetch").mockImplementation(
+ (async (input: RequestInfo | URL, init?: RequestInit) => {
+ const url = typeof input === "string" ? input : input instanceof URL ? input.href : input.url
+ if (url.endsWith("/register")) return this.handleRegister(url, init)
+ if (url.includes("/v1/chat/completions")) return this.handleChat(url, init)
+ throw new Error(`FakeGateway: unhandled URL ${url}`)
</file context>
Suggested change
| if (url.endsWith("/register")) return this.handleRegister(url, init) | |
| if (url.includes("/v1/chat/completions")) return this.handleChat(url, init) | |
| const request = new URL(url) | |
| if (request.pathname === "/register" && init?.method === "POST") return this.handleRegister(url, init) | |
| if (request.pathname === "/v1/chat/completions" && init?.method === "POST") return this.handleChat(url, init) |
| await registerWithGateway() | ||
|
|
||
| gateway.chatNext({ kind: "ok", content: "the answer is 42" }) | ||
| const response = await FreeTier.authorizedFetch(`${GATEWAY_URL}/v1/chat/completions`, chatRequestInit()) |
There was a problem hiding this comment.
P2: This test bypasses the configured @ai-sdk/openai-compatible model, so it cannot catch broken model construction, provider model selection, request serialization, or SDK response decoding. Drive a generation through the provider model instead.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/opencode/test/altimate/altimate-base-inference-e2e.test.ts, line 84:
<comment>This test bypasses the configured `@ai-sdk/openai-compatible` model, so it cannot catch broken model construction, provider model selection, request serialization, or SDK response decoding. Drive a generation through the provider model instead.</comment>
<file context>
@@ -0,0 +1,185 @@
+ await registerWithGateway()
+
+ gateway.chatNext({ kind: "ok", content: "the answer is 42" })
+ const response = await FreeTier.authorizedFetch(`${GATEWAY_URL}/v1/chat/completions`, chatRequestInit())
+
+ expect(response.status).toBe(200)
</file context>
| export function resetGatewayEnv(gatewayUrl: string): void { | ||
| delete process.env.ALTIMATE_BASE_GATEWAY_URL | ||
| delete process.env.ALTIMATE_FREE_GATEWAY_URL | ||
| process.env.ALTIMATE_BASE_GATEWAY_URL = gatewayUrl |
There was a problem hiding this comment.
P3: resetGatewayEnv() permanently mutates process env: it deletes ALTIMATE_FREE_GATEWAY_URL and sets ALTIMATE_BASE_GATEWAY_URL to the fake gateway URL, but no teardown restores the prior values. isolateAltimateBaseHome()'s afterAll only restores the ISOLATED_ENV keys, so in a shared bun test worker the fake gateway URL stays set and the legacy var stays deleted after the suite. Follow the file's own restoration pattern: capture the original values once (e.g., alongside the ISOLATED_ENV snapshot in isolateAltimateBaseHome) and restore them in the afterAll.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/opencode/test/altimate/_fixtures/altimate-base-harness.ts, line 59:
<comment>resetGatewayEnv() permanently mutates process env: it deletes ALTIMATE_FREE_GATEWAY_URL and sets ALTIMATE_BASE_GATEWAY_URL to the fake gateway URL, but no teardown restores the prior values. isolateAltimateBaseHome()'s afterAll only restores the ISOLATED_ENV keys, so in a shared `bun test` worker the fake gateway URL stays set and the legacy var stays deleted after the suite. Follow the file's own restoration pattern: capture the original values once (e.g., alongside the ISOLATED_ENV snapshot in isolateAltimateBaseHome) and restore them in the afterAll.</comment>
<file context>
@@ -0,0 +1,95 @@
+export function resetGatewayEnv(gatewayUrl: string): void {
+ delete process.env.ALTIMATE_BASE_GATEWAY_URL
+ delete process.env.ALTIMATE_FREE_GATEWAY_URL
+ process.env.ALTIMATE_BASE_GATEWAY_URL = gatewayUrl
+}
+
</file context>
| @@ -0,0 +1,672 @@ | |||
| # Altimate Base — E2E Test Suite: Spec, Harness Design, Parallel Partition | |||
|
|
|||
| Status: Phase 1 (research + design) complete. Not yet implemented. | |||
There was a problem hiding this comment.
P3: The doc's status header says "Not yet implemented" and describes itself as the contract for implementer agents, but every suite and fixture it specifies is implemented and shipped in this same PR. Its Deliverable 2 example and "Cross-file consent isolation" section also instruct each file to claim issueArmer() once at module scope, whereas the shipped harness uses a shared consented() singleton that claims the process's single armer — the exact collision this PR was built to fix. Update the status to reflect that the design is implemented and that the consent helper is a per-process consented() shared across files, not a per-file issueArmer(), so a future reader doesn't follow the stale design.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At docs/internal/2026-09-04-altimate-base-e2e-harness-plan.md, line 3:
<comment>The doc's status header says "Not yet implemented" and describes itself as the contract for implementer agents, but every suite and fixture it specifies is implemented and shipped in this same PR. Its Deliverable 2 example and "Cross-file consent isolation" section also instruct each file to claim `issueArmer()` once at module scope, whereas the shipped harness uses a shared `consented()` singleton that claims the process's single armer — the exact collision this PR was built to fix. Update the status to reflect that the design is implemented and that the consent helper is a per-process `consented()` shared across files, not a per-file `issueArmer()`, so a future reader doesn't follow the stale design.</comment>
<file context>
@@ -0,0 +1,672 @@
+# Altimate Base — E2E Test Suite: Spec, Harness Design, Parallel Partition
+
+Status: Phase 1 (research + design) complete. Not yet implemented.
+Scope: PR #1199, branch `codex/altimate-base-release-final`.
+Author: research/design pass, 2026-09-04. No test code was written by this pass — this
</file context>
| return new Response("", { status: 401 }) | ||
| case "server-error": | ||
| return new Response("upstream error", { status: mode.status ?? 500 }) | ||
| case "timeout": |
There was a problem hiding this comment.
P3: The timeout chat mode never settles when the request has no AbortSignal, and also hangs if the signal is already aborted before the listener attaches. Check the signal and reject immediately in those cases so a mis-scripted test degrades to a clear rejection instead of a hang until Bun's global timeout.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/opencode/test/altimate/_fixtures/fake-gateway.ts, line 151:
<comment>The `timeout` chat mode never settles when the request has no AbortSignal, and also hangs if the signal is already aborted before the listener attaches. Check the signal and reject immediately in those cases so a mis-scripted test degrades to a clear rejection instead of a hang until Bun's global timeout.</comment>
<file context>
@@ -0,0 +1,174 @@
+ return new Response("", { status: 401 })
+ case "server-error":
+ return new Response("upstream error", { status: mode.status ?? 500 })
+ case "timeout":
+ return new Promise<Response>((_resolve, reject) => {
+ init?.signal?.addEventListener("abort", () => reject(init.signal?.reason), { once: true })
</file context>
Suggested change
| case "timeout": | |
| case "timeout": { | |
| const signal = init?.signal | |
| if (signal?.aborted) return Promise.reject(signal.reason) | |
| return new Promise<Response>((_resolve, reject) => { | |
| signal?.addEventListener("abort", () => reject(signal.reason), { once: true }) | |
| }) | |
| } |
|
|
||
| expect(gateway.registerCalls).toHaveLength(1) | ||
| const sentVersion = gateway.registerCalls[0]!.cliVersion | ||
| expect(sentVersion).toBe(FreeTier.sanitizeCliVersion(Installation.VERSION)) |
There was a problem hiding this comment.
P3: The primary assertion in the cli_version test is a tautology: sentVersion is computed by the client as exactly sanitizeCliVersion(Installation.VERSION), so comparing it to the same re-called function can never fail independently and gives false confidence that the sanitizer contract is being exercised. The only meaningful check is the following regex. Make the assertion self-contained by hardcoding an expected value, or rely on the regex alone and drop the self-comparison.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/opencode/test/altimate/altimate-base-registration-gaps.test.ts, line 143:
<comment>The primary assertion in the cli_version test is a tautology: `sentVersion` is computed by the client as exactly `sanitizeCliVersion(Installation.VERSION)`, so comparing it to the same re-called function can never fail independently and gives false confidence that the sanitizer contract is being exercised. The only meaningful check is the following regex. Make the assertion self-contained by hardcoding an expected value, or rely on the regex alone and drop the self-comparison.</comment>
<file context>
@@ -0,0 +1,176 @@
+
+ expect(gateway.registerCalls).toHaveLength(1)
+ const sentVersion = gateway.registerCalls[0]!.cliVersion
+ expect(sentVersion).toBe(FreeTier.sanitizeCliVersion(Installation.VERSION))
+ // sanitizeCliVersion's contract: only these characters survive, capped at 32 chars, never empty.
+ expect(sentVersion).toMatch(/^[A-Za-z0-9._+-]{1,32}$/)
</file context>
| import fs from "node:fs" | ||
| import os from "node:os" | ||
| import path from "node:path" | ||
| import { consented } from "./_fixtures/altimate-base-harness" |
There was a problem hiding this comment.
P3: Use isolateAltimateBaseHome("altimate-base") here and remove the inline environment setup so this file has the same single isolation implementation as the new suites.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/opencode/test/altimate/altimate-base.test.ts, line 6:
<comment>Use `isolateAltimateBaseHome("altimate-base")` here and remove the inline environment setup so this file has the same single isolation implementation as the new suites.</comment>
<file context>
@@ -3,6 +3,7 @@ import { createHash, randomBytes } from "node:crypto"
import fs from "node:fs"
import os from "node:os"
import path from "node:path"
+import { consented } from "./_fixtures/altimate-base-harness"
const isolatedEnvironment = [
</file context>
6 tasks
anandgupta42
merged commit Sep 7, 2026
5cadd14
into
codex/altimate-base-release-final
32 checks passed
anandgupta42
added a commit
that referenced
this pull request
Sep 7, 2026
* feat: release Altimate Base hosted model Rebase of PR #1199 (codex/altimate-base-release-final) onto current origin/main. Squash-merged the 37 PR commits against main and resolved the single content conflict in packages/opencode/src/session/llm.ts: main's canonical `requestHeaders` (from mergeRequestHeaders, also used by the output-token budget estimator) is now wrapped with withManagedSessionHeaders() so Altimate Base (altimate-free provider) requests still get the session-scoped X-Session-Id header, instead of the PR's redundant inline header-object reconstruction. No other files had real conflicts; ci.yml/release.yml picked up both this branch's ALTIMATE_BASE_GATEWAY_URL env additions and main's unrelated changes cleanly via auto-merge. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VqnuBDGkh1ZT65Ti7e6DHZ * fix: address PR #1199 review threads — unforgeable consent gate + model-precedence fixes Rebased onto current origin/main (clean, no conflicts). Addresses all 8 unresolved review threads on the Altimate Base release PR. Security (P1, cubic + codex duplicate threads): the exported `ConsentCapabilityStore` let any in-process caller construct their own store, arm it with a fabricated token, and pass it straight to `registerAfterConsent()`, bypassing the disclosure gate entirely. Redesigned so `registerAfterConsent` takes a bare token and checks it against a private, module-scoped authority in `capability.ts` that is never exported. The only way to arm or redeem that authority is `issueArmer()`/`issueRedeemer()`, each claimable exactly once per process — claimed once by the TUI worker's consent gate at boot and once by `client.ts` at module load. A caller can still construct their own `ConsentCapabilityStore` (kept exported for direct unit tests of its TTL/eviction mechanics), but it is inert: a self-armed store only ever validates against itself, never against what `registerAfterConsent` actually checks. New tests prove both that a forged token is rejected and that no in-process caller can obtain a second armer/redeemer. Also fixes, per review: - Namespace projection: all consumers (`client.ts`, `worker.ts`, tests) now import `FreeTierCapability` via `export * as` and reference members through it, per AGENTS.md's flat-export convention (superseded by the security redesign above). - ACP default-model resolution (`acp/service.ts`): a project that sets `model: "altimate-free/altimate-base"` alongside any `provider` allowlist got a `defaultModel` resolved against the unfiltered provider map, even though the same allowlist had just excluded Altimate Base from the advertised snapshot — so ACP still selected and routed to it. Now resolves against the same filtered `snapshotProviders` used for the catalogue. - Preserve an explicit Big Pickle re-selection (`tui/context/local.tsx`, `tui/app.tsx`): a user who registers Altimate Base and later deliberately picks Big Pickle through `/model` had that choice silently overwritten on the next launch, because it persisted through the same fields the retired implicit default used. Adds a separate `explicitDefault` marker, set only by an interactive picker, that legacy migration now checks before treating a persisted Big Pickle selection as the old implicit default. - First-run picker race (`tui/app.tsx`, P2): the first-run picker effect was missing the `local.model.ready` gate its sibling migration effect already has, so it could misclassify a returning Big Pickle user as a fresh install if provider sync settled before `model.json`'s async read finished. - 401-counter reset (`altimate/free/client.ts`, P2): a non-401 response that also wasn't a 2xx (429/503/etc.) left the consecutive-401 counter untouched instead of resetting it, so an unrelated rate-limit/outage response between two real 401s could still push a healthy credential past the persistence threshold. - Wrong logout command in the disclosure (`tui/component/altimate-onboarding.tsx` + two docs pages): `/providers logout` is not a TUI slash command — only `/logout` exists, and it signs out of the paid gateway, not Altimate Base. Points to the actual CLI command, `altimate providers logout altimate-base`. Verification: typecheck clean across all 15 workspace packages; upstream marker check clean (`--base origin/main --strict`); altimate/free + acp + provider suites green (6542 tests, 0 fail); full tui package suite green (285 tests, 0 fail). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VqnuBDGkh1ZT65Ti7e6DHZ * fix: align altimate-base static limits to the gateway's served 131072/65536 `provider.ts`: update the altimate-base static model definition's declared `limit` from `{context: 65_536, output: 4_096}` to `{context: 131_072, output: 65_536}` to match what the gateway actually serves and advertises (Qwen3.8-27B natively supports 262144, so 131072 needs no YaRN scaling on the gateway side). This is the client's offline/fallback value only — the gateway is the source of truth for what it actually serves. Keep the two in sync so they never disagree. Output is a clean half of the context window rather than the model's theoretical max because this is a reasoning+coding model (reasoning tokens count toward output) and 65536 keeps a real free-tier cost guardrail while guaranteeing ample input room. Updates the matching assertion in provider.test.ts. This addresses the root cause of "Context budget exceeded" on Altimate Base's first message: the CLI's default agent prompt (~77-78K tokens) never fit inside the old declared 65,536-token window. This client-side change alone is not sufficient — the gateway must also actually serve context 131072 / output 65536 for the launch blocker to resolve end to end; that change is being made separately on the gateway side. The gateway endpoint embedded into release builds still comes only from the ALTIMATE_BASE_GATEWAY_URL build-time repo variable, never from source. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VqnuBDGkh1ZT65Ti7e6DHZ * feat: retire Big Pickle, default free slot to Altimate Base; accurate data-handling disclosure + docs; scrub model-name references - Remove Big Pickle as a NEW selectable option from the full model catalog (`dialog-model.tsx`); the migration path for users already on Big Pickle (detection + Altimate Base consent gate) is untouched. - Change the migration decline label to `No — pick something else` for both origins, and update the model-picker note to `free · no signup · rate limited` everywhere it appears (welcome picker, full catalog, `/connect`). - Replace the `ALTIMATE_BASE_DISCLOSURE` consent-gate text with an accurate, shorter disclosure (secrets are masked but shouldn't be relied on; usage is rate limited); the fuller per-install-identifier detail moves to the docs. - Rewrite the Altimate Base section of `providers.md` with an explicit Data handling note (logged/used to improve products including the model, secrets masked, pseudonymous not anonymous per the security FAQ, rate limited) and a contrast sentence pointing to the Altimate LLM Gateway for stronger data-handling guarantees. - Scrub the served model's name from public docs and source comments (`quickstart.md`, `provider.ts`), replacing it with generic phrasing. - Update `dialog-altimate-base.test.tsx` for the new disclosure text and the retired Big Pickle catalog entry. * chore: [AI] scrub model-family metadata from Altimate Base entry - `family: "qwen"` -> `family: "altimate"` on the Altimate Base catalog entry in `provider.ts`. Verified no behavior change: this model's providerID is `altimate-free`, so it never reaches the `providerID === "altimate-backend"` family-vendor switch in `session/system.ts` (prompt selection falls through to the `api.id` check instead), and `familyVendor()` does not map "qwen" to any vendor either way. - Update the test and fixture that pinned the old value: `provider.test.ts` (assertion + test title) and `dialog-altimate-base.test.tsx` (mock fixture). * test: hermetic e2e suite for Altimate Base (registration, catalog, inference, rate-limit/budget, error surfacing) (#1248) * test: add shared hermetic harness for Altimate Base e2e suites Foundation for a 6-suite parallel Altimate Base e2e test partition (see the design doc). Adds `test/altimate/_fixtures/fake-gateway.ts` (a `FakeGateway` that intercepts `fetch` via `spyOn(globalThis, "fetch")` — the repo's existing proven pattern, not a real HTTP server — implementing `/register` and `/v1/chat/completions` with controllable knobs for every failure mode the suites need: per-minute token rate-limit, both `budget_exceeded` variants, request-too-large, 401, 5xx, timeout, malformed JSON, and success) and `test/altimate/_fixtures/altimate-base-harness.ts` (isolated XDG/home bootstrap + gateway-env reset helpers, extracted from `altimate-base.test.ts`'s existing pattern so every suite shares one implementation). Adds `altimate-base-harness-smoke.test.ts` proving the harness works in both directions: a register -> `authorizedFetch` happy-path round trip, and one scripted failure knob (per-minute token rate-limit -> `describeRateLimit`'s non-retryable message). Does not add any of the 6 planned suite files themselves — those are a separate, parallel follow-up. Copies the design doc (`docs/internal/2026-09-04-altimate-base-e2e-harness-plan.md`) into the branch so it travels with the PR. Stacked on `codex/altimate-base-release-final` (#1199) since the harness targets that branch's 131072/65536 limits and Altimate Base code. * test: consolidate Altimate Base e2e suite + centralize test consent-arming Brings together 5 independently-written hermetic e2e suites for Altimate Base onto the shared harness branch (53 tests): - `altimate-base-registration-gaps.test.ts` (11) — HTTP/network/malformed register failure mapping, payload shape, retry idempotency - `altimate-base-catalog.test.ts` (9) — model catalog / provider isolation - `altimate-base-inference-e2e.test.ts` (5) — register -> list -> fetch round trip, placeholder-vs-real-key isolation - `altimate-base-rate-limit-messages.test.ts` (21) — throttle/budget/ request-too-large message mapping - `altimate-base-error-surfacing.test.ts` (7) — 5xx/timeout/abort/ malformed-body/401 pass-through at the inference layer All 5 (plus the two pre-existing files, `altimate-base.test.ts` and `altimate-base-harness-smoke.test.ts`) independently called `FreeTierCapability.issueArmer()` at module scope. That capability is process-global and throws on a second call, so running the directory in one `bun test` invocation — as CI does — threw "Altimate Base consent armer already issued for this process" once a second armer-calling file loaded into the same worker process (reproducible with just the two pre-existing files, before any of these suites existed). Fix: centralize arming in the shared harness (`_fixtures/altimate-base-harness.ts`) behind a new `consented()` helper that lazily calls `issueArmer()` exactly once per process and caches the returned armer in a module-level singleton. Because bun caches modules per process, every suite file that imports `consented()` shares that one cached armer regardless of load order or file count. This adds no way to reset, re-claim, or otherwise weaken the one-shot guarantee `issueArmer()` already enforces — it is a cache in front of the single legitimate call, not a new capability. All 7 armer-calling files now import and use the shared helper instead of claiming their own. Verified with `bun test --timeout 90000 test/altimate/` (the directory CI covers, at CI's timeout) from `packages/opencode`: 5103 pass, 0 fail, zero armer-collision errors, in one process invocation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VqnuBDGkh1ZT65Ti7e6DHZ --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: [AI] preserve explicit Big Pickle picks, apply provider policy to TUI fallback, and fix migration-decline routing Addresses four remaining unresolved review threads on the Altimate Base release PR (several sibling threads on the same functions were already fixed in an earlier pass): - `tui/context/local.tsx` `cycleFavorite()`: a deliberate favorite-cycle pick of Big Pickle after registering Altimate Base persisted through the same `model`/`recent` fields the retired implicit default uses, but bypassed the `explicitDefault` marker that `/model` already sets. Legacy migration then silently overwrote it on the next launch. Routes the pick through the shared `selectModel(..., { recent: true, explicit: true })` path instead of writing the store fields directly, so it is marked explicit like every other picker-driven selection. - `tui/context/local.tsx` `fallbackModel`: did not apply the managed-provider policy `Provider.defaultModel()` already enforces server-side, so a project `provider` block that excludes Altimate Base (e.g. `{ "openai": {} }`) could still have the TUI fall back to it through a persisted recent entry or the first-live-provider selection. Both paths now skip the managed model when `allowsManagedBaseDefault()` says the project has narrowed the allowlist. - `tui/component/altimate-onboarding.tsx`: `no()` for the migration-declined origin only cleared the dialog, leaving the user on the retired Big Pickle model with no way to pick anything else — even though the label read "No — pick something else". Routes a migration decline to the same curated picker a welcome-origin decline already uses (after persisting the refusal via `onDecline`), and collapses the now-identical ternary into a plain label. Updates the one existing test that encoded the old close-and-strand behavior. - `altimate/free/client.ts` `authorizedFetch`: after retrying with a concurrently rotated credential, a non-401 retry response never reset that credential's consecutive-401 counter (only the initial response's non-401 path did). A stale 401 recorded against it elsewhere could then survive a successful retry and later cross the rejection threshold on its own. Mirrors the initial response's reset for the retry response too. Verification: `bun run typecheck` clean; `@opencode-ai/tui`'s onboarding/local/ dialog-model-welcome suites (21 tests) and `@altimateai/altimate-code`'s altimate-base/acp/provider suites (218 tests, 1 pre-existing unrelated failure in altimate-base-catalog.test.ts's model-family assertion, tracked by a separate unresolved review thread) green; marker check clean (`--base origin/main --strict`). * fix: [AI] restore disclosure linkability line, isolate Altimate Base consent from public SDK context Two previously human-held decisions are now approved: - `packages/tui/src/component/altimate-onboarding.tsx` (`ALTIMATE_BASE_DISCLOSURE`): the on-screen consent text a user actually accepts before any registration request must itself disclose that requests are linkable across launches, not defer that to docs a user never sees before accepting. Restores "Logs are linked to a persistent per-installation identifier." before "Usage is rate limited." - Consent-gated registration is no longer reachable through the public SDK context. Previously `sdk.altimateBaseRegistration()` (the callback that arms consent and calls `/altimate/base/register`) was a plain property on the shared `useSDK()` context, exported as `@opencode-ai/tui/context/sdk` — any in-process consumer of that hook, including a plugin-rendered component, could call it directly and mint a Base install identifier / enable request logging without the disclosure dialog ever being shown or accepted. Moves the operation into a new `context/altimate-base-consent.tsx`, deliberately NOT listed in `package.json`'s `exports` map, so `@opencode-ai/tui/context/altimate-base-consent` cannot be resolved from outside this package at all (Node's exports field rejects unlisted subpaths). `app.tsx` still receives the host-injected operation on `TuiInput` and now provides it through this dedicated context instead of through `SDKProvider`. The two legitimate in-package readers — the consent dialog (which calls it, only after acceptance) and the provider picker (which only checks whether it exists, to decide whether to advertise Base setup) — read it from there. `useSDK()` itself no longer carries any property related to this operation. Adds `test/context/altimate-base-consent.test.tsx`, proving the public SDK context object has no such property (forged or otherwise) while the dedicated context does expose it and the legitimate accept flow can still call it; updates the one existing test harness that previously wired the registration callback through `SDKProvider`. Verification: `bun run typecheck` clean; full `@opencode-ai/tui` suite green (286 pass, 1 pre-existing skip, 0 fail, across 57 files, including the new isolation test and the updated onboarding harness). * chore: [AI] wrap the new dialog-provider availability check in altimate_change markers Marker Guard flagged `useAltimateBaseConsent()` in `createDialogProviderOptions()` as unmarked new code in this upstream-shared file. No behavior change. * fix: [AI] close two gaps automated review found in the prior fallback/migration fixes Two automated review findings on the just-pushed fixes, addressed immediately: - `tui/context/local.tsx` `fallbackModel`: the last-resort provider selection only excluded Altimate Base when a project provider allowlist disallowed it, but did not apply the allowlist to any OTHER provider — so it could still land on a connected provider the project never named either. Now filters every fallback candidate by the configured provider keys (mirroring `Provider.defaultModel()`'s `providerAllowed`), in addition to the existing Base-specific check. The `recent` scan is intentionally left as-is: matching `Provider.defaultModel()`'s own comment, a recent entry is the user's own past explicit pick and stays honored for every provider except the managed one, regardless of a later-narrowed allowlist. - `tui/component/altimate-onboarding.tsx` `yes()`: after a successful registration, `migrateLegacyDefault()` re-checks eligibility and can return `false` if a project allowlist or explicit model change landed while the request was in flight — but the success path ignored that result and unconditionally called `markSetupComplete()`, marking a user still on the retired Big Pickle model as ready. Now routes to the curated picker instead of marking setup complete when migration did not happen. Verification: `bun run typecheck` clean; full `@opencode-ai/tui` suite green (286 pass, 1 pre-existing skip, 0 fail). * fix: [AI] tune altimate-base sampling params to match the served model's family The gateway's sampling-tuning by model id in `ProviderTransform` never matched `altimate-base` (the id under which the hosted free model is registered), so it ran with default sampling instead of the values this family already gets elsewhere in the same file — the gateway itself does not force these on requests it doesn't recognize as needing them. - `temperature()`: altimate-base now returns 0.55, matching the row above it. - `topP()`: altimate-base now returns 1, matching the row above it. - `variants()`: altimate-base is now excluded from the reasoning-effort variant list, alongside the other ids already excluded there (matching behavior, not renamed logic). Matched on the literal model id (`id.includes("altimate-base")`) rather than importing a constant from `altimate/free/client.ts`, to keep this foundational, widely-imported file free of any new cross-domain dependency — not because of a confirmed import cycle (checked: `client.ts` and its transitive deps have no path back to `provider/`), but because a wrong call on a file this central is worse than the small duplication. Audited every other id-string check and adjacent reasoning/thinking-token handling in `packages/opencode/src` for the same gap; only these three needed a matching addition. Notably NOT touched: the `alibaba-cn`-specific `enable_thinking` body param (gated on that provider's specific transport quirk, not on any id string — extending it to a different, unverified gateway stack would be a guess) and the static `interleaved` capability on the altimate-base catalog entry (also provider/host-specific per the models registry, not inferrable from an id check, and changing it without confirming the actual gateway behavior risks a correctness regression in multi-turn reasoning replay). Also fixes the one now-in-scope pre-existing test failure: the catalog assertion pinned `model.family` to a value the production catalog no longer sets (scrubbed in bec2ae3); updated to match. Verification: `bun run typecheck` clean; full altimate-base + transform + provider + acp suites green (555 pass, 9 pre-existing skip, 0 fail). * chore: [AI] wrap the new altimate-base sampling checks in altimate_change markers Marker Guard flagged the temperature()/topP() additions as unmarked new code in this upstream-shared file (the variants() addition was already inside an existing marked block). No behavior change. * fix: [AI] make the Altimate Base consent test hermetic against cross-file credential leakage Root-caused the intermittent CI "TypeScript" job failure: `provider HttpApi > advertises Altimate Base for consent without marking it connected`. Pulled the actual failed CI run's log directly — the literal failure is `expect(isRecord(body) && Array.isArray(body.connected) && body.connected.includes("altimate-free")).toBe(false)` -> `Expected: false, Received: true`. Pre-existing since the test was added in `431a3b489b`, well before this branch's other work; confirmed by isolating the single test (passes) vs. running it alongside `test/altimate/*.test.ts` files that perform a REAL `FreeTier.registerAfterConsent()` (fails when they run first in the same `bun test` process). Mechanism: `FreeTierStore.credentialPath()` resolves through the process-wide, non-Instance-scoped `Global.Path.data` — not this test's own isolated `TestInstance` directory. A real registration performed by an earlier Altimate Base suite in the same shared Bun process writes a live credential there; this test never registers anything and reads `FreeTier.credentialsForLoad()` for real (no mock), so it picks up that leftover credential and the custom provider loader marks `altimate-free` `autoload: true`, landing it in `connected` depending on test-file execution order. Not a real production secret leak: `options.apiKey` for `altimate-free` is always `FreeTier.MANAGED_API_KEY_PLACEHOLDER`, never the real credential, regardless of this ordering issue — kept the `not.toContain("sk-")` guard in the test unchanged, since it is a real assertion worth having. Fix: clear any leftover `FreeTierStore` credential (`FreeTierStore.remove()`) at the top of this specific test, before it makes its request — a minimal, targeted isolation fix scoped to the one test that depends on a clean-slate credential store, rather than touching the shared `Global.Path` module (an earlier attempt at a deeper fix there — converting its module-level path consts to lazy getters — broke the eager one-time directory creation many unrelated tests depend on, causing 91 failures across the suite; reverted). Verification: `bun test packages/opencode/test/server/httpapi-provider.test.ts` — 6 pass, 0 fail; run alongside `test/altimate/altimate-base.test.ts` (which performs real registrations) — 40 pass, 0 fail, confirming the isolation now holds regardless of file order; `bun test packages/opencode/test/server/` — only two unrelated pre-existing local-sandbox failures (real ambient MCP config on this machine bleeding into `httpapi-mcp.test.ts`/experimental HttpApi tests, absent in CI); `bun run typecheck` clean. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue for this PR
Closes #1247
Type of change
What does this PR do?
Adds 53 hermetic e2e tests across 5 suites for Altimate Base's free-tier client (
src/altimate/free/*), consolidated onto the shared_fixtures/altimate-base-harness.ts+_fixtures/fake-gateway.tsharness already on this branch:altimate-base-registration-gaps.test.ts(11) — HTTP 4xx/5xx/network/malformed-JSON register failure mapping ontoRegistrationError, the exact request payload (hashed install secret,cli_version), and retry idempotency.altimate-base-catalog.test.ts(9) — model catalog / provider isolation (Provider.list()/Provider.defaultModel()/Provider.sort()) against a credential minted through the real consent + register path.altimate-base-inference-e2e.test.ts(5) — full register → provider-list →authorizedFetchround trip, plus the placeholder-vs-real-key and credential-storage isolation properties.altimate-base-rate-limit-messages.test.ts(21) — everydescribeRateLimit/describeRequestTooLargebranch: per-minute token throttle, burst throttle, wallet/global/unknown budget, request-too-large byte math, and the malformed/unrecognized fallback paths.altimate-base-error-surfacing.test.ts(7) — non-rate-limit inference-time failures reachauthorizedFetch's caller cleanly: 5xx pass-through, timeout/abort propagation, raw connection failure, malformed JSON body, and chat-time 401.All suites are fully hermetic —
fetchis injected via the sharedFakeGatewayfixture (spyOn(globalThis, "fetch")), so there is no live gateway, no credentials, and no network access anywhere in this PR. Each suite runs in its own isolated XDG/home tree (isolateAltimateBaseHome) so credential stores, config, and cache never collide across files or touch a real user directory. This is picked up by CI automatically via the existingtest/altimate/**path filters — no workflow change needed.The centralized-arming fix (the load-bearing change in this PR):
FreeTierCapability.issueArmer()is a process-global capability that throws on a second call in the same process (by design — it's the security property that makes Altimate Base's consent gate unforgeable, seesrc/altimate/free/capability.ts). Every one of these 5 new suites, plus the two pre-existing files (altimate-base.test.ts,altimate-base-harness-smoke.test.ts), independently calledissueArmer()at module scope.bun test test/altimate/loads multiple test files into one worker process, so this throws"Altimate Base consent armer already issued for this process"as soon as a second armer-calling file loads — reproducible with just the two pre-existing files, before any of these suites existed.Fixed by adding a
consented()helper to the shared harness (_fixtures/altimate-base-harness.ts) that lazily claimsissueArmer()exactly once per process and caches the returned armer in a module-level singleton. Because bun caches modules per process, every suite file that importsconsented()— regardless of load order or how many files load it — shares that one cached armer. All 7 armer-calling files (5 new + 2 existing) now go through this shared helper instead of claiming their own armer. This adds no way to reset, re-claim, or otherwise weaken the one-shot guaranteeissueArmer()already enforces; it is purely a cache in front of the single legitimate call, so the underlying security property (only one in-process caller can ever obtain the ability to arm the production consent authority) is unchanged — the existing "unforgeable consent" test inaltimate-base.test.ts(which asserts a secondissueArmer()/issueRedeemer()call throws) still passes unmodified.Live-prod smoke testing against a real gateway is intentionally out of scope here and tracked separately / non-blocking for this PR — everything in this PR is fetch-injected and hermetic.
This PR stacks on #1199 (Altimate Base hosted model release) — the base is
codex/altimate-base-release-finalso the diff here shows only the harness + tests, not #1199's feature changes. It should be retargeted tomainonce #1199 merges.How did you verify your code works?
Ran the full
test/altimate/directory in onebun testinvocation, matching CI's own invocation and timeout (bun test --timeout 90000, frompackages/opencode, per.github/workflows/ci.yml'stypescriptjob):Zero "consent armer already issued" errors — this is the actual acceptance bar, not file-by-file green (each file was also independently confirmed green: 11 + 9 + 5 + 21 + 7 = 53 new tests, all passing).
Also verified:
bun run typecheck(bun turbo typecheck) — clean, 13/13 tasks successful.bun run script/upstream/analyze.ts --markers --base origin/main --strict— clean, no unmarked upstream-shared changes.bunx prettier --checkon all changed files — clean.Screenshots / recordings
N/A — test-only change, no UI.
Checklist
Note
Low Risk
Test-only changes with no runtime behavior modifications; the shared
consented()helper preserves the one-shotissueArmer()security model while fixing multi-file test loading.Overview
Adds a hermetic Altimate Base e2e test layer built on shared
_fixtures/altimate-base-harness.ts(isolated XDG home, gateway env reset) and_fixtures/fake-gateway.ts(spyOn(globalThis, "fetch")for/registerand chat completions with scripted error modes). Five new suites cover registration failure gaps, provider catalog/defaultModel()behavior, register→Provider.list()→authorizedFetchinference, fulldescribeRateLimit/describeRequestTooLargebranches, and inference-time 5xx/timeout/network/malformed-JSON/401 surfacing; a small harness smoke file exercises the fixtures.Load-bearing harness fix: all suites (and the existing
altimate-base.test.ts) now mint consent via sharedconsented(), which lazily claimsFreeTierCapability.issueArmer()once per process sobun test test/altimate/does not crash when multiple files load in one worker.Also adds
docs/internal/2026-09-04-altimate-base-e2e-harness-plan.mdas the design contract. The planned context-clamp suite is not in this diff (still flagged in the doc). No production source changes; tests only, no CI workflow edits.Reviewed by Cursor Bugbot for commit d2973f4. Bugbot is set up for automated code reviews on this repo. Configure here.
Summary by cubic
Adds 53 hermetic e2e tests across 5 suites for Altimate Base's free-tier client (closes #1247), and centralizes the process-global
issueArmer()capability behind a sharedconsented()helper so all test files run together in one worker process without the "consent armer already issued" crash.Coverage
fetchvia the sharedFakeGatewayfixture with isolated XDG/home trees — no network, credentials, or live gateway.bun testinvocation at CI's timeout (5103 pass, 0 fail), and the diff is test-only since it stacks on feat: release Altimate Base hosted model #1199 — retarget tomainonce that merges.Consent armer
issueArmer()throws on a second in-process call by design; 7 files each claiming it at module scope crashed under bun's single-worker test loading.consented()lazily claims the armer once per process and caches it, so all 7 files share it — the unforgeable-consent guarantee is unchanged and still covered by an existing test.Written for commit d2973f4. Summary will update on new commits.