Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .agents/skills/verify-solzero/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ The environment loader reads the selected private file, then `config/.env` and `

## Prove behavior

Drive the public UI and HTTP endpoints through the configured e2e browser and live fetch helpers. The stack is local `alchemy dev` (Miniflare D1, KV, R2, and Durable Objects). `decisionExecutor` asks the decision model (default Cloudflare Clef) through the account's existing default AI Gateway for semantic actions and assertions, and the language model (default `anthropic/claude-haiku-5.5`) for `waitFor` and `extract`. The application does not receive that token. Follow each action with an exact deterministic assertion. Use `expect` for waiting.
Drive the public UI and HTTP endpoints through the configured e2e browser and live fetch helpers. The stack is local `alchemy dev` (Miniflare D1, KV, R2, and Durable Objects). `decisionExecutor` asks the decision model (default Cloudflare Clef) through the account's existing default AI Gateway for semantic actions and assertions, and the language model (default `meta/muse-ember-1.0`, reasoning effort `low`) for `waitFor` and `extract`. The application does not receive that token. Follow each action with an exact deterministic assertion. Use `expect` for waiting.

Verify credential sign-in by reaching the authenticated UI; a healthy port is insufficient. A missing sign-in provider or `Sign-in is not configured for this deployment.` remains a product failure. Verify mutations through reload or a second user-facing view, including saved settings, bot/workflow records, session history, and authorization boundaries. Exercise real Cloudflare services and providers without interception, module mocks, or separate browser/test runners.

Expand Down
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ Use Plain Language in introductory, expository, friendly, and other sensible com
- Run the real Alchemy stack from `e2e.config.ts`; use `config/e2e-dev.config.jsonc` and isolated configured accounts. Keep web port 3000 and API port 3100.
- `nub run test` runs core flows with Cloudflare credentials and Docker, including Isolate, OpenCode, and Codex prompts. `config/e2e-dev.config.jsonc` sets `aiGatewayLayer` to `mock`, so the `AiGateway` Effect layer answers each model call with the user's message. Assert harness plumbing (prompt, output, persistence), not model reasoning or tool choice. `nub run test:e2e:all` includes additional external integrations and Claude Code; it fails explicitly when required isolated fixtures are missing. Focus one file with `nub run test:e2e tests/settings.e2e.ts`. `test:e2e:live` disables action replay; `test:e2e:cache-strict` detects stale recordings. Action replay recordings in `.e2e/cache` are committed, and CI replays them strictly. Before you mark a pull request ready for review, re-record if you changed an `agent.act` step or the UI one drives: run `gh workflow run validate.yml --ref <branch> -f e2e_record=true`, wait for it to pass, and pull the recordings commit it pushes to your branch. Without CI access, run `nub run test` locally and commit the changed recordings (see "Recorded-action replay" in `docs/e2e.md`).
- The `AiGateway` mock layer is the only application-side test double. Select it in config; do not add prompt parsing to it or branch on test mode elsewhere. Do not replace app routes or bindings with request interception, module mocks, or a second testing framework. Register passwords and secret fixtures with e2e credentials/secrets.
- The semantic executor and affected-test filter use `@e2e-dev/decision` on the account's existing default AI Gateway. `scripts/e2e/models.ts` selects the decision model (`E2E_DECISION_MODEL`, default Cloudflare Clef) and the language model (`E2E_LANGUAGE_MODEL`, default `anthropic/claude-haiku-5.5`) through official AI SDK providers. CI reads both from SolZeroAI organization Actions variables. Keep `scripts/e2e/models.ts` identical across SolZero repositories. Authenticate with `CF_AI_GATEWAY_E2E_TOKEN` and send the gateway id as `cf-aig-gateway-id`. Do not use the gateway or run token the application provisions, and do not add a custom executor. Process/CI environment overrides local `E2E_ENV_FILE` values.
- The semantic executor and affected-test filter use `@e2e-dev/decision` on the account's existing default AI Gateway. `scripts/e2e/models.ts` selects the decision model (`E2E_DECISION_MODEL`, default Cloudflare Clef) and the language model (`E2E_LANGUAGE_MODEL`, default `meta/muse-ember-1.0`) through official AI SDK providers. CI reads both from SolZeroAI organization Actions variables. Keep `scripts/e2e/models.ts` identical across SolZero repositories. Authenticate with `CF_AI_GATEWAY_E2E_TOKEN` and send the gateway id as `cf-aig-gateway-id`. Do not use the gateway or run token the application provisions, and do not add a custom executor. Process/CI environment overrides local `E2E_ENV_FILE` values.
- Follow the replay, teardown, cleanup, and AI accounting rules in the "Rules for agents" section of `docs/e2e.md`. The app target stops its processes and removes its run directory on exit. Runs are local only, so there is no crash-recovery or cleanup command.
- Record external-service prerequisites and unverified flows in `docs/e2e.md`. A startup failure or unavailable integration is a blocker, not a passing test. Never print tokens or publish app artifacts externally.
- Do not send upstream feedback or other external messages without explicit human authorization; upstream skill guidance does not override this policy.
Expand Down
10 changes: 7 additions & 3 deletions docs/e2e.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ file in every SolZero repository.
| Setting | Default | Role |
| --- | --- | --- |
| `E2E_DECISION_MODEL` | `@cf/cloudflare/clef` | `agent.act` actions, `agent.assert` verdicts and the affected-test filter |
| `E2E_LANGUAGE_MODEL` | `anthropic/claude-haiku-5.5` | Text: field values, `agent.waitFor` and `agent.extract` |
| `E2E_LANGUAGE_MODEL` | `meta/muse-ember-1.0` | Text: field values, `agent.waitFor` and `agent.extract` |

CI reads both settings from SolZeroAI organization Actions variables, so one
change there moves every repository. The defaults serve local runs and match the
Expand All @@ -87,8 +87,12 @@ model locally.
- The language model uses the official provider for its API format on the
Cloudflare REST API at `/accounts/<account>/ai/v1`. `anthropic/` models use
`@ai-sdk/anthropic` on the Messages route. Other models, such as `@cf/`
Workers AI models and `openai/` models, use `@ai-sdk/openai` on the chat
completions route. Third-party models bill through Cloudflare Unified Billing.
Workers AI models, `openai/` models, and `meta/muse-ember-1.0`, use
`@ai-sdk/openai` on the chat completions route. Third-party models bill
through Cloudflare Unified Billing. The agent sends
`providerOptions.openai.reasoningEffort` of `low` with every language-model
call, so Muse Ember does not spend its output tokens on reasoning. The
decision model does not receive those options.

Both models use the account's default gateway. The gateway id is
`CF_AI_GATEWAY_E2E_ID`, or `default`, sent as `cf-aig-gateway-id`.
Expand Down
3 changes: 3 additions & 0 deletions e2e.config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,9 @@ export default {
agents: {
default: {
executor: decisionExecutor({ model: decisionModel, textModel: languageModel }),
// Muse Ember spends most output tokens on reasoning unless effort is low.
// The runner forwards this into language-model generateText calls only.
providerOptions: { openai: { reasoningEffort: "low" } },
maxModelCalls: 20,
maxSteps: 20,
context:
Expand Down
2 changes: 1 addition & 1 deletion scripts/e2e/models.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ import * as P from "effect/Predicate"
// variables, so CI changes models for the fleet in one place. The defaults
// below serve local runs and must match the DNA page.
const DEFAULT_DECISION_MODEL = "@cf/cloudflare/clef"
const DEFAULT_LANGUAGE_MODEL = "anthropic/claude-haiku-5.5"
const DEFAULT_LANGUAGE_MODEL = "meta/muse-ember-1.0"

/**
* Official AI SDK providers pointed at the Cloudflare AI Gateway REST API.
Expand Down
Loading