feat: add TypeSafe authentication and opt-in evaluations

Cargo.toml

@@ -77,6 +77,7 @@ ratatui = { version = "=0.30.2", default-features = false, features = ["crosster
 ratatui-image = { version = "=11.0.8", default-features = false, features = ["crossterm"], optional = true }
 reqwest = { version = "=0.13.5", default-features = false, features = ["blocking", "form", "rustls", "stream"] }
 rmcp = { version = "=3.2.0", default-features = false, features = ["auth", "client"] }
+rpassword = "=7.5.4"
 serde = { version = "=1.0.229", features = ["derive"] }
 serde_json = "=1.0.151"
 shlex = { version = "=2.0.1", optional = true }

docs/user/evaluations.md

@@ -0,0 +1,80 @@
+# Evaluate content with named questions
+
+Use `eval` to classify content, rate it against a rubric, or estimate whether a
+condition holds. One call evaluates multiple independent questions against the
+same supplied state and returns named answers.
+
+## Enable evaluations
+
+Evaluations are off by default. Enable them in your user configuration only:
+
+```sh
+kit config set experimental.eval true
+kit config set credential_store keychain
+kit auth login typesafe
+```
+
+These settings persist in `~/.kit/config.toml`: evaluations are enabled and
+login and subsequent Kit runs use the same persistent `keychain` credential
+store. Do not override that store when starting Kit. See
+[TypeSafe authentication](getting-started-and-configuration.md#typesafe-api-key)
+for the API key prerequisite and the file-store alternative; if using that
+alternative, persist both `credential_store` and `credential_dir` with
+`kit config set` so login and subsequent runs share the same store. A project's
+configuration cannot enable evaluations. You can use `TYPESAFE_API_KEY` instead
+of a saved key; a nonempty environment value takes precedence. Restart Kit after
+enabling the feature or adding a key. Without both opt-in and an available key,
+`eval` is not offered to the agent.
+
+**Each evaluation sends the supplied state and questions to TypeSafe and uses
+your TypeSafe quota.** Supply only content you intend to share. Kit does not
+choose content automatically, compact conversations, or filter other tool
+results through evaluations. Evaluations introduce no content logging or
+telemetry. As with other tool calls, inputs and results are part of the session.
+
+Disable with `kit config set experimental.eval false` and restart Kit.
+
+## Ask several questions in one call
+
+`eval` is a hidden Compose tool, not a separate Runlet expression:
+
+```text
+return eval({
+  state: "My invoice was charged twice. Please help.",
+  questions: {
+    urgent: { type: "noul", instructions: "Does this require immediate attention?" },
+    department: {
+      type: "choice",
+      instructions: "Which department should handle this?",
+      criteria: { billing: "Payments and invoices", support: "Product help" }
+    },
+    frustration: {
+      type: "score",
+      instructions: "How frustrated is the customer?",
+      criteria: ["Calm", "Frustrated", "Very angry"]
+    }
+  }
+})
+```
+
+- **Noul** returns `noul`, the probability of yes from 0 to 1. Optional `criteria`
+  can describe `true` and `false`.
+- **Choice** returns `choice`, option `probabilities`, and `confidence`. Supply
+  1–255 named options; a description can be `null`.
+- **Score** returns `score`, level `probabilities`, `legend`, and `confidence`.
+  Supply 2–10 ordered levels. Levels start at zero; scores can fall between them.
+
+State and instructions accept text, objects, or arrays. Each result includes the
+resolved Jev model version and input/output token usage. The selected model is
+`jev-latest`. Answers are probabilistic assessments, not guarantees. This tool
+does not generate arbitrary JSON schemas.
+
+A call accepts up to 64 questions and 256 KiB of encoded input. Results are
+limited to 1 MiB. Up to four evaluations run concurrently per runtime; each
+call has a 30-second deadline, including time waiting to start. Kit sends one
+request per call, does not split or batch calls, and never automatically retries.
+Cancellation stops waiting but cannot undo an evaluation already submitted;
+a cancelled, timed-out, or failed call may still use quota.
+
+See the [TypeSafe API reference](https://docs.typesafe.ai/api.md) for the question
+and answer definitions.

docs/user/getting-started-and-configuration.md

@@ -87,6 +87,36 @@ kit prompt --provider openrouter \
 
 Kit uses the CLI or TOML `model`; `OPENROUTER_MODEL` does not override that selection. The adapter also accepts `OPENROUTER_BASE_URL`, `OPENROUTER_APP_NAME`, `OPENROUTER_SITE_URL`, `OPENROUTER_MAX_COMPLETION_TOKENS`, `OPENROUTER_TEMPERATURE`, and `OPENROUTER_REASONING_EFFORT`. Its model-catalog lookup for the selected model's context length is best-effort, so a catalog failure does not by itself prevent normal provider usage.
 
+### TypeSafe API key
+
+TypeSafe authentication is BYOK (bring your own key). Create a key at
+<https://console.typesafe.ai/keys>, then enter it at the hidden terminal prompt:
+
+```sh
+kit auth login typesafe --credential-store keychain
+kit auth status typesafe --credential-store keychain
+kit auth logout typesafe --credential-store keychain
+```
+
+Use the same storage option for all three commands. You can instead
+use `--credential-store file --credential-dir /path/to/private-directory`.
+Login requires `keychain` or `file` to save your key. Enter the key only at the
+hidden prompt, not in command-line arguments or `config.toml`.
+
+Status gives a nonempty `TYPESAFE_API_KEY` environment variable precedence over a
+stored key; an empty variable does not override stored credentials. Login still
+prompts and saves a key when the variable is set, and warns about this precedence.
+Login checks the key with TypeSafe before saving it, without running inference
+or incurring inference charges. If authentication fails, your previously saved
+key is unchanged. Successful login confirms authentication. Status reports
+whether a key is configured; it does not check whether that key is still valid.
+
+Logout removes only the stored key and warns if `TYPESAFE_API_KEY` remains active.
+Revoke the key separately at <https://console.typesafe.ai/keys>; `--local-only`
+suppresses that reminder. These commands manage credentials only: TypeSafe is not
+a model provider, and this does not enable Jev inference, compaction, filtering,
+or behavior configuration.
+
 ### Speakeasy AI Control Plane
 
 Sign in through the Speakeasy dashboard, then use the same persistent credential
@@ -369,3 +399,10 @@ or task deadline.
 The setting is resolved once per Kit process. Native Kit ACP children and the
 TUI's Kit server receive the exact resolved value as a CLI argument, for both
 new and resumed sessions; their local TOML cannot override it.
+
+### Experimental evaluations
+
+`experimental.eval` defaults to `false`. Enable it in your user configuration to
+ask named classification, scoring, and yes/no questions with TypeSafe. A TypeSafe
+key is also required. Each call shares its supplied content with TypeSafe and
+uses your quota. See [Evaluations](evaluations.md) for setup, examples, and limits.

src/docs.rs

@@ -416,6 +416,7 @@ mod tests {
             vec![
                 "docs/user/agent-plugins.md",
                 "docs/user/compose-and-local-tools.md",
+                "docs/user/evaluations.md",
                 "docs/user/getting-started-and-configuration.md",
                 "docs/user/mcp.md",
                 "docs/user/migrating-from-claude-code-and-codex.md",