Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CLAUDE.md

Large diffs are not rendered by default.

51 changes: 51 additions & 0 deletions docs/api-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -516,6 +516,57 @@ All four enforce session ownership in multi-user mode; a foreign session id
answers `404 NOT_FOUND` (no existence leak), and profiles of two owners of the
same directory are distinct by construction.

## Custom Model Endpoints

Points a session's harness at a user-configured OpenAI-compatible endpoint —
local (llama.cpp, vLLM, DGX Spark) or cloud (Azure AI Foundry, OpenRouter) —
instead of its native cloud backend, gated by the opt-in
`customModelEndpointsEnabled` setting (default OFF). Endpoints are
machine-level infra, like remote/docker hosts: writes are admin-only in
multi-user mode. Design: [`custom-model-endpoints-plan.md`](custom-model-endpoints-plan.md);
user guide: [`custom-model-endpoints.md`](custom-model-endpoints.md).

- `GET /api/v1/model-endpoints` -> `CustomModelHost[]`, an unwrapped bare
array like every other list route (still riding the standard `{success,
data}` envelope on the wire — unwrap it the same way). Answers `[]` for a
non-admin in multi-user mode. `apiKey` is never returned; `apiKeySet:
boolean` reports whether one is stored, so a client can render "unchanged
if left blank" without ever holding the real value.
- `POST /api/v1/model-endpoints` with `{ id, label, baseUrl, apiKey?,
authStyle?, defaultModelId? }` creates one. `id` must match
`^[a-zA-Z0-9_-]+$`; `authStyle` is `bearer` (default) or `api-key`, never
both (a real server hung indefinitely when sent both headers on one
request); `baseUrl` must be `http(s)`, carry no embedded credentials, and
is refused if it points at (or resolves to) a link-local or
cloud-metadata address. `409 ALREADY_EXISTS` on a duplicate id.
- `PUT /api/v1/model-endpoints/:id` updates one. An **absent** `apiKey`
keeps the stored one rather than clearing it — the client never receives
the real value to resend deliberately unchanged, so omission is the only
way to say "leave it alone"; there is no way to clear a key back to unset
this way. `defaultModelId`, when set, must be one of that endpoint's own
`models` (`400 INVALID_INPUT` otherwise).
- `DELETE /api/v1/model-endpoints/:id` removes one.
- `POST /api/v1/model-endpoints/:id/discover-models` fetches the endpoint's
own `GET /v1/models` and stores the result as `models`, updating
`lastDiscoveredAt`. A `defaultModelId` that no longer appears in the fresh
list is dropped rather than carried forward invalid. Failures answer
`502 OPERATION_FAILED` with the underlying connection error, or a named
egress refusal if the resolved address turned out to be blocked. The same
refresh also runs automatically for every saved endpoint every 5 minutes
in the background (`refreshAllCustomModelHosts()`, `custom-model-routes.ts`,
started from `server.ts`), so there is no route for triggering "refresh
all" — one endpoint being unreachable on a cycle never blocks the others.
- `POST /api/v1/sessions/:id/custom-model` with `{ endpointId, modelId } |
{ clear: true }` applies (or clears) the session's selection and
**restarts the session's CLI process in place** — every supported harness
reads its endpoint config at process start, never per turn, so there is
no live hot-swap. A Claude session resumes its existing conversation
across the restart; pi/omp/grok additionally get a forced `--model`/`-m`
value, since for those three the config file alone does not select it.
`400 INVALID_INPUT` for a remote (SSH) or Docker session — both restart
their agent differently under the hood, and applying to one would report
success while changing nothing.

## Voice dictation

Browser dictation transcribed through this server's Claude Code login, i.e. the
Expand Down
8 changes: 8 additions & 0 deletions docs/custom-model-endpoints-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -208,6 +208,14 @@ extra per-model configuration on Codeman's side at all.

### 4. Toolbar UI

> **Superseded.** This section describes the toolbar-button design as originally
> planned. What actually shipped is a Run-menu picker instead: one generated entry
> per (capable harness, saved endpoint) pair directly in the existing `#runModeMenu`
> dropdown, rather than a separate `#customModelBtn`/`#customModelMenu` surface. See
> [`docs/custom-model-endpoints.md`](custom-model-endpoints.md#the-run-menu-picker)
> for the current design; the sections below (session-restart mechanics, security)
> remain accurate regardless of which UI calls the underlying route.

- New header/toolbar button (e.g. `#customModelBtn`, `btn-toolbar
btn-custom-model`), marker-hidden by default (`btn-custom-model--hidden`)
and revealed by `applyHeaderVisibilitySettings()` only when
Expand Down
81 changes: 70 additions & 11 deletions docs/custom-model-endpoints.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,20 +11,21 @@ company gateway) — anything answering `GET /v1/models` and
recipe confidence table, and security reasoning:
[`custom-model-endpoints-plan.md`](custom-model-endpoints-plan.md).

> **Status**: backend is implemented and tested (registry capability, the
> injection engine, the endpoint store + discovery route, the session
> restart route). The toolbar picker / settings UI described below as the
> intended surface is **not yet built** — until it lands, use the HTTP API
> directly (examples below). Antigravity has no known custom-endpoint
> mechanism and is not supported.
> **Status**: fully wired end to end — registry capability, the injection
> engine, the endpoint store + discovery route, the session restart route,
> a settings-panel CRUD surface, and the Run-menu picker described below.
> Antigravity has no known custom-endpoint mechanism and is not supported.
> The HTTP API (examples below) still works directly and is what the picker
> itself calls under the hood.

## Turning it on

App Settings → Agents & CLIs → **Custom Model Endpoints** (synced setting
`customModelEndpointsEnabled`, default **OFF**). Until the toolbar picker
lands, nothing reads this setting: the HTTP routes below work whether it is
on or off, and it exists now only so the picker has a switch to hang off
when it ships. The API equivalent:
App Settings → Models → **Custom model endpoints** (synced setting
`customModelEndpointsEnabled`, default **OFF**). Turning it on does two
things: it reveals the endpoint list/add/edit/discover panel in that same
settings section, and it makes the Run menu offer a generated entry per
(harness, endpoint) pair — see "The Run-menu picker" below. The API
equivalent:

```bash
curl -sk -X PUT https://localhost:3000/api/settings \
Expand All @@ -34,6 +35,9 @@ curl -sk -X PUT https://localhost:3000/api/settings \

## Adding an endpoint

Via App Settings → Models → Custom model endpoints → **+ Add endpoint**, or
directly:

```bash
curl -sk -X POST https://localhost:3000/api/model-endpoints \
-H 'Content-Type: application/json' \
Expand Down Expand Up @@ -62,6 +66,61 @@ configured, `PUT`/`DELETE /api/model-endpoints/:id` update or remove one.
Endpoint management is admin-only in multi-user mode, same as remote/docker
hosts — these are machine-level infra, not per-user settings.

`defaultModelId` names which discovered model the picker pre-marks for that
endpoint — the settings panel's Edit form exposes it as a select populated
from the endpoint's own discovered `models`, and the route refuses a value
that isn't one of them. It is applied automatically only when the endpoint
has exactly one discovered model (nothing to choose); with two or more it
is a pre-selection in the model-picker dialog below, never a silent default.
Re-discovering drops a default that no longer appears in the fresh list
rather than carrying an invalid one forward.

**Model lists refresh themselves.** A background sweep (`server.ts`,
`CUSTOM_MODEL_REDISCOVER_INTERVAL_MS`, every 5 minutes) re-discovers every
saved endpoint the same way the manual `POST .../discover-models` route
does, best-effort per endpoint — one being unreachable on a given cycle
never blocks the others. Off under `npm test`, same reasoning as the Codex
plan-usage poll it sits beside: no real network to hit, no server instance
to keep the timer alive for.

## The Run-menu picker

With the setting on and at least one endpoint carrying a discovered model,
the toolbar's Run dropdown grows a **Custom Endpoints** section: one entry
per (harness that can redirect to a custom endpoint, saved endpoint) pair,
e.g. "Claude Code (llama.cpp)". The harness list is read off the CLI
registry's own `capabilities.customModelInjection` at page render
(`window.__codemanCustomModelClis`, `server.ts`) — never a hardcoded id list
in the frontend — so a CLI whose injection recipe lands later shows up with
no frontend change, and Antigravity (`unsupported`) never does.

Picking an entry re-fetches the endpoint (`selectCustomModelEntry()`,
`session-ui.js`) rather than trusting anything cached from the dropdown's
own render — the model list can have changed via the 5-minute sweep above
or a settings-panel edit since the menu opened. With exactly one discovered
model it runs straight away; with two or more, a small modal
(`#customModelPickModal`) lists them and asks which one to use for this
launch, with the endpoint's `defaultModelId` marked but not auto-chosen —
the point of asking is letting one launch deliberately differ from the
saved default, not just confirming it. Whichever way the model was decided,
the launch itself runs a single session on that harness exactly the way its
own Run-menu entry would (same case creation, env overrides, everything),
then **waits for the new session to go idle** (`GET .../wait?until=idle`,
bounded at 20s — a normal 200 either way, never an error, per the wait
endpoint's own contract) before applying the endpoint and model to it via
the route below. That wait exists because a freshly launched CLI reports
itself as `busy` for its own startup (a boot spinner, a workspace-trust
check) well before the apply call would otherwise reach it, and the apply
route correctly refuses to restart a session mid-turn — a fresh boot looks
exactly like one from the outside. A session still busy after the wait
reaches the apply call anyway and gets that route's own honest
`SESSION_BUSY` error, now visible as a sticky toast with a close button
rather than a generic message that vanished in three seconds. It is a
one-off "try this endpoint" action, not a sticky mode: the plain Run button
still means "this harness, native cloud" afterward. Entries are hidden
entirely for a remote or Docker active case, since the apply route refuses
both (see the next section).

## Applying a model to a session

```bash
Expand Down
7 changes: 7 additions & 0 deletions docs/wiki/Agent-CLIs.md
Original file line number Diff line number Diff line change
Expand Up @@ -273,9 +273,16 @@ into the case's `.claude/settings.local.json` so that `/model` keeps working.
- **Shell** for the times you want a terminal on your phone with no agent at all. It is a
genuinely useful mode, not a fallback.

## Pointing one at your own server

Most of these harnesses can also run against a custom OpenAI-compatible endpoint instead of
their native cloud backend, for one session at a time, an opt-in feature covered in full on
[Custom Model Endpoints](Custom-Model-Endpoints).

## Read next

- [Core Concepts](Core-Concepts) - run modes versus location overlays.
- [Custom Model Endpoints](Custom-Model-Endpoints) - run a harness against your own server.
- [Settings Reference](Settings-Reference) - model, effort, and permission-mode settings.
- [Keeping Agents Running](Keeping-Agents-Running) - what idle detection does per mode.
- [Security](Security) - what skipping permission prompts actually means.
100 changes: 100 additions & 0 deletions docs/wiki/Custom-Model-Endpoints.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
# Custom Model Endpoints

Point a harness at your own OpenAI-compatible server instead of its native cloud backend, for
one session at a time. "Custom endpoint" covers **local** hardware (llama.cpp, Ollama, vLLM,
a home GPU rig, DGX Spark, Strix Halo) and **cloud** services (Azure AI Foundry's
OpenAI-compatible endpoint, OpenRouter, a company gateway) alike, anything answering
`GET /v1/models` and `POST /v1/chat/completions` in the standard shape.

**Off by default.** Turn it on in App Settings → Models → **Custom model endpoints**.

## Adding an endpoint

Still in App Settings → Models → Custom model endpoints:

1. **+ Add endpoint** — give it an id, a label, and the base URL (`http://192.168.1.50:8080`,
say). An API key is optional; most local servers don't check one.
2. **Discover** — fetches the endpoint's own model list over `GET /v1/models` and stores it.
3. Pick a **default model** from what was discovered. This is the model the Run-menu entry
applies directly when only one model is discovered; with two or more, it's just the one
pre-marked in the picker dialog described below, not a silent default.

Endpoint management is admin-only in multi-user mode, the same as remote hosts and Docker
hosts — these are machine-level infra, not a per-user setting.

**Model lists refresh themselves.** Every saved endpoint is re-discovered automatically every
5 minutes in the background, so a model the server starts serving later — or stops serving —
shows up without another manual click of **Discover**. One endpoint being unreachable on a
given cycle (powered off, wrong network) never blocks the others from refreshing.

## Running a session against one

With the setting on and at least one endpoint carrying a discovered model, the **Run**
dropdown grows a **Custom Endpoints** section: one entry per harness that can redirect to a
custom endpoint, per saved endpoint, e.g. "Claude Code (llama.cpp)". Picking one starts a
session on that harness exactly the way its own entry would. It is a one-off "try this
endpoint" action, not a sticky mode — the plain **Run** button still means "this harness,
native cloud" afterward, and a fresh session never inherits whatever the last one was
pointed at.

**Which model it uses depends on how many the endpoint has discovered.** With exactly one,
the session launches straight away on that model — nothing to choose. With two or more, a
small dialog asks which one to use for this launch before starting the session; the
endpoint's default model, if set, is marked but not auto-picked, so a launch can deliberately
use a different one without changing the saved default.

Applying a selection **restarts the harness's process in place** — same tab, same
conversation where the harness supports resuming one, fresh environment. That restart is
necessary, not incidental: every supported harness reads its endpoint config at process
start, never per turn, so there is no live hot-swap while a turn is running.

Picking an entry that launches a **brand-new** session waits (up to 20 seconds) for it to
finish its own startup before applying — a freshly started CLI reports itself as busy for its
boot sequence, and applying to a genuinely busy session is refused so a real, in-progress
turn is never interrupted out from under you. A session that is still busy after that wait
(a very slow-starting CLI, or one you started typing into right away) surfaces that refusal
as an ordinary error, which now stays on screen with a close button instead of vanishing
after a few seconds — read it, it names the actual reason rather than a generic failure.

Entries are hidden entirely for a session in a **remote (SSH) or Docker case** — support for
redirecting those hasn't landed yet, see below. The picker also only appears in the desktop
**Run** dropdown; the phone home screen builds its own run picker separately and does not
currently offer these entries.

## Which harnesses actually work

| Harness | Status |
| ------- | ------ |
| **Claude Code, opencode, Pi, Grok, OMP** | Verified end-to-end against a real local server. |
| **Codex** | Config is correct, but Codex only speaks the Responses API, which llama.cpp-style servers don't implement. A protocol gap, not a Codeman bug. |
| **Gemini** | Fails with an auth error gemini-cli raises once redirected. Unresolved; don't rely on it yet. |
| **DeepSeek** | Reaches the server but gets a consistent 404. Root cause not identified. |
| **Antigravity** | No known custom-endpoint mechanism at all. Not offered. |

Which harnesses show up in the Run-menu picker is read live off Codeman's own CLI registry,
not a fixed list here, so this table can go stale before this page does — a greyed-out or
missing entry is the more current answer.

## What it does not do

- **No remote or Docker sessions yet.** Both restart their agent differently under the hood
(reattaching a durable tmux session rather than relaunching the process), so redirecting
them needs its own plumbing that hasn't been built.
- **No live hot-swap mid-conversation.** Applying a selection always restarts the process.
- **No button to un-point a session from the UI yet.** Clearing back to native cloud is an
HTTP call (`POST .../custom-model {"clear": true}`) or deleting the session; the settings
panel manages saved endpoints, not what a running session is currently pointed at.
- **Nothing is shared with your real cloud credentials.** The endpoint's own key, if any,
never touches your Anthropic/OpenAI/Google login — a custom endpoint is a separate,
explicit choice per session.

## Security

An endpoint's base URL can't point at a link-local or cloud-metadata address (both at save
time and against the address it actually resolves to), the same guard Web Tabs uses for
saved dashboards. Endpoint records and any per-session config files a harness needs are
written with owner-only permissions. See
[custom-model-endpoints-plan.md](https://github.com/Ark0N/Codeman/blob/master/docs/custom-model-endpoints-plan.md)
in the repository for the full design reasoning, including why this feature closed a
pre-existing gap in how session environment overrides were guarded rather than opening a new
one.
4 changes: 4 additions & 0 deletions docs/wiki/Settings-Reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,10 @@ Model and effort are both **soft defaults**: the model is written into the case'
`.claude/settings.local.json` and effort is passed at start, so `/model` and `/effort`
inside a session override them at any time.

**Custom model endpoints** (off by default) adds a saved-endpoint list plus a matching
section to the Run dropdown, for pointing a harness at your own OpenAI-compatible server
instead of its native cloud backend. See [Custom Model Endpoints](Custom-Model-Endpoints).

### Agents & CLIs

| Setting | Notes |
Expand Down
1 change: 1 addition & 0 deletions docs/wiki/_Sidebar.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@

- [The Dashboard](The-Dashboard)
- [Agent CLIs](Agent-CLIs)
- [Custom Model Endpoints](Custom-Model-Endpoints)
- [Working With Files](Working-With-Files)
- [Input And Voice](Input-And-Voice)
- [Mobile Guide](Mobile-Guide)
Expand Down
10 changes: 10 additions & 0 deletions src/custom-model-hosts.ts
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,16 @@ export interface CustomModelHost {
authStyle?: CustomModelAuthStyle;
models?: string[];
lastDiscoveredAt?: string;
/**
* The model the Run-menu picker (docs/custom-model-endpoints-plan.md) applies when
* this endpoint is picked with no further choice — one generated menu entry per
* (CLI, endpoint) pair, not per (CLI, endpoint, model), so it needs a single answer.
* Must be a member of `models` when set; the picker falls back to `models[0]` when
* this is unset, and disables the entry entirely when `models` is empty (nothing to
* default to). Never auto-set on discovery — the previous default staying valid
* after a re-discover is a property worth keeping even if the model list changes.
*/
defaultModelId?: string;
}

export function customModelHostsPath(configDir: string): string {
Expand Down
Loading