diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 46a9b5f0..6be93983 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -1,7 +1,7 @@ { "$schema": "https://anthropic.com/claude-code/marketplace.schema.json", "name": "diffusers-workflow", - "description": "Skills for driving a diffusers-workflow server from Claude Code: one composition skill per model family.", + "description": "Skills for driving a diffusers-workflow server from Claude Code: a composition skill per model family, and one for cutting a series from them.", "owner": { "name": "Don Kackman" }, @@ -9,7 +9,7 @@ { "name": "dw", "source": "./plugins/dw", - "description": "Compose MiniMax H3 video, MiniMax Music 3 and LTX-2.5 workflows over a dw MCP server: which template fits which shape, the hard rules, cost, and how to judge the output. Prompt format comes from the vendors' own guides." + "description": "Compose MiniMax H3 video, MiniMax Music 3 and LTX-2.5 workflows over a dw MCP server, and cut a multi-episode series from them: which template fits which shape, the hard rules, cost, and how to judge the output. Prompt format comes from the vendors' own guides." } ] } diff --git a/CLAUDE.md b/CLAUDE.md index 21611686..4fe9d138 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -49,7 +49,10 @@ The stdio MCP server lives in `dw_mcp/` — see `dw_mcp/CLAUDE.md` and docs/MCP. `.claude-plugin/marketplace.json` publishes the `dw` plugin in `plugins/dw/`: one composition skill per model family (`minimax-h3`, `minimax-music3`, `ltx-2.5`) that -chooses a template for a request's shape and states the family's hard rules. Model +chooses a template for a request's shape and states the family's hard rules, plus +`series-episodes`, which is a shape above them - several episodes over one cast, +each cut and scored from those templates' runs. Every skill the directory holds is +named in `plugins/dw/README.md` and here, pinned by the same test. Model knowledge lives there and in the catalog, never in engine code; every number a skill states is pinned to a diffusers symbol by `tests/test_plugin_skills.py`. `plugin.json`'s version is the engine's, bumped by `scripts/release.sh`. Adding or re-auditing a family diff --git a/README.md b/README.md index 12a62a8a..b42bdfd6 100644 --- a/README.md +++ b/README.md @@ -83,7 +83,7 @@ once; without it the run fails partway through with a 401/403 from the Hub. ## Drive it from an agent -Then just ask. The agent has 55 tools covering the whole surface — the +Then just ask. The agent has 57 tools covering the whole surface — the workflow catalog, the real diffusers pipeline signatures, the job queue, the gallery, the model cache: diff --git a/docs/MCP.md b/docs/MCP.md index 6994890c..2be4b937 100644 --- a/docs/MCP.md +++ b/docs/MCP.md @@ -223,8 +223,8 @@ when no single workflow covers it. | `get_task(command)` | `command` | Get a task command's argument schema | | `list_models()` | — | List what the Hugging Face model cache holds, largest first | | `get_memory()` | — | Get the worker's VRAM and RAM statistics. `gpu_*` is the card; `host_memory_rss_mb` / `host_memory_peak_rss_mb` are the worker process's resident and high-water host memory, beside the machine's `host_memory_total_mb` / `host_memory_available_mb` - read both, since an offloading workflow keeps its weights in host RAM and the card says little about what it holds (a host field is absent, not null, where the platform cannot measure it). `host_pinned_reserved_mb` / `host_pinned_allocated_mb`, when present, are torch's pinned-host cache - the staging buffers group offloading moves weights through, part of `host_memory_rss_mb` and invisible in every `gpu_*` figure, which is what a worker holding GB after releasing every model is usually holding (#98). `live: true` means `info` was measured now and is the worker's own memory; only live readings are comparable with each other. `live: false` with `info: null` (and `stale: false`) means nothing has been measured because nothing is resident; `live: false` with a populated `info` is a cached earlier reading, with `reason` (`job_running`, `worker_stopped`, `worker_busy`, `worker_unreachable`) and `age_seconds` - one cached while a job loads a model understates what is resident, so ask again when the server is idle rather than comparing it with a live figure | -| `get_health()` | — | Check that the server is alive, and which machine answered: `version`, `device`, whether the worker process is up, the job running now and the queue depth | -| `get_server_info()` | — | What this installation can do and where it keeps things: `device` (the accelerator a run will use), `version`, the `workspace` this session is working in and the workflow/asset/output/prompt `directories` of *that* workspace, the bind address and port, whether a token is required, and whether MCP is mounted. Check the device before authoring - a CUDA-only choice (bitsandbytes, `torch.compile`, flash attention) is not available on an `mps` or `cpu` server | +| `get_health()` | — | Check that the server is alive, and which machine answered: `version`, `device`, whether a model process is currently resident (`worker_alive`), the job running now and the queue depth. `worker_alive: false` is the normal idle state on a server that has not run a job since startup or the last memory clear - not a fault - the on-demand worker starts with the next job (#206) | +| `get_server_info()` | — | What this installation can do and where it keeps things: `device` (the accelerator a run will use), `version`, the `workspace` this session is working in and the workflow/asset/output/prompt `directories` of *that* workspace, the bind address and port, whether a token is required, and whether MCP is mounted. Check the device before authoring - a CUDA-only choice (bitsandbytes, `torch.compile`, flash attention) is not available on an `mps` or `cpu` server. `runtime` (#222) reports the Python version, torch version and the CUDA version torch was built against, the NVIDIA driver version (when `nvidia-smi` is reachable), and the installed versions of diffusers, transformers, accelerate, bitsandbytes, peft, safetensors and sentencepiece (`null` for one not installed) - for diagnosing an environment mismatch between boxes without shelling in | | `list_jobs(limit=20, status=None, workspace=None)` | optional `limit` (newest N), `status` (one state or a comma-separated set of `queued`, `running`, `succeeded`, `failed`, `cancelled`), `workspace` | List queued, running and recent jobs, **newest first**. Bounded by default: the unbounded listing was over a client's tool-result limit on a server with a few months of history, which made it a tool that could not be called at all. `total` says how many matched and `truncated`/`next` say so when the answer was cut - raise `limit` or narrow with `status`. Without `workspace`, a named workspace lists its own jobs and the default one lists every job the server holds | | `list_gallery(limit=50, subfolder=None, only_orphans=False, workspace=None)` | `limit`, `subfolder`, `only_orphans`, `workspace` | List generated output files, newest first. A name is `//`, where `` may sit in the subfolder the step chose (`final/episode.mp4`); each entry carries `folder` (the workflow) and `subfolder` (by convention `final` or `intermediate`, `''` when the step chose none, any path the workflow wrote otherwise), and `subfolder="final"` lists only deliverables. Each entry also carries a ready-made `url`, already scoped to the workspace that made it - a hand-built `/outputs/` URL 404s for anything but the default workspace. `only_orphans=True` inverts the call: instead of files, it returns run directories with no media anywhere under them (`runs`, each `{name, mtime}`) - a run whose output was deleted before `delete_output` could remove it by name, or one that failed before writing anything; `subfolder` does not apply in this mode, and `name` is exactly what `delete_output` accepts (#170). `workspace` names the workspace for this one call without switching the session to it - the same pin `run_workflow` takes, so a job run into another workspace stays reachable from the session that queued it | | `get_gallery_metadata(name, envelope=False, workspace=None)` | `name`, `workspace` | Get the metadata embedded in a generated file — or, when `name` is an `asset:` reference, what an *input* asset holds (`source` says which; `job` is null for an asset). Reading an input's duration, frame count, fps and sample rate before a run is how a caller learns the `total_frames`, `fps` and `sample_rate` a workflow expects it to supply: the exact workflow and arguments that produced it, and, for audio/video, a `media` block (duration, rate, channels, fps, size, peak/mean dBFS). `envelope=true` adds `media.envelope` — `rms_dbfs` and `peak_dbfs` one entry per second — which is what locates something in a track rather than measuring the whole of it. `workspace` names the workspace for this one call without switching the session to it - the same pin `run_workflow` takes, so a job run into another workspace stays reachable from the session that queued it | @@ -234,6 +234,7 @@ when no single workflow covers it. | Tool | Arguments | Purpose | | --- | --- | --- | | `get_output_image(name, max_dimension=768, workspace=None)` | `name`, `max_dimension`, `workspace` | Look at a generated image, downscaled to `max_dimension` on its longest side. Returns the image plus a text part reporting `original_size`, `returned_size` and `bytes`, so a downscale is never silent. `workspace` names the workspace for this one call without switching the session to it - the same pin `run_workflow` takes, so a job run into another workspace stays reachable from the session that queued it | +| `get_output_audio(name, workspace=None)` | `name`, `workspace` | Listen to a generated audio output, as base64. Unlike `get_output_image` there is no downscale - a clip whose base64 size would exceed the same 4MB budget is refused outright rather than cut or transcoded, since a truncated clip is a different, misleading answer rather than a smaller correct one (#204). Use `download_output` or the gallery `url` for a longer file, and `get_gallery_metadata` for its duration and sample rate without fetching the bytes at all. `workspace` names the workspace for this one call without switching the session to it | | `get_output_text(name, max_characters=20000, workspace=None)` | `name`, `max_characters`, `workspace` | Read a text output — a prompt enhancement, or any step whose result is `text/plain` or JSON. Reports the file's real length and whether it was truncated. `workspace` names the workspace for this one call without switching the session to it - the same pin `run_workflow` takes, so a job run into another workspace stays reachable from the session that queued it | | `download_output(name, destination=None, overwrite=False, workspace=None)` | `name`, `destination`, `overwrite`, `workspace` | Save one output file to local disk, of any content type. `destination` may be a full path, a directory, or omitted to save under the output's own name in the current working directory; `~` expands and missing parent directories are created. `overwrite=True` is required to replace a file already at the resolved path. Over a `dw.serve --mcp` endpoint the file lands on the server, so the destination is confined to that workspace and a relative one is joined onto it. Returns nothing to the conversation but where the file landed — unlike the other media tools, the point is a file on disk, not a payload in context. Writes on the machine running the MCP server - over `dw.serve --mcp` that is the GPU box. A write that fails there (a path that exists only on the client, for instance) comes back as an error naming the server-side write and the client-side alternatives, not as an anonymous tool failure. `workspace` names the workspace for this one call without switching the session to it - the same pin `run_workflow` takes, so a job run into another workspace stays reachable from the session that queued it | | `delete_output(name, workspace=None)` | `name`, `workspace` | Permanently remove one generated file from the output directory. `workspace` names the workspace for this one call without switching the session to it - the same pin `run_workflow` takes, so a job run into another workspace stays reachable from the session that queued it | @@ -256,7 +257,7 @@ The session starts in `default` and stays there unless it is told otherwise. | `delete_workspace(name, acknowledged_cost=False)` | `name`, `acknowledged_cost` | Permanently delete a workspace and everything in it. Refuses without the acknowledgement, reporting what it would remove | | `list_assets()` | — | The input media on the server, each with the `asset:` reference a workflow argument carries. Look here before asking for a file - what a workflow needs may already be there. `libraries` names the roots searched and which are writable; `shadowed` lists names a nearer library hides | | `keep_output(name, asset_name=None, overwrite=False, shared=False, workspace=None)` | `name`, optional `asset_name`, `overwrite`, `shared`, `workspace` | Keep a generated file as an input asset under a stable `asset:` name, so a later workflow can rely on it. The copy happens on the server: nothing is downloaded or re-uploaded. `asset_name` may name a folder and takes the kept file's extension when it has none; `shared=true` keeps it in the library every workspace shares, which is where a recurring cast belongs. `workspace` names the workspace for this one call without switching the session to it - the same pin `run_workflow` takes, so a job run into another workspace stays reachable from the session that queued it | -| `upload_asset(file_path, asset_name=None, shared=False)` | `file_path` | Push a local image, video or audio file into the server's asset library and get back its `asset:` reference. The file is read from the machine the MCP server runs on, so this is how an input reaches a dw.serve running somewhere else. `asset_name` stores it under a readable name (`cast/priya-voice.wav`) instead of a random one; `shared=true` puts it in the library every workspace shares | +| `upload_asset(file_path=None, content=None, asset_name=None, shared=False)` | exactly one of `file_path` or `content` (base64), optional `asset_name`, `shared` | Push an image, video or audio file into the server's asset library and get back its `asset:` reference. `file_path` is read from the machine the MCP server runs on, so this is how an input reaches a dw.serve running somewhere else - but only when the caller shares a filesystem with that machine. `content` is the alternative for an agent that does not: the bytes travel inline, base64-encoded, in the tool call itself, capped at 4MB (well under `file_path`'s 200MB) because inline bytes compete with the calling agent's own context budget rather than being a bulk-transfer path (#203). `asset_name` is required with `content` (there is no file name to infer one from) and, either way, stores it under a readable name (`cast/priya-voice.wav`) instead of a random one; `shared=true` puts it in the library every workspace shares | | `delete_asset(name)` | `name` | Permanently remove one file from the asset library, by the name `list_assets` reports. Deletes from whichever library holds it - this workspace's own before the shared one; one from a read-only examples library is refused. Any workflow still carrying that `asset:` reference stops loading | | `save_workflow(name, workflow)` | `name`, `workflow` | Save a workflow into the server's writable workflow directory, overwriting any existing workflow of that name there. A name that currently resolves to a read-only source (an examples directory) is not overwritten - the copy lands in the writable directory and shadows it | | `delete_workflow(name)` | `name` | Permanently delete a stored workflow | @@ -270,7 +271,7 @@ references written in the same session. | Tool | Arguments | Purpose | | --- | --- | --- | -| `list_prompts()` | — | List the stored prompts with their text and descriptions | +| `list_prompts(tag=None, intended_model=None, include_text=False)` | optional `tag`, `intended_model`, `include_text` | List the stored prompts - description, intended model, tags and `text_chars`, bodies left out; `get_prompt` for one body | | `get_prompt(name)` | `name` | Get one stored prompt's full definition | | `get_prompt_schema()` | — | Get the JSON schema every stored prompt must satisfy. Its own route rather than a name under `/api/prompts`, so a prompt called `schema` cannot shadow it | | `save_prompt(name, prompt)` | `name`, `prompt` | Save a prompt, overwriting any prompt of that name. The server validates first, and refuses a `text` that itself begins with a reference prefix (`variable:`, `previous_result:`, `constant:`, `asset:`, `output:`, `prompt:`) | @@ -284,11 +285,12 @@ references written in the same session. | --- | --- | --- | | `run_workflow(workflow_path=None, inline_workflow=None, arguments=None, acknowledged_cost=False, workspace=None)` | exactly one of `workflow_path` (a catalog name from `list_workflows`, with or without `.json`, or a path to a workflow file on the server) or `inline_workflow`, optional `arguments`, `acknowledged_cost`, `workspace` | Queue a workflow for generation. Returns as soon as the job is queued. `workspace` names the workspace for this one call without switching the session to it - use it to pin a job whose `output:` or `asset:` references live in a workspace other than the session's - `acknowledged_cost` is `true` or the bound `{fingerprint, minutes, downloads}` from the validate plan; a 409 means the plan changed and the message carries the new estimate | | `get_job(job_id)` | `job_id` | Get a job's status, warnings, output manifest, error and traceback; each manifest entry's `subfolder` is the in-run subfolder the step declared - by convention `final` for the deliverable, `intermediate` for scratch, `''` for none. A running job also carries `progress` (below) | -| `get_job_workflow(job_id)` | `job_id` | The workflow the job actually ran. `realized: true` means every mutable input is pinned (arguments, seed, prompts, `output:latest`); `false` means the job predates run tracking and this is the definition as submitted. Pass it to `save_workflow` to keep it under a name | +| `get_job_workflow(job_id)` | `job_id` | The REST equivalent is `GET /api/jobs/{id}/workflow` (see [SERVER.md](SERVER.md#jobs-api)). The workflow the job actually ran. `realized: true` means every mutable input is pinned (arguments, seed, prompts, `output:latest`); `false` means the job predates run tracking and this is the definition as submitted. Pass it to `save_workflow` to keep it under a name | | `export_job(job_id, overwrite=False)` | `job_id`, `overwrite` | Gather one finished job into `/exports//` on the server: the realized workflow, the run's manifest, the job row, a README, and copies of the assets, earlier-run inputs and outputs. Returns the directory, a zip URL, the file list with sizes and the total. The three JSON files are in the zip, not repeated here - get_job_workflow and get_job serve them individually. **The directory is on the machine running the server**, like `download_output`'s destination - fetch the zip URL and unpack it into `exports/` under the session's working directory (a deliverable, not a temp file); the archive already unpacks into one folder named after the job id | | `get_job_events(job_id, after=-1, limit=200)` | `job_id`, `after`, `limit` | Get a page of a job's progress events | | `wait_for_job(job_id, timeout_seconds=20)` | `job_id`, `timeout_seconds` | Block until a job reaches a terminal status, or `timeout_seconds` elapses. **One call blocks for at most 55 seconds** — a larger `timeout_seconds` is clamped, not honoured, because no MCP client holds a tool call open for a generation's real runtime, so budget one call per ~55s of the job. Every reply carries `waited_seconds`, `timeout_requested_seconds`, `timeout_applied_seconds` and `timeout_capped`, so a capped return is distinguishable from an elapsed one. Use instead of hand-polling `get_job`/`get_job_events` in a loop; if it returns `still_running: true`, call it again. Returns a slim job - status, warnings, error, and the manifest once finished - without the arguments; `get_job` has those. A running job also carries `progress` (below) | | `cancel_job(job_id)` | `job_id` | Ask a queued or running job to stop | +| `clear_memory()` | — | Drop every loaded pipeline and the step cache, freeing VRAM/RAM immediately instead of waiting for the next job to evict one model for another. Also drops the step cache, so a seeded workflow that would otherwise reuse cached results regenerates on its next run. Refused with a 409 while a job is running or queued - the queue is FIFO, so wait for it to finish and retry rather than expecting this call to block until it does (#221) | | `rerun_job(job_id, acknowledged_cost=False, new_seed=False)` | `job_id`, `acknowledged_cost`, `new_seed` | Queue a fresh job from a previous job's stored specification. Costs GPU time, so it passes the same gate as `run_workflow`. `new_seed=true` draws a fresh seed into the workflow's seed variable — without it a seeded workflow's rerun repeats its arguments exactly and the step cache serves the whole run from the earlier one's files (`reused: true`), generating nothing. `get_job_workflow`'s `seed_variable` says whether there is one - `acknowledged_cost` is `true` or the bound `{fingerprint, minutes, downloads}` from the validate plan; a 409 means the plan changed and the message carries the new estimate | | `move_job(job_id, direction)` | `job_id`, `direction` (`up`\|`down`\|`front`\|`back`) | Reorder a queued job | @@ -451,18 +453,26 @@ default) for any server an MCP client can reach. events of a finished job (`MAX_PERSISTED_EVENTS` in the job history store). A job that ran before this feature existed returns an empty event list with a `note` explaining why. -- **Images only.** `get_output_image` decodes and returns images; it refuses - video and audio outputs. Use `get_gallery_metadata` to inspect other media - kinds. -- **Uploads read the MCP server's disk.** `upload_asset(file_path)` pushes a - local file into the asset library, but "local" means the machine `dw-mcp` - runs on. Over `dw.serve --mcp` that is the GPU box, so a file sitting on - the client's laptop is not reachable that way - put it on the server, or - give the workflow a URL (the arguments that take a path take a URL too). - On a `--mcp` endpoint `file_path` is also *confined* to the directories the - server works in (its workspace, workflows, assets, outputs and prompts), and - the refusal comes before the file is looked for, so the tool cannot be used - to probe which paths exist on the box (#138). +- **Images and audio only.** `get_output_image` decodes and returns images, + `get_output_audio` audio; both refuse video. Use `get_gallery_metadata` to + inspect other media kinds. `get_output_audio` also refuses a clip whose + base64 size would exceed its 4MB budget rather than truncating or + transcoding it (#204) - there is no `VideoContent` type in the MCP SDK to + extend this pattern to video, so a `get_output_video` is a harder, + unresolved design question rather than a mechanical port. +- **Uploads read the MCP server's disk, unless sent inline.** + `upload_asset(file_path)` pushes a local file into the asset library, but + "local" means the machine `dw-mcp` runs on. Over `dw.serve --mcp` that is + the GPU box, so a file sitting on the client's laptop is not reachable that + way - put it on the server, give the workflow a URL (the arguments that + take a path take a URL too), or use `upload_asset(content=..., asset_name=...)` + to send the bytes inline instead, base64-encoded in the call itself, capped + at 4MB (#203). On a `--mcp` endpoint `file_path` is also *confined* to the + directories the server works in (its workspace, workflows, assets, outputs + and prompts), and the refusal comes before the file is looked for, so the + tool cannot be used to probe which paths exist on the box (#138); `content` + reads no path and is not subject to this confinement, since no file on + either machine is ever named. `download_output` has the same asymmetry in the other direction: on a `--mcp` endpoint it writes on the GPU box, not the client's machine, and is confined to the workspace there. diff --git a/docs/SERVER.md b/docs/SERVER.md index 099d4187..2794131a 100644 --- a/docs/SERVER.md +++ b/docs/SERVER.md @@ -149,7 +149,7 @@ from another machine: | `POST /api/jobs` | Queue a run: `{"workflow_path": ...}` or an inline `{"workflow": {...}, "base_dir": ...}`, plus `arguments` for variable overrides. `workflow_path` accepts a stored workflow name as listed by `/api/workflows` (with or without `.json`, nested names included), or a relative/absolute path that still resolves under `--workflow-dir` - confined the same way the `/api/workflows` CRUD routes are; a path that names a real file outside that directory is rejected with 400, not opened. Answers with argument warnings from signature checking. Takes an optional `acknowledged_cost`: `true` is recorded as `acknowledged: boolean`; the object `{fingerprint, minutes, downloads}` from a validate answer's `plan` is `bound` - the server re-plans the run for the arguments given and answers **409** when the fingerprint differs or a repo in `downloads_required` is not in `downloads` (a download that has since vanished is not a refusal); the body is `{"detail": {message, reason: "fingerprint" \| "downloads" \| "unplannable", acknowledged, plan}}` with the current plan, so the caller re-quotes from it. `minutes` is recorded, never compared. Nothing is required: the web UI and every caller that sends nothing are `acknowledged: none`, and every job answer and history row carries `acknowledged` (and `acknowledged_cost` when bound). `POST /api/jobs/{id}/rerun` takes the same field and checks against the stored spec; a fresh seed does not change a fingerprint. | | `GET /api/jobs?workspace=&status=&limit=` | Queue + history summaries, oldest first, with `total` beside them. `status` narrows to one state or a comma-separated set (`queued`, `running`, `succeeded`, `failed`, `cancelled`; anything else is a 400); `limit` keeps the newest N, and `total` still reports how many matched, so a bounded answer cannot be mistaken for a complete one. No parameters means every job, which is what the web UI polls | | `GET /api/jobs/{id}` | Full detail: spec, events, manifest, error. A manifest entry for a step served from the step cache carries `reused: true`. Every entry carries `subfolder` - the in-run subfolder the step's `result.subfolder` chose, `''` for none. A `for_each` step appears in the manifest as its members (`shot@wide_open`, `shot@closeup`), because the manifest records what ran; the run's `workflow.json` keeps the `for_each` form, because it records what was asked | -| `GET /api/jobs/{id}/workflow` | The workflow the job ran: `{id, definition, realized, seed_variable}`. `seed_variable` names the variable a `new_seed` rerun would draw into (null when the workflow has none), read from the workflow as written rather than the realized copy, whose seed is pinned. `realized: true` is the copy the run itself wrote (`workflow.json` in its run directory), with arguments, seed, prompts and `output:latest` pinned; `false` falls back to the submitted definition, which is what a job from before run tracking has. 404 means neither is readable - the job itself still is | +| `GET /api/jobs/{id}/workflow` | The workflow the job ran: `{id, definition, realized, seed_variable}`. `seed_variable` names the variable a `new_seed` rerun would draw into (null when the workflow has none), read from the workflow as written rather than the realized copy, whose seed is pinned. `realized: true` is the copy the run itself wrote (`workflow.json` in its run directory), with arguments, seed, prompts and `output:latest` pinned; `false` falls back to the submitted definition, which is what a job from before run tracking has. 404 means neither is readable - the job itself still is. The equivalent MCP tool is `get_job_workflow` (see [MCP.md](MCP.md#diagnose)) | | `POST /api/jobs/{id}/export?workspace=&overwrite=` | Gather one finished job into `/exports//`: `workflow.json`, `manifest.json`, `job.json`, `README.md`, `assets/`, `inputs/`, `outputs/`. 201 with the file list, total bytes, anything it could not find, a `zip_url`, and the three JSON files inline. 404 unknown job, 409 for a job still running or an existing export without `overwrite` | | `GET /exports/{id}.zip?workspace=` | The same tree as one archive, built on request rather than kept as a second copy. Entries are named `/`. Ungated exactly as `/outputs` is | | `GET /api/jobs/{id}/events` | Server-sent events stream; `?after=N` / `Last-Event-ID` replay missed events, so reconnects are lossless. Every event carries `seq` and `at` - seconds since the job started (since it was queued, for the events before that) - so a step's cost is a subtraction: `step_start` to `generating` is what a reused pipeline still pays before it runs, `generating` to the first `pipeline_step` is the prompt and reference encoding | @@ -303,9 +303,26 @@ The editor's forms come from these; they are just as usable from scripts: also gets a warning saying so, since `0` alone does not distinguish a disabled cache from an empty one); `downloads_required`, each `model_name` the hub cache does not hold as - `{repo, gb}` (`gb` from the hub, `null` when it could not be asked - - `?sizes=false` skips the hub) and each `from_single_file` URL as - `{repo: null, url, gb: null}`; and `estimate`, `{minutes, basis, + `{repo, gb, gated, access_blocked}` (`gb` from the hub, `null` when it + could not be asked - `?sizes=false` skips the hub, and then `gated` and + `access_blocked` are `null` too); `gated` is the hub's own field for the + repo (`false`, `"auto"` or `"manual"`) or `null` when the lookup itself + failed for a reason other than the gate; `access_blocked` is `true` + when this box's Hugging Face token specifically has not been granted + access to a gated repo, `false` when the repo isn't gated or the token is + accepted, and `null` when it could not be determined either way. A gated + repo's own metadata is served by the hub regardless of this token's + access, so `model_info` succeeding proves nothing about the gate; a + gated entry gets a second, real check - a HEAD request against one of the + repo's own files - and it is *that* request's `GatedRepoError` that sets + `access_blocked: true` (the pre-flight signal for what would otherwise be + a 403 partway into a run, #186). `access_blocked` is `null` when there is + no file to probe or the probe itself fails for an unrelated reason (e.g. + offline) - "unknown" is not "not blocked". `validate_workflow`'s + `warnings` carries one line per entry with `access_blocked: true`. Each + `from_single_file` URL is `{repo: null, url, gb: null, gated: null, + access_blocked: null}`, since a direct file URL is never gated; and + `estimate`, `{minutes, basis, device, measured_on, partial, runs}` from this box's own history when it has one and otherwise from the workflow's `cost` block - `basis` is `observed` (the cold median of this server's own finished runs @@ -517,12 +534,22 @@ The editor's forms come from these; they are just as usable from scripts: measured because nothing is resident. health also reports `hostname`, `device` and whether `mcp` is mounted, so a remote client can tell which machine answered +- `POST /api/memory/clear` (#221) — drops every loaded pipeline and the step + cache, the same mechanism as the REPL's `memory clear`, and returns the + memory reading taken right after. Refused with 409 while a job is running + or queued - the queue is FIFO, so the caller retries once it finishes + rather than this call blocking until it does - `GET /api/server` — connection details for the Server page: `hostname`, `version`, `device`, the `bind_host`/`port`/`wildcard_bind` the server was started with, `auth_required` (whether a token is configured - never the - token itself), `mcp` (`mounted` plus its `path`), the machine's - non-loopback `addresses`, and the `directories` in use; a client composes - its URLs from an address, the port and the MCP path + token itself), `mcp` (`mounted` plus its `path`), the `directories` in use, + and `runtime` (#222) - Python version, torch version and the CUDA version + torch was built against, the NVIDIA driver version (via `nvidia-smi`, when + it's on PATH), and the installed versions of diffusers, transformers, + accelerate, bitsandbytes, peft, safetensors and sentencepiece (`null` for + one not installed) - for diagnosing an environment mismatch between boxes + without shelling in; the machine's non-loopback `addresses`; a client + composes its URLs from an address, the port and the MCP path ## Security model diff --git a/docs/TASKS.md b/docs/TASKS.md index cc3c3e0b..314ceee4 100644 --- a/docs/TASKS.md +++ b/docs/TASKS.md @@ -15,7 +15,9 @@ Tasks are utility operations that run outside of pipeline inference. Use them fo Any task that runs a model accepts a `"device"` argument to pin where it runs - useful for keeping a helper model (a captioner, an upscaler) off the accelerator a -loaded pipeline is using, or on a second one. +loaded pipeline is using, or on a second one. `"device"` is listed on every task's +schema, since it is always safe to pass: a task that runs no model - `slice_audio`, +`compose_text`, and the like - just ignores it. Task argument schemas are discoverable: `GET /api/tasks/{command}` on the [server](SERVER.md) returns each command's arguments read from its registered @@ -214,16 +216,19 @@ video generation" in the workflow guide): | `trim_frames` | No | Frames dropped from the head of every video after the first (default: 0) | | `crossfade_ms` | No | Equal-power crossfade at each audio seam, drawn from the trimmed material - no effect when `trim_frames` is 0, and validation warns when one is written there (default: 75) | | `audio_bleed_ms` | No | How long the outgoing video's tail rings on over the head of the next one, at seams with nothing trimmed to crossfade (default: 0, off) | +| `audio_bleed_gain_db` | No | Gain applied to the bled tail before it is added, in dB - negative ducks a tail that would otherwise push the seam over 0 dBFS (default: 0, unchanged) | | `seam_fade_ms` | No | Fade on each side of a seam that gets neither a crossfade nor a bleed - for tonal material, not for a continuous bed (default: 3, just enough not to click) | | `fps` | No | Frame rate of the videos - required to join audio when trimming, and the rate the joined file is written at unless `result.fps` overrides it | | `match_levels` | No | Even the shots' loudness out before joining - `"rms"` for perceived level (the measurement `get_gallery_metadata` reports as `mean_dbfs`), `"peak"` for the loudest sample. Off by default | -| `match_levels_dbfs` | No | The level `match_levels` moves every shot to (default: -1 dBFS for `peak`, -20 dBFS for `rms`) | +| `match_levels_dbfs` | No | The level `match_levels` moves every shot to (default: -1 dBFS for `peak`, -20 dBFS for `rms`). A shot that would clip at the target is held at -0.5 dBFS peak instead, reported as a `match_levels_held` warning with a per-shot log event | A video may also be named by path or URL, which is how shots an earlier run already wrote are joined without regenerating them - the file is read with the audio muxed into it, and its track is fitted to the frames' own duration so the codec's block padding does not walk the sound off the picture over a dozen -seams: +seams. A shot generated in memory through a `previous_result:` chain gets the +same fit, applied where the file is written rather than where it is decoded, +so per-shot drift does not accumulate across a cut the way it once did: ```json { @@ -279,7 +284,10 @@ a five-shot H3 sitcom cut, 700 ms still left a 44 dB hole at the worst seam; A bleed works because it copies ambience, which has no pitch and no attacks to give the copy away. It is the wrong tool for anything tonal - a copied musical -phrase or half-spoken word reads as a stutter whichever direction it runs. When a +phrase or half-spoken word reads as a stutter whichever direction it runs. +`bleed_join` checks the outgoing tail's spectral flatness and warns when it +looks tonal or speech-like rather than noise-like, so this failure mode +surfaces in the job's `warnings` list instead of only in the mix. When a shot ends on something tonal, either give the cut a continuous bed with `slice_audio` + `loop_audio` + `mix_audio` + `pair_audio`, which leaves no seam to treat at all, or fade the @@ -342,7 +350,7 @@ montage cut to a score wants: | `fade_color` | No | The RGB colour the fades come from and go to (default: black) | | `fps` | No | Frame rate of the videos - required to crossfade audio at a dissolve, and the rate the dissolved file is written at unless `result.fps` overrides it | | `match_levels` | No | Even the shots' loudness out before joining - `"rms"` or `"peak"`, as with [`concat_videos`](#concat_videos). Off by default | -| `match_levels_dbfs` | No | The level `match_levels` moves every shot to (default: -1 dBFS for `peak`, -20 dBFS for `rms`) | +| `match_levels_dbfs` | No | The level `match_levels` moves every shot to (default: -1 dBFS for `peak`, -20 dBFS for `rms`). A shot that would clip at the target is held at -0.5 dBFS peak instead, reported as a `match_levels_held` warning with a per-shot log event | Every seam shortens the result by one overlap, so eight 124-frame shots joined with 12-frame dissolves run 908 frames, not 992 - size a soundtrack slice to @@ -482,6 +490,49 @@ a length still passes the whole track along: | `start_frame` / `num_frames` / `fps` | One pair | The slice in video frames; `fps` is required, start and count may be omitted | | `sample_rate` | With a waveform | Sample rate of a directly passed waveform (files carry their own) | +### gain_audio + +Apply a gain, in decibels, to a region of an audio track - the rest of the +track passes through unchanged. The region is addressed the same way +`slice_audio`'s is, in seconds or in video frames, so ducking a scene under +another (lowering a dialogue track between two timestamps) is one step +instead of the `slice_audio` → `gain` (a whole-track `normalize_audio` on the +slice) → `mix_audio` → `rejoin` → `pair_audio` chain that used to be the only +way to gain part of a track rather than all of it. Unlike `slice_audio`, a +region reaching past the end of the track is clipped to it rather than +zero-padded - there is no silence there to gain, only the end of the real +material: + +```json +{ + "task": { + "command": "gain_audio", + "arguments": { + "audio": "./dialogue.wav", + "gain_db": -12, + "start_frame": 124, + "num_frames": 48, + "fps": 24 + } + }, + "result": { "content_type": "audio/wav", "sample_rate": 44100 } +} +``` + +| Argument | Required | Description | +| -------- | -------- | ----------- | +| `audio` | Yes | Path or URL of an audio file (or of a video file, whose soundtrack is taken), a waveform from a previous step, or an earlier step's video generated with a soundtrack (which brings its sample rate along) | +| `gain_db` | Yes | Gain to apply within the region, in decibels - negative ducks it, positive boosts it | +| `start_seconds` / `duration_seconds` | One pair | The region in seconds; either may be omitted | +| `start_frame` / `num_frames` / `fps` | One pair | The region in video frames; `fps` is required, start and count may be omitted | +| `sample_rate` | With a waveform | Sample rate of a directly passed waveform (files carry their own) | + +One pair is required — there is no separate "whole track" mode — but the +whole track is still one step: give just `start_seconds: 0` and leave +`duration_seconds` unset (or `start_frame: 0` + `fps` and leave `num_frames` +unset), which runs to the end of the track without needing to already know +how long that is. + ### crossfade_audio Join audio tracks with an equal-power crossfade. Each seam overlaps the two @@ -625,10 +676,13 @@ scene so the edits stop being audible: | `sample_rate` | With a waveform | Sample rate of a waveform passed directly; given for a file or a video it overrides the rate they carry | Laps are joined with an equal-power crossfade rather than butted together, so -the loop point is not a click and a tone with movement in it does not tick once -a second. The source is used whole every lap and only the last one is trimmed, -so the bed lands exactly on the requested length; a source longer than the -request is trimmed to it. +the loop point itself is not a click. That only smooths the seam: a transient +in the source (a hit, a swell) still recurs once per lap at full strength, so +the loop still reads as a level pulse at the lap rate — measured at 9.3 dB on +a source with one such transient. Pick a source with even internal level to +avoid the pulse; the crossfade does not remove it. The source is used whole +every lap and only the last one is trimmed, so the bed lands exactly on the +requested length; a source longer than the request is trimmed to it. The bed is laid under the cut with `mix_audio` and attached to the picture with `pair_audio`: @@ -687,6 +741,100 @@ into the next without the rate being restated: a `resample_audio` fed **Example:** [assemble-and-score.json](../workflows/templates/assemble-and-score.json) +### compress_audio + +Shape a track's dynamics with an envelope-follower - a compressor, a limiter +and a gate are the same algorithm with different knob settings, so one task +covers all three through `mode`: + +```json +{ + "task": { + "command": "compress_audio", + "arguments": { + "audio": "previous_result:mixed", + "threshold_dbfs": -18.0, + "ratio": 4.0, + "attack_ms": 10, + "release_ms": 100, + "sample_rate": 44100 + } + } +} +``` + +| Argument | Required | Description | +| -------- | -------- | ----------- | +| `audio` | Yes | Path or URL of an audio file (or of a video file, whose soundtrack is taken), a waveform from a previous step, or an earlier step's video generated with a soundtrack (which brings its sample rate along) | +| `threshold_dbfs` | Yes | The level the envelope is measured against, in dB below full scale. Cannot be above 0 | +| `ratio` | No | How hard the reduction is above the threshold, in `compress`/`gate` mode (default: 4.0). Ignored in `limit` mode, which always holds the signal at the threshold | +| `attack_ms` | No | How fast the envelope rises to a louder signal (default: 10.0). 0 means instantly | +| `release_ms` | No | How fast the envelope falls back after a louder signal ends (default: 100.0). 0 means instantly | +| `mode` | No | `compress` (turn down what's above the threshold), `limit` (hold the signal at the threshold), or `gate` (turn down what's below the threshold) (default: `compress`) | +| `sample_rate` | With a waveform | Sample rate of a directly passed waveform (files carry their own) | + +A silent track is returned unchanged. + +### filter_audio + +Run a track through a single biquad filter stage - trimming the frequencies a +mix doesn't need, or carving out room for another element: + +```json +{ + "task": { + "command": "filter_audio", + "arguments": { + "audio": "previous_result:world", + "cutoff_hz": 120, + "kind": "highpass", + "sample_rate": 44100 + } + } +} +``` + +| Argument | Required | Description | +| -------- | -------- | ----------- | +| `audio` | Yes | Path or URL of an audio file (or of a video file, whose soundtrack is taken), a waveform from a previous step, or an earlier step's video generated with a soundtrack (which brings its sample rate along) | +| `cutoff_hz` | Yes | The filter's corner frequency. Must be below the Nyquist frequency (half the sample rate) | +| `kind` | No | `lowpass`, `highpass`, `bandpass`, or `notch` (default: `lowpass`) | +| `q` | No | The filter's resonance/bandwidth (default: 0.707, a Butterworth response) | +| `sample_rate` | With a waveform | Sample rate of a directly passed waveform (files carry their own) | + +A silent track is returned unchanged. + +### analyze_audio + +Measure a track without changing it - peak and RMS level, crest factor, and a +rough low/mid/high spectral balance, the numbers a `compress_audio` or +`filter_audio` step downstream is tuned against rather than guessed at: + +```json +{ + "task": { + "command": "analyze_audio", + "arguments": { + "audio": "previous_result:mixed", + "sample_rate": 44100 + } + } +} +``` + +| Argument | Required | Description | +| -------- | -------- | ----------- | +| `audio` | Yes | Path or URL of an audio file (or of a video file, whose soundtrack is taken), a waveform from a previous step, or an earlier step's video generated with a soundtrack (which brings its sample rate along) | +| `sample_rate` | With a waveform | Sample rate of a directly passed waveform (files carry their own) | + +Returns a dict, not a track: `peak_dbfs`, `rms_dbfs`, `crest_factor_db`, +`low_dbfs` (20-250 Hz), `mid_dbfs` (250-4000 Hz), `high_dbfs` (4000-20000 Hz). +The three bands are each a share of the track's total power on the same +scale as `rms_dbfs` (their powers sum to it), so the loudest band sits near +`rms_dbfs` rather than tens of dB under it - comparable to `compress_audio`'s +`threshold_dbfs`. A silent track, or a band with no content at the track's +sample rate, reads as `null` rather than `-inf`. + ## Data Gathering ### gather_images @@ -1127,13 +1275,17 @@ Speak a line of text with a local text-to-speech model. The result is a waveform | Argument | Required | Description | | -------- | -------- | ----------- | -| `text` | Yes | The line to speak | +| `text` | One of `text`/`messages` | The line to speak | +| `messages` | One of `text`/`messages` | Chat-templated input for a model such as VibeVoice that takes a conversation rather than a bare string — a list of `{"role": ..., "content": ...}` dicts, passed straight through as the pipeline's `text_inputs` so the model's own chat template applies. A model with no chat template configured (Bark and friends) raises when handed this instead of `text` | | `model_name` | No | HuggingFace model ID (default: `suno/bark-small`) | | `voice_preset` | No | The speaker, for a model with presets — `v2/en_speaker_0` through `v2/en_speaker_9` for Bark. A model with no processor (a single-voice model such as `facebook/mms-tts-eng`) refuses a `voice_preset` with an error rather than ignoring it | +| `speaker_embedding` | No | A reference audio file (typically an `asset:` reference) whose voice a SpeechT5 model should speak in. Reduced to an x-vector with speechbrain's `spkrec-xvect-voxceleb` and injected into `forward_params` as `speaker_embeddings`. A model that isn't SpeechT5 refuses it the same way a single-voice model refuses `voice_preset` | | `forward_params` | No | Passed to the model's forward/generate call | | `generate_kwargs` | No | Ad-hoc generation settings for a generative model — `temperature`, `do_sample` | -The default is Bark because its voice presets give distinct speakers, which is what two characters in a scene need; `facebook/mms-tts-eng` is a quarter the size and a good override where one voice will do. `voice_preset` is a preprocessing argument — it selects the speaker before generation rather than parameterizing it — so naming it here is what makes it reach the processor. Passed through `forward_params` it would be dropped and every character would sound the same. +The command's own default is Bark because its voice presets give distinct speakers, which is what two characters in a scene need; `facebook/mms-tts-eng` is a quarter the size and a good override where one voice will do. [`generate-speech.json`](../workflows/templates/generate-speech.json) ships with `facebook/mms-tts-eng` as its own default instead, since a template is usually a single voice. `voice_preset` is a preprocessing argument — it selects the speaker before generation rather than parameterizing it — so naming it here is what makes it reach the processor. Passed through `forward_params` it would be dropped and every character would sound the same. + +`speaker_embedding` is the same kind of preprocessing argument for a SpeechT5 model (`microsoft/speecht5_tts`), which conditions its voice on an x-vector rather than a preset name. A VITS model's speaker is different again — a plain `speaker_id` int, passed through `forward_params` unchanged, since it was never a preprocessing argument and needs no argument of its own here. The result needs no `sample_rate`. A generated track carries the rate its model produced it at, and that beats the 44100 default; declaring one still wins over both, for a track whose rate was reported wrong. Every TTS model runs at a different rate, so a declared rate that does not match plays the speech at the wrong speed and pitch without ever failing. @@ -1159,6 +1311,32 @@ A speech model is worth releasing before a video model loads — set `release_mo - [generate-speech.json](../workflows/templates/generate-speech.json) — Speak a line and save it as a `.wav` - [voice-timbre-reference.json](../workflows/templates/minimax/voice-timbre-reference.json) — Generate a voice, then condition H3's `