Machines: detached job commands and MCP tools

README.md

@@ -124,7 +124,7 @@ See [quicksilverpro.io/dashboard#quickstart](https://quicksilverpro.io/dashboard
 
 ## QuickSilver boxes and agent setup (beta)
 
-The 0.5.2 client adds remote Linux boxes with QSP inference preinstalled. The
+The 0.6.0 client adds remote Linux boxes with QSP inference preinstalled. The
 Machines backend and approval page must be enabled for your account; endpoint
 routing and backend integration are release gates. This checkout can be installed
 with `pip install -e '.[mcp,shell]'` (Python 3.10+ for MCP). After release, bootstrap
@@ -154,7 +154,7 @@ No-TTY setup needs an existing Machines key and explicit local-write choices;
 `--yes` does not grant browser authorization, sharing or billing consent. Setup
 creates no box by default. Optional `--create NAME` requires a separate interactive
 confirmation after displaying effective prices/limits. If uv is unavailable,
-install persistently with pipx (`pipx install 'quicksilverpro[mcp,shell]==0.5.2'`)
+install persistently with pipx (`pipx install 'quicksilverpro[mcp,shell]==0.6.0'`)
 or pip, then pass `--no-install-cli`. Skills and MCP configs reference the absolute
 persistent executable for GUI PATH compatibility. Use `--store-key` to explicitly
 persist an environment-supplied Machines key for GUI agents; otherwise those
@@ -269,3 +269,74 @@ MIT.
 QuickSilver Pro is a product of MachineFi Inc. (68 Willow Rd, Menlo Park, CA).
 
 Links: [home](https://quicksilverpro.io) · [status](https://quicksilverpro.io/status) · [terms](https://quicksilverpro.io/terms) · [privacy](https://quicksilverpro.io/privacy)
+
+## Detached jobs
+
+
+For interactive commands, request `Accept: text/event-stream` on
+`POST /v1/machines/{mid}/exec`. Streams include `: keepalive` comments every
+15 seconds, including while commands are quiet. Ignore comment frames. Exec
+still has a 300-second total lifetime and stops its entire process group when
+the client disconnects. Buffered requests with `timeout_seconds > 90` are
+rejected before execution; use streaming or jobs. The CLI and MCP client
+select streaming automatically for timeouts over 60 seconds.
+
+For work that must survive a disconnected client, use detached jobs:
+
+```sh
+qsp box job start MACHINE_ID --command 'pytest > results.txt' --timeout 86400
+qsp box job ls MACHINE_ID
+qsp box job status MACHINE_ID JOB_ID
+qsp box job logs MACHINE_ID JOB_ID --tail 4096
+qsp box job logs MACHINE_ID JOB_ID -f --stream stderr
+qsp box job kill MACHINE_ID JOB_ID
+```
+
+`start` also accepts argv after `--`, `--argv-json`, `--cwd`, and repeatable
+`--env KEY=VALUE`. Commands run as `user`, inside `/home/user`, with the same
+environment restrictions as exec. Shell syntax requires explicit `command`.
+Do not use `nohup` or `&` inside exec to detach work.
+
+| Method | Path | Scope | Input/result |
+| --- | --- | --- | --- |
+| POST | `/v1/machines/{mid}/jobs` | `machines:exec` | Exec body; `timeout_seconds` defaults to 86400, max 259200; returns job status |
+| GET | `/v1/machines/{mid}/jobs` | `machines:read` | `{jobs: [...]}` |
+| GET | `/v1/machines/{mid}/jobs/{jid}` | `machines:read` | Status |
+| GET | `/v1/machines/{mid}/jobs/{jid}/logs` | `machines:read` | `stream=stdout|stderr`, `offset`, optional `tail` (0–65536 bytes), `follow=true` for SSE |
+| POST | `/v1/machines/{mid}/jobs/{jid}/kill` | `machines:exec` | Stops the job and its descendants |
+
+Dashboard sessions and scoped Machines keys use the existing ownership, machine
+restriction and human-control gates. Job operations do not create or resume a
+machine. Status contains `id`, `state` (`running`, `exited`, `killed`, `lost`),
+`exit_code`, Unix `started_at`/`ended_at`, and `max_runtime_seconds`. A killed or
+lost job may have no exit code. Starts are not idempotent: after an unknown
+outcome, list jobs before deciding whether to start again.
+
+Logs retain a circular 1 MiB per stream (2 MiB total per job). Responses
+contain base64 `data` and absolute byte cursors `offset`, `next_offset`,
+`start_offset`, `end_offset`. Each read returns at most 64 KiB. If a cursor is
+older than retained data, `offset` advances to `start_offset`; clients can detect
+the gap. Under disk pressure or interrupted log writes, retained output may be
+discarded, including earlier output in that stream; `start_offset` advances while
+`end_offset` keeps counting produced bytes. Empty terminal logs advance to the
+end so followers can finish. Poll `next_offset`, or follow SSE `logs` events with keepalive comments.
+Disconnecting log follow never kills a detached job. Clients may reconnect using
+the last cursor when authority refresh closes a follow stream. Logs are untrusted
+command output. Keep durable artifacts in the workspace. At most eight jobs run
+at once; history retains the latest 256 jobs, evicting the oldest completed job
+on the next start.
+
+Running jobs count as activity even while quiet, preventing idle auto-pause.
+Billing enforcement, the machine session runtime cap, and explicit owner pause
+still apply. Memory pause freezes jobs; restoring that memory within its
+24-hour retention resumes them. The guest service runtime limit uses the guest
+clock; it does not imply a 24-hour uninterrupted billable machine session.
+Cold restarts, Sleeping, expired memory checkpoints and named snapshot restores
+lose processes: old running records become `lost` when inspected. Paused status
+is the manager's last observed status; logs require a running machine.
+
+Retryable pre-exec failures include `exec_concurrency_limit` (429),
+`runner_not_ready` (409), and `exec_slot_race` (409). Check `retry_after` and the
+`Retry-After` header. `exec_outcome_unknown` and `job_outcome_unknown` must never
+be retried automatically. MCP equivalents are `box_job_start`, `box_job_list`,
+`box_job_status`, `box_job_logs` (bounded polling), and `box_job_kill`.

npm/README.md

@@ -10,3 +10,71 @@ This package is a thin launcher for the [QuickSilver Pro CLI](https://github.com
 - One OpenAI-compatible API for frontier and open models: https://quicksilverpro.io
 - QuickSilver Machines: cloud dev boxes with Claude Code, Codex, OpenCode and Hermes ready out of the box. Agents ready out of the box. One unified bill. Zero friction.
 - Docs: https://quicksilverpro.io/docs/machines/cli/
+
+## Detached jobs
+
+
+For interactive commands, request `Accept: text/event-stream` on
+`POST /v1/machines/{mid}/exec`. Streams include `: keepalive` comments every
+15 seconds, including while commands are quiet. Ignore comment frames. Exec
+still has a 300-second total lifetime and stops its entire process group when
+the client disconnects. Buffered requests with `timeout_seconds > 90` are
+rejected before execution; use streaming or jobs. The CLI and MCP client
+select streaming automatically for timeouts over 60 seconds.
+
+For work that must survive a disconnected client, use detached jobs:
+
+```sh
+qsp box job start MACHINE_ID --command 'pytest > results.txt' --timeout 86400
+qsp box job ls MACHINE_ID
+qsp box job status MACHINE_ID JOB_ID
+qsp box job logs MACHINE_ID JOB_ID --tail 4096
+qsp box job logs MACHINE_ID JOB_ID -f --stream stderr
+qsp box job kill MACHINE_ID JOB_ID
+```
+
+`start` also accepts argv after `--`, `--argv-json`, `--cwd`, and repeatable
+`--env KEY=VALUE`. Commands run as `user`, inside `/home/user`, with the same
+environment restrictions as exec. Shell syntax requires explicit `command`.
+Do not use `nohup` or `&` inside exec to detach work.
+
+| Method | Path | Scope | Input/result |
+| --- | --- | --- | --- |
+| POST | `/v1/machines/{mid}/jobs` | `machines:exec` | Exec body; `timeout_seconds` defaults to 86400, max 259200; returns job status |
+| GET | `/v1/machines/{mid}/jobs` | `machines:read` | `{jobs: [...]}` |
+| GET | `/v1/machines/{mid}/jobs/{jid}` | `machines:read` | Status |
+| GET | `/v1/machines/{mid}/jobs/{jid}/logs` | `machines:read` | `stream=stdout|stderr`, `offset`, optional `tail` (0–65536 bytes), `follow=true` for SSE |
+| POST | `/v1/machines/{mid}/jobs/{jid}/kill` | `machines:exec` | Stops the job and its descendants |
+
+Dashboard sessions and scoped Machines keys use the existing ownership, machine
+restriction and human-control gates. Job operations do not create or resume a
+machine. Status contains `id`, `state` (`running`, `exited`, `killed`, `lost`),
+`exit_code`, Unix `started_at`/`ended_at`, and `max_runtime_seconds`. A killed or
+lost job may have no exit code. Starts are not idempotent: after an unknown
+outcome, list jobs before deciding whether to start again.
+
+Logs retain a circular 1 MiB per stream (2 MiB total per job). Responses
+contain base64 `data` and absolute byte cursors `offset`, `next_offset`,
+`start_offset`, `end_offset`. Each read returns at most 64 KiB. If a cursor is
+older than retained data, `offset` advances to `start_offset`; clients can detect
+the gap. Poll `next_offset`, or follow SSE `logs` events with keepalive comments.
+Disconnecting log follow never kills a detached job. Clients may reconnect using
+the last cursor when authority refresh closes a follow stream. Logs are untrusted
+command output. Keep durable artifacts in the workspace. At most eight jobs run
+at once; history retains the latest 256 jobs, evicting the oldest completed job
+on the next start.
+
+Running jobs count as activity even while quiet, preventing idle auto-pause.
+Billing enforcement, the machine session runtime cap, and explicit owner pause
+still apply. Memory pause freezes jobs; restoring that memory within its
+24-hour retention resumes them. The guest service runtime limit uses the guest
+clock; it does not imply a 24-hour uninterrupted billable machine session.
+Cold restarts, Sleeping, expired memory checkpoints and named snapshot restores
+lose processes: old running records become `lost` when inspected. Paused status
+is the manager's last observed status; logs require a running machine.
+
+Retryable pre-exec failures include `exec_concurrency_limit` (429),
+`runner_not_ready` (409), and `exec_slot_race` (409). Check `retry_after` and the
+`Retry-After` header. `exec_outcome_unknown` and `job_outcome_unknown` must never
+be retried automatically. MCP equivalents are `box_job_start`, `box_job_list`,
+`box_job_status`, `box_job_logs` (bounded polling), and `box_job_kill`.

npm/package.json

@@ -1,6 +1,6 @@
 {
   "name": "quicksilverpro",
-  "version": "0.5.2",
+  "version": "0.6.0",
   "description": "QuickSilver Pro CLI (qsp): one API for frontier models, and cloud dev boxes your agents can run. Thin npm launcher for the Python CLI.",
   "bin": {
     "quicksilverpro": "bin/qsp.js",

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "quicksilverpro"
-version = "0.5.2"
+version = "0.6.0"
 description = "CLI for QuickSilver Pro — one OpenAI-compatible API for Claude, GPT, Gemini, Grok, GLM, DeepSeek, Qwen, Kimi and Muse chat plus FLUX image generation, on a single balance."
 readme = "README.md"
 requires-python = ">=3.9"

src/quicksilverpro/__init__.py

@@ -3,4 +3,4 @@
 See https://quicksilverpro.io for docs. `qsp --help` for quick reference.
 """
 
-__version__ = "0.5.2"
+__version__ = "0.6.0"

src/quicksilverpro/box.py

@@ -222,7 +222,7 @@ def execute(mid, guest_argv, argv_json, command, cwd, timeout, stream, as_json):
         argv = list(guest_argv)
     with MachinesClient() as client:
         args = dict(argv=argv, command=command, cwd=cwd, timeout_seconds=timeout)
-        if stream:
+        if stream or (timeout > 60 and not as_json):
             result = None
             with closing(client.events(mid, **args)) as events:
                 for event in events:
@@ -388,3 +388,64 @@ def doctor(mid, as_json):
         if mid:
             result["machine"] = client.get(mid)
         emit(result, as_json)
+
+
+@box.group('job', help='Run and inspect work that survives a disconnected client.')
+def job():
+    pass
+
+@job.command('start')
+@click.argument('mid')
+@click.argument('guest_argv', nargs=-1, type=click.UNPROCESSED)
+@click.option('--argv-json')
+@click.option('--command')
+@click.option('--cwd')
+@click.option('--env', 'env_values', multiple=True, help='KEY=VALUE environment override; repeatable.')
+@click.option('--timeout', type=click.IntRange(1,259200), default=86400, show_default=True)
+@json_option
+def job_start(mid, guest_argv, argv_json, command, cwd, env_values, timeout, as_json):
+    try:
+        argv=json.loads(argv_json) if argv_json is not None else list(guest_argv) if guest_argv else None
+        if argv_json is not None and guest_argv:raise ValueError()
+        env=dict(value.split('=',1) for value in env_values)
+    except ValueError:raise click.UsageError('Choose valid argv JSON or argv after --; env must be KEY=VALUE.')
+    with MachinesClient() as client:
+        emit(client.job_start(mid,argv=argv,command=command,cwd=cwd,env=env,timeout_seconds=timeout),as_json)
+
+@job.command('ls')
+@click.argument('mid')
+@json_option
+def job_list(mid, as_json):
+    with MachinesClient() as client:emit(client.job(mid),as_json)
+
+@job.command('status')
+@click.argument('mid')
+@click.argument('job_id')
+@json_option
+def job_status(mid, job_id, as_json):
+    with MachinesClient() as client:emit(client.job(mid,'status',job_id),as_json)
+
+@job.command('kill')
+@click.argument('mid')
+@click.argument('job_id')
+@json_option
+def job_kill(mid, job_id, as_json):
+    with MachinesClient() as client:emit(client.job(mid,'kill',job_id),as_json)
+
+@job.command('logs')
+@click.argument('mid')
+@click.argument('job_id')
+@click.option('--follow','-f',is_flag=True)
+@click.option('--stream',type=click.Choice(['stdout','stderr']),default='stdout')
+@click.option('--offset',type=click.IntRange(0),default=0)
+@click.option('--tail',type=click.IntRange(0,65536))
+@json_option
+def job_logs(mid, job_id, follow, stream, offset, tail, as_json):
+    import base64
+    options={'stream':stream,'offset':offset}
+    if tail is not None:options['tail']=tail
+    with MachinesClient() as client:
+        events=client.job_follow(mid,job_id,**options) if follow else iter([client.job(mid,'logs',job_id,**options)])
+        for event in events:
+            if as_json:emit(event,True,'job_logs')
+            else:click.echo(base64.b64decode(event['data']).decode('utf-8',errors='replace'),nl=False,err=stream=='stderr')

src/quicksilverpro/machines.py

@@ -155,7 +155,7 @@ def workspace_path(path):
     return str(PurePosixPath(path))
 
 
-def exec_body(argv=None, command=None, cwd=None, timeout_seconds=30, env=None):
+def exec_body(argv=None, command=None, cwd=None, timeout_seconds=30, env=None, _max_timeout=300):
     if (argv is None) == (command is None):
         raise MachineError("invalid_exec", "Provide exactly one of argv or command.", 2)
     if argv is not None and (
@@ -170,8 +170,8 @@ def exec_body(argv=None, command=None, cwd=None, timeout_seconds=30, env=None):
         raise MachineError(
             "invalid_exec", "command must contain 1..32768 characters without NUL.", 2
         )
-    if not isinstance(timeout_seconds, int) or not 1 <= timeout_seconds <= 300:
-        raise MachineError("invalid_exec", "Exec timeout must be 1..300 seconds.", 2)
+    if not isinstance(timeout_seconds, int) or not 1 <= timeout_seconds <= _max_timeout:
+        raise MachineError("invalid_exec", "Runtime must be 1..%s seconds." % _max_timeout, 2)
     body = {"timeout_seconds": timeout_seconds}
     body["argv" if argv is not None else "command"] = (
         argv if argv is not None else command
@@ -462,10 +462,10 @@ def exec_response(self, mid, body, stream=False):
                 # Exact public pre-execution rejection, never a generic conflict or stream error.
                 remaining = deadline - time.monotonic()
                 if (
-                    error.details.get("status_code") != 409
+                    error.details.get("status_code") not in (409, 429)
                     or error.code in ("outcome_unknown", "exec_outcome_unknown")
-                    or error.message
-                    not in ("Machine is not ready.", "Guest runner is not ready.")
+                    or (error.code not in ('runner_not_ready', 'exec_slot_race', 'exec_concurrency_limit') and error.message
+                    not in ("Machine is not ready.", "Guest runner is not ready."))
                     or remaining <= 0
                     or attempt == 9
                 ):
@@ -482,6 +482,14 @@ def exec_response(self, mid, body, stream=False):
 
     def execute(self, mid, **kwargs):
         body = exec_body(**kwargs)
+        if body['timeout_seconds'] > 60:
+            result = None
+            output = {'stdout': [], 'stderr': []}
+            for event in self.events(mid, **kwargs):
+                if event['event'] in output:output[event['event']].append(event['data'])
+                elif event['event'] == 'exit':result = event
+            if result is None:raise MachineError('outcome_unknown', 'Missing execution exit evidence.')
+            return {**{k:v for k,v in result.items() if k not in ('event','seq')}, **{k:''.join(v) for k,v in output.items()}}
         try:
             with self.exec_response(mid, body) as response:
                 response.read()
@@ -535,6 +543,8 @@ def events(self, mid, **kwargs):
                             "output_limit",
                             "Execution stream frame exceeds output limit.",
                         )
+                    if not line and not chunks:
+                        event, frame_size = 'message', 0
                     if line.startswith("event:"):
                         event = line[6:].strip()
                     elif line.startswith("data:"):
@@ -582,6 +592,43 @@ def events(self, mid, **kwargs):
                 "Execution stream interrupted or malformed; do not retry.",
             )
 
+    def job_start(self, mid, **kwargs):
+        kwargs.setdefault('timeout_seconds', 86400)
+        body = exec_body(**kwargs, _max_timeout=259200)
+        return self.request('POST', '/v1/machines/' + mid_value(mid) + '/jobs', json=body)
+
+    def job(self, mid, action='list', job_id=None, **options):
+        path = '/v1/machines/' + mid_value(mid) + '/jobs'
+        if action != 'list':
+            if not isinstance(job_id, str) or not re.fullmatch('[a-f0-9]{32}', job_id):
+                raise MachineError('invalid_job', 'Invalid job ID.', 2)
+            path += '/' + job_id
+        if action in ('kill', 'logs'):path += '/' + action
+        if action not in ('list', 'status', 'kill', 'logs'):
+            raise MachineError('invalid_job', 'Invalid job operation.', 2)
+        return self.request('POST' if action == 'kill' else 'GET', path, **({'json':{}} if action == 'kill' else {'params':options} if options else {}))
+
+    def job_follow(self, mid, job_id, **options):
+        import base64
+        if not re.fullmatch('[a-f0-9]{32}', job_id):raise MachineError('invalid_job', 'Invalid job ID.', 2)
+        self.ensure_active()
+        path = '/v1/machines/' + mid_value(mid) + '/jobs/' + job_id + '/logs'
+        # Reconnect with the absolute cursor: following never starts new work.
+        try:
+            while True:
+                with self.http.stream('GET', path, params={**options, 'follow':'true'}, timeout=35) as response:
+                    self.check(response)
+                    for line in response.iter_lines():
+                        if line.startswith('data: '):
+                            if len(line)>131072:raise MachineError('output_limit', 'Invalid job log frame.')
+                            event=json.loads(line[6:]);base64.b64decode(event['data'],validate=True)
+                            options.pop('tail',None);options['offset']=event['next_offset']
+                            yield event
+                            if event['state']!='running' and event['next_offset']>=event['end_offset']:return
+                self.ensure_active()
+        except (httpx.HTTPError, ValueError, KeyError, TypeError):
+            raise MachineError('job_logs_interrupted', 'Job log stream interrupted; resume from the last next_offset.', next_offset=options.get('offset',0)) from None
+
     def files(self, mid, path="."):
         return self.request(
             "GET",