keboola · padak · Aug 24, 2026
@@ -9,7 +9,7 @@ auth, and the concepts behind these routes live in
 [`web-server.md`](web-server.md); a running server serves the same spec
 interactively at `/docs` (Swagger) and `/openapi.json`.
 
-**231 operations** across **202 paths** and **30 routers**.
+**233 operations** across **204 paths** and **30 routers**.
 
 Paths are shown as the server registers them. Reaching them through the
 Node BFF (or single-process `--ui` mode) prefixes every path with `/api`.
@@ -166,7 +166,7 @@ Encrypt secret values for a specific project + component using the Keboola encry
 
 ## Data
 
-### `storage` (32 operations)
+### `storage` (34 operations)
 
 Buckets, tables, columns, files. Create, upload, download, describe, swap, delete. Mirrors `kbagent storage *`.
 
@@ -196,7 +196,9 @@ Buckets, tables, columns, files. Create, upload, download, describe, swap, delet
 | `POST` | `/storage/table-from-snapshot/{project}` | Create a NEW table from a snapshot |
 | `POST` | `/storage/tables/{project}/{table_id}/describe` | Set table description |
 | `POST` | `/storage/columns/{project}/{table_id}/describe` | Set column descriptions |
+| `POST` | `/storage/describe-batch/{project}` | Apply descriptions in bulk |
 | `POST` | `/storage/columns/{project}/describe-migrate` | Migrate legacy column descriptions |
+| `POST` | `/storage/tables/{project}/{table_id}/unload` | Export a table to a file |
 | `GET` | `/storage/files` | List storage files |
 | `POST` | `/storage/files/upload` | Upload a file to Storage |
 | `GET` | `/storage/files/{project}/{file_id}` | Get file detail |

@@ -106,8 +106,10 @@ read/audit endpoints, three deliberate gaps" below.
 
 Going the other way, several CLI surfaces are deliberately CLI-only: `sync`
 (filesystem-local by design), `permissions`, and `init`. The mirrors still
-considered missing are tracked in #657, and the fact that `permissions`
-constrains only `/auth/*` and not the other ~30 routers is #655.
+considered missing are tracked in #657 — `describe-batch` (inline payload),
+`unload-table` and `job run`'s idempotency fields came off that list in vNEXT;
+the fact that `permissions` constrains only `/auth/*` and not the other ~30
+routers is #655.
 
 Auto-generated OpenAPI spec at `/openapi.json`, Swagger UI at `/docs`.
 
@@ -572,6 +574,25 @@ for a human), then use `POST /auth/register-projects` — or `auth
 register-projects` on the CLI — to register the resulting session's projects
 for `serve` to use.
 
+### CLI options that a REST mirror deliberately drops
+
+*(since vNEXT — issue #657)*
+
+Three mirrors landed with a shape that is not a field-for-field copy of the CLI
+command, because the CLI's version of the option refers to the *caller's*
+filesystem — which, over `serve`, is the server's:
+
+| CLI | REST | Difference |
+|---|---|---|
+| `storage describe-batch --from-file PATH` | `POST /storage/describe-batch/{project}` | Sections travel **inline in the body** (`buckets` / `tables` / `columns` / `branch_id`). Same validator (`services/_describe_batch_input.py`), so both surfaces accept and reject exactly the same documents, whole, before the first write. |
+| `storage unload-table [--download] [--output]` | `POST /storage/tables/{project}/{table_id}/unload` | No `download` / `output`. The response carries the Storage `file_id`; fetch the bytes with `GET /storage/files/{project}/{file_id}/download`. That is also the only shape that works for `file_type: "parquet"`, whose export is sliced. |
+| `job run --idempotency-key KEY [--force-rerun]` | `POST /jobs/{project}/run` | Same two fields, now on the body. Retrying a POST is the canonical case #427's store was built for, so the mirror omitting them left the caller who needs them most without them. Dedup is scoped to the **served** config dir, i.e. per machine. |
+
+One error-shape note: a malformed `describe-batch` body answers **422**
+(`HTTP_ERROR`) where the CLI exits **2** (`INVALID_ARGUMENT`). The message is
+the same one #645 writes — the offending key, its actual type, and an example —
+but the code differs, so branch on the status for this one, not the code.
+
 ### Manage tokens are per-request
 
 Operations that need a Keboola Manage API token (`org setup`,

@@ -4506,6 +4506,19 @@ shapes.
   keep) must be in its allow list, or the server will not start either.
   Pass `--config-dir` to `serve` itself: the root-level `kbagent --config-dir
   ... serve` sets the dir for the CLI invocation, not for the server process.
+- **Three `serve` mirrors landed in vNEXT (#657) with a deliberately different
+  shape from their CLI command** -- `POST /storage/describe-batch/{project}`
+  takes the `buckets`/`tables`/`columns` sections INLINE in the body (no
+  `--from-file` path: a REST caller's filesystem is not the server's), and it
+  answers **422** for a malformed document where the CLI exits 2
+  `INVALID_ARGUMENT` -- same message, different code, so branch on the status.
+  `POST /storage/tables/{project}/{table_id}/unload` has no `--download` /
+  `--output`; it returns the Storage `file_id` and you fetch the bytes with
+  `GET /storage/files/{project}/{file_id}/download` (the only shape that works
+  for `file_type: "parquet"`, whose export is sliced). `POST /jobs/{p}/run`
+  finally accepts `idempotency_key` / `force_rerun` -- retrying a POST is
+  exactly what #427's store is for, and before vNEXT the REST caller could not
+  reach it; dedup is scoped to the SERVED config dir, i.e. per machine.
 - **A deny policy does NOT firewall the whole REST surface.**
   `permissions set --mode deny --deny cli:write` blocks `POST
   /auth/register-projects` (HTTP 403, `error_code: PERMISSION_DENIED`) but

@@ -31,6 +31,14 @@ class JobRun(BaseModel):
     variable_values_id: str | None = None
     no_variables: bool = False
     mode: str = DEFAULT_JOB_MODE
+    # Client-side de-duplication (#427, exposed over REST by #657). Retrying a
+    # POST is the canonical case the store was built for -- a timed-out or
+    # replayed request must not start a second job -- so the REST mirror
+    # dropping these two fields left the one caller that needs them most
+    # without them. The store is keyed inside the SERVED config dir, so dedup
+    # is per-machine, exactly as on the CLI.
+    idempotency_key: str | None = None
+    force_rerun: bool = False
 
 
 class JobTerminate(BaseModel):
@@ -93,7 +101,14 @@ def run(
     log_tail_lines: int = DEFAULT_LOG_TAIL_LINES,
     registry: ServiceRegistry = Depends(get_registry),
 ) -> dict[str, Any]:
-    """Run a component configuration and optionally wait for completion. Mirrors `kbagent job run`."""
+    """Run a component configuration and optionally wait for completion.
+
+    Mirrors `kbagent job run`, including its client-side idempotency
+    (`idempotency_key` / `force_rerun`, #427): replaying this POST under the
+    same key returns the recorded job instead of starting a second one, unless
+    that job failed or `force_rerun` is set. Reusing a key for a *different*
+    component/config raises rather than return the wrong job.
+    """
     return registry.job.run_job(
         alias=project,
         component_id=body.component_id,
@@ -107,6 +122,8 @@ def run(
         poll_strategy=poll_strategy,
         log_tail_lines=log_tail_lines,
         mode=body.mode,
+        idempotency_key=body.idempotency_key,
+        force_rerun=body.force_rerun,
     )
 
 

@@ -15,6 +15,47 @@
 router = APIRouter(prefix="/storage", tags=["storage"])
 
 
+class DescribeBatch(BaseModel):
+    """Inline payload for the bulk-documentation mirror (issue #657).
+
+    The CLI takes a YAML *path*; a REST caller has a JSON *body*, and asking
+    it to first write a file onto the server's filesystem would be both
+    awkward and a way to smuggle a path in. So the three sections travel in
+    the body instead, and everything else -- the rules, the messages, the
+    reject-whole-before-first-write guarantee -- is literally the shared
+    validator (`services/_describe_batch_input.py`).
+
+    The sections are typed ``Any`` on purpose. Declaring them
+    ``dict[str, str]`` would have pydantic reject a wrong shape first, with
+    its generic 422, discarding the whole point of #645's messages: the key
+    that is wrong, its actual type, and a copy-pasteable example.
+    """
+
+    buckets: Any = None
+    tables: Any = None
+    columns: Any = None
+    branch_id: int | None = None
+
+
+class UnloadTable(BaseModel):
+    """Body for the `storage unload-table` mirror (issue #657).
+
+    The CLI's `--download` / `--output` are deliberately absent: they write to
+    the *caller's* filesystem, which for a REST caller is the server's.
+    The route returns the Storage file id instead, and
+    `GET /storage/files/{project}/{file_id}/download` fetches the bytes --
+    which is also the only shape that works for `parquet`, whose export is
+    sliced and has no single-file download even on the CLI.
+    """
+
+    columns: list[str] | None = None
+    limit: int | None = None
+    tags: list[str] | None = None
+    file_type: str = "csv"
+    keep_slices: bool = False
+    branch_id: int | None = None
+
+
 class CreateBucket(BaseModel):
     stage: str
     name: str
@@ -580,6 +621,36 @@ def describe_columns(
     )
 
 
+@router.post("/describe-batch/{project}", summary="Apply descriptions in bulk")
+def describe_batch(
+    project: str,
+    body: DescribeBatch,
+    registry: ServiceRegistry = Depends(get_registry),
+) -> dict[str, Any]:
+    """Apply bucket/table/column descriptions in one call.
+
+    Mirrors `kbagent storage describe-batch --from-file`, with the document
+    inline instead of on disk. Same validation, so the whole payload is
+    rejected before the first write if any section has the wrong shape; a
+    per-item API failure afterwards is accumulated into `errors` and does not
+    stop the remaining items.
+
+    A shape error answers **422** (`HTTP_ERROR`) rather than the CLI's exit-2
+    `INVALID_ARGUMENT` -- it is a malformed request body, the same class of
+    problem FastAPI already answers 422 for on this surface. The message is
+    the CLI's, naming the offending key and its actual type.
+    """
+    document = {"buckets": body.buckets, "tables": body.tables, "columns": body.columns}
+    try:
+        return registry.storage.describe_batch_document(
+            alias=project,
+            document=document,
+            branch_id=body.branch_id,
+        )
+    except ValueError as exc:
+        raise HTTPException(status_code=422, detail=str(exc)) from None
+
+
 @router.post("/columns/{project}/describe-migrate", summary="Migrate legacy column descriptions")
 def describe_migrate(
     project: str,
@@ -597,6 +668,39 @@ def describe_migrate(
     )
 
 
+@router.post("/tables/{project}/{table_id:path}/unload", summary="Export a table to a file")
+def unload_table(
+    project: str,
+    table_id: str,
+    body: UnloadTable | None = None,
+    registry: ServiceRegistry = Depends(get_registry),
+) -> dict[str, Any]:
+    """Export a table into a Storage File and return its metadata.
+
+    Mirrors `kbagent storage unload-table`. The sync `GET
+    /storage/table-preview/...` covers small reads; this covers the big ones,
+    where the async export + a presigned Storage file is the whole point.
+
+    Nothing is written to the server's filesystem: the response carries the
+    `file_id`, and `GET /storage/files/{project}/{file_id}/download` fetches
+    the bytes. `file_type: "parquet"` produces a SLICED file with no
+    single-file download -- use `GET /storage/files/{project}/{file_id}` and
+    work with the slices.
+    """
+    params = body or UnloadTable()
+    return registry.storage.unload_table_to_file(
+        alias=project,
+        table_id=table_id,
+        columns=params.columns,
+        limit=params.limit,
+        tags=params.tags,
+        download=False,
+        branch_id=params.branch_id,
+        file_type=params.file_type,
+        keep_slices=params.keep_slices,
+    )
+
+
 # Registered AFTER the more specific /columns/.../describe route above: the
 # greedy {table_id:path} would otherwise shadow that POST and swallow a
 # ".../describe" suffix as part of the table id.

@@ -180,6 +180,41 @@ def _columns_section(raw: dict[str, Any]) -> dict[str, dict[str, str]]:
     return parsed
 
 
+def parse_describe_batch_document(raw: Any, source_label: str) -> DescribeBatchInput:
+    """Validate an already-loaded describe-batch document.
+
+    Split out from :func:`parse_describe_batch_file` so the REST mirror
+    (``POST /storage/describe-batch/{project}``, issue #657) validates the
+    JSON body through exactly the same rules as the CLI's YAML file. Two
+    copies of these rules would drift, and the half that drifted would be the
+    one that lets a malformed section reach the write loop.
+
+    Args:
+        raw: The parsed document -- YAML from a file, or a JSON request body.
+        source_label: What to name in an error message ("describe.yml",
+            "request body"). Only ever used for the top-level shape error;
+            per-section errors already name their own key.
+
+    Returns:
+        The validated sections, descriptions coerced to ``str``. Absent,
+        ``None`` and empty sections come back empty.
+
+    Raises:
+        ValueError: The document is not a mapping at the top level, or any
+            section/entry has a wrong non-empty shape, a null description, or
+            a duplicate coerced key.
+    """
+    if _is_empty(raw):
+        return DescribeBatchInput()
+    if not isinstance(raw, dict):
+        raise _shape_error(source_label, _TOP_LEVEL_SUBJECT, raw, _TOP_LEVEL_EXAMPLE)
+    return DescribeBatchInput(
+        buckets=_scalar_section(raw, "buckets"),
+        tables=_scalar_section(raw, "tables"),
+        columns=_columns_section(raw),
+    )
+
+
 def parse_describe_batch_file(from_file: Path) -> DescribeBatchInput:
     """Read and validate a describe-batch YAML file.
 
@@ -191,10 +226,9 @@ def parse_describe_batch_file(from_file: Path) -> DescribeBatchInput:
         ``None`` and empty sections come back empty.
 
     Raises:
-        ValueError: The file is missing, is not valid YAML, is not a mapping
-            at the top level, or any section/entry has a wrong non-empty
-            shape, a null description, or a duplicate coerced key. The command
-            layer maps this to ``INVALID_ARGUMENT`` and exit 2.
+        ValueError: The file is missing, is not valid YAML, or fails any check
+            in :func:`parse_describe_batch_document`. The command layer maps
+            this to ``INVALID_ARGUMENT`` and exit 2.
     """
     import yaml
 
@@ -204,12 +238,4 @@ def parse_describe_batch_file(from_file: Path) -> DescribeBatchInput:
         raw = yaml.safe_load(from_file.read_text(encoding="utf-8"))
     except yaml.YAMLError as exc:
         raise ValueError(f"Batch file is not valid YAML: {exc}") from None
-    if _is_empty(raw):
-        return DescribeBatchInput()
-    if not isinstance(raw, dict):
-        raise _shape_error(from_file.name, _TOP_LEVEL_SUBJECT, raw, _TOP_LEVEL_EXAMPLE)
-    return DescribeBatchInput(
-        buckets=_scalar_section(raw, "buckets"),
-        tables=_scalar_section(raw, "tables"),
-        columns=_columns_section(raw),
-    )
+    return parse_describe_batch_document(raw, from_file.name)