feat: Implement AllowlistCORSMiddleware for private network access co…

CHANGELOG.md

@@ -27,6 +27,39 @@ project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ---
 
+## [1.0.0rc3] - 2026-10-30
+
+The third release candidate of `10xgraph-api` 1.0. It requires `10xgraph` 1.0.0rc3 (stores have
+`aping()`) and is the `1.0.0rc2` code plus the changes below. `10xgraph-client` stays at 1.0.0-rc.2.
+
+### Added
+
+- The server refuses to start while a configured backend is unreachable: the checkpointer's
+  Postgres (query and schema version) and Redis, the store (when it has `aping`/`ping`), the
+  API's Redis (`redis` in `10xgraph.json` or `REDIS_URL`) and the rate limiter's own Redis. It
+  retries with a short backoff for `STARTUP_CHECK_TIMEOUT_SECONDS` (default 30), so services
+  started together can come up in any order, then exits non-zero with each failed component, its
+  `host:port` (no credentials) and the error. This applies in production too, so a broken
+  connection string fails the deploy instead of the first request. `STARTUP_CHECKS=false` skips
+  it. Before, the server started anyway and, with a fail-closed Redis rate limiter, answered every
+  request except `/ping` with 429.
+
+### Changed
+
+- A rate-limiter backend outage with `fail_open: false` now answers `503` with error code
+  `RATE_LIMIT_UNAVAILABLE` and `Retry-After: 5` instead of `429 RATE_LIMIT_EXCEEDED`; 429 is kept
+  for a real limit hit. Clients that retried on 429 now also get 503 during a Redis outage and
+  should retry on it too. A custom backend can signal the same with
+  `RateLimitDecision.backend_unavailable()`; backends that return only allow/deny work unchanged.
+  The Redis backend logs a traceback for the first error of an outage and one line per error
+  after it, instead of a traceback per request.
+
+### Fixed
+
+- The hosted playground can reach a local `10xgraph dev` server: CORS answers the browser's
+  private-network preflight (`Access-Control-Request-Private-Network`) for origins in `ORIGINS`.
+  Before, it answered 400 and the browser reported a CORS error.
+
 ## [1.0.0rc2] - 2026-10-30
 
 The second release candidate of `10xgraph-api` 1.0. It requires `10xgraph` 1.0.0rc2 and is aligned

CLAUDE.md

@@ -5,13 +5,13 @@ core framework see `agentflow/CLAUDE.md`; for the TS client, docs, or playground
 for the monorepo overview see the workspace-root `CLAUDE.md`.
 
 - Package name (PyPI): `10xgraph-api` (formerly `10xscale-agentflow-cli`, last release 0.6.x)
-- Version: `1.0.0rc2` (`pyproject.toml`; 0.7.0 was the first release under the 10xGraph name). `CLI_VERSION`
+- Version: `1.0.0rc3` (`pyproject.toml`; 0.7.0 was the first release under the 10xGraph name). `CLI_VERSION`
   and `tenxgraph_api.__version__` are single-sourced from the installed distribution metadata
   (falling back to `pyproject.toml` only for a non-installed source checkout).
 - Requires: Python >= 3.12 · Status: `5 - Production/Stable`
 - Console entry points: `10xgraph = tenxgraph_api.cli.main:main`, plus the deprecated alias
   `agentflow = tenxgraph_api.cli.main:legacy_main` (prints a notice to stderr; removed in 2.0)
-- Depends on the core framework: `10xgraph>=1.0.0rc2,<2.0` (import `tenxgraph`). Must not be
+- Depends on the core framework: `10xgraph>=1.0.0rc3,<2.0` (import `tenxgraph`). Must not be
   installed alongside the old `10xscale-agentflow`: both provide an `agentflow` module.
 
 ## Rename compatibility (kept until 2.0)

README.md

@@ -527,7 +527,7 @@ sdist + wheel, checks the distribution metadata, and creates a GitHub Release wi
 notes and the artifacts attached. PyPI publishing is manual (`make publish`).
 
 ```bash
-git tag v1.0.0rc2 && git push origin v1.0.0rc2
+git tag v1.0.0rc3 && git push origin v1.0.0rc3
 ```
 
 ---

RELEASE_NOTES.md

@@ -1,4 +1,30 @@
-# 10xgraph-api 1.0.0rc2
+# 10xgraph-api 1.0.0rc3
+
+The third 1.0 release candidate: the `1.0.0rc2` code plus a startup check for every configured backend, a clear 503 when the rate limiter's backend is down, and a CORS fix so the hosted playground can reach a local `10xgraph dev` server.
+
+```bash
+pip install -U --pre "10xgraph-api==1.0.0rc3"
+```
+
+This pulls `10xgraph` 1.0.0rc3. `10xgraph-client` stays at 1.0.0-rc.2 (no client changes).
+
+## Before you upgrade (changes in rc3)
+
+- **The server refuses to start while a configured backend is unreachable**, in development and production. It checks the checkpointer's Postgres (query and schema version) and Redis, the memory store (`aping()`, now required in `10xgraph` 1.0.0rc3), the API's Redis and the rate limiter's Redis. It retries for `STARTUP_CHECK_TIMEOUT_SECONDS` (default 30), then exits naming each failed backend and its `host:port` (no credentials). Set `STARTUP_CHECKS=false` to skip it.
+- **A rate-limiter backend outage answers 503, not 429.** With `fail_open: false` and Redis unreachable, requests get `503 RATE_LIMIT_UNAVAILABLE` with `Retry-After: 5`. A real limit hit is still `429 RATE_LIMIT_EXCEEDED`. Retry on 503 as well as 429.
+- **Running `10xgraph dev` on your machine against Docker backends:** publish Postgres, Redis and Qdrant on `127.0.0.1` and use `localhost` in `.env`. The compose service names only resolve inside the Docker network, and the startup check now reports this at boot.
+
+## What changed in rc3
+
+- The hosted playground (`https://play.10xgraph.com`) can reach a local `10xgraph dev` server: CORS answers the browser's private-network preflight for origins in `ORIGINS` (before, it answered 400 and the browser reported a CORS error). Chrome also asks the user to allow access to the local machine; the playground now shows the steps.
+- A custom rate-limit backend can signal an outage with `RateLimitDecision.backend_unavailable()`.
+- The Redis rate-limit backend logs one traceback per outage instead of one per request.
+
+See [CHANGELOG.md](CHANGELOG.md) for the full list. The rc2 notes follow.
+
+---
+
+## 10xgraph-api 1.0.0rc2
 
 The second 1.0 release candidate: it is the `1.0.0rc1` code plus graceful shutdown, stricter request validation, 409 answers for writes during a run, idempotency keys, JWT key rotation and a redesigned eval and config experience.
 

SECURITY.md

@@ -7,7 +7,7 @@
 | `1.0.x` (`10xgraph-api`) | Yes |
 | `< 1.0` | No |
 
-`10xgraph-api` is at 1.0 (release candidate `1.0.0rc2`). Security fixes are applied to the
+`10xgraph-api` is at 1.0 (release candidate `1.0.0rc3`). Security fixes are applied to the
 latest published release only; there are no long-term support branches. The old
 `10xscale-agentflow-cli` package is no longer published; upgrade to `10xgraph-api` to receive
 fixes.

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "10xgraph-api"
-version = "1.0.0rc2"
+version = "1.0.0rc3"
 description = "Production server for 10xGraph agents (formerly 10xscale-agentflow-cli). Serves a compiled graph over REST, SSE, WebSocket, realtime audio and AG-UI with JWT or custom auth, scoped authorization on every endpoint, owner-only threads and rate limiting, and generates Docker Compose and Kubernetes files. Includes the 10xgraph CLI. MIT licensed, self-hosted."
 readme = "README.md"
 license = "MIT"
@@ -60,7 +60,7 @@ classifiers = [
 dependencies = [
     # This release needs core 1.0 (import `tenxgraph`). The explicit rc in the
     # specifier lets pip and uv pick the core pre-release until 1.0.0 is out.
-    "10xgraph>=1.0.0rc2,<2.0",
+    "10xgraph>=1.0.0rc3,<2.0",
     # Pre-1.0 and major-version-risky deps are capped; floors are the versions
     # the suite is verified against.
     "fastapi>=0.116,<1.0",
@@ -105,7 +105,7 @@ media = [
 # Pillow for image resize / EXIF / format conversion in core's media processor.
 # Without it, uploaded images pass through unchanged.
 images = [
-    "10xgraph[images]>=1.0.0rc2,<2.0",
+    "10xgraph[images]>=1.0.0rc3,<2.0",
 ]
 gcloud = [
     "google-cloud-logging",
@@ -312,6 +312,8 @@ convention = "google"
 [tool.pytest.ini_options]
 env = [
     "ENVIRONMENT=pytest",
+    # Tests build the app without real backends; the startup gate has its own tests.
+    "STARTUP_CHECKS=false",
 ]
 testpaths = [
     "tests",
@@ -374,7 +376,7 @@ ENVIRONMENT = "pytest"
 
 [dependency-groups]
 dev = [
-    "10xgraph>=1.0.0rc2",
+    "10xgraph>=1.0.0rc3",
     "snowflakekit",
     "pytest>=8.4.2",
     "pytest-asyncio>=1.2.0",

tenxgraph_api/cli/config_editor/schema.py

@@ -301,7 +301,10 @@
                 "default": True,
                 "label": "Fail open",
                 "type": "bool",
-                "help": "Allow requests when the backend errors. Off means deny.",
+                "help": (
+                    "Allow requests when the backend errors. Off means deny them with 503 "
+                    "RATE_LIMIT_UNAVAILABLE."
+                ),
             },
             {
                 "path": ["rate_limit", "trusted_proxy_headers"],
@@ -949,6 +952,20 @@
                     "docs enabled. Unset means on in production; false only warns."
                 ),
             },
+            {
+                "name": "STARTUP_CHECKS",
+                "purpose": (
+                    "Refuse to start while a configured Postgres, Redis or store is unreachable, "
+                    "in development and production. Defaults to true; false skips the check."
+                ),
+            },
+            {
+                "name": "STARTUP_CHECK_TIMEOUT_SECONDS",
+                "purpose": (
+                    "Seconds startup keeps retrying an unreachable backend before it gives up. "
+                    "Defaults to 30."
+                ),
+            },
         ],
     },
     {

tenxgraph_api/cli/main.py

@@ -49,7 +49,7 @@ def execute(self, *args: Any, **kwargs: Any) -> int:
                     hints=(
                         "Run `10xgraph audit` to inspect installed package compatibility.",
                         "Upgrade with `python -m pip install --upgrade "
-                        '"10xgraph>=1.0.0rc2,<2"`.',
+                        '"10xgraph>=1.0.0rc3,<2"`.',
                     ),
                 ) from exc
             implementation = getattr(module, class_name)

tenxgraph_api/cli/templates/skills/10xgraph/references/api-settings-and-middleware.md

@@ -19,6 +19,8 @@ Important variables:
 - `DOCS_PATH`
 - `REDOCS_PATH`
 - `REDIS_URL`
+- `STARTUP_CHECKS`
+- `STARTUP_CHECK_TIMEOUT_SECONDS`
 - `SENTRY_DSN`
 - `JWT_SECRET_KEY`
 - `JWT_ALGORITHM`
@@ -64,6 +66,14 @@ default in production (`STRICT_PRODUCTION_CHECKS=false` downgrades them to warni
 - docs endpoints enabled
 - `ALLOWED_HOST=*`
 
+## Startup Checks
+
+In every mode, the server refuses to start while a configured backend is unreachable: the
+checkpointer's Postgres and Redis, the store (when it has `aping`/`ping`), `REDIS_URL` (or
+`redis` in `10xgraph.json`) and the rate limiter's Redis. It retries for
+`STARTUP_CHECK_TIMEOUT_SECONDS` (30) first, then exits with each failed component, its
+`host:port` and the error. `STARTUP_CHECKS=false` skips the check.
+
 ## Sentry
 
 `SENTRY_DSN` enables Sentry setup through the API config module. Keep error reporting optional and safe when unset.

tenxgraph_api/cli/templates/skills/10xgraph/references/rate-limiting.md

@@ -78,7 +78,7 @@ single Redis operation, which prevents concurrent requests from racing past the
 | `trusted_proxy_headers` | boolean | `false` | Use `X-Forwarded-For` as the client IP (only behind a trusted proxy). |
 | `redis.url` | string | `null` | Redis URL; required for the Redis backend. Supports `${ENV_VAR}` expansion. |
 | `redis.prefix` | string | `"10xgraph:rate-limit"` | Prefix for all Redis keys. |
-| `fail_open` | boolean | `true` | On Redis errors, allow (`true`) or deny (`false`) requests. |
+| `fail_open` | boolean | `true` | On Redis errors, allow (`true`) or deny (`false`, 503) requests. |
 
 ## Identity Modes
 
@@ -107,7 +107,7 @@ Every response includes:
 | `X-RateLimit-Remaining` | Requests remaining in the current window. |
 | `X-RateLimit-Reset` | Unix timestamp for the window reset estimate. |
 | `X-RateLimit-Reset-After` | Seconds until the window reset estimate. |
-| `Retry-After` | Present on `429` responses only. |
+| `Retry-After` | Present on `429` and `503` responses only. |
 
 When the limit is exceeded, 10xGraph returns `429 Too Many Requests`:
 
@@ -127,6 +127,24 @@ When the limit is exceeded, 10xGraph returns `429 Too Many Requests`:
 }
 ```
 
+When the backend cannot be reached and `fail_open` is `false`, the request is denied with
+`503 Service Unavailable` and `Retry-After: 5`, not 429, so clients can tell an outage from a
+limit hit:
+
+```json
+{
+  "error": {
+    "code": "RATE_LIMIT_UNAVAILABLE",
+    "message": "Rate limiter unavailable: its backend could not be reached.",
+    "retry_after_seconds": 5
+  },
+  "metadata": {
+    "request_id": "request-id",
+    "status": "error"
+  }
+}
+```
+
 ## Custom Backend
 
 Implement `BaseRateLimitBackend` and bind the instance through InjectQ, then set
@@ -154,6 +172,9 @@ class MyRateLimitBackend(BaseRateLimitBackend):
         return None
 ```
 
+When the backend's store cannot be reached and the request must not pass, return
+`RateLimitDecision.backend_unavailable()`; the middleware answers it with the 503 above.
+
 ```json
 {
   "rate_limit": {

tenxgraph_api/src/app/core/config/settings.py

@@ -160,6 +160,19 @@ class Settings(BaseSettings):
     # ``redis_key_prefix`` option so its cache keys are separated too.
     REDIS_KEY_PREFIX: str = ""
 
+    #################################
+    ###### Startup checks ###########
+    #################################
+    # At startup, before serving, ping every configured backend (the checkpointer's Postgres
+    # and Redis, the store, the API's Redis, the rate limiter's Redis) and refuse to start when
+    # one stays unreachable. Applies in development and production, so a wrong connection
+    # string fails the boot (and the deploy) instead of the first request. False skips it.
+    STARTUP_CHECKS: bool = True
+    # How long, in seconds, startup keeps retrying a failing backend (with a short backoff)
+    # before it gives up, so services started together (docker compose) can come up in any
+    # order. Each attempt is bounded by the readiness probe's own per-check timeout.
+    STARTUP_CHECK_TIMEOUT_SECONDS: float = Field(default=30.0, ge=0.0)
+
     #################################
     ###### Shutdown #################
     #################################

tenxgraph_api/src/app/core/config/setup_middleware.py

@@ -7,9 +7,10 @@
 from fastapi.middleware.trustedhost import TrustedHostMiddleware
 from fastapi.responses import JSONResponse
 from injectq import InjectQ
-from starlette.datastructures import MutableHeaders
+from starlette.datastructures import Headers, MutableHeaders
 from starlette.middleware.cors import CORSMiddleware
 from starlette.requests import Request
+from starlette.responses import Response
 from starlette.types import ASGIApp, Message, Receive, Scope, Send
 
 from tenxgraph_api.src.app.core.exceptions.server_errors import UnhandledErrorMiddleware
@@ -76,6 +77,21 @@ def _is_health_check(scope: Scope) -> bool:
     StaleStateError = None  # type: ignore[assignment]
 
 
+class AllowlistCORSMiddleware(CORSMiddleware):
+    """``CORSMiddleware`` that grants private-network access only to allowed origins.
+
+    Starlette adds ``Access-Control-Allow-Private-Network: true`` to every preflight that asks
+    for it, including one it refuses for its origin. That refusal still stands (400, no
+    ``Access-Control-Allow-Origin``), but an origin outside ``ORIGINS`` should not get the grant.
+    """
+
+    def preflight_response(self, request_headers: Headers) -> Response:
+        response = super().preflight_response(request_headers)
+        if not self.is_allowed_origin(origin=request_headers["origin"]):
+            del response.headers["Access-Control-Allow-Private-Network"]
+        return response
+
+
 class InsecureCorsConfigError(RuntimeError):
     """Raised when production is configured with wildcard CORS + credentials."""
 
@@ -507,11 +523,13 @@ def setup_middleware(
     app.add_middleware(RequestIDMiddleware)
 
     app.add_middleware(
-        CORSMiddleware,
+        AllowlistCORSMiddleware,
         allow_origins=origins,
         allow_credentials=allow_credentials,
         allow_methods=["*"],
         allow_headers=["*"],
+        # Answer the Private Network Access preflight the hosted playground sends to a local server.
+        allow_private_network=True,
     )
 
     logger.debug("Middleware set up")

tenxgraph_api/src/app/core/middleware/rate_limit/base.py

@@ -2,20 +2,39 @@
 from dataclasses import dataclass
 
 
+# Retry-After (seconds) sent with the 503 answered while the backend cannot be reached.
+UNAVAILABLE_RETRY_AFTER_SECONDS = 5
+
+
 @dataclass(frozen=True)
 class RateLimitDecision:
-    """Result of a single rate-limit check."""
+    """Result of a single rate-limit check.
+
+    ``unavailable`` marks a denial caused by the backend (it could not be reached), not by the
+    limit: the middleware then answers 503 ``RATE_LIMIT_UNAVAILABLE`` instead of 429, with
+    ``reset_after`` as its ``Retry-After``. Backends that never set it keep working unchanged.
+    """
 
     allowed: bool
     remaining: int
     reset_after: int
+    unavailable: bool = False
+
+    @classmethod
+    def backend_unavailable(
+        cls, retry_after: int = UNAVAILABLE_RETRY_AFTER_SECONDS
+    ) -> "RateLimitDecision":
+        """A denial because the backend failed; answered 503 with ``Retry-After``."""
+        return cls(allowed=False, remaining=0, reset_after=retry_after, unavailable=True)
 
 
 class BaseRateLimitBackend(ABC):
     """Abstract base class for rate-limit backends.
 
     Users can implement this class and bind an instance into InjectQ when they
-    need a backend other than the built-in memory or Redis implementations.
+    need a backend other than the built-in memory or Redis implementations. A backend that
+    cannot reach its store and must not let the request through returns
+    :meth:`RateLimitDecision.backend_unavailable`.
     """
 
     @abstractmethod