Enhance documentation for rate limiting and backend checks

src/content/docs/client/error-handling.md

@@ -167,6 +167,8 @@ const result = await retryWithBackoff(() =>
 
 The server returns 429 when the `rate_limit` window is used up, or when one user has more graph runs in flight than `runs.max_concurrent_per_user` allows (default 10). The rate-limit response includes a `Retry-After` header in seconds and ends its message with `Retry after <n>s.` The run-cap response has no `Retry-After` header. The client has no class for 429, so it throws a plain `TenxGraphError` with `statusCode` 429, and it does not keep the response headers.
 
+If the server's rate-limit backend (Redis) is down and `fail_open` is `false`, it answers `503` with `RATE_LIMIT_UNAVAILABLE` and `Retry-After: 5` instead. Retry on 503 the same way you retry on 429. Added after 1.0.0rc2.
+
 ```ts
 import { TenxGraphError } from '10xgraph-client';
 

src/content/docs/concepts/extensibility.md

@@ -140,10 +140,13 @@ class PineconeStore(BaseStore):
     async def aupdate(self, config, memory_id, content, metadata=None, **kwargs): ...
     async def adelete(self, config, memory_id, **kwargs): ...
     async def aforget_memory(self, config, **kwargs): ...
+    async def aping(self) -> None: ...   # raise if the backend is unreachable; read-only and cheap
 
 compiled = graph.compile(store=PineconeStore())
 ```
 
+`aping` is abstract: a store that does not implement it raises `TypeError` when created. Return when the backend is reachable and raise otherwise; an in-process store just returns. `10xgraph-api` calls it for `/health/ready` and at startup. Added after 1.0.0rc2.
+
 ### BaseEmbedding: custom embedding model
 
 An embedding model converts text into vectors for storage and search. You extend `BaseEmbedding` when you need an embedding provider other than the framework defaults or when you want to use a fine-tuned embedding model specific to your domain.

src/content/docs/concepts/serving-agents.md

@@ -264,7 +264,8 @@ from tenxgraph_api.src.app.core.middleware.rate_limit.base import (
 class CustomRateLimitBackend(BaseRateLimitBackend):
     async def check(self, key: str, *, limit: int, window: int) -> RateLimitDecision:
         # Atomically count the request, then return a decision.
-        # allowed=False produces a 429 response.
+        # allowed=False produces a 429 response. Return
+        # RateLimitDecision.backend_unavailable() for a 503 when your store is down.
         return RateLimitDecision(allowed=True, remaining=limit - 1, reset_after=window)
 
     async def close(self) -> None:

src/content/docs/integrations/postgres-and-redis.md

@@ -66,6 +66,8 @@ volumes:
 
 Start them with `docker compose up -d`.
 
+If you run `10xgraph dev` on the host, publish the ports on `127.0.0.1` (`"127.0.0.1:5432:5432"`) and use `localhost`, not the service name, in your connection URLs. The server's startup check fails at boot otherwise. See [Run `10xgraph dev` against Docker backends](/docs/server/run-the-server#run-10xgraph-dev-against-docker-backends).
+
 ## Create a checkpointer and resume a thread
 
 Pass the checkpointer to `compile`, then invoke the graph with the same `thread_id` to continue a conversation. The tables are created on first use, so no manual migration step is needed for a new database.

src/content/docs/project/upgrade-to-1.0.md

@@ -100,6 +100,11 @@ npm install 10xgraph-client@next
 - **Skill and memory text is no longer run through `str.format`.** Only your own
   `system_prompt` is interpolated, and a bad placeholder is logged instead of failing the run.
   Remove any reliance on `{placeholder}` expansion inside skill or memory text.
+- **`BaseStore.aping()` is now abstract (added after 1.0.0rc2).** A custom store must implement
+  `async def aping(self) -> None`: return when its backend is reachable, raise when it is not.
+  Keep it read-only and cheap, because `10xgraph-api` calls it on `/health/ready` and at
+  startup. An in-process store just returns. A store without it raises `TypeError` when
+  created. `QdrantStore`, `Mem0Store` and `InMemoryStore` implement it already.
 - **`aforget_memory` differs by store.** `QdrantStore` applies `memory_type`, `category` and
   `filters`. `Mem0Store` raises `ValueError` if any of them is passed. The `qdrant` extra needs
   `qdrant-client>=1.10.0` and a Qdrant server 1.10 or newer.
@@ -297,6 +302,19 @@ npm install 10xgraph-client@next
   in development. Raise it, or set it to `null`. Only the core tool and node timeouts
   (`tool_timeout`, `node_timeout`) lost their defaults.
 
+### Startup check and rate limiter outages
+
+Added after 1.0.0rc2.
+
+- **The server refuses to start while a configured backend is unreachable.** At boot it pings
+  the checkpointer's Postgres and Redis, the API's Redis, the rate limiter's own Redis and the
+  store, retrying for `STARTUP_CHECK_TIMEOUT_SECONDS` (30), in development and production. A
+  wrong connection string now fails the deploy. Set `STARTUP_CHECKS=false` to skip it. See the
+  [production checklist](/docs/server/production-checklist#startup-check).
+- **A rate limiter outage with `fail_open: false` answers `503` `RATE_LIMIT_UNAVAILABLE`**
+  (with `Retry-After: 5`) instead of `429 RATE_LIMIT_EXCEEDED`. Clients that retry on 429 should
+  retry on 503 too. See [Rate limiting](/docs/server/rate-limiting).
+
 ### Auth and authorization
 
 - **Custom authorization backends are isolated by default.** A backend that implements only

src/content/docs/reference/api-cli/environment.md

@@ -57,6 +57,19 @@ The settings model allows extra variables, so unknown names in the environment a
 
 ---
 
+## Startup check variables
+
+Added after 1.0.0rc2.
+
+| Variable | Default | Description |
+| --- | --- | --- |
+| `STARTUP_CHECKS` | `true` | Before serving, ping every configured backend and refuse to start while one stays unreachable. Applies in development and production. `false` skips the check and logs a warning. |
+| `STARTUP_CHECK_TIMEOUT_SECONDS` | `30` | Seconds the check keeps retrying an unreachable backend (with a short backoff) before it gives up, so services started together can come up in any order. Must be 0 or more. |
+
+The check covers the checkpointer's Postgres (a query plus the schema version) and its Redis cache, the API's Redis (`redis` in `10xgraph.json`, else `REDIS_URL`), the rate limiter's Redis when it has its own URL, and the memory store through `aping()`. It does not check the model provider. When a backend stays down, the server exits with a message that names each backend and its `host:port`, never credentials. See [Startup check](/docs/server/production-checklist#startup-check) and [Backend unreachable at startup](/docs/troubleshooting/api-server#issue-backend-unreachable-at-startup).
+
+---
+
 ## Security headers
 
 Applied by the security-headers middleware when `SECURITY_HEADERS_ENABLED` is true.

src/content/docs/reference/api-cli/rate-limiting.md

@@ -26,7 +26,7 @@ The `rate_limit` block in `10xgraph.json` turns on a sliding-window limiter for
 | `redis` | object or string | `null` | Redis connection for the `"redis"` backend. Either `{"url": "...", "prefix": "..."}` or a bare URL string (uses default prefix). The URL supports `$ENV_VAR` and `${ENV_VAR}` expansion. |
 | `redis.url` | string | `null` | Redis connection URL. Required for `"redis"` backend unless a Redis client is bound in InjectQ. |
 | `redis.prefix` | string | `"10xgraph:rate-limit"` | Key prefix for all Redis entries. The default changed from `"agentflow:rate-limit"` in version 0.7.0, so counters restart once on upgrade. |
-| `fail_open` | boolean | `true` | When `true`, requests are allowed if the Redis backend is unreachable (fail-open). When `false`, requests are denied (fail-closed). Only applies to `"redis"` backend. |
+| `fail_open` | boolean | `true` | When `true`, requests are allowed if the Redis backend is unreachable (fail-open). When `false`, requests are denied with `503` `RATE_LIMIT_UNAVAILABLE` (fail-closed). Only applies to `"redis"` backend. |
 
 Config load validates all values. Invalid `by`, `backend`, non-positive `requests` or `window`, `trusted_proxy_hops` < 1, invalid networks in `trusted_proxies`, or an unresolved `$ENV_VAR` in the Redis URL raise `ValueError` and prevent the server from starting. A `rate_limit` block with `enabled: false` behaves as if the block were absent.
 
@@ -72,7 +72,7 @@ Every HTTP response includes these headers:
 | `X-RateLimit-Remaining` | Requests remaining in the current window. |
 | `X-RateLimit-Reset` | Unix timestamp when the window resets. |
 | `X-RateLimit-Reset-After` | Seconds until the window resets. |
-| `Retry-After` | Present on `429` responses only; equals `X-RateLimit-Reset-After`. |
+| `Retry-After` | Present on `429` responses, equal to `X-RateLimit-Reset-After`, and on `503` `RATE_LIMIT_UNAVAILABLE` responses (`5`). |
 
 When a client exceeds the limit, the server responds with status `429 Too Many Requests` and the following body:
 
@@ -92,6 +92,22 @@ When a client exceeds the limit, the server responds with status `429 Too Many R
 }
 ```
 
+When the Redis backend cannot be reached and `fail_open` is `false`, the server responds with `503 Service Unavailable` and `Retry-After: 5` instead of 429. A real limit hit stays 429. Added after 1.0.0rc2.
+
+```json
+{
+  "error": {
+    "code": "RATE_LIMIT_UNAVAILABLE",
+    "message": "Rate limiter unavailable: its backend could not be reached.",
+    "retry_after_seconds": 5
+  },
+  "metadata": {
+    "request_id": "...",
+    "status": "error"
+  }
+}
+```
+
 ## WebSocket handshakes and graph runs
 
 WebSocket handshakes bypass HTTP middleware, so the server re-applies the same check when a client connects to `/v1/graph/ws` or `/v1/graph/live`. A handshake counts like an HTTP request against the same bucket and backend. If the bucket is full, the handshake is rejected with close code `1013` (Try Again Later) and the reason `Rate limit exceeded`.
@@ -110,7 +126,7 @@ The separate `websocket.max_connections` and `websocket.max_connections_per_user
 
 ## Custom backend
 
-Subclass `BaseRateLimitBackend` when neither built-in backend fits, for example to call an external quota service. Both methods are abstract. `check` must atomically record the request and return a `RateLimitDecision` with `allowed`, `remaining` and `reset_after` (seconds). `close` releases resources at shutdown.
+Subclass `BaseRateLimitBackend` when neither built-in backend fits, for example to call an external quota service. Both methods are abstract. `check` must atomically record the request and return a `RateLimitDecision` with `allowed`, `remaining` and `reset_after` (seconds). `close` releases resources at shutdown. A backend that cannot reach its store and must not let the request through returns `RateLimitDecision.backend_unavailable()` (optional `retry_after` seconds, default 5); the middleware answers it with the 503 above. Added after 1.0.0rc2.
 
 ```python title="my_backend.py"
 import time

src/content/docs/reference/client/errors.md

@@ -177,6 +177,12 @@ try {
 }
 ```
 
+### Rate limiter backend down (503)
+
+Added after 1.0.0rc2.
+
+When the server's rate-limit backend is unreachable and `fail_open` is `false`, it answers 503 with `error.code` `RATE_LIMIT_UNAVAILABLE` and `Retry-After: 5`, not 429. The client throws a `ServerError` with `statusCode` 503. Retry after a short delay, as for 429.
+
 ### ServerError
 
 `ServerError` covers status 500, 502, 503 and 504 when the error code does not match a more specific class. Its `errorCode` is the server's code, or `INTERNAL_SERVER_ERROR` when absent, and `statusCode` is the actual status.

src/content/docs/reference/error-codes.md

@@ -347,6 +347,8 @@ The server maps exceptions to responses in `tenxgraph_api/src/app/core/exception
 | `StateFormatVersionError` | 500 (message says to upgrade) |
 | `ResumeValidationError` | 422 |
 
+`RATE_LIMIT_UNAVAILABLE` is a `503` from the rate limiter, not a core exception: the `rate_limit` backend could not be reached and `fail_open` is `false`. It carries `Retry-After: 5`. The code `RATE_LIMIT_EXCEEDED` (429) stays for a real limit hit. Added after 1.0.0rc2.
+
 A `429` is not a core exception either. The server returns it when the `rate_limit` window is used up (`error.code` `RATE_LIMIT_EXCEEDED`, with a `Retry-After` header) or when one user has more than `runs.max_concurrent_per_user` runs in flight (standard body, code `HTTPException`, no `Retry-After`).
 
 In production the message of a 5xx response is replaced by a generic text and the real error goes to the server log. Messages of 4xx `HTTPException` errors are kept. Handled 4xx errors are logged at `WARNING` with no traceback, and 5xx errors at `ERROR` with the traceback, so alert rules for 4xx traffic must match `WARNING` (`10xgraph-api` 1.0.0rc2). A missing checkpointer answers `503` (`Checkpointer is not configured`), a thread state update with a bad `context` answers `422`, and a message id missing from a thread answers `404`.

src/content/docs/reference/python/memory-stores.md

@@ -373,10 +373,13 @@ Abstract base class that every store backend implements. Every method takes a `c
 | `adelete` | `async (config, memory_id, **kwargs) -> Any` | Delete a memory by ID. |
 | `aforget_memory` | `async (config, **kwargs) -> Any` | Delete memories for the `user_id` in `config`. `thread_id` is an optional extra filter. `QdrantStore` also narrows by the keyword arguments `memory_type`, `category` and `filters` (combined with AND; a `filters` value that is not a string, integer or boolean raises `ValueError`). `Mem0Store` can delete only by user, so it raises `ValueError` when any of the three is passed. |
 | `asetup` | `async () -> Any` | Create the collection or other backing resources. Implemented by `QdrantStore`; the base class raises `NotImplementedError`. |
+| `aping` | `async () -> None` | Abstract. Return when the store's backend is reachable, raise when it is not. Must be read-only and cheap, because a server may call it on every readiness probe. An in-process store just returns. `10xgraph-api` calls it for `/health/ready` and at startup. A custom store that does not implement it raises `TypeError` when created. Added after 1.0.0rc2. |
 | `arelease` | `async () -> None` | Release connections. Not abstract, but the base implementation raises `NotImplementedError`; `Mem0Store` overrides it. |
 
 There is no bulk `abatch_store`; call `astore` in a loop or with `asyncio.gather`.
 
+`QdrantStore.aping()` lists the collections on the server, `Mem0Store.aping()` lists the collections of its vector store, and `InMemoryStore.aping()` returns at once. Added after 1.0.0rc2.
+
 ### Sync wrappers
 
 ```python

src/content/docs/reference/rest-api/conventions.md

@@ -157,4 +157,4 @@ These are the status codes the server returns. Routes document their own specifi
 | `429` | Rate limit exceeded. Body has `error.code` `RATE_LIMIT_EXCEEDED`; the `Retry-After` and `X-RateLimit-*` headers say when to retry. The same status is returned when a user has more than `runs.max_concurrent_per_user` runs in flight (default 10); that body has code `HTTPException` and `Retry-After: 5`. The model provider rate limiting the server also returns `429`, with code `PROVIDER_RATE_LIMITED` and a `Retry-After` taken from the provider (5 seconds if it sends none) |
 | `499` | The client closed the request while it waited for a busy thread. The response is never read |
 | `500` | Unexpected server error. Body has code `INTERNAL_SERVER_ERROR`; in production the message is generic |
-| `503` | Checkpointer or storage temporarily unavailable (code `SERVICE_UNAVAILABLE` for a database or cache driver error), with `Retry-After`. Readiness answers 503 while the server is shutting down |
+| `503` | Checkpointer or storage temporarily unavailable (code `SERVICE_UNAVAILABLE` for a database or cache driver error), with `Retry-After`. Readiness answers 503 while the server is shutting down. A rate limiter whose backend is unreachable with `fail_open: false` also answers 503, with code `RATE_LIMIT_UNAVAILABLE` and `Retry-After: 5` (added after 1.0.0rc2) |

src/content/docs/reference/rest-api/ping.md

@@ -66,6 +66,8 @@ curl -i http://127.0.0.1:8000/health/ready
 | `503` | At least one check is `fail`. `status` is `not_ready`. The checkpointer check fails when the database schema version is below what the code expects (an upgrade whose migration has not run). |
 | `503` | The server is shutting down. `status` is `draining` and `checks` is empty. No component is checked. |
 
+The server runs the same checks once at boot (the [startup check](/docs/server/production-checklist#startup-check)). There an unreachable backend stops the server from starting; here it only turns readiness to 503, so traffic stops until the backend returns. Added after 1.0.0rc2.
+
 Once the process gets SIGTERM, the server marks itself as draining. Readiness answers `503` from then on, so the load balancer stops sending new requests while open runs finish. `/ping` stays `200` until the process exits.
 
 The response is a plain JSON object, not the standard `data` and `metadata` envelope.

src/content/docs/server/cors-and-security-headers.md

@@ -52,6 +52,12 @@ export ORIGINS="https://yourdomain.com, https://app.yourdomain.com"
 export CORS_ALLOW_CREDENTIALS="true"
 ```
 
+### Private-network preflight for the hosted playground
+
+Added after 1.0.0rc2.
+
+When a public page such as `https://play.10xgraph.com` calls a server on `localhost`, Chrome first sends a preflight with `Access-Control-Request-Private-Network`. The API answers it with `Access-Control-Allow-Private-Network: true`, but only for an origin listed in `ORIGINS`. For any other origin the preflight is refused as before and the header is not sent. The browser also asks the user for permission; see [Allow the playground to reach your local server](/docs/server/playground#allow-the-playground-to-reach-your-local-server).
+
 ### Host validation
 
 The `ALLOWED_HOST` setting (comma-separated, default `*`, no spaces after the commas) is checked by the `TrustedHostMiddleware`. It rejects requests to hostnames not in the list, protecting against Host header injection attacks. The `/ping` health check is exempt so container orchestration probes work.

src/content/docs/server/playground.md

@@ -40,6 +40,23 @@ If the browser tab does not open automatically, copy the playground URL from the
 
 `10xgraph-api` 0.7.0 opens the same playground at its previous address, `playground-463bd.web.app`. Both addresses serve the same app.
 
+## Allow the playground to reach your local server
+
+Added after 1.0.0rc2.
+
+The playground is a public HTTPS page and your server runs on your own machine. Chrome's Local Network Access feature therefore asks you for permission the first time the page calls `localhost`, and the playground makes one early request to your server so the prompt appears on the Connect page. Click Allow.
+
+Chrome keeps two separate permissions: one for your own machine (loopback addresses such as `localhost` and `127.0.0.1`, the Permissions API name `loopback-network`) and one for the local network (`local-network`). Both appear in the site settings. For `10xgraph dev` on `127.0.0.1`, the loopback one must be Allowed.
+
+If you dismissed or denied the prompt, the Connect page shows **Your browser blocked the local server**. To fix it:
+
+1. Click the site settings icon left of the address bar.
+2. Find the local network or loopback access entry and set it to Allow. Chrome may label it differently by version.
+3. Reload the page and click Try again.
+4. Check that `10xgraph dev` is still running (`curl http://127.0.0.1:8000/ping`).
+
+The server must also allow the playground origin: the API answers the browser's private-network preflight only for origins in `ORIGINS` (see [CORS](/docs/server/cors-and-security-headers#private-network-preflight-for-the-hosted-playground)). A server reached over a public HTTPS URL does not need any of this.
+
 ## Verify the playground works
 
 Once your browser opens the playground, you should land on the **Connect** page. The playground has already received your local API URL from the `backendUrl` query parameter, so the connection is usually pre-filled.