diff --git a/src/content/docs/client/error-handling.md b/src/content/docs/client/error-handling.md index deb5f66..dd400ee 100644 --- a/src/content/docs/client/error-handling.md +++ b/src/content/docs/client/error-handling.md @@ -167,6 +167,8 @@ const result = await retryWithBackoff(() => The server returns 429 when the `rate_limit` window is used up, or when one user has more graph runs in flight than `runs.max_concurrent_per_user` allows (default 10). The rate-limit response includes a `Retry-After` header in seconds and ends its message with `Retry after s.` The run-cap response has no `Retry-After` header. The client has no class for 429, so it throws a plain `TenxGraphError` with `statusCode` 429, and it does not keep the response headers. +If the server's rate-limit backend (Redis) is down and `fail_open` is `false`, it answers `503` with `RATE_LIMIT_UNAVAILABLE` and `Retry-After: 5` instead. Retry on 503 the same way you retry on 429. Added after 1.0.0rc2. + ```ts import { TenxGraphError } from '10xgraph-client'; diff --git a/src/content/docs/concepts/extensibility.md b/src/content/docs/concepts/extensibility.md index 510c246..55ef8ae 100644 --- a/src/content/docs/concepts/extensibility.md +++ b/src/content/docs/concepts/extensibility.md @@ -140,10 +140,13 @@ class PineconeStore(BaseStore): async def aupdate(self, config, memory_id, content, metadata=None, **kwargs): ... async def adelete(self, config, memory_id, **kwargs): ... async def aforget_memory(self, config, **kwargs): ... + async def aping(self) -> None: ... # raise if the backend is unreachable; read-only and cheap compiled = graph.compile(store=PineconeStore()) ``` +`aping` is abstract: a store that does not implement it raises `TypeError` when created. Return when the backend is reachable and raise otherwise; an in-process store just returns. `10xgraph-api` calls it for `/health/ready` and at startup. Added after 1.0.0rc2. + ### BaseEmbedding: custom embedding model An embedding model converts text into vectors for storage and search. You extend `BaseEmbedding` when you need an embedding provider other than the framework defaults or when you want to use a fine-tuned embedding model specific to your domain. diff --git a/src/content/docs/concepts/serving-agents.md b/src/content/docs/concepts/serving-agents.md index 62d5c4c..8c333ad 100644 --- a/src/content/docs/concepts/serving-agents.md +++ b/src/content/docs/concepts/serving-agents.md @@ -264,7 +264,8 @@ from tenxgraph_api.src.app.core.middleware.rate_limit.base import ( class CustomRateLimitBackend(BaseRateLimitBackend): async def check(self, key: str, *, limit: int, window: int) -> RateLimitDecision: # Atomically count the request, then return a decision. - # allowed=False produces a 429 response. + # allowed=False produces a 429 response. Return + # RateLimitDecision.backend_unavailable() for a 503 when your store is down. return RateLimitDecision(allowed=True, remaining=limit - 1, reset_after=window) async def close(self) -> None: diff --git a/src/content/docs/integrations/postgres-and-redis.md b/src/content/docs/integrations/postgres-and-redis.md index 5aee160..cdcbbe2 100644 --- a/src/content/docs/integrations/postgres-and-redis.md +++ b/src/content/docs/integrations/postgres-and-redis.md @@ -66,6 +66,8 @@ volumes: Start them with `docker compose up -d`. +If you run `10xgraph dev` on the host, publish the ports on `127.0.0.1` (`"127.0.0.1:5432:5432"`) and use `localhost`, not the service name, in your connection URLs. The server's startup check fails at boot otherwise. See [Run `10xgraph dev` against Docker backends](/docs/server/run-the-server#run-10xgraph-dev-against-docker-backends). + ## Create a checkpointer and resume a thread Pass the checkpointer to `compile`, then invoke the graph with the same `thread_id` to continue a conversation. The tables are created on first use, so no manual migration step is needed for a new database. diff --git a/src/content/docs/project/upgrade-to-1.0.md b/src/content/docs/project/upgrade-to-1.0.md index ddbfdce..ede12d1 100644 --- a/src/content/docs/project/upgrade-to-1.0.md +++ b/src/content/docs/project/upgrade-to-1.0.md @@ -100,6 +100,11 @@ npm install 10xgraph-client@next - **Skill and memory text is no longer run through `str.format`.** Only your own `system_prompt` is interpolated, and a bad placeholder is logged instead of failing the run. Remove any reliance on `{placeholder}` expansion inside skill or memory text. +- **`BaseStore.aping()` is now abstract (added after 1.0.0rc2).** A custom store must implement + `async def aping(self) -> None`: return when its backend is reachable, raise when it is not. + Keep it read-only and cheap, because `10xgraph-api` calls it on `/health/ready` and at + startup. An in-process store just returns. A store without it raises `TypeError` when + created. `QdrantStore`, `Mem0Store` and `InMemoryStore` implement it already. - **`aforget_memory` differs by store.** `QdrantStore` applies `memory_type`, `category` and `filters`. `Mem0Store` raises `ValueError` if any of them is passed. The `qdrant` extra needs `qdrant-client>=1.10.0` and a Qdrant server 1.10 or newer. @@ -297,6 +302,19 @@ npm install 10xgraph-client@next in development. Raise it, or set it to `null`. Only the core tool and node timeouts (`tool_timeout`, `node_timeout`) lost their defaults. +### Startup check and rate limiter outages + +Added after 1.0.0rc2. + +- **The server refuses to start while a configured backend is unreachable.** At boot it pings + the checkpointer's Postgres and Redis, the API's Redis, the rate limiter's own Redis and the + store, retrying for `STARTUP_CHECK_TIMEOUT_SECONDS` (30), in development and production. A + wrong connection string now fails the deploy. Set `STARTUP_CHECKS=false` to skip it. See the + [production checklist](/docs/server/production-checklist#startup-check). +- **A rate limiter outage with `fail_open: false` answers `503` `RATE_LIMIT_UNAVAILABLE`** + (with `Retry-After: 5`) instead of `429 RATE_LIMIT_EXCEEDED`. Clients that retry on 429 should + retry on 503 too. See [Rate limiting](/docs/server/rate-limiting). + ### Auth and authorization - **Custom authorization backends are isolated by default.** A backend that implements only diff --git a/src/content/docs/reference/api-cli/environment.md b/src/content/docs/reference/api-cli/environment.md index 318a72b..5dacae8 100644 --- a/src/content/docs/reference/api-cli/environment.md +++ b/src/content/docs/reference/api-cli/environment.md @@ -57,6 +57,19 @@ The settings model allows extra variables, so unknown names in the environment a --- +## Startup check variables + +Added after 1.0.0rc2. + +| Variable | Default | Description | +| --- | --- | --- | +| `STARTUP_CHECKS` | `true` | Before serving, ping every configured backend and refuse to start while one stays unreachable. Applies in development and production. `false` skips the check and logs a warning. | +| `STARTUP_CHECK_TIMEOUT_SECONDS` | `30` | Seconds the check keeps retrying an unreachable backend (with a short backoff) before it gives up, so services started together can come up in any order. Must be 0 or more. | + +The check covers the checkpointer's Postgres (a query plus the schema version) and its Redis cache, the API's Redis (`redis` in `10xgraph.json`, else `REDIS_URL`), the rate limiter's Redis when it has its own URL, and the memory store through `aping()`. It does not check the model provider. When a backend stays down, the server exits with a message that names each backend and its `host:port`, never credentials. See [Startup check](/docs/server/production-checklist#startup-check) and [Backend unreachable at startup](/docs/troubleshooting/api-server#issue-backend-unreachable-at-startup). + +--- + ## Security headers Applied by the security-headers middleware when `SECURITY_HEADERS_ENABLED` is true. diff --git a/src/content/docs/reference/api-cli/rate-limiting.md b/src/content/docs/reference/api-cli/rate-limiting.md index d474a13..c931cf3 100644 --- a/src/content/docs/reference/api-cli/rate-limiting.md +++ b/src/content/docs/reference/api-cli/rate-limiting.md @@ -26,7 +26,7 @@ The `rate_limit` block in `10xgraph.json` turns on a sliding-window limiter for | `redis` | object or string | `null` | Redis connection for the `"redis"` backend. Either `{"url": "...", "prefix": "..."}` or a bare URL string (uses default prefix). The URL supports `$ENV_VAR` and `${ENV_VAR}` expansion. | | `redis.url` | string | `null` | Redis connection URL. Required for `"redis"` backend unless a Redis client is bound in InjectQ. | | `redis.prefix` | string | `"10xgraph:rate-limit"` | Key prefix for all Redis entries. The default changed from `"agentflow:rate-limit"` in version 0.7.0, so counters restart once on upgrade. | -| `fail_open` | boolean | `true` | When `true`, requests are allowed if the Redis backend is unreachable (fail-open). When `false`, requests are denied (fail-closed). Only applies to `"redis"` backend. | +| `fail_open` | boolean | `true` | When `true`, requests are allowed if the Redis backend is unreachable (fail-open). When `false`, requests are denied with `503` `RATE_LIMIT_UNAVAILABLE` (fail-closed). Only applies to `"redis"` backend. | Config load validates all values. Invalid `by`, `backend`, non-positive `requests` or `window`, `trusted_proxy_hops` < 1, invalid networks in `trusted_proxies`, or an unresolved `$ENV_VAR` in the Redis URL raise `ValueError` and prevent the server from starting. A `rate_limit` block with `enabled: false` behaves as if the block were absent. @@ -72,7 +72,7 @@ Every HTTP response includes these headers: | `X-RateLimit-Remaining` | Requests remaining in the current window. | | `X-RateLimit-Reset` | Unix timestamp when the window resets. | | `X-RateLimit-Reset-After` | Seconds until the window resets. | -| `Retry-After` | Present on `429` responses only; equals `X-RateLimit-Reset-After`. | +| `Retry-After` | Present on `429` responses, equal to `X-RateLimit-Reset-After`, and on `503` `RATE_LIMIT_UNAVAILABLE` responses (`5`). | When a client exceeds the limit, the server responds with status `429 Too Many Requests` and the following body: @@ -92,6 +92,22 @@ When a client exceeds the limit, the server responds with status `429 Too Many R } ``` +When the Redis backend cannot be reached and `fail_open` is `false`, the server responds with `503 Service Unavailable` and `Retry-After: 5` instead of 429. A real limit hit stays 429. Added after 1.0.0rc2. + +```json +{ + "error": { + "code": "RATE_LIMIT_UNAVAILABLE", + "message": "Rate limiter unavailable: its backend could not be reached.", + "retry_after_seconds": 5 + }, + "metadata": { + "request_id": "...", + "status": "error" + } +} +``` + ## WebSocket handshakes and graph runs WebSocket handshakes bypass HTTP middleware, so the server re-applies the same check when a client connects to `/v1/graph/ws` or `/v1/graph/live`. A handshake counts like an HTTP request against the same bucket and backend. If the bucket is full, the handshake is rejected with close code `1013` (Try Again Later) and the reason `Rate limit exceeded`. @@ -110,7 +126,7 @@ The separate `websocket.max_connections` and `websocket.max_connections_per_user ## Custom backend -Subclass `BaseRateLimitBackend` when neither built-in backend fits, for example to call an external quota service. Both methods are abstract. `check` must atomically record the request and return a `RateLimitDecision` with `allowed`, `remaining` and `reset_after` (seconds). `close` releases resources at shutdown. +Subclass `BaseRateLimitBackend` when neither built-in backend fits, for example to call an external quota service. Both methods are abstract. `check` must atomically record the request and return a `RateLimitDecision` with `allowed`, `remaining` and `reset_after` (seconds). `close` releases resources at shutdown. A backend that cannot reach its store and must not let the request through returns `RateLimitDecision.backend_unavailable()` (optional `retry_after` seconds, default 5); the middleware answers it with the 503 above. Added after 1.0.0rc2. ```python title="my_backend.py" import time diff --git a/src/content/docs/reference/client/errors.md b/src/content/docs/reference/client/errors.md index 11946ff..050afdb 100644 --- a/src/content/docs/reference/client/errors.md +++ b/src/content/docs/reference/client/errors.md @@ -177,6 +177,12 @@ try { } ``` +### Rate limiter backend down (503) + +Added after 1.0.0rc2. + +When the server's rate-limit backend is unreachable and `fail_open` is `false`, it answers 503 with `error.code` `RATE_LIMIT_UNAVAILABLE` and `Retry-After: 5`, not 429. The client throws a `ServerError` with `statusCode` 503. Retry after a short delay, as for 429. + ### ServerError `ServerError` covers status 500, 502, 503 and 504 when the error code does not match a more specific class. Its `errorCode` is the server's code, or `INTERNAL_SERVER_ERROR` when absent, and `statusCode` is the actual status. diff --git a/src/content/docs/reference/error-codes.md b/src/content/docs/reference/error-codes.md index 4976df8..9e4db8e 100644 --- a/src/content/docs/reference/error-codes.md +++ b/src/content/docs/reference/error-codes.md @@ -347,6 +347,8 @@ The server maps exceptions to responses in `tenxgraph_api/src/app/core/exception | `StateFormatVersionError` | 500 (message says to upgrade) | | `ResumeValidationError` | 422 | +`RATE_LIMIT_UNAVAILABLE` is a `503` from the rate limiter, not a core exception: the `rate_limit` backend could not be reached and `fail_open` is `false`. It carries `Retry-After: 5`. The code `RATE_LIMIT_EXCEEDED` (429) stays for a real limit hit. Added after 1.0.0rc2. + A `429` is not a core exception either. The server returns it when the `rate_limit` window is used up (`error.code` `RATE_LIMIT_EXCEEDED`, with a `Retry-After` header) or when one user has more than `runs.max_concurrent_per_user` runs in flight (standard body, code `HTTPException`, no `Retry-After`). In production the message of a 5xx response is replaced by a generic text and the real error goes to the server log. Messages of 4xx `HTTPException` errors are kept. Handled 4xx errors are logged at `WARNING` with no traceback, and 5xx errors at `ERROR` with the traceback, so alert rules for 4xx traffic must match `WARNING` (`10xgraph-api` 1.0.0rc2). A missing checkpointer answers `503` (`Checkpointer is not configured`), a thread state update with a bad `context` answers `422`, and a message id missing from a thread answers `404`. diff --git a/src/content/docs/reference/python/memory-stores.md b/src/content/docs/reference/python/memory-stores.md index bdf4a49..4e2ee06 100644 --- a/src/content/docs/reference/python/memory-stores.md +++ b/src/content/docs/reference/python/memory-stores.md @@ -373,10 +373,13 @@ Abstract base class that every store backend implements. Every method takes a `c | `adelete` | `async (config, memory_id, **kwargs) -> Any` | Delete a memory by ID. | | `aforget_memory` | `async (config, **kwargs) -> Any` | Delete memories for the `user_id` in `config`. `thread_id` is an optional extra filter. `QdrantStore` also narrows by the keyword arguments `memory_type`, `category` and `filters` (combined with AND; a `filters` value that is not a string, integer or boolean raises `ValueError`). `Mem0Store` can delete only by user, so it raises `ValueError` when any of the three is passed. | | `asetup` | `async () -> Any` | Create the collection or other backing resources. Implemented by `QdrantStore`; the base class raises `NotImplementedError`. | +| `aping` | `async () -> None` | Abstract. Return when the store's backend is reachable, raise when it is not. Must be read-only and cheap, because a server may call it on every readiness probe. An in-process store just returns. `10xgraph-api` calls it for `/health/ready` and at startup. A custom store that does not implement it raises `TypeError` when created. Added after 1.0.0rc2. | | `arelease` | `async () -> None` | Release connections. Not abstract, but the base implementation raises `NotImplementedError`; `Mem0Store` overrides it. | There is no bulk `abatch_store`; call `astore` in a loop or with `asyncio.gather`. +`QdrantStore.aping()` lists the collections on the server, `Mem0Store.aping()` lists the collections of its vector store, and `InMemoryStore.aping()` returns at once. Added after 1.0.0rc2. + ### Sync wrappers ```python diff --git a/src/content/docs/reference/rest-api/conventions.md b/src/content/docs/reference/rest-api/conventions.md index 1bd7e28..60477b9 100644 --- a/src/content/docs/reference/rest-api/conventions.md +++ b/src/content/docs/reference/rest-api/conventions.md @@ -157,4 +157,4 @@ These are the status codes the server returns. Routes document their own specifi | `429` | Rate limit exceeded. Body has `error.code` `RATE_LIMIT_EXCEEDED`; the `Retry-After` and `X-RateLimit-*` headers say when to retry. The same status is returned when a user has more than `runs.max_concurrent_per_user` runs in flight (default 10); that body has code `HTTPException` and `Retry-After: 5`. The model provider rate limiting the server also returns `429`, with code `PROVIDER_RATE_LIMITED` and a `Retry-After` taken from the provider (5 seconds if it sends none) | | `499` | The client closed the request while it waited for a busy thread. The response is never read | | `500` | Unexpected server error. Body has code `INTERNAL_SERVER_ERROR`; in production the message is generic | -| `503` | Checkpointer or storage temporarily unavailable (code `SERVICE_UNAVAILABLE` for a database or cache driver error), with `Retry-After`. Readiness answers 503 while the server is shutting down | +| `503` | Checkpointer or storage temporarily unavailable (code `SERVICE_UNAVAILABLE` for a database or cache driver error), with `Retry-After`. Readiness answers 503 while the server is shutting down. A rate limiter whose backend is unreachable with `fail_open: false` also answers 503, with code `RATE_LIMIT_UNAVAILABLE` and `Retry-After: 5` (added after 1.0.0rc2) | diff --git a/src/content/docs/reference/rest-api/ping.md b/src/content/docs/reference/rest-api/ping.md index 1bfcd64..bd2461a 100644 --- a/src/content/docs/reference/rest-api/ping.md +++ b/src/content/docs/reference/rest-api/ping.md @@ -66,6 +66,8 @@ curl -i http://127.0.0.1:8000/health/ready | `503` | At least one check is `fail`. `status` is `not_ready`. The checkpointer check fails when the database schema version is below what the code expects (an upgrade whose migration has not run). | | `503` | The server is shutting down. `status` is `draining` and `checks` is empty. No component is checked. | +The server runs the same checks once at boot (the [startup check](/docs/server/production-checklist#startup-check)). There an unreachable backend stops the server from starting; here it only turns readiness to 503, so traffic stops until the backend returns. Added after 1.0.0rc2. + Once the process gets SIGTERM, the server marks itself as draining. Readiness answers `503` from then on, so the load balancer stops sending new requests while open runs finish. `/ping` stays `200` until the process exits. The response is a plain JSON object, not the standard `data` and `metadata` envelope. diff --git a/src/content/docs/server/cors-and-security-headers.md b/src/content/docs/server/cors-and-security-headers.md index ee159a5..0b40b40 100644 --- a/src/content/docs/server/cors-and-security-headers.md +++ b/src/content/docs/server/cors-and-security-headers.md @@ -52,6 +52,12 @@ export ORIGINS="https://yourdomain.com, https://app.yourdomain.com" export CORS_ALLOW_CREDENTIALS="true" ``` +### Private-network preflight for the hosted playground + +Added after 1.0.0rc2. + +When a public page such as `https://play.10xgraph.com` calls a server on `localhost`, Chrome first sends a preflight with `Access-Control-Request-Private-Network`. The API answers it with `Access-Control-Allow-Private-Network: true`, but only for an origin listed in `ORIGINS`. For any other origin the preflight is refused as before and the header is not sent. The browser also asks the user for permission; see [Allow the playground to reach your local server](/docs/server/playground#allow-the-playground-to-reach-your-local-server). + ### Host validation The `ALLOWED_HOST` setting (comma-separated, default `*`, no spaces after the commas) is checked by the `TrustedHostMiddleware`. It rejects requests to hostnames not in the list, protecting against Host header injection attacks. The `/ping` health check is exempt so container orchestration probes work. diff --git a/src/content/docs/server/playground.md b/src/content/docs/server/playground.md index 1d29cce..29a2b6d 100644 --- a/src/content/docs/server/playground.md +++ b/src/content/docs/server/playground.md @@ -40,6 +40,23 @@ If the browser tab does not open automatically, copy the playground URL from the `10xgraph-api` 0.7.0 opens the same playground at its previous address, `playground-463bd.web.app`. Both addresses serve the same app. +## Allow the playground to reach your local server + +Added after 1.0.0rc2. + +The playground is a public HTTPS page and your server runs on your own machine. Chrome's Local Network Access feature therefore asks you for permission the first time the page calls `localhost`, and the playground makes one early request to your server so the prompt appears on the Connect page. Click Allow. + +Chrome keeps two separate permissions: one for your own machine (loopback addresses such as `localhost` and `127.0.0.1`, the Permissions API name `loopback-network`) and one for the local network (`local-network`). Both appear in the site settings. For `10xgraph dev` on `127.0.0.1`, the loopback one must be Allowed. + +If you dismissed or denied the prompt, the Connect page shows **Your browser blocked the local server**. To fix it: + +1. Click the site settings icon left of the address bar. +2. Find the local network or loopback access entry and set it to Allow. Chrome may label it differently by version. +3. Reload the page and click Try again. +4. Check that `10xgraph dev` is still running (`curl http://127.0.0.1:8000/ping`). + +The server must also allow the playground origin: the API answers the browser's private-network preflight only for origins in `ORIGINS` (see [CORS](/docs/server/cors-and-security-headers#private-network-preflight-for-the-hosted-playground)). A server reached over a public HTTPS URL does not need any of this. + ## Verify the playground works Once your browser opens the playground, you should land on the **Connect** page. The playground has already received your local API URL from the `backendUrl` query parameter, so the connection is usually pre-filled. diff --git a/src/content/docs/server/production-checklist.md b/src/content/docs/server/production-checklist.md index 6d58faa..d6117b1 100644 --- a/src/content/docs/server/production-checklist.md +++ b/src/content/docs/server/production-checklist.md @@ -30,6 +30,7 @@ With `MODE=production`, the server does these for you. You do not configure them - Scopes graph introspection (`GET /v1/graph`, `/v1/graph/tools`, `/v1/graph:StateSchema`) to callers with the `graph:read` scope, unless you set `graph_introspection` to `"open"`. - Defaults authorization to ownership, so each user reaches only their own threads. - Turns debug mode off. +- Pings every configured backend at boot and refuses to start while one is unreachable (see [Startup check](#startup-check)). Added after 1.0.0rc2. - Ships a hardened image when you use `10xgraph build`: non-root user, Gunicorn with `TenxGraphUvicornWorker` and long graceful timeouts, health check on `/ping`. On SIGTERM `/health/ready` answers 503 while open runs finish. For why these matter, see the [security model](/docs/production/security-model) and [scaling and reliability](/docs/production/scaling). @@ -188,6 +189,14 @@ Expected tables in the `public` schema (or the `schema=` you pass): `threads`, ` `/health/ready` answers 503 while the database schema version is below what the code expects, so a pod started before its migration ran stays out of rotation. +### Startup check + +At boot the server pings every backend it is configured to use, in development and production: the checkpointer's Postgres (a query and the schema version) and its Redis cache, the API's Redis (run limits, ownership cache, token revocation), the rate limiter's Redis when it has its own URL, and the memory store through `aping()`. It retries with a short backoff for `STARTUP_CHECK_TIMEOUT_SECONDS` (30 by default), so containers started together can come up in any order. If a backend is still down, the server exits with a message naming each backend and its `host:port` (no credentials). A broken connection string therefore fails the deploy instead of the first request, and the orchestrator restarts the container until the backends are up. + +`STARTUP_CHECKS=false` skips the check. It runs the same probes as `/health/ready`, with a different outcome: at startup an unreachable backend stops the server, while at runtime `/health/ready` answers 503 so the load balancer stops sending traffic until the backend returns. See [Ping and health](/docs/reference/rest-api/ping#get-healthready). + +Added after 1.0.0rc2. + ### Redis caching Redis acts as a hot cache in front of Postgres. `PgCheckpointer` requires a Redis connection, but the durable data lives in Postgres. After a Redis connection failure, `PgCheckpointer` skips Redis for a short cool-down (`redis_breaker_cooldown`, 5 seconds by default, `0` disables it) and uses Postgres at once, so a Redis outage does not slow every run step. @@ -330,6 +339,8 @@ done # After threshold, requests should return 429 Too Many Requests ``` +With `fail_open` set to `false` and the Redis backend unreachable, the API answers `503` with `RATE_LIMIT_UNAVAILABLE` and `Retry-After: 5` instead of 429. See [Rate limiting](/docs/server/rate-limiting). Added after 1.0.0rc2. + ## 8. Proxy buffering and request handling When running behind a reverse proxy (nginx, HAProxy, AWS ALB), ensure it forwards headers correctly: @@ -500,6 +511,7 @@ curl http://127.0.0.1:8000/ping | One worker sees state, another does not | Workers are not sharing the same backend | Point all instances at the same Postgres and Redis | | CORS requests are blocked | `ORIGINS` does not include the client domain | Add the domain to `ORIGINS`. A public API can set `ORIGINS=*` with `CORS_ALLOW_CREDENTIALS=false` and `STRICT_PRODUCTION_CHECKS=false` | | `409 thread is busy` on a thread | Another run is still active, and the 30 s wait ran out | Wait for the run to end, use `on_busy: "cancel"`, or use one thread per conversation | +| Server exits at boot with `Refusing to start: ... backend(s) unreachable` | A configured Postgres, Redis or store did not answer within `STARTUP_CHECK_TIMEOUT_SECONDS` | Fix the URL or start the backend. `STARTUP_CHECKS=false` skips the check | | Rate limiting not working | No `rate_limit` block, or Redis is down (with `fail_open` true, requests pass) | Verify `rate_limit.redis.url` and Redis connectivity | ## Related pages diff --git a/src/content/docs/server/rate-limiting.md b/src/content/docs/server/rate-limiting.md index 28841ce..5fc7f76 100644 --- a/src/content/docs/server/rate-limiting.md +++ b/src/content/docs/server/rate-limiting.md @@ -190,6 +190,8 @@ container = InjectQ() container.bind_instance(BaseRateLimitBackend, MyRateLimitBackend()) ``` +When your backend cannot reach its store and the request must not pass, return `RateLimitDecision.backend_unavailable()` instead of a plain denial. The middleware then answers `503` with `RATE_LIMIT_UNAVAILABLE` (see [What clients see when the backend is down](#what-clients-see-when-the-backend-is-down)). It takes an optional `retry_after` in seconds (default 5). Backends that return only allow or deny keep working unchanged. Added after 1.0.0rc2. + Bind the instance under `BaseRateLimitBackend`: that is the key the server looks up. If `backend` is `"custom"` and nothing is bound there, startup fails. Then in `10xgraph.json`: @@ -345,7 +347,27 @@ Every response also includes these headers so clients can track their quota: | `X-RateLimit-Remaining` | Requests remaining in the current window | | `X-RateLimit-Reset` | Unix timestamp for the window reset | | `X-RateLimit-Reset-After` | Seconds until the window resets | -| `Retry-After` | Seconds to wait before retrying (only on `429`) | +| `Retry-After` | Seconds to wait before retrying (only on `429` and `503`) | + +### What clients see when the backend is down + +With the Redis backend and `fail_open: false`, a Redis outage denies requests with `503 Service Unavailable`, not 429, so clients can tell an outage from a limit hit. The response carries `Retry-After: 5`: + +```json +{ + "error": { + "code": "RATE_LIMIT_UNAVAILABLE", + "message": "Rate limiter unavailable: its backend could not be reached.", + "retry_after_seconds": 5 + }, + "metadata": { + "request_id": "abc123", + "status": "error" + } +} +``` + +A real limit hit stays `429 RATE_LIMIT_EXCEEDED`. With `fail_open: true` requests pass during the outage. The Redis backend logs a traceback for the first error of an outage and one line for each error after it. Clients that retried on 429 should retry on 503 too. Added after 1.0.0rc2. Clients can check `X-RateLimit-Remaining` before sending the next request to avoid hitting the limit, and use `Retry-After` to back off intelligently. diff --git a/src/content/docs/server/run-the-server.md b/src/content/docs/server/run-the-server.md index 1d1b386..1200a33 100644 --- a/src/content/docs/server/run-the-server.md +++ b/src/content/docs/server/run-the-server.md @@ -52,6 +52,43 @@ The server binds to `http://127.0.0.1:8000` and restarts whenever you edit any P Flags: `--config`/`-c`, `--host`/`-H`, `--port`/`-p`, `--reload/--no-reload`, `--open/--no-open`, `--verbose`/`-v` and `--quiet`/`-q`. The commands `10xgraph api` and `10xgraph play` were removed in `10xgraph-api` 0.9.0; use `10xgraph dev`. +### Run `10xgraph dev` against Docker backends + +Added after 1.0.0rc2. + +When Postgres, Redis or Qdrant run in Docker Compose and `10xgraph dev` runs on your host, the host reaches them through published ports. Publish each port on `127.0.0.1` only, so nothing else on your network can connect: + +```yaml +services: + postgres: + image: postgres:17-alpine + ports: + - "127.0.0.1:5432:5432" + redis: + image: redis:7-alpine + ports: + - "127.0.0.1:6379:6379" + qdrant: + image: qdrant/qdrant:latest + ports: + - "127.0.0.1:6333:6333" + - "127.0.0.1:6334:6334" +``` + +In `.env`, use `localhost` as the host, not the compose service names. `postgres`, `redis` and `qdrant` resolve only inside the Docker network, that is, for a container of the same compose project: + +```bash +# On the host (10xgraph dev) +DATABASE_URL=postgresql://:@localhost:5432/ +REDIS_URL=redis://:@localhost:6379/0 +QDRANT_URL=http://localhost:6333 + +# Inside a container on the compose network +# DATABASE_URL=postgresql://:@postgres:5432/ +``` + +If the URLs still use a service name, the [startup check](/docs/server/production-checklist#startup-check) fails at boot and names the unreachable `host:port`. See [Backend unreachable at startup](/docs/troubleshooting/api-server#issue-backend-unreachable-at-startup). + ## Verify the server is running Open another terminal and ping the server to confirm it is reachable: @@ -267,6 +304,8 @@ Confirm that: Test with: `curl http://127.0.0.1:8000/ping` +If the server itself exits at boot with `Refusing to start: ... backend(s) unreachable`, a configured Postgres, Redis or store is down or its URL is wrong. See [Backend unreachable at startup](/docs/troubleshooting/api-server#issue-backend-unreachable-at-startup). + ## Next steps - **Configure the server:** See [Configure the server](/docs/server/configure) for auth, checkpointing, rate limiting, and per-environment settings. diff --git a/src/content/docs/troubleshooting/api-server.md b/src/content/docs/troubleshooting/api-server.md index a215842..dbf544f 100644 --- a/src/content/docs/troubleshooting/api-server.md +++ b/src/content/docs/troubleshooting/api-server.md @@ -47,6 +47,37 @@ flowchart TD - test the graph import manually: `python -c "from graph.react import app; print(app)"` - verify required provider keys and dependencies are installed before startup +### Issue: backend unreachable at startup + +Added after 1.0.0rc2. + +**Symptoms** + +- the server exits at boot after about 30 seconds (`STARTUP_CHECK_TIMEOUT_SECONDS`) with a message like: + +```text +Refusing to start: 1 configured backend(s) unreachable after 30s: + - redis (127.0.0.1:6399): ConnectionError: Connection refused +Start Redis/Postgres/Qdrant, or set STARTUP_CHECKS=false to skip this check. +``` + +- the checkpointer line can name two hosts, for example `checkpointer (postgres db:5432, redis cache:6379)` +- earlier log lines say `Startup check: ... is unreachable (attempt N)` while it retries + +**Likely causes** + +- the backend is not running or not published on the host and port named in the message +- running `10xgraph dev` on the host with `.env` URLs that use Docker service names (`postgres`, `redis`, `qdrant`), which resolve only inside the Docker network +- a wrong password, port or database in the connection string. The message shows `host:port` only, never credentials + +**Fix** + +- start the backend, or correct the URL. For Docker backends and `10xgraph dev` on the host, publish the ports on `127.0.0.1` and use `localhost` in `.env` (see [Run `10xgraph dev` against Docker backends](/docs/server/run-the-server#run-10xgraph-dev-against-docker-backends)) +- raise `STARTUP_CHECK_TIMEOUT_SECONDS` if the backends are slow to come up +- `STARTUP_CHECKS=false` skips the check; use it only when you accept that a down backend shows up at the first request instead +- with `10xgraph dev` (reload on), the reloader keeps waiting for a file change after the failure; fix the cause and save a file, or restart the command +- in production the orchestrator restarts the container until the backends are up. The same checks run behind `/health/ready`, which answers 503 at runtime while a backend is down + ### Issue: `RuntimeError: Refusing to start: the following routes are not protected` **Symptoms** @@ -289,8 +320,27 @@ Two different limits answer 429: - wait for the `Retry-After` interval, or for a running request to finish, then retry - raise `rate_limit.requests` or `runs.max_concurrent_per_user` in `10xgraph.json` if the traffic is legitimate - set `trusted_proxies` when the server sits behind a proxy +- a rate limiter whose Redis is down answers `503 RATE_LIMIT_UNAVAILABLE` (not 429) when `fail_open` is `false`; see the entry below - on the TypeScript client this is a plain `TenxGraphError` with `statusCode` 429; see [Error handling](/docs/client/error-handling#rate-limits-429) +### Issue: `503 RATE_LIMIT_UNAVAILABLE` + +Added after 1.0.0rc2. + +**Symptoms** + +- requests fail with status `503`, `error.code` `RATE_LIMIT_UNAVAILABLE`, the message `Rate limiter unavailable: its backend could not be reached.` and `Retry-After: 5` + +**Cause** + +`rate_limit.fail_open` is `false` and the rate limiter's backend (Redis) is unreachable. This is an outage, not a limit hit; a real limit hit stays `429 RATE_LIMIT_EXCEEDED`. + +**Fix** + +- restore the Redis the rate limiter uses (`rate_limit.redis.url`) and retry after `Retry-After` +- clients that retry on 429 should retry on 503 too +- set `fail_open` to `true` to let requests through during an outage. `/ping` and `/health/ready` are never rate limited + ### Issue: the error message is generic in production **Symptoms** diff --git a/src/content/docs/troubleshooting/client.md b/src/content/docs/troubleshooting/client.md index ece6846..6f7bbf4 100644 --- a/src/content/docs/troubleshooting/client.md +++ b/src/content/docs/troubleshooting/client.md @@ -72,6 +72,8 @@ A `TenxGraphError` with `statusCode` 429 means the server refused the call to li - **Rate limit.** `errorCode` is `RATE_LIMIT_EXCEEDED` and the message ends with `Retry after s.` The server also sent a `Retry-After` header, but the client does not keep response headers on the error, so read the seconds from the message. - **Per-user run cap.** One user has `runs.max_concurrent_per_user` runs in flight (default 10). `errorCode` is `HTTPException` and there is no `Retry-After`. Wait for a run to finish, for example close streams you no longer read. +A related case is `503` with `errorCode` `RATE_LIMIT_UNAVAILABLE`: the server's rate-limit backend is unreachable and `fail_open` is `false`. It is an outage, not a limit hit; retry after about 5 seconds. Added after 1.0.0rc2. + The client does not retry 429 on its own. Back off in your app (see [Error handling](/docs/client/error-handling#rate-limits-429)). If every user hits the limit at once from behind a proxy, the server is counting the proxy's IP: set `trusted_proxies` on the server ([API server troubleshooting](/docs/troubleshooting/api-server)). ## Fix browser requests that fail while curl works @@ -281,6 +283,7 @@ These codes appear in error responses the client surfaces as typed errors. The f | `STORAGE_NOT_FOUND_000` | The thread or stored resource does not exist. | Check the `thread_id` and that a checkpointer is enabled. | | `STORAGE_TRANSIENT_000` | A temporary storage failure, such as an unreachable database. | Retry after a short delay. | | `VALIDATION_ERROR` | The request body failed validation (HTTP 422). | Check the message format and `config` shape against [Invoke and stream](/docs/client/invoke-agent). | +| `RATE_LIMIT_UNAVAILABLE` | The rate limiter's backend is unreachable and `fail_open` is `false` (HTTP 503, `Retry-After: 5`). Added after 1.0.0rc2. | Retry after a short delay and restore the server's rate-limit Redis. See [Fix 429 responses](#fix-429-responses). | | `RATE_LIMIT_EXCEEDED` | The server's rate limit is used up (HTTP 429). Arrives as a plain `TenxGraphError`, not a typed subclass. | Wait for the `Retry-After` interval, which the message also states, then retry. See [Fix 429 responses](#fix-429-responses). | ## Related pages diff --git a/src/content/docs/troubleshooting/playground.md b/src/content/docs/troubleshooting/playground.md index fb2b768..451ff28 100644 --- a/src/content/docs/troubleshooting/playground.md +++ b/src/content/docs/troubleshooting/playground.md @@ -39,6 +39,30 @@ The playground is hosted externally. `10xgraph dev` does not run a separate loca - test that URL with `curl http://127.0.0.1:8000/ping` (use your own host and port) - verify the browser URL contains the correct `backendUrl` +## Issue: "Your browser blocked the local server" + +Added after 1.0.0rc2. + +**Symptoms** + +- the Connect page shows **Your browser blocked the local server** +- or the connection fails with `Connection failed ... (network or CORS)` +- the browser console shows `Permission was denied for this request to access the `loopback` address space` +- `curl http://127.0.0.1:8000/ping` works from the terminal + +**Cause** + +Chrome's Local Network Access blocks a public page (`https://play.10xgraph.com`) from calling your machine until you allow it. Chrome has two separate permissions, one for your own machine (loopback: `localhost`, `127.0.0.1`; Permissions API name `loopback-network`) and one for the local network (`local-network`). For `10xgraph dev` on `127.0.0.1` the loopback one must be Allowed. + +**Fix** + +1. Click the site settings icon left of the address bar. +2. Set the local network or loopback access entry to Allow. Chrome may label it differently by version; both permissions show in the site settings. +3. Reload the playground and click Try again. +4. If it still fails, confirm the playground origin is in the server's `ORIGINS`. The API answers the private-network preflight only for listed origins. + +See [Allow the playground to reach your local server](/docs/server/playground#allow-the-playground-to-reach-your-local-server). + ## Issue: playground loads but requests fail silently **Symptoms**