diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index db55af5..58522e2 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -7,7 +7,7 @@ Thank you for your interest in contributing! This document covers everything you ```bash git clone https://github.com/bedrock-python/redis-client-kit.git cd redis-client-kit -uv sync --group dev +uv sync --all-extras --group dev uv run pre-commit install --hook-type commit-msg ``` diff --git a/Makefile b/Makefile index a08eadc..3c221ba 100644 --- a/Makefile +++ b/Makefile @@ -1,7 +1,7 @@ .PHONY: test test-unit test-integration fmt check build install docs-serve docs-build clean install: - uv sync --group dev + uv sync --all-extras --group dev fmt: uv run ruff format . diff --git a/README.md b/README.md index 3f3358d..106cd80 100644 --- a/README.md +++ b/README.md @@ -303,7 +303,7 @@ Full documentation: [bedrock-python.github.io/redis-client-kit](https://bedrock- - [Quick Start Guide](https://bedrock-python.github.io/redis-client-kit/guide/quickstart/) - [Configuration Guide](https://bedrock-python.github.io/redis-client-kit/guide/configuration/) - [Advanced Usage](https://bedrock-python.github.io/redis-client-kit/guide/advanced/) -- [API Reference](https://bedrock-python.github.io/redis-client-kit/reference/api/) +- [API Reference](https://bedrock-python.github.io/redis-client-kit/reference/) ## Contributing diff --git a/docs/agents.md b/docs/agents.md index 8885bb4..82adc3b 100644 --- a/docs/agents.md +++ b/docs/agents.md @@ -225,7 +225,7 @@ The rest lives one import deeper. | `redis_client_kit.utils` | — | the three exported helpers, plus `mask_redis_kwargs(kwargs)` for logging | | `redis_client_kit.settings` | `settings` | `BaseRedisSettings`, `RedisConnectionSettings`, `RedisClusterSettings`, `RedisPoolSettings`, `RedisRetrySettings`, `RedisSSLSettings`, `RedisResponseSettings` | | `redis_client_kit.metrics` | `metrics` | `RedisMetrics`, `REDIS_COMMAND_DURATION_BUCKETS` | -| `redis_client_kit.providers` | `providers` | `AsyncRedisProvider` | +| `redis_client_kit.providers` | `providers` | `AsyncRedisProvider(check_health_on_startup=True, provide_default_metrics=True)` | Each optional module raises `ImportError` at import time when its extra is missing, naming the extra. The root package imports none of them. @@ -276,10 +276,16 @@ container = make_async_container(AsyncRedisProvider(), AppProvider()) # this o client = await container.get(AsyncRedisClient) ``` -`AsyncRedisProvider` is `Scope.APP`, provides the client as an `AsyncIterator` so the -container closes it on teardown, and also provides `RedisMetricsProtocol | None` as `None` -so a container without metrics still resolves. Both of those facts have consequences — -see rules 15 to 17. +`AsyncRedisProvider` is `Scope.APP` and provides the client as an `AsyncIterator`, so the +container closes it on teardown. Two keyword-only constructor arguments decide what it +registers: + +| Argument | Default | Effect | +|---|---|---| +| `check_health_on_startup` | `True` | pings Redis before yielding the client, and raises when it does not answer; `False` registers the factory that yields immediately | +| `provide_default_metrics` | `True` | provides `RedisMetricsProtocol | None` as `None` so a container without metrics resolves; `False` leaves that type to your own provider | + +See rules 15 to 17. ## Rules that hold or break the code @@ -331,29 +337,38 @@ see rules 15 to 17. pydantic, prometheus-client or dishka; `from redis_client_kit.settings import …` raises `ImportError` naming the extra when it is missing. Do not guard the root import. -15. **Register `AsyncRedisProvider()` before your own metrics provider.** It provides a - default `RedisMetricsProtocol | None` of `None`, and in Dishka the last provider to - claim a type wins — put it second and your metrics are silently dropped, leaving an - uninstrumented client. The annotation on your factory must be exactly +15. **Register `AsyncRedisProvider()` before your own metrics provider, or turn its + default off.** It provides a default `RedisMetricsProtocol | None` of `None`, and in + Dishka the last provider to claim a type wins — put it second and your metrics are + silently dropped, leaving an uninstrumented client. + `AsyncRedisProvider(provide_default_metrics=False)` registers no default, so order + stops mattering. Either way the annotation on your factory must be exactly `RedisMetricsProtocol | None`; `RedisMetricsProtocol` is a different key. -16. **`AsyncRedisProvider.get_redis()` is not reachable through a container.** Both it and - `get_redis_with_health_check()` provide the same type, so only one survives - registration — the health-check one. There is no supported way to ask for the other. -17. **The provider's startup health check does not fail startup.** It retries - `check_async_redis_health`, which returns `False` instead of raising, so the retry loop - falls straight through and the container yields a client that cannot reach Redis. Call - `check_async_redis_health` yourself and act on the result if startup must fail. +16. **The provider registers one client factory, chosen at construction.** + `AsyncRedisProvider()` registers `get_redis_with_health_check()`; + `AsyncRedisProvider(check_health_on_startup=False)` registers `get_redis()` instead. + They provide the same type, so registering both would leave only the second — the + constructor picks one. +17. **The provider's startup health check fails startup.** It pings up to three times + with exponential backoff — 1 s, then 2 s — and raises `ConnectionError` when Redis + never answers, closing the client it built. Resolving `AsyncRedisClient` is what + triggers it, so that is where the error surfaces. Use + `AsyncRedisProvider(check_health_on_startup=False)` when a missing Redis must not + block startup. 18. **A `RedisMetrics` instance owns global Prometheus names.** Building a second one with the same prefix raises a duplicate-timeseries `ValueError` from the default registry. Build one per process and inject it. -19. **Instrumented pool gauges are partial.** `redis_pool_size` is always `0` on the async - client — it reads a pool attribute `redis-py` 8 does not have — and cluster clients of - either flavour record no pool statistics at all. Command counts, durations and error - counts are recorded everywhere. -20. **A `RuntimeError` translated into `ConnectionError` is counted as a success.** The - uvloop "transport is closed" path converts the error before the metrics branch is - reached, so it lands in `redis_commands_total{status="success"}` and never in - `redis_connection_errors_total`. Do not alert on the absence of that counter. +19. **Cluster clients record no pool statistics.** Single-node clients, async and sync, + report `redis_pool_size` and `redis_pool_checked_out` from the pool's own containers + before every command; `InstrumentedRedisCluster` reports neither. Command counts, + durations and error counts are recorded everywhere. +20. **A uvloop closed-transport `RuntimeError` reaches you as `redis-py`'s + `ConnectionError`.** `execute_command` translates it so `redis-py` can retry or + reconnect, and records it under the type you catch: + `redis_commands_total{status="error"}` and + `redis_connection_errors_total{error_type="ConnectionError"}`. Catch + `redis.exceptions.ConnectionError`, not `RuntimeError`. Any other `RuntimeError` is + re-raised unchanged and counted under `RuntimeError`. ## Common mistakes @@ -432,6 +447,7 @@ Python, by Pydantic or by `redis-py`. | `pydantic.ValidationError` from `BaseRedisSettings` | a field out of range, an unknown keyword, a missing `key_prefix`, or one of the validator's three checks | | `AttributeError` from the factory | a settings object missing an attribute the protocols name | | `redis.exceptions.*` from the client | everything at run time: `ConnectionError`, `TimeoutError`, `ResponseError`, `RedisClusterException`, `ClusterDownError` and the rest of `redis-py`'s tree, all under `RedisError` | +| builtin `ConnectionError` from `AsyncRedisProvider` | Redis did not answer within the startup health check's three attempts — Python's `ConnectionError`, not `redis.exceptions.ConnectionError`, so it is not caught by `except RedisError` | | `ImportError` from an optional submodule | the extra is not installed; the message names it | `check_*_redis_health` and `close_*_redis_client` convert `redis-py`'s errors into a @@ -449,10 +465,3 @@ Fetch a page when the task is the one named beside it. | [Advanced](guide/advanced.md) | writing a `RedisMetricsProtocol`, Dishka wiring, cluster and TLS deployment notes | | [API reference](reference/index.md) | an exact signature or docstring — HTML only, see above | | [Changelog](changelog.md) | what changed between versions | - -One caveat about the three guide pages and the home page: their `BaseRedisSettings(...)` -calls predate the grouped settings model and pass flat keywords — `host=`, `retry_enabled=`, -`cluster_mode=`, `ssl=True` — that the model now rejects with `extra_forbidden`. The -protocols, the factory arguments, the metric names and the prose around those snippets are -current; the constructor calls are not. Take the settings shapes from this page or from -`README.md`, and read those pages for the parts they are still the only source of. diff --git a/docs/guide/advanced.md b/docs/guide/advanced.md index 09b338d..ed030d1 100644 --- a/docs/guide/advanced.md +++ b/docs/guide/advanced.md @@ -117,6 +117,7 @@ from dishka import make_async_container, Provider, Scope, provide from redis_client_kit import AsyncRedisClient from redis_client_kit.providers import AsyncRedisProvider from redis_client_kit.config import RedisSettingsProtocol +from redis_client_kit.settings import BaseRedisSettings, RedisConnectionSettings class SettingsProvider(Provider): scope = Scope.APP @@ -124,8 +125,8 @@ class SettingsProvider(Provider): @provide def get_redis_settings(self) -> RedisSettingsProtocol: return BaseRedisSettings( - host="localhost", - port=6379, + key_prefix="myapp", + connection=RedisConnectionSettings(host="localhost", port=6379), ) # Create container @@ -140,6 +141,12 @@ async with container() as ctx: await redis_client.set("key", "value") ``` +`AsyncRedisProvider()` verifies the connection before it yields the client and raises +`ConnectionError` when Redis stays unreachable, so the container fails to start rather +than handing out a client that cannot answer. Pass +`AsyncRedisProvider(check_health_on_startup=False)` for the faster startup that does not +contact Redis at all. + ### With Metrics ```python @@ -149,7 +156,7 @@ class MetricsProvider(Provider): scope = Scope.APP @provide - def get_redis_metrics(self) -> RedisMetricsProtocol | None: + def get_redis_metrics(self) -> RedisMetricsProtocol | None: # this exact annotation return PrometheusRedisMetrics() container = make_async_container( @@ -159,6 +166,21 @@ container = make_async_container( ) ``` +The annotation has to be exactly `RedisMetricsProtocol | None`; plain +`RedisMetricsProtocol` is a different key and the provider will not see it. + +`AsyncRedisProvider` also provides `RedisMetricsProtocol | None` as `None` so a container +without a metrics provider still resolves, and in dishka the last provider to claim a type +wins. Register `AsyncRedisProvider()` first, as above, or turn its default off: + +```python +container = make_async_container( + MetricsProvider(), + AsyncRedisProvider(provide_default_metrics=False), + SettingsProvider(), +) +``` + ### Automatic Lifecycle Management `AsyncRedisProvider` handles: @@ -170,8 +192,8 @@ container = make_async_container( ```python # Provider automatically: # 1. Creates client -# 2. Retries connection (max 3 attempts) -# 3. Checks health +# 2. Pings Redis, retrying up to 3 times with exponential backoff +# 3. Raises ConnectionError if it never answers # 4. Yields client # 5. Closes safely on exit @@ -185,22 +207,28 @@ async with container() as ctx: ### Cluster Configuration ```python -from redis_client_kit.settings import BaseRedisSettings +from redis_client_kit.settings import BaseRedisSettings, RedisClusterSettings settings = BaseRedisSettings( - cluster_mode=True, - cluster_nodes=[ - "node1.example.com:6379", - "node2.example.com:6379", - "node3.example.com:6379", - ], - require_full_coverage=True, # Fail if not all slots covered - read_from_replicas=False, # Read from replicas for better perf + key_prefix="myapp", + cluster=RedisClusterSettings( + enabled=True, + nodes=[ + "node1.example.com:6379", + "node2.example.com:6379", + "node3.example.com:6379", + ], + require_full_coverage=True, # Fail if not all slots covered + read_from_replicas=False, # Read from replicas for better perf + ), ) client = create_async_redis_client(settings) ``` +Every node string needs an explicit port; `node1.example.com` and `redis://node1` raise +`ValueError` when the factory parses them. + ### Cluster Health Checks ```python @@ -218,9 +246,12 @@ Enable reading from replicas for read-heavy workloads: ```python settings = BaseRedisSettings( - cluster_mode=True, - cluster_nodes=["..."], - read_from_replicas=True, # Distribute reads across replicas + key_prefix="myapp", + cluster=RedisClusterSettings( + enabled=True, + nodes=["node1.example.com:6379"], + read_from_replicas=True, # Distribute reads across replicas + ), ) ``` @@ -231,19 +262,22 @@ settings = BaseRedisSettings( ```python from pathlib import Path +from redis_client_kit.settings import RedisConnectionSettings, RedisSSLSettings + settings = BaseRedisSettings( - host="redis.prod.example.com", - port=6380, # Secure port - - # TLS configuration - ssl=True, - ssl_cert_reqs="required", - ssl_ca_certs=str(Path("/certs/ca.pem")), - ssl_certfile=str(Path("/certs/client-cert.pem")), - ssl_keyfile=str(Path("/certs/client-key.pem")), - - # Password authentication - password="secret-password", + key_prefix="myapp", + connection=RedisConnectionSettings( + host="redis.prod.example.com", + port=6380, # Secure port + password="secret-password", # Password authentication + ), + ssl=RedisSSLSettings( + enabled=True, + cert_reqs="required", + ca_certs=str(Path("/certs/ca.pem")), + certfile=str(Path("/certs/client-cert.pem")), + keyfile=str(Path("/certs/client-key.pem")), + ), ) client = create_async_redis_client(settings) @@ -256,14 +290,17 @@ redis-client-kit validates PEM files on startup: ```python # Validates PEM format and base64 content settings = BaseRedisSettings( - ssl=True, - ssl_ca_certs="/path/to/ca.pem", # Must be valid PEM + key_prefix="myapp", + ssl=RedisSSLSettings( + enabled=True, + cert_reqs="required", # required once SSL is enabled + ca_certs="/path/to/ca.pem", # Must be valid PEM + ), ) -# Raises ValueError if invalid: -# - Invalid PEM format -# - Invalid base64 content -# - File not found +# create_async_redis_client then raises: +# - ValueError on an invalid PEM format or invalid base64 content +# - FileNotFoundError on a missing file ``` ## Connection Resilience @@ -271,11 +308,16 @@ settings = BaseRedisSettings( ### Retry Logic ```python +from redis_client_kit.settings import RedisRetrySettings + settings = BaseRedisSettings( - retry_enabled=True, - retry_max_attempts=5, - retry_backoff_base=0.2, - retry_backoff_cap=2.0, + key_prefix="myapp", + retry=RedisRetrySettings( + enabled=True, + max_attempts=5, # 0, the default, retries nothing + backoff_base=0.2, + backoff_cap=2.0, + ), ) # Retries with exponential backoff: @@ -289,16 +331,23 @@ settings = BaseRedisSettings( ### Connection Pool Tuning ```python +import socket + +from redis_client_kit.settings import RedisPoolSettings + settings = BaseRedisSettings( - max_connections=50, # Pool size - socket_timeout=5.0, # Command timeout - socket_connect_timeout=2.0, # Connection timeout - socket_keepalive=True, # TCP keepalive - socket_keepalive_options={ # TCP settings - socket.TCP_KEEPIDLE: 1, - socket.TCP_KEEPINTVL: 1, - socket.TCP_KEEPCNT: 3, - }, + key_prefix="myapp", + pool=RedisPoolSettings( + max_connections=50, # Pool size + socket_timeout=5.0, # Command timeout + socket_connect_timeout=2.0, # Connection timeout + socket_keepalive=True, # TCP keepalive + socket_keepalive_options={ # TCP settings + socket.TCP_KEEPIDLE: 1, + socket.TCP_KEEPINTVL: 1, + socket.TCP_KEEPCNT: 3, + }, + ), ) ``` @@ -306,15 +355,19 @@ settings = BaseRedisSettings( ```python settings = BaseRedisSettings( + key_prefix="myapp", health_check_interval=30, # Ping every 30 seconds ) -# Set to 0 to disable: +# Set to 0 or None to disable: settings = BaseRedisSettings( + key_prefix="myapp", health_check_interval=0, # No health checks ) ``` +This is redis-py's per-connection ping interval, not the `check_*_redis_health` function. + ## Error Handling ### UVLoop RuntimeError Translation @@ -397,7 +450,8 @@ async with client.pipeline() as pipe: ```python # Decode responses only when needed settings = BaseRedisSettings( - decode_responses=False, # Return bytes (faster) + key_prefix="myapp", + response=RedisResponseSettings(decode_responses=False), # Return bytes (faster) ) # Manual decoding when needed @@ -445,21 +499,23 @@ logging.getLogger("redis_client_kit").setLevel(logging.DEBUG) ### Test Configuration ```python -class TestSettings(BaseRedisSettings): - host: str = "localhost" - port: int = 6380 # Different port - db: int = 15 # High DB number - - socket_timeout: float = 1.0 - retry_enabled: bool = False # Fail fast in tests - - decode_responses: bool = True +def test_settings() -> BaseRedisSettings: + return BaseRedisSettings( + key_prefix="test", + connection=RedisConnectionSettings( + host="localhost", + port=6380, # Different port + db=15, # High DB number + ), + pool=RedisPoolSettings(socket_timeout=1.0), + retry=RedisRetrySettings(enabled=False), # Fail fast in tests + response=RedisResponseSettings(decode_responses=True), + ) # Use in tests @pytest.fixture async def redis_client(): - settings = TestSettings() - client = create_async_redis_client(settings) + client = create_async_redis_client(test_settings()) yield client await client.flushdb() # Clean up await client.aclose() @@ -478,8 +534,11 @@ def redis_container(): @pytest.fixture async def redis_client(redis_container): settings = BaseRedisSettings( - host=redis_container.get_container_host_ip(), - port=int(redis_container.get_exposed_port(6379)), + key_prefix="test", + connection=RedisConnectionSettings( + host=redis_container.get_container_host_ip(), + port=int(redis_container.get_exposed_port(6379)), + ), ) client = create_async_redis_client(settings) yield client @@ -489,5 +548,5 @@ async def redis_client(redis_container): ## Next Steps - [Configuration Guide](configuration.md) — Complete settings reference -- [API Reference](../reference/) — Full API documentation +- [API Reference](../reference/index.md) — Full API documentation - [GitHub Repository](https://github.com/bedrock-python/redis-client-kit) — Source code and issues diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index 4eb8c46..d6a945c 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -19,25 +19,42 @@ The easiest way to configure redis-client-kit is using `BaseRedisSettings`: pip install redis-client-kit[settings] ``` +The model is **grouped, not flat**. Six group models go in as objects — `connection`, +`cluster`, `pool`, `retry`, `ssl` and `response` — alongside the top-level `key_prefix`, +`health_check_interval` and `metrics_enabled`. It forbids extra keywords, so a flat +`BaseRedisSettings(host="localhost")` raises `ValidationError: Extra inputs are not +permitted`, and `key_prefix` is required. + +```python +from redis_client_kit.settings import ( + BaseRedisSettings, + RedisConnectionSettings, + RedisPoolSettings, + RedisResponseSettings, +) + +settings = BaseRedisSettings( + key_prefix="myapp", + connection=RedisConnectionSettings(host="localhost", port=6379), + pool=RedisPoolSettings(max_connections=10, socket_timeout=5.0), + response=RedisResponseSettings(decode_responses=True, encoding="utf-8"), + health_check_interval=30, +) +``` + +`key_prefix` is required but never read by this library — it is a value you carry and +apply to your own keys. + +Subclass it when you want different defaults or extra fields of your own: + ```python -from redis_client_kit.settings import BaseRedisSettings +from pydantic import Field class MySettings(BaseRedisSettings): - host: str = "localhost" - port: int = 6379 - password: str | None = None - db: int = 0 - - # Connection pool - max_connections: int = 10 - socket_timeout: float = 5.0 - - # Health checks - health_check_interval: int = 30 - - # Response format - decode_responses: bool = True - encoding: str = "utf-8" + key_prefix: str = "myapp" + connection: RedisConnectionSettings = Field( + default_factory=lambda: RedisConnectionSettings(host="redis.internal"), + ) ``` ## Connection Settings @@ -45,31 +62,49 @@ class MySettings(BaseRedisSettings): ### Basic Connection ```python +from redis_client_kit.settings import BaseRedisSettings, RedisConnectionSettings + settings = BaseRedisSettings( - host="redis.example.com", - port=6379, - password="secret", - db=0, - client_name="myapp", - protocol=2, # RESP protocol version (2 or 3) + key_prefix="myapp", + connection=RedisConnectionSettings( + host="redis.example.com", + port=6379, + password="secret", # stored as SecretStr, read with get_password() + db=0, + client_name="myapp", + protocol=2, # RESP protocol version (2 or 3) + ), ) ``` ### From Environment Variables +`BaseRedisSettings` sets no `env_prefix` and no `env_nested_delimiter`, so out of the box +the names are the bare field names and a group is one JSON document: + +```bash +KEY_PREFIX=myapp +HEALTH_CHECK_INTERVAL=15 +CONNECTION='{"host": "redis.example.com", "port": 6379}' +``` + +Subclass it for the usual per-field shape: + ```python from pydantic_settings import SettingsConfigDict class Settings(BaseRedisSettings): model_config = SettingsConfigDict( - env_prefix="REDIS_", # Read from REDIS_HOST, REDIS_PORT, etc. + env_prefix="REDIS_", + env_nested_delimiter="__", env_file=".env", ) # Set environment variables: -# REDIS_HOST=redis.example.com -# REDIS_PORT=6379 -# REDIS_PASSWORD=secret +# REDIS_KEY_PREFIX=myapp +# REDIS_CONNECTION__HOST=redis.example.com +# REDIS_CONNECTION__PORT=6379 +# REDIS_CONNECTION__PASSWORD=secret settings = Settings() ``` @@ -79,16 +114,23 @@ settings = Settings() Configure connection pool behavior: ```python +import socket + +from redis_client_kit.settings import RedisPoolSettings + settings = BaseRedisSettings( - max_connections=20, # Maximum connections in pool - socket_timeout=5.0, # Socket operation timeout (seconds) - socket_connect_timeout=5.0, # Connection timeout (seconds) - socket_keepalive=True, # Enable TCP keepalive - socket_keepalive_options={ # TCP keepalive options - socket.TCP_KEEPIDLE: 1, - socket.TCP_KEEPINTVL: 1, - socket.TCP_KEEPCNT: 3, - }, + key_prefix="myapp", + pool=RedisPoolSettings( + max_connections=20, # Maximum connections in pool + socket_timeout=5.0, # Socket operation timeout (seconds) + socket_connect_timeout=5.0, # Connection timeout (seconds) + socket_keepalive=True, # Enable TCP keepalive + socket_keepalive_options={ # TCP keepalive options + socket.TCP_KEEPIDLE: 1, + socket.TCP_KEEPINTVL: 1, + socket.TCP_KEEPCNT: 3, + }, + ), ) ``` @@ -97,44 +139,66 @@ settings = BaseRedisSettings( Configure automatic retry with exponential backoff: ```python +from redis_client_kit.settings import RedisRetrySettings + settings = BaseRedisSettings( - retry_enabled=True, - retry_max_attempts=3, # Max retry attempts - retry_backoff_base=0.1, # Base delay in seconds - retry_backoff_cap=1.0, # Maximum delay in seconds + key_prefix="myapp", + retry=RedisRetrySettings( + enabled=True, + max_attempts=3, # Max retry attempts + backoff_base=0.1, # Base delay in seconds + backoff_cap=1.0, # Maximum delay in seconds + ), ) ``` +`max_attempts` defaults to `0`, and `enabled=True` on its own retries nothing: the `Retry` +object is handed to redis-py only when both `enabled` and `max_attempts` are truthy. + Retry logic uses exponential backoff: ``` -delay = min(backoff_base * (2 ** attempt), backoff_cap) +delay = min(backoff_cap, backoff_base * (2 ** failures)) ``` +It covers connection failures — redis-py retries on `ConnectionError`, `TimeoutError` and +`socket.timeout`. A `ResponseError` from a bad command is raised on the first try. + ## SSL/TLS Settings Configure secure connections: ```python +from redis_client_kit.settings import RedisSSLSettings + settings = BaseRedisSettings( - ssl=True, - ssl_cert_reqs="required", # "required", "optional", or "none" - ssl_ca_certs="/path/to/ca.pem", - ssl_certfile="/path/to/cert.pem", - ssl_keyfile="/path/to/key.pem", + key_prefix="myapp", + ssl=RedisSSLSettings( + enabled=True, + cert_reqs="required", # "required", "optional", or "none" + ca_certs="/path/to/ca.pem", + certfile="/path/to/cert.pem", + keyfile="/path/to/key.pem", + ), ) ``` +`cert_reqs` is mandatory once `enabled` is set: `ssl.enabled` with `cert_reqs=None` raises +`ValueError` from the model validator. + ### SSL Certificate Validation -redis-client-kit validates PEM files automatically when SSL is enabled: +redis-client-kit validates PEM files when the client is built, not when it connects: ```python from pathlib import Path settings = BaseRedisSettings( - ssl=True, - ssl_cert_reqs="required", - ssl_ca_certs=str(Path("/certs/ca.pem").absolute()), + key_prefix="myapp", + ssl=RedisSSLSettings( + enabled=True, + cert_reqs="required", + ca_certs=str(Path("/certs/ca.pem").absolute()), + ), ) ``` @@ -143,27 +207,35 @@ settings = BaseRedisSettings( Configure Redis Cluster: ```python +from redis_client_kit.settings import RedisClusterSettings + settings = BaseRedisSettings( - cluster_mode=True, - cluster_nodes=[ - "node1.example.com:6379", - "node2.example.com:6379", - "node3.example.com:6379", - ], - require_full_coverage=True, # Require all slots covered - read_from_replicas=False, # Read from replicas + key_prefix="myapp", + cluster=RedisClusterSettings( + enabled=True, + nodes=[ + "node1.example.com:6379", + "node2.example.com:6379", + "node3.example.com:6379", + ], + require_full_coverage=True, # Require all slots covered + read_from_replicas=False, # Read from replicas + ), ) ``` +Every node string needs an explicit port; `node1.example.com` raises `ValueError` when the +factory parses it. `connection.db` must be `0` when the cluster is enabled. + ### Cluster Discovery -If `cluster_nodes` is empty, uses `host:port` as the entry point: +If `cluster.nodes` is empty, uses `connection.host:port` as the entry point: ```python settings = BaseRedisSettings( - cluster_mode=True, - host="cluster-entry.example.com", - port=6379, + key_prefix="myapp", + cluster=RedisClusterSettings(enabled=True), + connection=RedisConnectionSettings(host="cluster-entry.example.com", port=6379), ) ``` @@ -172,9 +244,14 @@ settings = BaseRedisSettings( Configure response format: ```python +from redis_client_kit.settings import RedisResponseSettings + settings = BaseRedisSettings( - decode_responses=True, # Return strings instead of bytes - encoding="utf-8", # String encoding + key_prefix="myapp", + response=RedisResponseSettings( + decode_responses=True, # Return strings instead of bytes + encoding="utf-8", # String encoding + ), ) ``` @@ -190,10 +267,12 @@ await client.get("key") # Returns b"value" (bytes) ## Health Check Settings -Configure health check behavior: +`health_check_interval` is redis-py's per-connection ping interval, not the +`check_*_redis_health` function. `0` and `None` both disable it. ```python settings = BaseRedisSettings( + key_prefix="myapp", health_check_interval=30, # Seconds between health checks (0 to disable) ) ``` @@ -283,78 +362,75 @@ settings = MySettings( client = create_async_redis_client(settings) ``` +Every attribute the protocols name has to be there: the factory reads all of them and +raises `AttributeError` on the first one missing. The protocols are not +`@runtime_checkable`, so `isinstance(settings, RedisSettingsProtocol)` raises `TypeError`. + ## Configuration Best Practices ### Production Settings ```python -class ProductionSettings(BaseRedisSettings): - # Connection - host: str = Field(default="redis.prod.example.com") - port: int = 6379 - password: SecretStr # Use SecretStr for passwords - - # Pool - max_connections: int = 50 # Higher for production - socket_timeout: float = 5.0 - socket_connect_timeout: float = 2.0 - socket_keepalive: bool = True - - # Retry - retry_enabled: bool = True - retry_max_attempts: int = 5 # More retries - retry_backoff_base: float = 0.2 - retry_backoff_cap: float = 2.0 - - # Security - ssl: bool = True - ssl_cert_reqs: str = "required" - - # Health - health_check_interval: int = 30 - - # Response - decode_responses: bool = True +def production_settings(password: str) -> BaseRedisSettings: + return BaseRedisSettings( + key_prefix="myapp", + connection=RedisConnectionSettings( + host="redis.prod.example.com", + port=6379, + password=password, # stored as SecretStr + ), + pool=RedisPoolSettings( + max_connections=50, # Higher for production + socket_timeout=5.0, + socket_connect_timeout=2.0, + socket_keepalive=True, + ), + retry=RedisRetrySettings( + enabled=True, + max_attempts=5, # More retries + backoff_base=0.2, + backoff_cap=2.0, + ), + ssl=RedisSSLSettings(enabled=True, cert_reqs="required"), + response=RedisResponseSettings(decode_responses=True), + health_check_interval=30, + ) ``` ### Development Settings ```python -class DevelopmentSettings(BaseRedisSettings): - host: str = "localhost" - port: int = 6379 - password: str | None = None - - # Smaller pool for dev - max_connections: int = 5 - - # No SSL in dev - ssl: bool = False - - # Decode responses for easier debugging - decode_responses: bool = True +def development_settings() -> BaseRedisSettings: + return BaseRedisSettings( + key_prefix="myapp", + connection=RedisConnectionSettings(host="localhost", port=6379), + pool=RedisPoolSettings(max_connections=5), # Smaller pool for dev + response=RedisResponseSettings(decode_responses=True), + ) ``` ### Testing Settings ```python -class TestSettings(BaseRedisSettings): - host: str = "localhost" - port: int = 6380 # Different port - db: int = 15 # Use high DB number - - # Fast timeouts for tests - socket_timeout: float = 1.0 - socket_connect_timeout: float = 1.0 - - # No retries in tests - retry_enabled: bool = False - - decode_responses: bool = True +def test_settings() -> BaseRedisSettings: + return BaseRedisSettings( + key_prefix="test", + connection=RedisConnectionSettings( + host="localhost", + port=6380, # Different port + db=15, # Use high DB number + ), + pool=RedisPoolSettings( + socket_timeout=1.0, # Fast timeouts for tests + socket_connect_timeout=1.0, + ), + retry=RedisRetrySettings(enabled=False), # No retries in tests + response=RedisResponseSettings(decode_responses=True), + ) ``` ## Next Steps - [Quick Start](quickstart.md) — Basic usage examples - [Advanced Usage](advanced.md) — Metrics, instrumentation, DI -- [API Reference](../reference/) — Complete API documentation +- [API Reference](../reference/index.md) — Complete API documentation diff --git a/docs/guide/quickstart.md b/docs/guide/quickstart.md index 9212f57..aab5627 100644 --- a/docs/guide/quickstart.md +++ b/docs/guide/quickstart.md @@ -18,14 +18,17 @@ pip install redis-client-kit[settings] ### Async Client +`BaseRedisSettings` is grouped: connection, pool, retry, SSL and response options each +go in as their own model, and `key_prefix` is required. + ```python from redis_client_kit import create_async_redis_client -from redis_client_kit.settings import BaseRedisSettings +from redis_client_kit.settings import BaseRedisSettings, RedisConnectionSettings # Configure settings = BaseRedisSettings( - host="localhost", - port=6379, + key_prefix="myapp", + connection=RedisConnectionSettings(host="localhost", port=6379), ) # Create client @@ -44,9 +47,12 @@ await client.aclose() ```python from redis_client_kit.sync import create_redis_client -from redis_client_kit.settings import BaseRedisSettings +from redis_client_kit.settings import BaseRedisSettings, RedisConnectionSettings -settings = BaseRedisSettings(host="localhost", port=6379) +settings = BaseRedisSettings( + key_prefix="myapp", + connection=RedisConnectionSettings(host="localhost", port=6379), +) client = create_redis_client(settings) client.set("key", "value") @@ -61,10 +67,12 @@ client.close() By default, Redis returns bytes. Enable `decode_responses` to get strings: ```python +from redis_client_kit.settings import RedisResponseSettings + settings = BaseRedisSettings( - host="localhost", - port=6379, - decode_responses=True, # Return strings instead of bytes + key_prefix="myapp", + connection=RedisConnectionSettings(host="localhost", port=6379), + response=RedisResponseSettings(decode_responses=True), # strings instead of bytes ) client = create_async_redis_client(settings) @@ -125,11 +133,15 @@ async with redis_client(settings) as client: Connection pooling is enabled by default: ```python +from redis_client_kit.settings import RedisPoolSettings + settings = BaseRedisSettings( - host="localhost", - port=6379, - max_connections=20, # Pool size - socket_timeout=5.0, # Socket timeout in seconds + key_prefix="myapp", + connection=RedisConnectionSettings(host="localhost", port=6379), + pool=RedisPoolSettings( + max_connections=20, # Pool size + socket_timeout=5.0, # Socket timeout in seconds + ), ) client = create_async_redis_client(settings) @@ -200,4 +212,4 @@ top = await client.zrange("scores", 0, -1, withscores=True) - [Configuration Guide](configuration.md) — Learn about all settings - [Advanced Usage](advanced.md) — Clusters, SSL, metrics -- [API Reference](../reference/) — Complete API documentation +- [API Reference](../reference/index.md) — Complete API documentation diff --git a/docs/index.md b/docs/index.md index cc0765b..0d026a5 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,11 +1,11 @@ # redis-client-kit -Production-ready Redis client library for Python with optional Pydantic settings, OpenTelemetry instrumentation, and Dishka dependency injection support. +Production-ready Redis client library for Python with optional Pydantic settings, Prometheus metrics, and Dishka dependency injection support. ## Why redis-client-kit? -- **Zero Dependencies** — Core library only depends on `redis>=7.1.0` -- **Optional Features** — Add Pydantic, OpenTelemetry, or Dishka only when you need them +- **Zero Dependencies** — Core library only depends on `redis>=4.5.0,<9.0.0` +- **Optional Features** — Add Pydantic, Prometheus, or Dishka only when you need them - **Zero Overhead** — Plain redis-py clients when metrics aren't provided - **Production Ready** — Battle-tested with 90%+ test coverage - **Type Safe** — Full type hints with protocols for flexibility @@ -50,19 +50,23 @@ pip install redis-client-kit[providers] pip install redis-client-kit[all] ``` -**Requirements:** Python 3.11+ +**Requirements:** Python 3.10+ ## Quick Example ```python from redis_client_kit import create_async_redis_client -from redis_client_kit.settings import BaseRedisSettings +from redis_client_kit.settings import ( + BaseRedisSettings, + RedisConnectionSettings, + RedisResponseSettings, +) -# Configure +# Configure: settings are grouped, and key_prefix is required settings = BaseRedisSettings( - host="localhost", - port=6379, - decode_responses=True, + key_prefix="myapp", + connection=RedisConnectionSettings(host="localhost", port=6379), + response=RedisResponseSettings(decode_responses=True), ) # Create client @@ -90,7 +94,7 @@ redis-client-kit follows a layered architecture: ``` ┌─────────────────────────────────────────┐ -│ Optional Modules (settings, etc.) │ [settings], [instrumentation], [providers] +│ Optional Modules (settings, etc.) │ [settings], [metrics], [providers] ├─────────────────────────────────────────┤ │ Factory Functions & Lifecycle │ create_*, check_*, close_* ├─────────────────────────────────────────┤ diff --git a/pyproject.toml b/pyproject.toml index 62ae576..33bc0b5 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -140,8 +140,6 @@ omit = [ "*/test_*.py", "*/_deps.py", "redis_client_kit/metrics/redis.py", - "redis_client_kit/providers/redis.py", - "redis_client_kit/providers/utils.py", ] [tool.coverage.report] diff --git a/redis_client_kit/aio/instrumented.py b/redis_client_kit/aio/instrumented.py index 0254be4..7b80393 100644 --- a/redis_client_kit/aio/instrumented.py +++ b/redis_client_kit/aio/instrumented.py @@ -15,6 +15,27 @@ logger = logging.getLogger(__name__) +def _translate_closed_transport(error: RuntimeError) -> RedisConnectionError | None: + """Translate a uvloop closed-transport RuntimeError into a Redis ConnectionError. + + uvloop raises RuntimeError when the transport is closed but still being used. + redis-py only handles its own ConnectionError (retry or reconnect), so the error + is translated. Returns None for any other RuntimeError. + """ + message = str(error).lower() + if "the handler is closed" in message or "transport is closed" in message: + return RedisConnectionError(str(error)) + return None + + +def _record_error(metrics: RedisMetricsProtocol, error: BaseException) -> None: + """Record an error without letting the metrics backend break the command.""" + try: + metrics.record_error(error_type=type(error).__name__) + except Exception: + logger.exception("Failed to record Redis error metrics") + + class InstrumentedRedis(Redis): """Redis client with Prometheus metrics collection.""" @@ -32,10 +53,12 @@ async def execute_command(self, *args: Any, **options: Any) -> Any: pool = self.connection_pool if pool: try: - # redis-py async pool has _all_connections and _in_use_connections + # redis-py async pool has _available_connections and _in_use_connections + available: list[Any] = getattr(pool, "_available_connections", []) + in_use: set[Any] = getattr(pool, "_in_use_connections", set()) self._metrics.record_pool_stats( - pool_size=len(getattr(pool, "_all_connections", [])), - pool_checked_out=len(getattr(pool, "_in_use_connections", [])), + pool_size=len(available) + len(in_use), + pool_checked_out=len(in_use), ) except Exception: logger.exception("Failed to record Redis pool metrics") @@ -43,18 +66,15 @@ async def execute_command(self, *args: Any, **options: Any) -> Any: try: return await super().execute_command(*args, **options) except RuntimeError as e: - # uvloop raises RuntimeError when transport is closed but still being used. - # We translate it to ConnectionError so redis-py can handle it (retry or reconnect). - if "the handler is closed" in str(e).lower() or "transport is closed" in str(e).lower(): - raise RedisConnectionError(str(e)) from e + status = "error" + translated = _translate_closed_transport(e) + _record_error(self._metrics, translated or e) + if translated is not None: + raise translated from e raise except Exception as e: status = "error" - error_type = type(e).__name__ - try: - self._metrics.record_error(error_type=error_type) - except Exception: - logger.exception("Failed to record Redis error metrics") + _record_error(self._metrics, e) raise finally: duration = time.perf_counter() - start @@ -86,18 +106,15 @@ async def execute_command(self, *args: Any, **options: Any) -> Any: try: return await super().execute_command(*args, **options) except RuntimeError as e: - # uvloop raises RuntimeError when transport is closed but still being used. - # We translate it to ConnectionError so redis-py can handle it (retry or reconnect). - if "the handler is closed" in str(e).lower() or "transport is closed" in str(e).lower(): - raise RedisConnectionError(str(e)) from e + status = "error" + translated = _translate_closed_transport(e) + _record_error(self._metrics, translated or e) + if translated is not None: + raise translated from e raise except Exception as e: status = "error" - error_type = type(e).__name__ - try: - self._metrics.record_error(error_type=error_type) - except Exception: - logger.exception("Failed to record Redis error metrics") + _record_error(self._metrics, e) raise finally: duration = time.perf_counter() - start diff --git a/redis_client_kit/providers/redis.py b/redis_client_kit/providers/redis.py index 394e82f..2ee6073 100644 --- a/redis_client_kit/providers/redis.py +++ b/redis_client_kit/providers/redis.py @@ -9,7 +9,7 @@ from ..aio import AsyncRedisClient, check_async_redis_health, close_async_redis_client, create_async_redis_client from ..config import RedisSettingsProtocol from ..protocols import RedisMetricsProtocol -from ._deps import Provider, Scope, provide +from ._deps import Provider, Scope from .utils import retry_async_connection, safe_async_cleanup logger = logging.getLogger(__name__) @@ -18,17 +18,39 @@ class AsyncRedisProvider(Provider): # type: ignore[misc] """Dishka provider for Redis dependencies. - Provides two Redis client options: - - get_redis(): Simple client without startup health check - - get_redis_with_health_check(): Client with connection verification and retries + Provides one ``AsyncRedisClient``, chosen when the provider is constructed: - Choose get_redis() for faster startup when Redis availability is not critical. - Choose get_redis_with_health_check() for guaranteed connection on startup. + - ``AsyncRedisProvider()`` verifies the connection on startup with retries and + raises when Redis stays unreachable, so startup fails instead of handing out + a client that cannot answer. + - ``AsyncRedisProvider(check_health_on_startup=False)`` yields the client without + contacting Redis, for a faster startup when Redis availability is not critical. + + It also provides ``RedisMetricsProtocol | None`` as ``None`` so a container with no + metrics provider still resolves. Dishka lets the last registered provider of a type + win, so this default overrides a metrics provider registered before it: either + register this provider first, or construct it with ``provide_default_metrics=False``. """ scope = Scope.APP # type: ignore[misc] - @provide # type: ignore[misc] + def __init__( + self, + *, + check_health_on_startup: bool = True, + provide_default_metrics: bool = True, + ) -> None: + """Register the client factory, and the default metrics factory when asked to. + + Args: + check_health_on_startup: Verify the connection before yielding the client + provide_default_metrics: Provide ``RedisMetricsProtocol | None`` as ``None`` + """ + super().__init__() + self.provide(self.get_redis_with_health_check if check_health_on_startup else self.get_redis) + if provide_default_metrics: + self.provide(self.get_default_metrics) + async def get_redis( self, redis_settings: RedisSettingsProtocol, @@ -57,7 +79,6 @@ async def get_redis( exception_type=RedisError, ) - @provide # type: ignore[misc] async def get_redis_with_health_check( self, redis_settings: RedisSettingsProtocol, @@ -76,14 +97,22 @@ async def get_redis_with_health_check( Configured and verified AsyncRedisClient instance Raises: - Exception: If connection fails after max retry attempts + ConnectionError: If Redis is still unreachable after the last attempt """ client = create_async_redis_client(redis_settings, metrics=metrics) - await retry_async_connection( - connect_func=lambda: check_async_redis_health(client), - service_name="Redis", - ) + try: + await retry_async_connection( + connect_func=lambda: check_async_redis_health(client), + service_name="Redis", + ) + except BaseException: + await safe_async_cleanup( + cleanup_func=functools.partial(close_async_redis_client, client), + service_name="Redis client", + exception_type=RedisError, + ) + raise try: yield client @@ -94,7 +123,6 @@ async def get_redis_with_health_check( exception_type=RedisError, ) - @provide # type: ignore[misc] def get_default_metrics(self) -> RedisMetricsProtocol | None: """Provide default None for metrics if not provided in container.""" return None diff --git a/redis_client_kit/providers/utils.py b/redis_client_kit/providers/utils.py index 18e1f59..77a45cf 100644 --- a/redis_client_kit/providers/utils.py +++ b/redis_client_kit/providers/utils.py @@ -13,25 +13,44 @@ async def retry_async_connection( max_attempts: int = 3, backoff_base: float = 1.0, ) -> None: - """Retry async connection with exponential backoff.""" + """Retry async connection with exponential backoff. + + An attempt fails when connect_func raises or returns a falsy value; both are + retried with the same backoff. + + Args: + connect_func: Callable returning True once the service is reachable + service_name: Name used in log messages and in the raised error + max_attempts: Number of attempts before giving up + backoff_base: Base of the exponential backoff in seconds + + Raises: + ConnectionError: If no attempt reported success and none of them raised. + Exception: The error raised by the last attempt, if it raised one. + """ for attempt in range(1, max_attempts + 1): + error: Exception try: if await connect_func(): logger.info("%s connected successfully", service_name) return + error = ConnectionError(f"{service_name} did not report a healthy connection") except Exception as e: - if attempt == max_attempts: - raise - wait_time = backoff_base * (2 ** (attempt - 1)) - logger.warning( - "%s connection failed (attempt %d/%d), retrying in %ss: %s", - service_name, - attempt, - max_attempts, - wait_time, - e, - ) - await asyncio.sleep(wait_time) + error = e + + if attempt == max_attempts: + raise error + + wait_time = backoff_base * (2 ** (attempt - 1)) + logger.warning( + "%s connection failed (attempt %d/%d), retrying in %ss: %s", + service_name, + attempt, + max_attempts, + wait_time, + error, + ) + await asyncio.sleep(wait_time) async def safe_async_cleanup( diff --git a/redis_client_kit/settings/redis.py b/redis_client_kit/settings/redis.py index 6c6264e..cc997cc 100644 --- a/redis_client_kit/settings/redis.py +++ b/redis_client_kit/settings/redis.py @@ -116,4 +116,12 @@ def _validate_cluster_and_tls(self) -> "BaseRedisSettings": return self -__all__ = ["BaseRedisSettings"] +__all__ = [ + "BaseRedisSettings", + "RedisClusterSettings", + "RedisConnectionSettings", + "RedisPoolSettings", + "RedisResponseSettings", + "RedisRetrySettings", + "RedisSSLSettings", +] diff --git a/redis_client_kit/sync/instrumented.py b/redis_client_kit/sync/instrumented.py index a29ca3a..5522d61 100644 --- a/redis_client_kit/sync/instrumented.py +++ b/redis_client_kit/sync/instrumented.py @@ -15,6 +15,26 @@ logger = logging.getLogger(__name__) +def _translate_closed_transport(error: RuntimeError) -> RedisConnectionError | None: + """Translate a closed-transport RuntimeError into a Redis ConnectionError. + + redis-py only handles its own ConnectionError (retry or reconnect), so the error + is translated. Returns None for any other RuntimeError. + """ + message = str(error).lower() + if "the handler is closed" in message or "transport is closed" in message: + return RedisConnectionError(str(error)) + return None + + +def _record_error(metrics: RedisMetricsProtocol, error: BaseException) -> None: + """Record an error without letting the metrics backend break the command.""" + try: + metrics.record_error(error_type=type(error).__name__) + except Exception: + logger.exception("Failed to record Redis error metrics") + + class InstrumentedRedis(Redis): """Redis client with Prometheus metrics collection.""" @@ -46,17 +66,15 @@ def execute_command(self, *args: Any, **options: Any) -> Any: try: return super().execute_command(*args, **options) except RuntimeError as e: - # Handle closed connection errors - if "the handler is closed" in str(e).lower() or "transport is closed" in str(e).lower(): - raise RedisConnectionError(str(e)) from e + status = "error" + translated = _translate_closed_transport(e) + _record_error(self._metrics, translated or e) + if translated is not None: + raise translated from e raise except Exception as e: status = "error" - error_type = type(e).__name__ - try: - self._metrics.record_error(error_type=error_type) - except Exception: - logger.exception("Failed to record Redis error metrics") + _record_error(self._metrics, e) raise finally: duration = time.perf_counter() - start @@ -88,17 +106,15 @@ def execute_command(self, *args: Any, **options: Any) -> Any: try: return super().execute_command(*args, **options) except RuntimeError as e: - # Handle closed connection errors - if "the handler is closed" in str(e).lower() or "transport is closed" in str(e).lower(): - raise RedisConnectionError(str(e)) from e + status = "error" + translated = _translate_closed_transport(e) + _record_error(self._metrics, translated or e) + if translated is not None: + raise translated from e raise except Exception as e: status = "error" - error_type = type(e).__name__ - try: - self._metrics.record_error(error_type=error_type) - except Exception: - logger.exception("Failed to record Redis error metrics") + _record_error(self._metrics, e) raise finally: duration = time.perf_counter() - start diff --git a/redis_client_kit/utils.py b/redis_client_kit/utils.py index 91f2b0d..a0d55cf 100644 --- a/redis_client_kit/utils.py +++ b/redis_client_kit/utils.py @@ -116,3 +116,11 @@ def mask_redis_kwargs(kwargs: dict[str, object]) -> dict[str, object]: if masked.get("password"): masked["password"] = "********" return masked + + +__all__ = [ + "build_base_redis_kwargs", + "build_redis_retry", + "mask_redis_kwargs", + "parse_redis_url_node", +] diff --git a/tests/unit/aio/test_instrumented_exceptions.py b/tests/unit/aio/test_instrumented_exceptions.py index 593e60e..9289405 100644 --- a/tests/unit/aio/test_instrumented_exceptions.py +++ b/tests/unit/aio/test_instrumented_exceptions.py @@ -21,7 +21,7 @@ async def test__instrumented_redis__pool_stats_exception__logs_and_continues(moc # Create mock pool that raises exception when accessing attributes mock_pool = MagicMock() - type(mock_pool)._all_connections = property(lambda self: (_ for _ in ()).throw(Exception("Pool error"))) + type(mock_pool)._available_connections = property(lambda self: (_ for _ in ()).throw(Exception("Pool error"))) client.connection_pool = mock_pool # Act diff --git a/tests/unit/aio/test_metrics.py b/tests/unit/aio/test_metrics.py index af69332..c21b878 100644 --- a/tests/unit/aio/test_metrics.py +++ b/tests/unit/aio/test_metrics.py @@ -17,9 +17,9 @@ async def test__instrumented_redis__execute_command_success__records_metrics_and ) -> None: # Arrange client = InstrumentedRedis(metrics=mock_metrics) - client.connection_pool = MagicMock() - client.connection_pool._all_connections = [1, 2, 3] - client.connection_pool._in_use_connections = [1] + # The real redis-py async pool: _available_connections is a list, _in_use_connections a set. + client.connection_pool._available_connections = [1, 2] + client.connection_pool._in_use_connections = {3} # Act with patch("redis.asyncio.Redis.execute_command", new_callable=AsyncMock) as mock_execute: @@ -32,6 +32,28 @@ async def test__instrumented_redis__execute_command_success__records_metrics_and mock_metrics.record_command.assert_called_with(command="SET", status="success", duration=pytest.approx(0, abs=1)) +@pytest.mark.asyncio +async def test__instrumented_redis__real_connection_pool__reports_non_zero_pool_size( + mock_metrics: MagicMock, +) -> None: + # Arrange + client = InstrumentedRedis(metrics=mock_metrics) + pool = client.connection_pool + pool._available_connections.append(pool.make_connection()) + pool._in_use_connections.add(pool.make_connection()) + + # Act + with patch("redis.asyncio.Redis.execute_command", new_callable=AsyncMock) as mock_execute: + mock_execute.return_value = "OK" + await client.execute_command("PING") + + # Assert + mock_metrics.record_pool_stats.assert_called_with(pool_size=2, pool_checked_out=1) + + # Cleanup + await pool.disconnect() + + @pytest.mark.asyncio async def test__instrumented_redis__execute_command_error__records_error_metrics( mock_metrics: MagicMock, diff --git a/tests/unit/aio/test_uvloop_handling.py b/tests/unit/aio/test_uvloop_handling.py index 9bf14a1..a23701a 100644 --- a/tests/unit/aio/test_uvloop_handling.py +++ b/tests/unit/aio/test_uvloop_handling.py @@ -42,6 +42,68 @@ async def test__instrumented_redis_cluster__uvloop_handler_closed_error__transla assert "the handler is closed" in str(exc_info.value).lower() +@pytest.mark.asyncio +async def test__instrumented_redis__uvloop_handler_closed_error__records_error_metrics() -> None: + # Arrange + mock_metrics = MagicMock() + client = InstrumentedRedis(host="localhost", port=6379, metrics=mock_metrics) + msg = ( + "unable to perform operation on ; the handler is closed" + ) + + # Act + with ( + patch("redis.asyncio.Redis.execute_command", side_effect=RuntimeError(msg)), + pytest.raises(RedisConnectionError), + ): + await client.execute_command("GET", "key") + + # Assert + mock_metrics.record_error.assert_called_once_with(error_type="ConnectionError") + assert mock_metrics.record_command.call_args.kwargs["status"] == "error" + + +@pytest.mark.asyncio +async def test__instrumented_redis_cluster__uvloop_handler_closed_error__records_error_metrics() -> None: + # Arrange + mock_metrics = MagicMock() + msg = ( + "unable to perform operation on ; the handler is closed" + ) + + # Act + with patch("redis.asyncio.cluster.RedisCluster.__init__", return_value=None): + client = InstrumentedRedisCluster(metrics=mock_metrics) + + with ( + patch("redis.asyncio.cluster.RedisCluster.execute_command", side_effect=RuntimeError(msg)), + pytest.raises(RedisConnectionError), + ): + await client.execute_command("GET", "key") + + # Assert + mock_metrics.record_error.assert_called_once_with(error_type="ConnectionError") + assert mock_metrics.record_command.call_args.kwargs["status"] == "error" + + +@pytest.mark.asyncio +async def test__instrumented_redis__other_runtime_error__records_error_metrics() -> None: + # Arrange + mock_metrics = MagicMock() + client = InstrumentedRedis(host="localhost", port=6379, metrics=mock_metrics) + + # Act + with ( + patch("redis.asyncio.Redis.execute_command", side_effect=RuntimeError("some other error")), + pytest.raises(RuntimeError), + ): + await client.execute_command("GET", "key") + + # Assert + mock_metrics.record_error.assert_called_once_with(error_type="RuntimeError") + assert mock_metrics.record_command.call_args.kwargs["status"] == "error" + + @pytest.mark.asyncio async def test__instrumented_redis__other_runtime_error__preserves_original_exception() -> None: # Arrange diff --git a/tests/unit/providers/test_provider_utils.py b/tests/unit/providers/test_provider_utils.py new file mode 100644 index 0000000..1e75848 --- /dev/null +++ b/tests/unit/providers/test_provider_utils.py @@ -0,0 +1,79 @@ +"""Tests for the connection retry helper used by the Dishka providers.""" + +from unittest.mock import AsyncMock, patch + +import pytest + +from redis_client_kit.providers.utils import retry_async_connection, safe_async_cleanup + + +@pytest.mark.asyncio +async def test__retry_async_connection__connect_succeeds__returns_without_sleeping() -> None: + # Arrange + connect = AsyncMock(return_value=True) + + # Act + with patch("redis_client_kit.providers.utils.asyncio.sleep", new_callable=AsyncMock) as mock_sleep: + await retry_async_connection(connect_func=connect, service_name="Redis") + + # Assert + assert connect.await_count == 1 + mock_sleep.assert_not_awaited() + + +@pytest.mark.asyncio +async def test__retry_async_connection__connect_always_returns_false__raises_connection_error() -> None: + # Arrange + connect = AsyncMock(return_value=False) + + # Act & Assert + mock_sleep = AsyncMock() + with ( + patch("redis_client_kit.providers.utils.asyncio.sleep", mock_sleep), + pytest.raises(ConnectionError, match="Redis did not report a healthy connection"), + ): + await retry_async_connection(connect_func=connect, service_name="Redis", max_attempts=3) + + assert connect.await_count == 3 + assert [call.args[0] for call in mock_sleep.await_args_list] == [1.0, 2.0] + + +@pytest.mark.asyncio +async def test__retry_async_connection__connect_recovers__returns_after_backoff() -> None: + # Arrange + connect = AsyncMock(side_effect=[False, True]) + + # Act + with patch("redis_client_kit.providers.utils.asyncio.sleep", new_callable=AsyncMock) as mock_sleep: + await retry_async_connection(connect_func=connect, service_name="Redis", backoff_base=0.5) + + # Assert + assert connect.await_count == 2 + mock_sleep.assert_awaited_once_with(0.5) + + +@pytest.mark.asyncio +async def test__retry_async_connection__connect_always_raises__reraises_original_error() -> None: + # Arrange + connect = AsyncMock(side_effect=OSError("no route to host")) + + # Act & Assert + with ( + patch("redis_client_kit.providers.utils.asyncio.sleep", new_callable=AsyncMock), + pytest.raises(OSError, match="no route to host"), + ): + await retry_async_connection(connect_func=connect, service_name="Redis", max_attempts=2) + + assert connect.await_count == 2 + + +@pytest.mark.asyncio +async def test__safe_async_cleanup__cleanup_raises__swallows_error() -> None: + # Arrange + cleanup = AsyncMock(side_effect=RuntimeError("close failed")) + + # Act + await safe_async_cleanup(cleanup_func=cleanup, service_name="Redis client", exception_type=OSError) + + # Assert + cleanup.assert_awaited_once() diff --git a/tests/unit/providers/test_providers_init.py b/tests/unit/providers/test_providers_init.py index 2efeb91..25e074d 100644 --- a/tests/unit/providers/test_providers_init.py +++ b/tests/unit/providers/test_providers_init.py @@ -6,17 +6,24 @@ import pytest +def _unload_providers(monkeypatch: pytest.MonkeyPatch) -> None: + """Drop the providers package from sys.modules for the duration of one test. + + monkeypatch restores every entry afterwards, so the modules the rest of the + suite already imported keep pointing at the real ones. + """ + for module in [key for key in sys.modules if key.startswith("redis_client_kit.providers")]: + monkeypatch.delitem(sys.modules, module) + + def test__providers_init__dishka_not_installed__raises_import_error(monkeypatch: pytest.MonkeyPatch) -> None: # Arrange - # Remove redis_client_kit.providers from modules if already imported - modules_to_remove = [key for key in sys.modules if key.startswith("redis_client_kit.providers")] - for module in modules_to_remove: - del sys.modules[module] + _unload_providers(monkeypatch) # Mock _deps to simulate dishka not installed mock_deps = MagicMock() mock_deps.HAS_DISHKA = False - sys.modules["redis_client_kit.providers._deps"] = mock_deps + monkeypatch.setitem(sys.modules, "redis_client_kit.providers._deps", mock_deps) # Act & Assert with pytest.raises(ImportError, match="dishka not installed"): @@ -25,18 +32,15 @@ def test__providers_init__dishka_not_installed__raises_import_error(monkeypatch: def test__providers_init__dishka_installed__imports_successfully(monkeypatch: pytest.MonkeyPatch) -> None: # Arrange - # Remove redis_client_kit.providers from modules if already imported - modules_to_remove = [key for key in sys.modules if key.startswith("redis_client_kit.providers")] - for module in modules_to_remove: - del sys.modules[module] + _unload_providers(monkeypatch) # Mock _deps to simulate dishka installed mock_deps = MagicMock() mock_deps.HAS_DISHKA = True mock_deps.AsyncRedisProvider = MagicMock - sys.modules["redis_client_kit.providers._deps"] = mock_deps - sys.modules["redis_client_kit.providers.redis"] = mock_deps - sys.modules["redis_client_kit.providers.utils"] = mock_deps + monkeypatch.setitem(sys.modules, "redis_client_kit.providers._deps", mock_deps) + monkeypatch.setitem(sys.modules, "redis_client_kit.providers.redis", mock_deps) + monkeypatch.setitem(sys.modules, "redis_client_kit.providers.utils", mock_deps) # Act import redis_client_kit.providers # noqa: F401, PLC0415 diff --git a/tests/unit/providers/test_redis_provider.py b/tests/unit/providers/test_redis_provider.py new file mode 100644 index 0000000..6774d00 --- /dev/null +++ b/tests/unit/providers/test_redis_provider.py @@ -0,0 +1,115 @@ +"""Tests for AsyncRedisProvider registration and startup behaviour.""" + +from unittest.mock import AsyncMock, MagicMock, patch + +import pytest +from dishka import Provider, Scope, make_async_container, provide + +from redis_client_kit import AsyncRedisClient +from redis_client_kit.aio import InstrumentedRedis +from redis_client_kit.config import RedisSettingsProtocol +from redis_client_kit.protocols import RedisMetricsProtocol +from redis_client_kit.providers import AsyncRedisProvider + + +def _settings_provider(settings: MagicMock) -> Provider: + provider = Provider(scope=Scope.APP) + provider.provide(lambda: settings, provides=RedisSettingsProtocol) + return provider + + +class _MetricsProvider(Provider): + scope = Scope.APP + + @provide + def metrics(self) -> RedisMetricsProtocol | None: + return MagicMock(spec=RedisMetricsProtocol) + + +@pytest.mark.asyncio +async def test__provider__redis_unreachable__fails_container_startup(mock_redis_settings: MagicMock) -> None: + # Arrange + container = make_async_container(AsyncRedisProvider(), _settings_provider(mock_redis_settings)) + + # Act & Assert + with ( + patch("redis_client_kit.providers.redis.check_async_redis_health", new_callable=AsyncMock) as mock_health, + patch("redis_client_kit.providers.utils.asyncio.sleep", new_callable=AsyncMock), + ): + mock_health.return_value = False + + with pytest.raises(ConnectionError, match="Redis did not report a healthy connection"): + await container.get(AsyncRedisClient) + + assert mock_health.await_count == 3 + await container.close() + + +@pytest.mark.asyncio +async def test__provider__redis_reachable__yields_client(mock_redis_settings: MagicMock) -> None: + # Arrange + container = make_async_container(AsyncRedisProvider(), _settings_provider(mock_redis_settings)) + + # Act + with patch("redis_client_kit.providers.redis.check_async_redis_health", new_callable=AsyncMock) as mock_health: + mock_health.return_value = True + client = await container.get(AsyncRedisClient) + + # Assert + assert client is not None + mock_health.assert_awaited_once() + await container.close() + + +@pytest.mark.asyncio +async def test__provider__health_check_disabled__skips_the_check(mock_redis_settings: MagicMock) -> None: + # Arrange + container = make_async_container( + AsyncRedisProvider(check_health_on_startup=False), + _settings_provider(mock_redis_settings), + ) + + # Act + with patch("redis_client_kit.providers.redis.check_async_redis_health", new_callable=AsyncMock) as mock_health: + client = await container.get(AsyncRedisClient) + + # Assert + assert client is not None + mock_health.assert_not_awaited() + await container.close() + + +@pytest.mark.asyncio +async def test__provider__default_metrics_disabled__keeps_metrics_provider_registered_first( + mock_redis_settings: MagicMock, +) -> None: + # Arrange + container = make_async_container( + _MetricsProvider(), + AsyncRedisProvider(check_health_on_startup=False, provide_default_metrics=False), + _settings_provider(mock_redis_settings), + ) + + # Act + client = await container.get(AsyncRedisClient) + + # Assert + assert isinstance(client, InstrumentedRedis) + await container.close() + + +@pytest.mark.asyncio +async def test__provider__registered_first__keeps_metrics_provider(mock_redis_settings: MagicMock) -> None: + # Arrange + container = make_async_container( + AsyncRedisProvider(check_health_on_startup=False), + _MetricsProvider(), + _settings_provider(mock_redis_settings), + ) + + # Act + client = await container.get(AsyncRedisClient) + + # Assert + assert isinstance(client, InstrumentedRedis) + await container.close() diff --git a/tests/unit/sync/test_sync_metrics.py b/tests/unit/sync/test_sync_metrics.py index 1f3acee..61d2bcb 100644 --- a/tests/unit/sync/test_sync_metrics.py +++ b/tests/unit/sync/test_sync_metrics.py @@ -214,6 +214,58 @@ def test__instrumented_redis_cluster__runtime_error_handler_closed__converts_to_ client.execute_command("SET", "key", "val") +def test__instrumented_redis__runtime_error_handler_closed__records_error_metrics( + mock_metrics: MagicMock, +) -> None: + # Arrange + client = InstrumentedRedis(metrics=mock_metrics) + + # Act + with patch("redis.Redis.execute_command") as mock_execute: + mock_execute.side_effect = RuntimeError("the handler is closed") + with pytest.raises(RedisConnectionError): + client.execute_command("SET", "key", "val") + + # Assert + mock_metrics.record_error.assert_called_once_with(error_type="ConnectionError") + assert mock_metrics.record_command.call_args.kwargs["status"] == "error" + + +def test__instrumented_redis_cluster__runtime_error_handler_closed__records_error_metrics( + mock_metrics: MagicMock, +) -> None: + # Arrange + with patch("redis.cluster.RedisCluster.__init__", return_value=None): + client = InstrumentedRedisCluster(metrics=mock_metrics, host="localhost", port=6379) + + # Act + with patch("redis.cluster.RedisCluster.execute_command") as mock_execute: + mock_execute.side_effect = RuntimeError("the handler is closed") + with pytest.raises(RedisConnectionError): + client.execute_command("SET", "key", "val") + + # Assert + mock_metrics.record_error.assert_called_once_with(error_type="ConnectionError") + assert mock_metrics.record_command.call_args.kwargs["status"] == "error" + + +def test__instrumented_redis__other_runtime_error__records_error_metrics( + mock_metrics: MagicMock, +) -> None: + # Arrange + client = InstrumentedRedis(metrics=mock_metrics) + + # Act + with patch("redis.Redis.execute_command") as mock_execute: + mock_execute.side_effect = RuntimeError("some other error") + with pytest.raises(RuntimeError): + client.execute_command("SET", "key", "val") + + # Assert + mock_metrics.record_error.assert_called_once_with(error_type="RuntimeError") + assert mock_metrics.record_command.call_args.kwargs["status"] == "error" + + def test__instrumented_redis_cluster__record_error_exception__logs_and_reraises_original( mock_metrics: MagicMock, ) -> None: diff --git a/zensical.toml b/zensical.toml index a00b86e..1249d78 100644 --- a/zensical.toml +++ b/zensical.toml @@ -1,6 +1,6 @@ [project] site_name = "redis-client-kit" -site_description = "Redis client with optional Pydantic and OpenTelemetry support" +site_description = "Redis client with optional Pydantic, Prometheus, and Dishka support" site_author = "Alex Shalaev" repo_name = "bedrock-python/redis-client-kit" repo_url = "https://github.com/bedrock-python/redis-client-kit"