Skip to content

The health check is a PING, and PING says healthy for a full noeviction server and a read-only replica #31

Description

@AlexeyShalaev

Measured on 0.1.4 against three redis:7-alpine containers: a healthy primary, a primary started with --maxmemory 1mb --maxmemory-policy noeviction and then filled, and a replica of the healthy one. Each was asked check_async_redis_health and then a write-then-read of one namespaced key with a TTL:

primary                          PING health=True   write+read probe: ok (2.2 ms)
maxmemory 1mb, noeviction, full  PING health=True   write+read probe: OutOfMemoryError: command not allowed when used memory > 'maxmemory'.
replica of the primary           PING health=True   write+read probe: ReadOnlyError: You can't write against a read only replica.
paused primary                   PING health=False  decided in 0.50 s

The check is a PING, and PING answers PONG from a server that cannot take a write: one that is out of memory under noeviction (every cache and idempotency SET fails from now on), and a replica that a failover or a DNS mistake pointed the client at (every write fails, every read is stale). Both are the states in which a service that uses Redis for anything but reads is down, and both report healthy, so readiness stays green and the pods keep taking traffic they cannot serve. Rule 10 says the check "never says maybe"; here it says yes when the answer is no.

What I think it needs: an opt-in deeper probe on both health functions, something like check_async_redis_health(client, *, probe="ping" | "write") with "ping" staying the default so nothing changes for existing callers. The write probe does SET <key_prefix>:health:<uuid> 1 EX <n> followed by GET and DEL (or just the SET with EX and let it expire), and returns False on ReadOnlyError, OutOfMemoryError and any other ResponseError the way it already returns False on connection trouble, still never raising, still inside the socket timeout. The key needs the settings' key_prefix, which the function does not receive today, so either the prefix becomes an argument or the probe key is passed in. On a cluster the write lands on one slot, which is a partial answer; the docs should say so and leave cluster slot coverage (CLUSTER INFO cluster_state:ok) as a separate follow-up rather than fold it in here. Rule 10, the health rows in the API table and the health section of the guide follow.

Lab: health_lab.py in https://github.com/bedrock-python/bedrock-python.github.io/tree/docs/production-python-series/docs/blog/lab/2026-09-07-redis-health-checks.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions