rivet

Rivet is a task queue for distributed systems, written in Rust. Rivet gives effectively-once execution when a worker fails. A chaos harness tests this guarantee.

What it guarantees

Exactly-once delivery is not possible. Rivet does not make this claim. Rivet gives you two guarantees that work together:

• Delivery is at-least-once. A worker can receive one task many times.
• The side effect is exactly-once. An idempotency key protects the side effect. Delivery 2 to delivery N is a no-op. Each no-op returns the result of delivery 1.

Rivet uses two mechanisms. Each mechanism does a different job. Neither mechanism is sufficient alone.

A fencing token protects the queue state. The technical name for the fencing token is lease_epoch. The fencing token stops a stalled worker from acking a task that now belongs to a different worker.

The idempotency key protects the side effect. A stalled worker can have a request in flight. You cannot recall this request. The idempotency key stops the request from landing twice.

Quickstart

Do these steps to start rivet:

1. Start the database.

   docker compose up -d rivet-db

2. Run the tests.

   cargo test

The committed .env file supplies DATABASE_URL. You do not need to set it yourself. Expect 67 checks: 18 against the in-memory store, 21 against the Postgres store, 8 multi-worker, 7 idempotency, 10 in the chaos harness, and 3 for the retry backoff.

If you do not have Docker, use the in-memory backend. The in-memory backend needs no database.

cargo test --test memory_store

Run a worker

Add a task, then start a worker for it. Use two terminals, because the worker runs until you stop it.

cargo run -- enqueue --queue demo --kind noop --idem-key job-1
cargo run -- worker  --queue demo --lease-ttl-secs 5 --concurrency 4
cargo run -- stats   --queue demo

Each flag also reads an environment variable. The names are RIVET_QUEUE, RIVET_LEASE_TTL_SECS, RIVET_CONCURRENCY, RIVET_RETRY_BACKOFF_SECS, RIVET_RETRY_BACKOFF_CAP_SECS, RIVET_POOL_SIZE, and RIVET_OWNER.

A task that fails waits before the next attempt. The wait is a random value between zero and min(cap, backoff * 2^attempts). The random part keeps the tasks that failed together from coming back together.

The binary knows three demo kinds. noop always succeeds. fail always fails, which shows you the retry path and the dead-letter path. effect calls the effect service, and it is the kind with a side effect to protect.

An enqueue with a key that you used before prints duplicate and does not add a second task. A producer that retries is normal, and it is not an error. The same key with different work is an error, because that is a bug in the producer.

See the guarantee

This sequence kills a worker between the side effect and the ack, which is the window that makes duplicates. Do these steps:

1. Start the two databases and the effect service.

   docker compose up -d rivet-db effect-db
   cargo run -p effect-svc

2. Add one task, in a second terminal.

   cargo run -- enqueue --queue demo --kind effect --idem-key pay-1 \
       --payload '{"amount":100}'

3. Run a worker that aborts after the effect, before the ack.

   RIVET_FAILPOINT=after_effect_before_ack:1.0 \
       cargo run -- worker --queue demo --lease-ttl-secs 2

   The worker stops with SIGABRT. The effect is done and the queue does not know, so rivet stats --queue demo still shows the task as ready.

4. Run a normal worker. The task is delivered a second time and the handler runs again, in full.

   cargo run -- worker --queue demo --lease-ttl-secs 5
   curl 'http://127.0.0.1:8081/verify'

Two deliveries. One effect. The task reads done.

The dead-letter queue

A task goes to the dead-letter queue when its attempts are exhausted. A claim buries it, so you do not need a reaper. Use sweep only when no worker polls the queue.

cargo run -- dlq --queue demo list
cargo run -- dlq --queue demo show 42
cargo run -- dlq --queue demo requeue 42
cargo run -- dlq --queue demo sweep
cargo run -- dlq --queue demo purge

The last_error field says why the task died. It ends with max attempts exceeded (handler error) if the handler gave an error each time, or max attempts exceeded (no ack, worker lost) if no worker ever answered. Under chaos the second one is the usual reason, and it is not a defect.

A requeue keeps the idempotency key of the task. If the effect already happened, the new attempt gets a replay and the effect does not happen a second time. This is correct, and it surprises operators.

A purge deletes the rows, so it releases the idempotency keys. Purge only tasks that are older than the dedup retention time of the effect service. If you purge sooner, the same key can make a second effect.

The chaos run

This is the part that turns the guarantee into a measurement. The harness fills a queue, starts worker processes, and then spends a minute breaking them while they work. Afterwards it reads both databases and checks four invariants.

docker compose up --build --abort-on-container-exit --exit-code-from chaos

Exit 0 means every invariant held. --exit-code-from chaos is not optional: it is what makes the exit code the verifier's verdict rather than compose's own.

Workers are broken in three ways. A random worker is SIGKILLed and restarted. A worker aborts itself at a named point, which is how the window between the effect and the ack gets hit on purpose rather than by luck. And a worker is SIGSTOPped for longer than its lease and then continued, which is a worker that is absent but not dead, the case a lease alone cannot tell from a crash. One seed drives all of it, so a run that fails can be repeated.

The harness only ever signals processes it started itself, and only through a pidfd. A raw pid is not a stable handle: a worker can abort on its own and be reaped between the moment a target is chosen and the moment the signal is sent, and Linux reuses pids. A pidfd refers to one specific process, and once that process is gone a signal through it fails instead of reaching a stranger.

The report at the end:

tasks enqueued        5002
chaos events           227   (kill 71 / failpoint 101 / zombie 55)
effects written       5000
tasks done            5000
duplicate effects        0   <- S1
done without effect      0   <- S2
tasks still pending      0   <- L1
dead-lettered            2   (0 worker lost, 2 poison)
fenced acks              7   <- proof F2 fired and was handled

The last line is the one that matters most, and a zero there fails the run. Zero fenced acks means no worker ever tried to write to a task it had lost, so the fencing token was never actually tested and everything above it is a statement about a system that was not under the pressure it claims to survive.

To run the harness outside a container, against the two development databases:

cargo build --workspace     # the harness starts the `rivet` binary, so build it
cargo run -p effect-svc     # in another terminal
cargo run -p chaos -- --tasks 500 --workers 4 --duration-secs 15

Note the first line. cargo run -p chaos builds the harness and the rivet library but not the rivet binary, so without it the workers can be from an earlier build. The harness warns when it sees that. Also note that the run starts by emptying both databases, since a count of effects from an earlier run says nothing about this one. Pass --keep to leave them alone.

The benchmark

The harness also measures. Build with --release first, or the number you get is the speed of a debug build.

cargo build --release --workspace
./target/release/chaos bench                    # 1, 4 and 16 workers
docker compose run chaos bench --workers 16

Each run writes bench/results.md and prints the same text. Every latency comes from the task rows and not from anything a worker recorded. End to end is updated_at - created_at. Service time is updated_at - visible_at + lease_ttl, because a claim sets visible_at = now() + lease_ttl and the ack leaves it alone. Both ends are written by the database server, so the clock of the machine that runs the harness cannot reach the result.

The default run inserts the whole workload before the workers start. A full backlog is what measures throughput. It is not what measures latency: under a backlog, end to end is mostly the time a task waited its turn. Use --rate to offer load below capacity, and read the end-to-end columns then.

./target/release/chaos bench --workers 4 --tasks 3000 --rate 300

On a 4-thread laptop that also runs Postgres: 315 task/s at one worker, 654 at four, and 4.3 ms p50 service time. Offered at 300 task/s to four workers, end to end is 9.3 ms p50 and 76.2 ms p99. The 16-worker row moved a long way between runs on this machine, because 16 workers there compete with the database they are measuring. bench/results.md carries the machine, the Postgres version, the payload size, the concurrency and the pool size, since a throughput number without those is not a number anybody can repeat.

Status

Sections 0 to 7 are complete. These sections give you the core state machine, an in-memory store, a Postgres store, one conformance suite that both stores must pass, the multi-worker runtime, the rivet binary, the effect service, the retry policy, the dead-letter queue, the chaos harness with its verifier, and the benchmark.

Design

rivet::conformance is a public module. Every rule for the storage layer is in this module, as a generic check. A new Store implementation inherits the full suite. You do not need to test a new store in isolation.

The state machine has three states: pending, done, and dead. There is no claimed state. A claimed task is a pending task with a visible_at value in the future.

This design combines two conditions into one indexed check: a task that is runnable now, and a task with an expired lease. Crash recovery uses the normal claim path for this reason. Rivet needs no reaper daemon and no leader election.

In the worker, the batch size and the concurrency are one value. A worker that claimed 10 tasks and ran them one after the other would let the lease of task 10 run out while tasks 1 to 9 were busy. A second worker would then claim task 10. The worker would make the duplicate delivery that this design prevents.

The effect service and the queue use two different databases. A crash between the effect and the ack cannot be removed, so one store must be the answer to "did the effect happen?". It is the store that holds the effect. The effect row and its dedup row are written in one transaction. Two writes would move the duplicate one layer down and not remove it.

A repeated key gets 200 and the answer the first delivery got, with the header idempotent-replay: true. It does not get 409. A replay is the system working as designed, and a client that reads it as an error fails a task that is in fact complete.

A guard runs beside each handler. The guard stops the handler for one of two reasons: a heartbeat came back fenced, or a local deadline passed. The local deadline is a monotonic timer, and it is the only clock a worker reads. It stops work early, and it never gives a time value to the database. Every decision timestamp comes from the database server.

Licence

Apache-2.0.

See LICENSE.txt