Skip to content
BranchesTags

Repository files navigation

Deadman

A distributed job scheduler in C++20. Submit a job over HTTP and it runs — even if the worker executing it is killed mid-run.

A deadman's switch fires when the operator stops signalling. That is what a lease heartbeat is: a worker holds a job only while it keeps saying it is alive, and the moment it goes quiet the job is reclaimed by someone else. The name is the architecture.

Verified at 100,000 jobs with 70 worker processes hard-killed mid-run: every job completed, none lost, 98 jobs executed twice and absorbed, zero duplicate side effects. Numbers and method in BENCHMARKS.md.

Contents

What it does

  • HTTP API — submit, inspect, list, cancel and retry jobs, with an Idempotency-Key header so a client that times out and retries does not create a second job.
  • Postgres as the source of truth. Claims are atomic via SELECT ... FOR UPDATE SKIP LOCKED, so N workers behave like N workers rather than queueing behind each other on one row.
  • Leases with heartbeats. A worker holds a time-boxed claim, renewed while its handler runs. Kill the worker and the lease lapses; a reaper returns the job to the ready stream and someone else finishes it.
  • Retries with fixed or exponential backoff and full jitter, bounded by the job's own max_attempts, then a dead-letter entry that keeps the reason.
  • Delayed jobs, held in a Redis sorted set and promoted when due.
  • Kafka transport — workers join a consumer group; offsets commit after execution, never on receipt.
  • A chaos harness that kills worker processes with taskkill /F while load runs, then measures whether the guarantee held.

The guarantee, stated precisely

At-least-once delivery plus idempotent execution, producing effectively-once behaviour. Not exactly-once. That distinction is the point of the project, not a hedge:

  • Offsets commit after execution, so a worker killed mid-job has not told Kafka it finished and the work comes back. The cost is duplicates.
  • A reaped lease does not increment the attempt count, because nobody knows whether the handler ran before the worker died. So a job genuinely can execute twice — 98 of them did in the 100k run.
  • Absorbing that is the handler's job. The demo handler writes its side effect under a WHERE NOT EXISTS guard in the same transaction as the effect itself, which is the only arrangement that actually holds. See decision 012, written after a chaos run lost a side effect to a Redis-based version and showed exactly why.

The short version: a side effect can be made exactly-once only against a store you can transact with. Against one you cannot — an email gateway, a payment API — at-least-once plus a provider-side idempotency key is the best available.

Architecture

HTTP API ──▶ Postgres            (state, source of truth)
   │
   └──────▶ Kafka jobs.ready ──▶ worker fleet ──▶ handlers
                                      │  │
                    Redis ◀───────────┘  └──────▶ Kafka jobs.dead
              (leases, dedup,
               delayed ZSET)
                    │
                 reaper ──▶ requeues expired leases and due jobs

Postgres holds state. Kafka is only transport; Redis is a cache plus a lease register. Neither is consulted to decide whether a job ran — which is what makes it safe to lose either of them.

Prerequisites

Requirement Notes
Windows 10/11 x64 The chaos harness is Windows-native (taskkill, PowerShell). The engine itself is portable C++20.
Visual Studio 2022 Community edition is fine. Install the Desktop development with C++ workload.
CMake 3.21+ Ships with VS 2022, or install standalone and put it on PATH.
Docker Desktop Supplies Postgres, Redis and Kafka. Must be running before you build or test.
vcpkg Dependency manager. Setup below.
Git For cloning.

Check what you have:

cmake --version
docker version
git --version

Setup

1. Clone

git clone https://github.com/codecommander03/Deadman.git
cd Deadman

2. Install vcpkg

Skip if you already have it and VCPKG_ROOT is set.

git clone https://github.com/microsoft/vcpkg.git C:\vcpkg
C:\vcpkg\bootstrap-vcpkg.bat

# Persist for future shells
[Environment]::SetEnvironmentVariable("VCPKG_ROOT", "C:\vcpkg", "User")
$env:VCPKG_ROOT = "C:\vcpkg"

3. Install the static libpqxx

This step is required and easy to miss. The project pulls most dependencies automatically from vcpkg.json, but libpqxx must come from a different triplet:

cd C:\vcpkg
.\vcpkg.exe install libpqxx:x64-windows-static-md
cd <back to the repo>

Why: vcpkg builds the x64-windows (DLL) libpqxx with CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS, so its import library exports the MSVC STL inline functions it instantiated — including std::basic_string_view<char>'s members. Any test executable that links both that and GoogleTest fails with a wall of LNK2005: ... already defined. The static build has no import library and exports nothing, so the duplicate COMDATs merge normally. Full reasoning in decision 013.

If your vcpkg lives somewhere other than C:\vcpkg, pass the prefix at configure time:

-DDEADMAN_PQXX_ROOT="D:/vcpkg/installed/x64-windows-static-md"

4. Start the infrastructure

docker compose -f deploy/docker-compose.yml up -d

This brings up Postgres (host port 55432), Redis (56379) and Kafka (59092), applies deploy/schema.sql, and creates the jobs.ready and jobs.dead topics with explicit partition counts. Non-default ports, so it cannot collide with anything already installed on the host.

Wait for all three to report healthy:

docker ps --format "{{.Names}}`t{{.Status}}"

Build

cmake -S . -B build `
  -DCMAKE_TOOLCHAIN_FILE="C:/vcpkg/scripts/buildsystems/vcpkg.cmake" `
  -DVCPKG_TARGET_TRIPLET=x64-windows

cmake --build build --config Debug

The first configure takes a while — vcpkg builds Drogon, librdkafka, redis-plus-plus and their dependencies (OpenSSL among them). Subsequent builds are fast.

For benchmarks and the chaos harness, build optimised:

cmake --build build --config RelWithDebInfo

Binaries land in build\Debug\ and build\RelWithDebInfo\:

Binary Purpose
deadmand.exe HTTP API + an embedded worker pool
deadman_worker.exe execution-only worker process
deadman_load.exe bulk load generation and the Stage 5 verification queries
deadman_bench.exe Google Benchmark suites

Run

Start the server:

.\build\RelWithDebInfo\deadmand.exe
deadmand starting
  postgres : postgresql://deadman:deadman@localhost:55432/deadman
  redis    : tcp://127.0.0.1:56379
  kafka    : localhost:59092 (6 partitions)
  handlers : 5 registered
  workers  : 4 threads
listening on http://localhost:8080

Submit a job from another shell:

curl.exe -s -X POST localhost:8080/jobs -H "Content-Type: application/json" -d '{\"type\":\"noop\"}'
curl.exe -s localhost:8080/stats

Add execution capacity with separate worker processes:

.\build\RelWithDebInfo\deadman_worker.exe

Stop with Ctrl+C. Both binaries shut down gracefully — in-flight jobs finish rather than being abandoned.

Built-in handlers

Type Behaviour
noop does nothing; the scheduler-only baseline
sleep sleeps for payload milliseconds
flaky throws with payload% probability
always-fails always throws; drives the dead-letter path
side-effect writes one guarded row; the handler the chaos run verifies

HTTP API

Method Path Notes
POST /jobs {type, payload?, max_attempts?, run_at_ms?}. Honours Idempotency-Key. 201 on create, 200 on a recognised repeat
GET /jobs/{id} one job
GET /jobs?state=&type=&limit=&offset= filtered, paged list
DELETE /jobs/{id} cancel; moves to dead, does not delete the row
POST /jobs/{id}/retry re-books a pending or failed job. A dead job returns 409 — submit a new one
GET /stats engine counters and job counts
GET /health liveness

Timestamps are integer milliseconds since the Unix epoch, not ISO-8601, so there is no timezone or precision ambiguity for a client to get wrong.

# submit, idempotently
curl -s -X POST localhost:8080/jobs \
  -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: order-42' \
  -d '{"type":"sleep","payload":"250","max_attempts":3}'

# submit for 30 seconds from now
curl -s -X POST localhost:8080/jobs \
  -H 'Content-Type: application/json' \
  -d "{\"type\":\"noop\",\"run_at_ms\":$(( ($(date +%s) + 30) * 1000 ))}"

curl -s localhost:8080/jobs/<id>
curl -s 'localhost:8080/jobs?state=dead&limit=10'
curl -s -X DELETE localhost:8080/jobs/<id>

Testing

ctest --test-dir build -C Debug --output-on-failure

141 tests. Tests needing Postgres, Redis or Kafka skip rather than fail when it is absent, so the suite still runs with nothing started — but start the compose stack to actually exercise them.

Suite Covers
core_test state machine (every illegal edge too), retry policies, engine lifecycle
job_store_contract_test one contract suite run against both InMemoryJobStore and PostgresJobStore
redis_test leases, compare-and-set renewal, delayed-job set, dedup keys
kafka_test consumer group, at-least-once redelivery, partition assignment, DLQ

The contract suite is the load-bearing one: it is what stops the in-memory store from drifting into something with more convenient semantics than the real one. It has already caught two divergences.

Full HTTP lifecycle check against a running server (30 assertions):

.\scripts\smoke_api.ps1

Note: stop deadmand before running ctest. A running server executes jobs out of the same database the contract tests use and will make them fail.

Chaos harness

cmake --build build --config RelWithDebInfo
.\scripts\chaos.ps1 -Jobs 100000 -Workers 4 -KillRate 0.30

Submits the jobs, starts worker processes, and kills each one independently with probability KillRate every few seconds — taskkill /F /T, so no unwinding and no chance to release a lease. Killed workers are replaced to keep capacity roughly constant. Then it stops killing, lets the survivors drain, and verifies:

  • no job left outstanding
  • every job completed
  • every job's side effect fired
  • no side effect fired twice

It also reports how many jobs were delivered more than once. That number is what makes the run meaningful: without duplicate deliveries actually occurring, "no duplicate side effects" would only mean the case was never exercised.

Smaller, faster run:

.\scripts\chaos.ps1 -Jobs 2000 -Workers 3 -KillRate 0.4 -KillIntervalSec 4

Parameters: -Jobs -Workers -KillRate -KillIntervalSec -TimeoutSec -LeaseMs -ThreadsPerWorker -BuildDir.

Benchmarks

.\build\RelWithDebInfo\deadman_bench.exe --benchmark_min_time=0.35s

Results and the machine they came from are in BENCHMARKS.md. Do not compare them across machines.

Configuration

All optional; every one has a working default.

Variable Default Used by
DEADMAN_DATABASE_URL postgresql://deadman:deadman@localhost:55432/deadman all
DEADMAN_REDIS_URL tcp://127.0.0.1:56379 all
DEADMAN_KAFKA_BROKERS localhost:59092 all
DEADMAN_PORT 8080 deadmand
DEADMAN_WORKER_THREADS 4 deadmand, deadman_worker
DEADMAN_WORKER_NAME random deadman_worker
DEADMAN_LEASE_MS 5000 deadman_worker
DEADMAN_STORE_POOL 4 deadman_worker
DEADMAN_HANDLER_POOL 3 deadman_worker
DEADMAN_TEST_DATABASE_URL same as above contract tests

Pool sizes matter: pool size × worker processes must fit inside Postgres's max_connections, which the compose stack raises to 300 for exactly this reason.

Troubleshooting

LNK2005: "std::basic_string_view<char>::..." already defined The static libpqxx is missing. Run step 3 of Setup, then delete build/ and reconfigure — libpqxx_DIR is cached and will otherwise keep pointing at the DLL build.

error during connect: ... docker_engine Docker Desktop is not running. Start it and wait for the whale icon to settle.

FATAL: sorry, too many clients already More worker processes than max_connections allows. Lower DEADMAN_STORE_POOL and DEADMAN_HANDLER_POOL, or kill leftover workers: Get-Process deadman_worker | Stop-Process -Force.

Failed to resolve 'kafka:9092' The host port must map to Kafka's HOST listener (59092:19092), not the internal one. Recreate the container: docker compose -f deploy/docker-compose.yml up -d --force-recreate kafka.

Contract tests fail with wrong counts A deadmand or deadman_worker is running against the same database and eating the tests' jobs. Stop it before running ctest.

Kafka tests skip with "topics missing" docker compose -f deploy/docker-compose.yml up kafka-init

Reset everything

docker compose -f deploy/docker-compose.yml down -v
docker compose -f deploy/docker-compose.yml up -d

Repository layout

include/deadman/   public headers
src/               implementation, plus the four binaries
tests/             GoogleTest suites, including the JobStore contract suite
bench/             Google Benchmark suites
deploy/            docker-compose and schema
scripts/           API smoke test and chaos harness
docs/              PLAN.md, DECISIONS.md, FUTURE.md

Reading

  • docs/DECISIONS.md — every design decision with the alternatives and what each one costs. Decision 012 is the interesting one.
  • docs/PLAN.md — six stages, their acceptance gates, and the evidence each gate passed on.
  • BENCHMARKS.md — numbers, with the machine they came from and a plain statement of where the system stops scaling.
  • docs/FUTURE.md — deliberately out of scope, and why.

Licence

MIT. See LICENSE.

About

Distributed job scheduler in C++20. Kill a worker mid-job — the job still runs exactly once.

Resources

Readme
MIT license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages