solid_queue benchmark

Benchmarks solid_queue under realistic load, comparing the official gem against a local fork. It writes traceable JSON results and markdown/HTML comparison reports.

Prerequisites

• mise for the pinned Ruby version
• Docker for MySQL 8.0 and Postgres 16
• libpq dev headers on the host, to compile the pg gem's native extension (libpq-dev on Debian/Ubuntu, postgresql via Homebrew on macOS)

Workers run on the host. Only the database (MySQL or Postgres) is containerized.

Setup

mise install
mise run setup

If mise reports that mise.toml is not trusted, run mise trust once in the checkout, then retry.

setup bundles the default upstream source and pulls both the MySQL and Postgres images.

Run a Benchmark

# latest official gem, against MySQL (default)
bin/bench run baseline --source upstream

# same run against Postgres, for a like-for-like comparison
bin/bench run baseline --source upstream --database postgres

# pinned RubyGems release
bin/bench run baseline --source upstream@1.2.4

# local fork; must be clean unless --allow-dirty is passed
bin/bench run sprawl --source path:~/Projects/solid_queue

Each run starts a fresh database volume (MySQL or Postgres, per --database; default mysql), loads schema from the selected gem source, starts Solid Queue workers, runs the scenario, waits for drain, and writes results/<run-id>/result.json plus logs. Results include the resolved gem version, the database engine used, the process launcher, and, for path: sources, the git SHA.

Scenarios

Scenario	Params (--set key=val)	Measures
baseline	jobs=20000 rate=500 work_ms=50 duration=60	Steady-state throughput and latency. jobs=0 measures idle polling for duration seconds.
baseline_limited	jobs=1000 rate=500 work_ms=50	Same job work as baseline, with a class-level limits_concurrency limit of 1 for blocked-execution and throttling cost.
sprawl	seeds=100 fanout=50 depth=2 work_ms=10	Fan-out burst where each job enqueues fanout children down to depth. Defaults total 255,100 jobs; reduce params for quick runs.

Examples:

bin/bench run baseline --source upstream --profile smoke --set jobs=100 --set rate=100 --set work_ms=0 --timeout 180
bin/bench run baseline --source upstream --set jobs=1000 --set rate=500 --set work_ms=50
bin/bench run baseline_limited --source upstream --set jobs=1000 --set rate=500 --set work_ms=50
bin/bench run sprawl --source path:~/Projects/solid_queue --set seeds=5 --set fanout=10 --set depth=1

Profiles and Topology

Profiles live in profiles/*.yml and bundle database resource limits with worker topology so comparisons stay comparable, independent of which engine a run targets. The default profile pins the database container to 1 CPU / 1 GB, with 10 worker processes x 2 threads and 1 dispatcher.

bin/bench run baseline --source upstream --profile default
bin/bench run baseline --source upstream --profile smoke
bin/bench run baseline --source upstream --workers 50 --db-cpus 2 --db-memory 2g
bin/bench run baseline --source upstream --database postgres --profile default

bin/bench compare refuses profile mismatches unless --force is passed. Comparing a MySQL run against a Postgres run under the same profile is not treated as a mismatch — the report simply shows which engine each side used.

By default, the harness uses Solid Queue's fork supervisor. On macOS with Postgres, the harness uses direct launching instead: each worker and dispatcher is started as a fresh Ruby process to avoid the pg/libpq crashes seen when forked children connect after Rails has booted. The chosen launcher is recorded in result.json under profile.process_launcher, so mixed launcher comparisons are guarded like other topology changes. Override with BENCH_SOLID_QUEUE_LAUNCHER=supervisor or BENCH_SOLID_QUEUE_LAUNCHER=direct when needed.

Compare Runs

bin/bench list
bin/bench compare results/<A>/result.json results/<B>/result.json

Reports are written to reports/<A>__vs__<B>/report.md and report.html. They include which database engine each run used, metric deltas, DB CPU and ready-depth overlay charts, and top SQL statements by total database time.

Use --force only when you intentionally want to compare different profiles:

bin/bench compare results/<A>/result.json results/<B>/result.json --force

Repeat Runs

bin/bench run baseline --source upstream --repeat 3

Repeats keep every raw result and print median throughput across completed runs.

Metrics

• Throughput, including a per-second series
• Latency percentiles for enqueue-to-start and enqueue-to-finish
• Database container CPU samples from docker stats
• Queue depth samples for ready, scheduled, claimed, blocked, failed, and completed jobs
• Top statement digests from MySQL performance_schema or Postgres pg_stat_statements

Adding a Scenario

1. Add defaults, validation, and expected-total logic in lib/bench/scenarios.rb.
2. Add the enqueue branch in harness/script/drive.rb.
3. Add any job class needed under harness/app/jobs/.

Scenario params become CLI-settable through repeated --set key=val flags.

Development

mise run test
mise run smoke

mise run test runs orchestrator unit tests. mise run smoke runs a 100-job upstream benchmark against the smoke profile.