chore(benchmarks): add the observability-overhead harness and its findings

[lesnik512]

Why

Issues #184, #185, #186 and #187 all rest on measurements. Without the harness in the repo, those numbers are a claim in an issue body that nobody can check, re-run on their own hardware, or update when sentry-sdk or the OTel instrumentation changes.

The findings themselves, in one line: on a do-nothing endpoint through uvicorn, the full stack costs 70% of throughput (7978 → 2389 RPS), roughly half of that is recoverable, and OpenTelemetry is twice Sentry — which is not the ordering I expected, and is the reason #184 and #185 exist.

Design

Two suites, three measurement modes, one process per scenario.

• sentry_scenarios.py — one sentry_sdk.init() knob at a time.
• stack_scenarios.py — the lite-bootstrap instruments, configured through FastAPIBootstrapper, so the numbers are what a real service pays rather than what a synthetic wiring pays.
• run.py drives the ASGI app in-process (isolates library cost, inflates the ratio); run_http.py drives uvicorn with ab -k (real ratio, noisier); micro.py / verify.py / profile_one.py attribute a per-request total to individual calls.

Three shapes are load-bearing and are not obvious from reading the files:

• One subprocess per scenario. sentry_sdk.init() monkeypatches Starlette.__call__, Middleware.__init__ and logging.Logger.callHandlers; set_tracer_provider is set-once; the Prometheus registry is global. None of it can be undone, so scenarios cannot share a process.
• Scenarios that simulate a fix that does not exist. errors_only_skip_txn, otel_sampler and otel_exclude_spans monkeypatch the library to measure what a proposed change would be worth before anyone writes it. That is where the numbers in OpenTelemetry sampler is not configurable #184, Pass exclude_spans to FastAPIInstrumentor so ASGI send/receive spans are opt-out #185 and ASGI integrations build and discard a Transaction per request when tracing is disabled (~34 µs) getsentry/sentry-python#7400 come from.
• verify.py exists because speed alone does not settle anything. It captures a real error event with an incoming sentry-trace header and prints what survived. That is how http_methods_to_capture=() was caught silently breaking distributed trace linkage, and how the upstream proposal was shown to lose nothing.

benchmarks/ has no __init__.py, so coverage does not walk into it and the 100% gate is untouched — the same mechanism that already excludes scripts/. The pyproject.toml change is three ruff per-file-ignores (INP001, T201, SLF001): standalone scripts that print tables and read sentry-sdk internals on purpose.

Non-goals

• No library change. Nothing under lite_bootstrap/ moves. The improvements stay as issues.
• No CI job. These are wall-clock benchmarks on shared runners; a gate built on them would flap. They are run by hand when someone is asking the question.
• Not a docs page. Document what the observability stack costs in RPS #187 is the user-facing writeup; benchmarks/README.md is the lab notebook that would source it.

Verification

Everything in the README was re-measured from this branch after the restructure. Sentry ablation reproduced within ~2% (+60.9 µs vs the +61.3 µs in the note), the stack breakdown within ~3%, and verify.py reproduced all four rows of the trade-off table exactly, including errors_only_no_txn losing the incoming trace and errors_only_skip_txn keeping it.

• just lint-ci — clean (ruff format, ruff check, ty, eof-fixer)
• just test — 242 passed, coverage still 100.00%
• Every entry point exercised: both suites through run.py, run_http.py against uvicorn, micro.py, verify.py, profile_one.py including --callers, and repro_sentry_txn.py in all three modes

One thing to push back on

benchmarks/README.md is prose, and the repo's rule is that a fact lives in lite_bootstrap/, in a named test, or in README.md/docs/. My argument for the fourth home: it documents libraries this repo depends on rather than mechanism this repo owns, none of it is derivable by reading lite_bootstrap/, and it sits next to the code that produced it. If that does not hold, the alternative is to strip it to a usage-only README and let #187 carry the findings into docs/.