Enable PR-level performance quality gates

[igoragoli]

What does this PR do?

Adds a PR-level performance quality gate for microbenchmarks: microbenchmarks-check-big-regressions.

The job runs after microbenchmarks complete and fails if any benchmark regresses by more than 20%, using bp-runner bp-runner.fail-on-regression.yml --debug.

Motivation:

APMSP-2545 Setup pre-release and PR level quality gates for Ruby

Change log entry

None.

Additional Notes:

Can we bypass this?
Yes. I added a comment with directions on how to do it:

dd-trace-rb/.gitlab/benchmarks.yml

Lines 315 to 317 in 940bd7e

	# Verify that the microbenchmarks-check-big-regressions CI job has passed. If any regression happened, merging this PR will be blocked.
	# If bypassing is necessary, see https://datadoghq.atlassian.net/wiki/x/8YFzMwE for more details.
	microbenchmarks-check-big-regressions:

What does this command even mean?

• bp-runner is the tool we use to orchestrate benchmarks and provide common actions across repos (such as this one of comparing candidates against baselines against a regression threshold of 20%).
• bp-runner.fail-on-regression.yml is the config file which tells bp-runner what to do. It's on https://github.com/DataDog/benchmarking-platform/blob/dd-trace-rb/bp-runner.fail-on-regression.yml, but we could alternatively move it to this repo :)

Why 20%?

• This is a default limit for regressions on a single PR.
• We could have fixed limits via SLOs, like dd-trace-py does.
• We could also shrink this regression threshold and make it more ambitious, but I prefer to make sure benchmarking jobs are up and running correctly and fast first and make this threshold more agressive later.

How to test the change?

microbenchmarks-check-big-regressions running after microbenchmarks in CI: https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-rb/-/jobs/1592412230

[github-actions]

Thank you for updating Change log entry section 👏

^{Visited at: 2026-04-09 14:50:40 UTC}

[igoragoli]

• Enable PR-level performance quality gates #5571 👈 (View in Graphite)
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[pr-commenter]

Benchmarks

Benchmark execution time: 2026-04-14 12:56:25

Comparing candidate commit 515d6bd in PR branch augusto/enable-perf-quality-gates with baseline commit 3260714 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 45 metrics, 1 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

• 🟩 = significantly better candidate vs. baseline
• 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

[datadog-official]

✅ Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 95.35% (-0.02%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 515d6bd | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!}