Skip to content

CI: benchmark diff of pull requests by benchstat - #45

Merged
Insei merged 1 commit into
mainfrom
bench-ci
Oct 16, 2026
Merged

Insei merged 1 commit into
mainfrom
bench-ci

Conversation

@Insei

@Insei Insei commented Oct 16, 2026

Copy link
Copy Markdown
Owner

Benchmarks of every pull request compared with the target branch, like in gerpo.

  • .github/workflows/bench-diff.yml runs scripts/bench-diff.sh: the benchmarks of . and ./drivers on the head and on main, 10 runs each, compared by benchstat.
  • The summary is posted to the pull request as one comment that is updated after every push, and to the summary of the job; the raw results are uploaded as an artifact.
  • New BenchmarkParse_Drivers: the usual chain of tag, yaml and env drivers on a config with nested structs, lists, maps and a secret (~60–80 µs, 648 allocs per Parse locally).

The job does not fail on regressions: the numbers of shared CI machines are noisy, the comment is for the review.

…nch compared by benchstat

Signed-off-by: Dmitriy Aleksandrov <goodmobiledevices@gmail.com>
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (5a8136c) to head (fc5fa14).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff            @@
##              main       #45   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           30        30           
  Lines         2571      2571           
=========================================
  Hits          2571      2571           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions

Copy link
Copy Markdown
Contributor

Benchmark diff: 88bafad → 0eda20c (10 runs)

goos: linux
goarch: amd64
pkg: github.com/insei/tinyconf
cpu: AMD EPYC 9V74 80-Core Processor                
                    │  base.txt   │              head.txt              │
                    │   sec/op    │   sec/op     vs base               │
Parse-4               5.976µ ± 0%   6.029µ ± 1%  +0.88% (p=0.001 n=10)
Parse_SubConfig-4     2.168µ ± 0%   2.217µ ± 0%  +2.26% (p=0.000 n=10)
Current-4             42.67n ± 0%   42.87n ± 2%  +0.46% (p=0.000 n=10)
Current_SubConfig-4   76.97n ± 0%   77.18n ± 0%       ~ (p=0.105 n=10)
geomean               454.2n        458.6n       +0.96%

                    │    base.txt    │               head.txt                │
                    │      B/op      │     B/op      vs base                 │
Parse-4               4.899Ki ± 0%     4.899Ki ± 0%       ~ (p=1.000 n=10) ¹
Parse_SubConfig-4     1.367Ki ± 0%     1.367Ki ± 0%       ~ (p=1.000 n=10) ¹
Current-4               0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=10) ¹
Current_SubConfig-4     0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=10) ¹
geomean                            ²                 +0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                    │   base.txt   │              head.txt               │
                    │  allocs/op   │ allocs/op   vs base                 │
Parse-4               139.0 ± 0%     139.0 ± 0%       ~ (p=1.000 n=10) ¹
Parse_SubConfig-4     44.00 ± 0%     44.00 ± 0%       ~ (p=1.000 n=10) ¹
Current-4             0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
Current_SubConfig-4   0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
geomean                          ²               +0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

pkg: github.com/insei/tinyconf/drivers
                │  head.txt   │
                │   sec/op    │
Parse_Drivers-4   50.42µ ± 1%

                │   head.txt   │
                │     B/op     │
Parse_Drivers-4   32.04Ki ± 0%

                │  head.txt  │
                │ allocs/op  │
Parse_Drivers-4   645.0 ± 0%

@Insei
Insei merged commit ba64f6e into main Oct 16, 2026
4 checks passed
@Insei
Insei deleted the bench-ci branch October 16, 2026 12:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants