Initial commit: add CockroachDB benchmark wrapper - #1
Open
sayalibhavsar wants to merge 3 commits into
Open
Conversation
Add wrapper scripts, configuration, and documentation for running CockroachDB benchmarks in the CPT pipeline.
📝 WalkthroughSummary by CodeRabbit
WalkthroughAdds a Bash-based CockroachDB benchmark wrapper. It manages tooling and cluster lifecycle, runs configured workloads, validates iteration results, aggregates metrics, documents operation, adds platform dependencies, and includes GPLv2 licensing. ChangesCockroachDB benchmark execution
Estimated code review effort: 4 (Complex) | ~45 minutes Suggested reviewers: Sequence Diagram(s)sequenceDiagram
participant BenchmarkUser
participant cockroachdb_run
participant test_tools
participant CockroachDB
participant PCP
participant ResultSchema
BenchmarkUser->>cockroachdb_run: Provide benchmark options
cockroachdb_run->>test_tools: Acquire and install shared tools
cockroachdb_run->>CockroachDB: Download, start, and validate cluster
cockroachdb_run->>PCP: Start collection when enabled
cockroachdb_run->>CockroachDB: Execute workloads
CockroachDB-->>cockroachdb_run: Return ops/sec output
cockroachdb_run->>ResultSchema: Validate iteration results
cockroachdb_run->>cockroachdb_run: Aggregate and save reports
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (4)
result_schema.py (1)
16-17: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winConstrain
Average/Deviationto non-negative.Both represent ops/sec and a percentage deviation, neither of which is meaningfully negative. Add
ge=0to catch malformed/parsing-error data (e.g. a negative value slipping through) at validation time instead of silently passing.diff
- Average: float = pydantic.Field(allow_inf_nan=False) - Deviation: float = pydantic.Field(allow_inf_nan=False) + Average: float = pydantic.Field(allow_inf_nan=False, ge=0) + Deviation: float = pydantic.Field(allow_inf_nan=False, ge=0)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@result_schema.py` around lines 16 - 17, Update the Average and Deviation fields in the result schema to enforce a minimum value of zero by adding the appropriate ge=0 validation constraint while preserving the existing allow_inf_nan=False behavior.cockroachdb/openmetrics_cockroachdb_reset.txt (1)
1-8: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueReset template declares metrics never pushed by the wrapper.
running,numthreads,runtime,throughput, andlatencyare reset here butcockroachdb_run's PCP block only ever pushesiteration,concurrency, andaverage(results2pcp_add_valuecalls). These extra fields look like leftover boilerplate from another wrapper's template rather than metrics this benchmark actually reports. Trim the template to the fields actually produced, or wire up the missing pushes if they're intended.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@cockroachdb/openmetrics_cockroachdb_reset.txt` around lines 1 - 8, Update the cockroachdb reset template to contain only the metrics emitted by cockroachdb_run: iteration, concurrency, and average. Remove running, numthreads, runtime, throughput, and latency unless corresponding results2pcp_add_value pushes are intentionally added.cockroachdb/cockroachdb_run (2)
198-212: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winFixed
sleep 5for cluster readiness.A hard-coded sleep is fragile under load (this executes once per workload×concurrency×iteration). A short poll loop against
node statuswould be more reliable and faster on average.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@cockroachdb/cockroachdb_run` around lines 198 - 212, Replace the fixed sleep in start_cockroachdb with a short bounded polling loop that repeatedly runs cockroach_bin node status until the cluster is ready or the timeout is reached. Keep the existing failure path through exit_out with status 103 when readiness is not achieved, and retain the success message after readiness is confirmed.
180-193: 🔒 Security & Privacy | 🔵 Trivial | ⚖️ Poor tradeoffNo integrity verification for downloaded CockroachDB binary.
The tarball is fetched over HTTPS but never checksummed against CockroachDB's published SHA256SUMS before extraction and execution. Worth adding a checksum check as a supply-chain hardening measure.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@cockroachdb/cockroachdb_run` around lines 180 - 193, Update the CockroachDB download flow before tar extraction to retrieve CockroachDB’s published SHA256SUMS, compute the downloaded tarball’s SHA-256 digest, and verify it matches the expected checksum for the selected tarball. On missing or mismatched checksums, call exit_out with a clear integrity-verification failure and avoid extracting or installing the archive; preserve the existing wget/curl fallback and successful installation flow.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@cockroachdb/cockroachdb_run`:
- Around line 167-171: Update the cached-install logic around cockroach_dir so
the cache is version-specific and cannot reuse a CockroachDB build created for a
different --cockroach_version. Incorporate the requested version into the
installation path or validate the existing installation’s version before
returning, while preserving reuse when the cached version matches.
- Around line 350-380: Adjust the averaging logic surrounding the iteration loop
to track the count of successfully parsed rval values, incrementing it only
after a valid parse. Use that count for samples, including the trimmed-mean
calculation and its high/low exclusions, while preserving the existing behavior
when all iterations parse successfully.
- Around line 310-317: Update both test_header_info invocations in the
CockroachDB result-generation flow to include Start_Date and End_Date in the
--field_header list, preserving the existing fields and ordering otherwise.
Ensure the generated CSV header matches the start_time and end_time result
columns consumed by csv_to_json and Cockroachdb_Results.
In `@license`:
- Around line 1-17: Rename the lowercase license header-notice template to a
distinct, non-colliding filename such as HEADER_NOTICE, while preserving its
contents and updating any references to the current license filename. Keep the
existing full LICENSE file unchanged.
---
Nitpick comments:
In `@cockroachdb/cockroachdb_run`:
- Around line 198-212: Replace the fixed sleep in start_cockroachdb with a short
bounded polling loop that repeatedly runs cockroach_bin node status until the
cluster is ready or the timeout is reached. Keep the existing failure path
through exit_out with status 103 when readiness is not achieved, and retain the
success message after readiness is confirmed.
- Around line 180-193: Update the CockroachDB download flow before tar
extraction to retrieve CockroachDB’s published SHA256SUMS, compute the
downloaded tarball’s SHA-256 digest, and verify it matches the expected checksum
for the selected tarball. On missing or mismatched checksums, call exit_out with
a clear integrity-verification failure and avoid extracting or installing the
archive; preserve the existing wget/curl fallback and successful installation
flow.
In `@cockroachdb/openmetrics_cockroachdb_reset.txt`:
- Around line 1-8: Update the cockroachdb reset template to contain only the
metrics emitted by cockroachdb_run: iteration, concurrency, and average. Remove
running, numthreads, runtime, throughput, and latency unless corresponding
results2pcp_add_value pushes are intentionally added.
In `@result_schema.py`:
- Around line 16-17: Update the Average and Deviation fields in the result
schema to enforce a minimum value of zero by adding the appropriate ge=0
validation constraint while preserving the existing allow_inf_nan=False
behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 3cd79350-8317-4962-9d0d-fd968f012bc4
📒 Files selected for processing (8)
LICENSEREADME.mdcockroachdb.jsoncockroachdb/cockroachdb_runcockroachdb/openmetrics_cockroachdb_reset.txtcockroachdb/test_optslicenseresult_schema.py
kdvalin
suggested changes
Aug 3, 2026
Collaborator
Author
|
CSV, PCP output and deviation under 3% |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@cockroachdb/cockroachdb_run`:
- Around line 363-371: Update the statistics calculation around avg, sumsq, and
stddev to compute and retain a high-precision mean for the sum-of-squares loop,
while continuing to round avg only for CSV output. Use the unrounded mean when
calculating each deviation and preserve the existing output formatting and
percentage-deviation logic.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 8ab38b86-b4ea-4cf6-8e57-da34da42473e
📒 Files selected for processing (1)
cockroachdb/cockroachdb_run
Comment on lines
+363
to
+371
| avg=$(echo "scale=1; $sum/$to_times_to_run" | bc | sed 's/^\./0./') | ||
| if [[ $to_times_to_run -gt 1 ]]; then | ||
| sumsq=0 | ||
| for v in $values; do | ||
| sumsq=$(echo "$sumsq + ($v - $avg) * ($v - $avg)" | bc) | ||
| done | ||
| stddev=$(echo "scale=2; sqrt($sumsq / $to_times_to_run)" | bc -l | sed 's/^\./0./') | ||
| if [[ $(echo "$avg > 0" | bc) -eq 1 ]]; then | ||
| deviation=$(echo "scale=2; $stddev * 100 / $avg" | bc | sed 's/^\./0./') |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Default bc scale:"
printf 'scale=1; 3/2\n(.5)*(.5)\n' | bc
echo "Arbitrary-precision bc scale:"
printf 'scale=20; (.5)*(.5)\n' | bc -lRepository: redhat-performance/cockroachdb-wrapper
Length of output: 234
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf 'Locate file and relevant context:\n'
git ls-files | grep -F 'cockroachdb/cockroachdb_run' || true
printf '\nRelevant excerpt:\n'
sed -n '340,390p' cockroachdb/cockroachdb_run
printf '\nParse and compute current vs high-precision formulas for a sample sequence:\n'
python3 - <<'PY'
from decimal import Decimal, getcontext
values = ["1", "2"]
to_times_to_run = len(values)
# Current parser-style behavior: sum as Decimal, divide with scale=1, then compute sumsq using Decimal
# without any additional scaling/multiplication preservation, matching bc's default scale.
scale = 1
sum_total = Decimal(0)
for v in values:
sum_total += Decimal(v)
avg = (sum_total / Decimal(to_times_to_run)).quantize(Decimal(10) ** -scale)
sumsq = Decimal(0)
for v in values:
term = avg - Decimal(v)
prod = term * term
# emulate bc default truncation of the product fractional digits at current scale
prod = prod.quantize(Decimal(10) ** -scale)
sumsq += prod
# If using arbitrary-precision intermediate math for the same mean:
prod_precise = ((avg - Decimal("1")) * (avg - Decimal("1")) +
(avg - Decimal("2")) * (avg - Decimal("2")))
sumsq_precise_prod = prod_precise.quantize(Decimal(10) ** -scale)
print(f"values={values}")
print(f"avg={avg}")
print(f"current sumsq (quantized products)={sumsq}")
print(f"current stddev rounded to scale=2={round(float(sumsq / Decimal(to_times_to_run)) ** 0.5, 2)}")
print(f"current coefficient-of-variation rounded to scale=2=")
print(f"{round(float(sumsq / Decimal(to_times_to_run)) ** 0.5 * Decimal(100) / avg, 2)}")
print(f"precise product sum before rounding={prod_precise}")
print(f"precise product sum rounded to scale=1={sumsq_precise_prod}")
PY
printf '\nCheck where CSV/schema results are consumed:\n'
rg -n "devi|avg|result_schema|csv" -S .Repository: redhat-performance/cockroachdb-wrapper
Length of output: 2339
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf 'result_schema locations:\n'
git ls-files | grep -Fx 'result_schema.py' || true
printf '\nresult_schema excerpts:\n'
for f in $(git ls-files | grep -Fx 'result_schema.py'); do
echo "--- $f"
sed -n '1,220p' "$f"
done
printf '\nRelevant lines around results_file writes in cockroachdb_run:\n'
sed -n '310,410p' cockroachdb/cockroachdb_run
printf '\nFind result_schema usages:\n'
rg -n "result_schema|deviation|Average|Average|Average|Deviation|Average,Deviation" -S .Repository: redhat-performance/cockroachdb-wrapper
Length of output: 6271
Compute deviation using the full average internally.
avg is only a display-rounded value, but the standard-deviation calculation uses it as the mean. Use a high-precision mean for the sum-of-squares division before rounding avg for the CSV output.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@cockroachdb/cockroachdb_run` around lines 363 - 371, Update the statistics
calculation around avg, sumsq, and stddev to compute and retain a high-precision
mean for the sum-of-squares loop, while continuing to round avg only for CSV
output. Use the unrounded mean when calculating each deviation and preserve the
existing output formatting and percentage-deviation logic.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
##Description
Breaking out CockroachDB benchmark from phoronix-test suite
Clerical Stuff
This closes #
Relates to JIRA: RPOPC-1288
Test Artifacts:
https://gist.github.com/sayalibhavsar/c4d945f15bd5afc35ba3a7bfb2cbba04
https://gist.github.com/sayalibhavsar/9cb24048f581c2efaeba18b0b67c2f5c