feat: add weekly OSS benchmark workflow

[kisernl]

Summary

Adds a weekly aggregate OSS benchmark workflow to computesdk/benchmarks so it runs from master against the weekly branch and commits weekly outputs without overwriting the existing per-category results/ and root SVGs.

• weekly-oss-benchmarks.yml triggers Sundays at 10am UTC (or manually), checks out computesdk/benchmarks@weekly into oss/, runs the benchmark matrix, merges results, generates charts, and commits weekly outputs to results-weekly/, weekly-*.svg, and .oss-benchmark-revision-weekly.
• Weekly results are kept separate from the per-category results/ directories and root SVGs so a weekly run cannot clobber daily/push outputs.
• Helper scripts added/updated to support the aggregate workflow:
  • scripts/build-daily-matrix.mjs
  • scripts/oss-workflow-defaults.mjs (fixed --root/--kind flag parsing and added FALLBACK_DEFAULTS for ai-gateway)
• Burst-only sandbox: the sandbox case runs tti.bench.ts once with --concurrency 100, matching the current results/burst_tti output.
• DAX runs mirror the per-category workflow: LIGHTNING_INSTANCE_TYPE=cpu-8, ARKER_PLATFORMS=graviton4, and dedicated Blaxel DAX credentials are remapped onto BL_API_KEY/BL_WORKSPACE.
• Vault credential loading uses a grep-compatible regex built from provider-vars.json (with DAX-specific BL_DAX_* keys appended), matching load-vault-secrets.sh's expected $1 regex argument.
• namespacelabs/nscloud-setup@v0 and id-token: write are added to the benchmark job so nsc vault works.
• Removed the legacy benchmarks/src/ingest.ts and Ingest results step; benchmark runs stream to the platform orchestrator directly.
• generate-pricing-svg.ts now reads live scores from results/burst_tti/latest.json.

actionlint and pnpm typecheck pass.

Link to Devin session: https://app.devin.ai/sessions/b5022ba352b848c696ded905bf862867
Requested by: @kisernl

---

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[open-cla]

Contributor License Agreement

All contributors are covered by a CLA.

[superagent-security]

Superagent found 2 security concern(s).

[superagent-security]

P2: Workflow grants contents:write to all jobs including benchmark runners

Top-level contents:write applies to all jobs, including untrusted benchmark runners.

Scope contents:write to the collect job only; use read permissions for other jobs.

AI prompt

Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior.

<file name=".github/workflows/weekly-oss-benchmarks.yml">
<violation number="1" location=".github/workflows/weekly-oss-benchmarks.yml:68">
<priority>P2</priority>
<title>Workflow grants contents:write to all jobs including benchmark runners</title>
<evidence>Top-level `permissions:
  contents: write` applies to every job in the workflow, including the `benchmark` job (line 154) which checks out and executes arbitrary code from the `computesdk/benchmarks` `weekly` branch. If that upstream repository is compromised, the injected code would execute with repository write access, raising blast radius.</evidence>
<recommendation>Move `contents: write` to only the `collect` job that commits results. Set `permissions: contents: read` at the workflow level and override with `permissions: contents: write` on the `collect` job.</recommendation>
</violation>
</file>

[superagent-security]

P2: Matrix values from external repo interpolated directly into shell scripts

Matrix values from upstream repo are interpolated into shell run blocks without env indirection.

Use env variables to pass matrix values into shell scripts instead of direct expression interpolation.

AI prompt

Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior.

<file name=".github/workflows/weekly-oss-benchmarks.yml">
<violation number="1" location=".github/workflows/weekly-oss-benchmarks.yml:189">
<priority>P2</priority>
<title>Matrix values from external repo interpolated directly into shell scripts</title>
<evidence>The `benchmark` job embeds `${{ matrix.kind }}` and `${{ matrix.provider }}` directly inside `run:` shell scripts (e.g. `case "${{ matrix.kind }}" in` at line 189, `manifest_kind="${{ matrix.kind }}"` at line 192, `if [ "${{ matrix.provider }}" = "all" ]` at line 195, and `--arg provider "${{ matrix.provider }}"` at line 200). These values are produced by `build-daily-matrix.mjs` from provider manifests (`benchmarks/*/providers.ts`) residing in the `computesdk/benchmarks` `weekly` branch. If that upstream repo is compromised, shell metacharacters injected into provider names would execute as commands in this workflow.</evidence>
<recommendation>Pass matrix values through job-level `env:` variables instead of interpolating them directly into shell scripts. For example, add `env: BENCH_KIND: ${{ matrix.kind }}, BENCH_PROVIDER: ${{ matrix.provider }}` to the job and reference `$BENCH_KIND` and `$BENCH_PROVIDER` in the script.</recommendation>
</violation>
</file>