Skip to content

[CI] Weekly overview snapshot sweep for /overview freshness / [CI] 每周 overview snapshot 定时 sweep #2586

Description

@edwingao28

Problem

/overview shows the latest run per model×hardware (30 combos), but there is no scheduled benchmark run — data only lands when a PR touches a recipe. Cells go stale unevenly, and cross-hardware numbers come from different dates/batches (parent: #2304).

Plan

Weekly cron sweep, 1 best config per combo, same batch → cross-hardware comparable:

  • New schedule: workflow reusing e2e-tests.yml + ingest-results dispatch (app side needs zero changes — changelog-metadata falls back to the run name)
  • Lowest effective priority: schedule events get no event bump in ci-priority.yaml, so PR/main sweeps preempt
  • Saturday 06:00 UTC, off-peak
  • Partial-failure runs still ingest (app skips failed rows)

Cost: pilot ~18 node-hours/week; full 30-combo scope ~8–40 node-hours/week ≈ 0.3–2% of current CI load (~2,000–2,500 job-hours/wk measured).

Rollout

  • Pilot: dsv4 + qwen3.5 on b200/b300 (4 configs, 37 jobs), 2–3 weeks
  • Extend: mi355x + minimaxm3/kimik2.5
  • Extend: gb200/gb300 (3-node pools — keep 1 snapshot job at a time)
  • Recipe bring-up for missing combos tracked separately

Accepted trade-off

/overview is latest-run-wins: a scheduled run replaces the displayed number. Accepted for the pilot (option A); revisit source tagging if a bad scheduled run ever regresses a cell.


问题

/overview 每个 model×hardware 格子只显示最新一次 run(共 30 个 combo),但目前没有任何定时 benchmark——数据只在 PR 动到 recipe 时产生。格子新鲜度参差,且跨硬件数字来自不同日期/批次(父 issue:#2304)。

方案

每周 cron sweep,每个 combo 跑 1 个 best config,同批次 → 跨硬件可比:

  • 新增 schedule: workflow,复用 e2e-tests.yml + ingest-results dispatch(app 侧零改动——changelog-metadata 缺失时回退到 run name)
  • 实际最低优先级:schedule 事件在 ci-priority.yaml 中无 event 加分,PR/main sweep 天然抢占
  • 周六 06:00 UTC,避开高峰
  • 部分失败的 run 仍然 ingest(app 跳过失败行)

成本: pilot 约 18 node-hours/周;全量 30-combo 约 8–40 node-hours/周 ≈ 当前 CI 负载(实测 ~2,000–2,500 job-hours/周)的 0.3–2%。

推进

  • Pilot:dsv4 + qwen3.5 on b200/b300(4 configs,37 jobs),2–3 周
  • 扩展:mi355x + minimaxm3/kimik2.5
  • 扩展:gb200/gb300(3 节点池——同时最多 1 个 snapshot job)
  • 缺 recipe 的 combo 单独跟踪

已接受的取舍

/overview 是 latest-run-wins:定时 run 会替换展示数字。pilot 阶段接受(方案 A);若定时 run 拉低格子数字再考虑 source tag。

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status
    In Progress

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions