ci(cli): validate installed Eval frameworks - #3188
Conversation
Run Harbor and Pier against the same immutable CLI tarball after cross-platform spec and prerequisite checks. Use deterministic local tasks so the release gate exercises the installed relay, Docker environment, verifier result, artifacts, and cleanup without provider credentials. Preserve Pier's framework-owned log mounts when adding user mounts; otherwise an explicit empty mount list replaces the paths required for subject scope, rewards, and collected artifacts. Generated-by: OpenAI Codex
|
Warning Review limit reached
Next review available in: 4 minutes Limit details: You’ve used all 3 included reviews currently available under your plan. Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughSummaryThis PR closes the immutable CLI tarball release-validation gap. It validates Eval specifications and Python framework environments before execution. It runs deterministic Harbor and Pier Docker cells against the installed tarball. It preserves Pier framework mounts, execution scope, verifier rewards, and collected artifacts. The PR extends the existing release and smoke-validation paths. It adds one standalone Eval release-validation script and one CI job. It does not replace the production execution system or add public exports. The solution is the smallest coherent approach shown by the current diff. The added script is necessary for offline installation, deterministic fixtures, timeout handling, credential sanitization, framework-output checks, artifact checks, and cleanup. No deletion or simplification is evident without weakening framework-specific regression coverage. Complexity delta
The total maintenance complexity increases. The increase is justified by the broader immutable-tarball release gate. No safe deletion is evident from the current diff. ValidationThe PR includes:
The final status of required checks is unverified because direct check results are not provided. Review-relevant risks
The person performing the merge must review the final diff. A maintainer makes the final determination. WalkthroughChangesEval release validation
Estimated code review effort: 4 (Complex) | ~45 minutes Merge Risk: 🔵 Low · up to The release gate now validates installed Eval frameworks and Docker cells, but floating runtime and transitive dependency resolution can make results vary for identical CLI artifacts; merge is reasonable with explicit owner awareness or follow-up to pin or constrain those inputs. Sequence Diagram(s)sequenceDiagram
participant CI
participant ReleaseValidator
participant EvalCLI
participant Harbor
participant Pier
participant Docker
CI->>ReleaseValidator: run release:cli:eval
ReleaseValidator->>EvalCLI: install and invoke CLI
EvalCLI->>Harbor: execute Eval cell
EvalCLI->>Pier: execute Eval cell
Harbor->>Docker: run trial containers
Pier->>Docker: run trial containers
ReleaseValidator->>EvalCLI: inspect results and artifacts
ReleaseValidator->>Docker: verify cleanup
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
PR Summary by QodoValidate installed CLI with Harbor and Pier Eval frameworks
AI Description
Diagram
High-Level Assessment
Files changed (6)
|
Code Review by Qodo
1.
|
![qodo-code-review[bot]](https://avatars.githubusercontent.com/in/484649?s=60&v=4){kind=small}
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (1)
scripts/release-cli-eval-package.mjs (1)
249-264: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUse the isolated environment for the fixture git commands (disposition: follow-up).
These
gitcalls passprocess.env. The hostHOMEstays in scope, so user or system git configuration applies. Settings such ascommit.gpgsign,core.hooksPath, orinit.templateDircan make the commit fail or add unexpected files, which breaks the determinism this fixture provides.isolatedEnvironmentalready produces a cleanHOME; reuse it here instead of adding new configuration flags.Source: Path instructions
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 17eb6f79-3d4d-4fe0-befe-0b91d7e72ab3
📒 Files selected for processing (6)
.github/workflows/cli-package-validation.ymlpackage.jsonpackages/eval/src/__tests__/lifecycle-boundaries.test.tspackages/eval/src/harness-executor.tsscripts/release-cli-eval-package.mjsscripts/smoke-release-cli-package.mjs
Included review availability: Your plan includes up to 3 reviews per rolling hour; 0 remain after this review.
Reserve Pier's framework-owned log subtrees before Docker composition so configured mounts cannot shadow verifier rewards or collected artifacts. Keep framework failures authoritative when summary or diagnostic evidence is malformed or unreadable, and create the deterministic Git fixture under the same isolated environment as the installed candidate. Generated-by: OpenAI Codex
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
packages/eval/src/__tests__/lifecycle-boundaries.test.ts (1)
1377-1403: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winRetain the Harbor regression case alongside the Pier case. The current test only exercises
createPierExecutorand asserts Pier’s log mounts and framework environment. The remaining Harbor tests do not cover these Harbor-specific assertions.Source: Path instructions
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: ac6429d6-2bf5-46c4-8822-99278eafe4b6
📒 Files selected for processing (6)
package.jsonpackages/eval/src/__tests__/lifecycle-boundaries.test.tspackages/eval/src/harness-executor.tsscripts/release-cli-eval-package.mjsscripts/release-cli-eval-support.mjsscripts/release-cli-eval-support.test.mjs
🚧 Files skipped from review as they are similar to previous changes (1)
- package.json
Included review availability: Your plan includes up to 3 reviews per rolling hour; 0 remain after this review.
Use an exact predicate for the observable Pier mount rejection so path punctuation cannot weaken the regression check.
|
The current PR changes have been manually reviewed by me. I take responsibility for this result and decide to merge them. |
1 participant
English
Summary
Extend the immutable CLI tarball gate with cross-platform Eval spec and Python-framework preflight checks, plus one Ubuntu job that completes deterministic Harbor and Pier Docker cells without provider credentials.
The real Pier cell exposed that an explicit mount list replaced framework-owned log mounts. Preserve those mounts and inventory the Pier artifact layout so execution scope, verifier rewards, and collected subject output survive the installed-package path.
Refs #3166
Verification
AI use
Select exactly one:
Tool(s) and scope: OpenAI Codex implemented the release validation, diagnosed the Pier integration failure, added the root fix and tests, and ran verification. M4n5ter is the human contributor of record.
Checklist
Does this PR entail a change in behavior?
中文
摘要
扩展同一份不可变 CLI tarball 的发布门禁：三平台验证 Eval spec 与 Python framework preflight，并增加一个 Ubuntu job，使用相互隔离的 Python 3.12 环境顺序完成无 provider 凭证的 Harbor 和 Pier 真实 Docker cell。
真实 Pier cell 暴露出显式 mounts 会替换 framework 自有日志挂载。现在适配层保留这些挂载，并按 Pier 的实际目录回收 artifacts，确保安装包路径下的执行作用域、verifier reward 和 subject 输出都能完整落盘。
关联 #3166。验证项与 AI 披露以英文区为准。