feat(benchmark): add family-local qualification

[chaofengw-nv]

Background

PR #1093 isolated model families but removed the centralized Accuracy path. This draft restores family-local Accuracy and Performance qualification, including HF conversion comparisons, without a central model registry or device policy in model configuration.

Exit Criteria

• Discover explicit family Accuracy/Performance files; support exact model selection and multiple suites/cases without enrolling L0-only manifests.
• Reuse common Python by default and allow family-owned environment preparation without modifying the common environment.
• Compare GPT-2 conversion with HF and measure TensorRT against an actual compiled HF reference with matching outputs and stable timings.
• Preserve preparation and report evidence, including failure paths, while keeping model-specific behavior inside the family.

Implementation

• Add trtmc-qualify plan/prepare/run/resume/report, reusable suite definitions, family executors, and JSON/HTML reports. All preparation precedes measurement.
• Discover optional family prepare_environment.py scripts and propagate selected build/reference interpreters. GPT-2 can provision a private CUDA reference without changing the common build environment.
• Resolve an immutable HF checkpoint and build a fresh run-owned bundle. Accuracy preparation validates and saves selected dataset samples; execution/resume use this saved selection.
• Keep GPT-2 reference execution, continuation-parity gates, and ten-sample stability evaluation family-local. Match generated tokens and GPU identity, verify compiled graphs, and reject compilation during timed iterations. Retry one unstable pair; preserve inconclusive status and both attempts.
• Own executor/preparation process groups. Timeout and SIGINT/SIGTERM terminate descendants before subsequent work is launched.
• Use generated run-local IDs, repository revision evidence, interpreter/package receipts, and local file-state checks. Do not hash source, dependencies, weights, or bundles.
• Keep optional device-run speed targets separate from family observation verdicts. Package the qualification catalog and scripts. Extend the architecture guard to reject reverse family Python imports from the benchmark application.

Change categories

• Model or runtime behavior
• Public API
• ABI
• Bundle or artifact format
• Dependencies
• Documentation only
• CI or developer tooling

Validation

Commands and Results

On 230f9d11410183d26c71cd135956980d48899728:

• PYTHONPATH=core/builder:apps/benchmark:. python3 -m pytest -q apps/benchmark/trtmc_benchmark/tests/test_qualification.py apps/benchmark/trtmc_benchmark/tests/test_qualification_process.py families/gpt2/tests/qualification/test_executor.py families/gpt2/tests/qualification/test_measurement_stability.py families/gpt2/tests/test_e2e.py tools/tests/test_new_ci.py apps/benchmark/trtmc_benchmark/tests/test_perf_matrix.py: 129 passed, 3 GPU-dependent tests skipped.
• CI_BASE_REF=aefbd09639f2bf036130719554a0df1f80580f94 PYTHONPATH=core/builder:apps/benchmark:. python3 -m tools.community_ci source-quality --base aefbd09639f2bf036130719554a0df1f80580f94: passed, including 125 tests, architecture/family validation, lint, and complexity checks.
• git diff --check aefbd09639f2bf036130719554a0df1f80580f94...HEAD: passed.
• Regression tests exercise real executor/environment descendants and SIGINT/SIGTERM cancellation, frozen Accuracy samples, modified prepared files, isolated Python environments, and the real family entry point through the scheduler/report boundary for inconclusive results. GPU acquisition/build is stubbed in the sample-preparation test.

Hardware, Environment, and Revisions

• Current revision: 230f9d11410183d26c71cd135956980d48899728; clean source checkout, Linux x86_64, Python 3.12.3. This revision's recorded checks are CPU/source checks, not GPU or premerge qualification.
• Earlier target-hardware evidence at ce7a13336309a6ae02bb58f544deda7937af3f50: GPT-2 continuation parity passed and compiled-HF Performance produced a valid stable comparison. That evidence does not validate this new revision.
• The earlier comparison used GPT-2 checkpoint 607a30d783dfa663caf39e06633721c8d4cfcd7e, FP32, Torch 2.14.0+cu130, Transformers 5.2.0, and the retained MMLU five-shot continuation dataset. Continuation parity is not an MMLU answer score.

Not Run / Remaining Gaps

• GPU Accuracy/Performance has not been rerun after these review fixes. The three GPU-dependent pytest cases were skipped because the local check lacks the required runtime/checkpoint setup.
• Full wheel build/install qualification and protected exact-head premerge remain unverified. No current-head GPU or protected premerge pass is claimed.
• Remaining historical benchmarks and device-specific CI scheduling are not implemented by this draft.

Notes For Future Readers

• Start with apps/benchmark/qualification/README.md, then the process/environment boundary, GPT-2 executor, and packaging. No Design document is included.
• Existing draft runs without input-file state receipts and saved Accuracy samples require fresh preparation; there is no legacy receipt conversion.
• Execution uses POSIX process groups; family workers must not detach into separate sessions. File-state checks detect ordinary local modifications/replacements, not hostile tampering. Runs must stay in their original trusted storage.
• Family-only dependencies, reference behavior, and acceptance criteria remain family-owned. Reports are artifacts, not GitHub Pages content.
• Review findings on the preceding head were addressed without relaxing Accuracy or Performance criteria. Keep the PR draft pending fresh GPU evidence and subsequent review.

Risk level

• Low
• Medium
• High

This adds process lifecycle management, environment provisioning, and resumable artifact handling to a shared application. CPU regressions cover the reported failure modes, but package and current-head GPU/protected qualification are still outstanding.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}