Update Qwen3.5 validation sample count

[jepio]

Summary

• update the MLPerf Training v6.1 Qwen3.5 GRPO compliance rules for open and closed submissions
• require eval_samples to equal 251 instead of 256

Rationale

Validation of the reference dataset removed five invalid validation tasks. The qualified validation dataset now contains 251 samples, so compliant benchmark logs report eval_samples: 251. The compliance checker must use the validated dataset size.

The corresponding dataset update is tracked in NVIDIA-NeMo/RL#3601.

Verification

• parsed both updated YAML rule files with PyYAML
• verified both eval_samples checks require exactly 251
• ran git diff --check

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[ShriyaRishab]

WG meeting discussion 9/3/26:

1. @jepio can you please update https://github.com/mlcommons/training/tree/master/llm_post_training and https://github.com/mlcommons/training_policies/blob/master/training_rules.adoc to reflect this change as well?
2. Does it impact RCPs or convergence in any way? Can you share data for the same?
3. Also, the number 251 is not very well formed (not a power of 2) so how are incomplete batches handled?

[RissyRan]

+1 for the 251 samples issue. Or could we add 5 extra/duplicate samples to be as 256?

[jepio]

Does it impact RCPs or convergence in any way? Can you share data for the same?

The 0.69 target accuracy requires solving 177+ problems when the validation size was 256, and now with 251 requires solving 174+. Since the 5 removed environments were not solvable, a previously passing implementation would still be able to solve 177+, but now out of 251, which corresponds to an accuracy of 0.705.

We kept the target accuracy and validation start step the same, and since 0.705 > 0.69 the convergence still happens at the same step. We didn't re-run an RCP study without late eval, so I can't say what happens earlier.

Also, the number 251 is not very well formed (not a power of 2) so how are incomplete batches handled?

@RissyRan @ShriyaRishab 251 is the size of the validation dataset and is used for inference/generation only, it never passes through the trainer. Inference is not batched so there is no practical reason it needs to be power-of-2 - agent turns are never synchronized and may come at any time, and have different durations.

Do you know of any RL frameworks that have a limitation on validation dataset size? If not then there's no reason to impose a stricter requirement to the dataset size.