Make it easier to reproduce CI steps

[potiuk]

Our CI is designed in the way so that all the tests should be locally reproducible. Generally speaking (with some very weird exceptions related to resource constraints like memory, kernel used and local networking / filesystem setup) - the tests that we run in CI use breeze CI image as a base for containers where they run and they use breeze commands to run the steps.

It sometimes requires some prerequisites (like building an image) and in many cases the flags to breeze commands in CI are passed through environment variables, so it is not straightforward how to reproduce things out-of-the-box - you need to stitch together a few different sources of information.

Also, we have breeze ci-image load method that has --from-pr and --from-job parameters that can be used to pull the image that was used in CI for a given PR or job and use it locally - for even greater reproducibility because you would not need to build the image locally (providing that you run the tests on the same architecture).

It would be really great if each CI step would print the exact instructions on how to reproduce it - with all the prerequisites and such. This should be rather uncomplicated. We just need to:

• determine if image is needed (we already have that as we run appropriate check in each command)
• generate the right breeze ci-image build and breeze ci-image load in this case
• convert all the environment variables that are passed to CI jobs and steps into --flags including parallelism if needed (number of cores)
• clearly print in prominent place "HOW TO REPRODUCE IT LOCALLY' section - visually distinct with color and separation so that it is clearer what commands need to be run to reproduce the step.

[bugraoz93]

This would be amazing to have!

[aaron-wolmutt]

I like this. Especially the ability to print the docker build command and which options were used. I think a maybe a --show-image-build-command would be useful for reproducibility.

[suii2210]

Reproducing CI locally is technically possible today, but in practice it often means digging through multiple places (CI logs, env vars, Breeze docs, job configs) and mentally reconstructing the exact command. Having a clearly printed “HOW TO REPRODUCE LOCALLY” section per CI step would remove a lot of guesswork.

I especially like the idea of:

Translating CI environment variables into explicit breeze flags

Printing the exact breeze ci-image build/load command used (including --from-pr / --from-job)

Making the output visually distinct so it’s hard to miss in logs

This would significantly lower the barrier for new contributors and make debugging CI failures much faster and more deterministic. Happy to help test or iterate on this once implementation starts.

[aaron-wolmutt]

It looks like environment variables and Docker commands are printed when you use the --verbose flag ie,

breeze ci-image build --verbose

You could maybe redirect the verbose output and save it with the steps.

[XPE-7]

Good point about --verbose already surfacing a lot of this information today. That makes me think the core problem here may be less about missing data and more about making the reproduction path explicit and easy to spot, without requiring contributors to know which flags to use or how to mentally reconstruct the command.
A possible direction (happy to adjust based on feedback) could be:

• reusing the existing verbose output internally,
• but emitting a clearly marked “HOW TO REPRODUCE LOCALLY” section for relevant CI steps,
• optionally gated behind a focused flag rather than always-on verbosity.

I can try to put together a small, scoped implementation and tests that build on the current behavior rather than duplicating it.

[potiuk]

I would say we can print “HOW TO REPRODUCE LOCALLY” always in verbose, if CI env variable is set.

Also additionally, we could print it with clearing the ci-group (and defer it to be printed when we exit the command) - this way this HOW_TO_REPRODUCE will be visible directly in the logs, without having to unfold the folded ci group.

https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-commands#grouping-log-lines

[XPE-7]

thanks for the clarification @potiuk . I’ll start with a small, scoped change that reuses the existing verbose output and prints a clearly marked “HOW TO REPRODUCE LOCALLY” section when CI is set, emitted outside grouped logs so it’s immediately visible in CI output. I’ll keep the scope limited to a single CI step initially and build on the current behavior rather than duplicating logic.

[XPE-7]

Hi potiuk,

I’ve opened a PR #60901 that implements the reproduction hint for ci selective-check as discussed.

The change is intentionally minimal:

• Prints a clear “HOW TO REPRODUCE LOCALLY” message only when running in CI with --verbose
• Output is emitted at the end, outside collapsed CI groups, so it’s immediately visible in logs
• Added focused tests covering CI+verbose, non-verbose, and non-CI cases

No other behavior was changed.

When you get a chance, could you please take a look so the CI tests can run?
Happy to adjust anything if you feel something should be done differently.

Thanks for your guidance and time.