[jinyouzhi]

Stacked on #9077 (which is stacked on #9009). Until those merge, this branch also contains their commits; only the last commit (Run the whole static rollout decode step inside the CUDA graph) is new here.

Summary

On the static graph path (use_graph_capture=True without continuous_batch_size) only the model forward is captured. After every replay the host chooses the next token, writes it to the output, advances the position / cache position / attention mask, and calls eos_mask.all(), which synchronizes with the device on every generated token.

This PR captures the whole decode step:

	Before	After
Captured	Model forward	Forward + repetition penalty + token selection (argmax or HF warpers + multinomial) + EOS bookkeeping + position / mask / output-buffer update
Host work per token	Token selection, tensor updates, eos_mask.all() sync	One replay_graph
EOS check	Every step (device sync)	Every 16 steps, on a device-side finished flag; rows that finished keep receiving pad, as before
Graph signature	Shapes, device, dtype, parameter layout	Same, plus sampling settings, repetition penalty and EOS ids

• The RNG generator state is restored after warmup so the captured graph draws the same random numbers as generate(); graph-sampled output matches the eager path bitwise for a fixed seed.
• Other generation_config settings stay unsupported on the graph path, as before.
• Docs: inference-engine.rst updated.

Relation to #8602

Independent of segment KI: everything is plain PyTorch ops, so any HF model on the graph path gets it. #8602 rewrites the same decode loop and adds a decode_step_graph CUDA kernel that does in-graph token selection only for batch size 1; this PR covers every batch size, so that kernel would no longer be needed. Whichever lands second rebases.

Tests

• New test_static_graph_stops_once_every_row_has_produced_eos: forces every row to hit EOS on its third token and checks that graph output equals the eager rollout and that the graph is replayed fewer than max_new_tokens - 1 times (i.e. the periodic EOS check stops decoding).
• Existing greedy / sampling / repetition-penalty parity tests against model.generate() cover the in-graph selection.
• tests/unit/v1/test_hybrid_engine_rollout_v1.py: 26 passed, 1 skipped (pretrained-model test, needs DEEPSPEED_PRETRAINED_ROLLOUT_MODEL)
• tests/unit/runtime/rollout/test_hybrid_engine_rollout.py: 63 passed, 1 skipped

Hardware: 1x NVIDIA H20, torch 2.13.0+cu130, transformers 5.17.

Performance

Decode throughput is unchanged where decode is GPU-bound (Llama-3.1-8B, H20, prompt 128, 512 new tokens, greedy, EOS disabled; decode tok/s):

batch	#9077	this PR
1	118	119
4	357	357
16	774	775
64	1087	1094

The gain comes from removing per-token host work and the per-step sync. It grows as the forward gets faster and host time becomes a larger share of the step, and it lets EOS-terminated rollouts stop without a sync on every token.