[Doc] Fold corrected TRL interoperability end-to-end section into trl_interop tutorial

[coder-jayp]

This PR extends the existing trl_interop.py tutorial with a corrected
end-to-end section (Part 4) that demonstrates the complete TorchRL ↔ TRL
user journey.

Specific changes:

• Adds Part 4 to trl_interop.py covering the full preference-pair pipeline:
  collect (prompt + chosen + rejected) into a ReplayBuffer, bridge to a trl
  trainer via TorchRLBufferDataset, and score rollouts with HFRewardModelWrapper
  using the canonical ("tokens", "full") / ("masks", "all_attention_mask") keys.
• Shows LLMCollector as the production data-collection path.
• Removes the standalone coding_grpo_trl.py file.
• Removes the coding_grpo_trl entry from index.rst and llms.rst.

Motivation and Context:

Addresses review feedback on the previous version. The new section:

• Preserves full conversational context (prompt is no longer discarded).

• Exercises the actual TorchRL-to-TRL boundary as described in the RFC.

• Runs in CI using toy models only (no network access or GPU required).

• I have raised an issue to propose this change (required for new features and bug fixes)

Types of changes

What types of changes does your code introduce? Remove all that do not apply:

• Documentation (update in the documentation)

Checklist

Go over all the following points, and put an x in all the boxes that apply.
If you are unsure about any of these, don't hesitate to ask. We are here to help!

• I have read the CONTRIBUTION guide (required)
• My change requires a change to the documentation.
• I have updated the tests accordingly (required for a bug fix or a new feature).
• I have updated the documentation accordingly.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/4161

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 4 New Failures

As of commit d9d3a53 with merge base 60b1d01 ():

NEW FAILURES - The following jobs have failed:

• Libs Tests on Linux / unittests-gym (3.10, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t 321f4e952715260fced01d3923fc670b78784c328920f6b096cdcacbe9391238 /exec failed with exit code 1
• Lint / lint-done (gh)
• Lint / python-source-and-configs / linux-job (gh)
  RuntimeError: Command docker exec -t 29afdd81f3443353bf2f441fafbf410b37bf4c53d542790a3ccf3d003b149909 /exec failed with exit code 1
• Unit-tests on Linux / tests-cpu (3.10, mp) / linux-job (gh)
  test/test_loggers.py::TestMLFlowLogger::test_log_hparams

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[coder-jayp]

@torchrlbot reviewer @vmoens

[github-actions]

Requested review from @vmoens (requested by @coder-jayp).

[gtnv]

set() changes TensorDict's process-global mode and LIST_TO_STACK. the change lasts for the rest of the process. can we scope it to buffer construction?

[coder-jayp]

Updated. Scoped using with set_list_to_stack(True): around the ReplayBuffer population loop so it doesn't leak globally

[vmoens]

Can you rebase?

[coder-jayp]

Done @vmoens

[coder-jayp]

cc @vmoens

[vmoens]

The real-TRL example is useful and fits the interoperability RFC, but the current tutorial does not yet demonstrate the user journey that motivates TorchRL.

It round-trips a static HF dataset through TorchRL, discards the prompt when constructing reward-model examples, skips the preference-pair creation step attributed to LLMCollector, and finally scores random tokens outside the canonical History/Tokens representation.

The real path is also skipped by both docs and tutorial CI. Could we either fold a corrected end-to-end section into the existing interop tutorial, or position this within a coherent LLM tutorial track, while preserving conversational context and exercising the actual TorchRL-to-TRL boundary?

[vmoens]

that's not amazing for a tutorial, how do we avoid that kind of guard and block?

[vmoens]

Suggested change

	# ----------------------------------------------------
	# ---------------------------------------------------

[coder-jayp]

@vmoens Removed the standalone file and folded a corrected end-to-end section (Part 4) into the existing trl_interop.py. Full prompt context included, LLMCollector shown as the production path, and HFRewardModelWrapper uses the canonical ("tokens", "full") / ("masks", "all_attention_mask") keys. Runs in CI with toy models only.

[coder-jayp]

cc @vmoens @theap06

[vmoens]

Thanks for iterating on this. I pushed a small cleanup to make the scope and contracts accurate: LLMCollector produces rollouts rather than preference labels, RewardTrainer and GRPOTrainer have different dataset requirements, and the toy reward model was not actually trained by the preceding example. The final section now presents the preference buffer as an interoperability boundary rather than claiming a complete end-to-end pipeline.

At a higher level, I still think this raises an organizational question for TorchRL. This is useful as a focused recipe, but our LLM story is still forming and I would rather see it become part of a broader sequence covering collection/conversation state, preference construction, training, and scoring. In particular, a follow-up should explore how this composes with History and the existing conversational abstractions instead of establishing standalone string fields as the apparent workflow. I don’t think that necessarily needs to block this PR once CI is green, but I would like us to agree on that broader direction so this does not remain an isolated pattern.

cc @theap06: we need and RFC on our LLM APIs to clean it up, make it more suitable for large scale training (incl. distributed APIs, titan/megatron integration, etc). One thing I'm particularly worried/thinking about is how do we guarantee that we have one way and only one way of doing things such that it's not confusing for users.

I think the tutorials should be the last step of our LLM post training integration.

[coder-jayp]

@vmoens I completely agree. This tutorial serves as a focused recipe for bridging to TRL right now, but it makes total sense that it should eventually integrate with the History abstractions.

Once the larger LLM API RFC is finalized (especially around distributed APIs and unified conversational state), we can revisit and update these tutorials so they demonstrate the single, canonical workflow.