[BugFix] Do not invent vLLM prompt log-probabilities

[YeonwooSung]

Description

vLLMWrapper requested SamplingParams.prompt_logprobs but the vLLM V1 generate path often returns nothing useful. The wrapper then wrote zero-filled prompt log-probs so LogProbs.full could be prompt || response. Zero log-prob is probability 1, so GRPO / KL callers that read log_probs.prompt (or full as if it contained prompt scores) got a silently wrong tensor.

This PR implements option 2 from #4226:

• If the engine returned usable prompt_logprobs, keep using them.
• If it did not (None, empty tensors, all-zero padded placeholders), leave LogProbs.prompt unset and set LogProbs.full to the response only.
• Warn once via torchrl_logger.
• Do not replace missing prompt scores with zeros.

The generate=False prompt-scoring path is unchanged. Prefix-cache zero-padding inside _build_prompt_logprobs (partial real returns) is also unchanged.

Motivation and Context

close #4226

The issue asked maintainers to choose between plumbing real vLLM prompt_logprobs and omitting a silently wrong tensor. There is still no maintainer reply. This is the fail-closed contract: do not invent values. If vLLM starts populating prompt_logprobs on generate, they are stored as before.

• I have raised an issue to propose this change (required for new features and bug fixes)

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• Documentation (update in the documentation)

Callers that assumed log_probs.prompt is always a dense tensor of prompt length, or that full is always prompt || response, will now see prompt is None and full == response when the engine omitted scores. That is the previous silent-zero case, not a change when vLLM actually returns prompt log-probs.

Checklist

• I have read the CONTRIBUTION guide (required)
• My change requires a change to the documentation.
• I have updated the tests accordingly (required for a bug fix or a new feature).
• I have updated the documentation accordingly.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/4255

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

⚠️ 16 Awaiting Approval

As of commit 3e61bbe with merge base c5ca88e ():

AWAITING APPROVAL - The following workflows need approval before CI can run:

• Build Aarch64 Linux Wheels (gh)
• Build Linux Wheels (gh)
• Build M1 Wheels (gh)
• Build Windows Wheels (gh)
• Continuous Benchmark (PR) (gh)
• Examples Tests on Linux (gh)
• Generate documentation (gh)
• Habitat Tests on Linux (gh)
• Libs Tests on Linux (gh)
• Lint (gh)
• LLM Tests on Linux (gh)
• Push Binary Nightly (gh)
• SOTA Tests on Linux (gh)
• Tutorials Tests on Linux (gh)
• Unit-tests on Linux (gh)
• Validate Test Partitioning (gh)

This comment was automatically generated by Dr. CI and updates every 15 minutes.