feat(client): add dynamo_chat transport + routed_experts to renderer generate

[biswapanda]

Description

Adds a dynamo_chat transport to the renderer-based generate() client so it can run against NVIDIA Dynamo, which serves no /inference/v1/generate route. Selected per-call via transport=; defaults to the existing vLLM path, so behavior is unchanged unless opted in.

Two transports:

• vllm_generate (default): unchanged — messages → render_ids() → POST /inference/v1/generate → parse_response() (vLLM TITO surface).
• dynamo_chat: messages → render_ids() → POST /v1/chat/completions with nvext.token_data (pre-tokenized prompt) + nvext.extra_fields=["engine_data"]. Completion token IDs and logprobs are read back from nvext.engine_data.

Dynamo wire shape (_post_dynamo_chat)

Mirrors the verifiers token client so the payload is identical whether a rollout goes through the token client or the renderer client. nvext.token_data (Dynamo skips tokenization when present); cache_salt → nvext.cache_salt, priority → nvext.agent_hints.priority; a single placeholder user message; sampling remap (max_tokens → max_completion_tokens, logprobs=N → logprobs=true + top_logprobs=N); passthrough fields ride the Dynamo allowlist. Tools are baked into token_data by the renderer (not sent on the wire).

routed_experts (MoE expert replay) — now surfaced on dynamo_chat

(Supersedes the earlier "routed_experts intentionally NOT surfaced" note — it now is.) parse reads routed_experts from nvext.routed_experts (or nvext.engine_data.routed_experts) and maps it to the downstream RoutedExpertsPayload {data, shape, start, dtype}. The Dynamo worker returns full-sequence routing with start=0; the renderer row-trims the leading prompt rows only when the caller explicitly sets routed_experts_prompt_start — a first-turn request with no caller start stays full-sequence with start=0 (no phantom prefix). Completion logprobs prefer nvext.engine_data.completion_logprobs (the same authoritative source as the engine token IDs) over the chat echo; a present-but-empty engine list is authoritative and does not fall back to chat.

Other

• Public RendererTransport = Literal["vllm_generate", "dynamo_chat"] alias. A present-but-empty completion_token_ids is a valid zero-token completion; only a fully absent field raises. Multimodal renderers raise NotImplementedError on dynamo_chat (vLLM path / token-client TITO remain available for VLMs).

Type of Change

• New feature (non-breaking change which adds functionality)

Review

Codex adversarial review: SIGN-OFF (F1/F2/F3 + the N1 logprob-presence finding resolved; head 5f2a914). All review threads resolved.

Testing

tests/test_client.py covers the Dynamo request body shape (priority/detokenize/sampling remap), routed_experts parse + row-trim (explicit prompt_start vs first-turn full-sequence), engine-logprob preference incl. present-but-empty, and missing/empty completion IDs.

---

Note

Medium Risk
New Dynamo wire/parse path affects completion IDs, logprobs, and MoE routed_experts for opt-in callers; default vLLM behavior is unchanged but misconfigured Dynamo responses fail at runtime.

Overview
Adds an opt-in dynamo backend to renderer generate() via a new transport argument (default vllm), so rollouts can target NVIDIA Dynamo without changing existing vLLM TITO behavior.

Wire/parse logic moves into _VllmGenerateTransport and _DynamoChatTransport, with responses normalized through _WireResult. The Dynamo path posts to /v1/chat/completions with nvext.token_data and reads nvext.engine_data.completion_token_ids / logprobs (engine channel wins over chat echo). It maps cache_salt, priority, and routed_experts_prompt_start into nvext, forwards sampling params via a denylist, and surfaces routed_experts via _normalize_routed_experts plus optional client-side _trim_dynamo_routed_experts. Large expert blobs use the same zero-copy JSON strip as vLLM (_parse_dynamo_response). Multimodal on Dynamo raises NotImplementedError.

tests/test_client.py adds broad Dynamo coverage (body shape, nvext merge, errors, logprob alignment, routing trim).

^{Reviewed by Cursor Bugbot for commit b9d25b1. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Add Dynamo chat-completions transport and routed_experts support to generate()

• Adds a pluggable transport architecture to renderers/client.py with _VllmGenerateTransport (existing behavior) and _DynamoChatTransport (new) strategies selected via a transport parameter on generate().
• The Dynamo transport posts to /chat/completions with nvext.token_data, routes cache_salt and priority into nvext, maps logprobs to OpenAI-style fields, and drops vLLM-only keys via a denylist.
• Response parsing prefers nvext.engine_data for completion_token_ids and completion_logprobs, normalizes routed_experts into a typed struct, and preserves large base64 blobs as zero-copy memoryview objects.
• Adds client-side trimming of routed_experts prompt rows when the worker did not apply routed_experts_prompt_start.
• Risk: Dynamo transport raises NotImplementedError for multimodal inputs, RuntimeError when completion IDs are absent or logprobs length mismatches — these are new runtime failure modes with no fallback.

^{Macroscope summarized b9d25b1.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit fec0a81. Configure here.}