Add request/response logging hook to native model client

[andreatgretel]

Priority Level

Medium (Nice to have)

Is your feature request related to a problem? Please describe.

With the migration from litellm to the native model client in DD 0.5.5, there's no built-in way to capture full HTTP request/response payloads for LLM API calls. Previously, litellm's callback system allowed registering custom loggers that could observe every request and response. This was useful for debugging prompt issues, auditing API calls, and troubleshooting generation failures at scale.

Describe the solution you'd like

Expose a hook or callback mechanism on the native model client that allows consumers to observe request/response data. This could be:

1. httpx event_hooks: The native client uses httpx internally. httpx supports event_hooks={"request": [...], "response": [...]} on the client constructor. Exposing a way to inject these hooks (e.g. via ModelConfig or the ModelClient factory) would let downstream tools register logging callbacks without modifying DD internals.

2. A dedicated logging transport: An optional httpx transport wrapper that logs request/response bodies before forwarding, similar to how ThrottledModelClient wraps with throttling.

3. RunConfig option: A debug_log_requests: bool field on RunConfig that enables built-in request/response logging to a configurable path, handled entirely within DD.

Option 1 is the most flexible (consumers control what's logged). Option 3 is the most user-friendly.

Describe alternatives you've considered

• Keeping litellm as an optional dependency just for debug logging - adds unnecessary dependency weight
• Monkey-patching the httpx client after DD creates it - fragile and version-dependent
• Using Python logging at DEBUG level to capture httpx traffic - httpx logs at TRACE level which is very verbose and not structured

Agent Investigation

Explored DD 0.5.5 codebase:

• Native client lives in packages/data-designer-engine/src/data_designer/engine/models/clients/
• ThrottledModelClient wraps ModelClient with per-request hooks (throttle acquire/release) - same pattern could wrap with logging
• clients/retry.py builds httpx RetryTransport - the transport layer is already composable
• No existing event_hooks, logging hooks, or callback mechanism on the client
• clients/types.py defines ChatCompletionRequest/Response which contain the structured data that would need to be logged

Additional context

The previous litellm-based logger captured: timestamp, model name, full prompt messages, request kwargs, complete response body, and token usage per API call. Whatever replaces it should capture equivalent data. This is particularly useful when running large-scale generation jobs where individual request failures need to be diagnosed.

Checklist

• I've reviewed existing issues and the documentation
• This is a design proposal, not a "please build this" request

[andreatgretel]

Related: #386 tracks adding structured metadata (model, token counts, latency, retries) to log records. That's the consumer side of the hook proposed here - worth considering them together.