Skip to content

History entries with missing outputs: Imp invents a filler DSPy doesn't; native replay drops answers #261

Description

@deepfates

What's true now. When a History entry has no value for an output field, Imp renders that field as "Not supplied for this conversation history message." (lib/imp/adapter/chat.ex ~1170, introduced in d2e6881). The resulting assistant message puts invented words in the model's own voice.

DSPy 3.2.1 doesn't do this. Its ChatAdapter renders a missing history output as None (dspy/adapters/chat_adapter.py ~204, outputs.get(k, None)), which was confirmed by running ChatAdapter.format in tmp/dspy-parity-venv. DSPy's "Not supplied…" text is its filler for demos (dspy/adapters/base.py:451), not for history.

Separately, the native-tool replay path (render_native_tool_history_turn) shows only next_thought as the assistant's content. So a generic Imp.Predict that sets tool_calls_field and has no next_thought loses its recorded answer when history is replayed natively.

Question. What should a history entry with missing outputs render as? Three options:

  • DSPy's None, for parity;
  • the user message alone, with no assistant message, if the entry is not an exchange;
  • the current filler.

Parity says follow DSPy unless there's a recorded reason not to, and there's no recorded reason for the filler.

Done when.

  • The chosen rendering is tested and matches the parity differentials, or the difference is recorded with its reason.
  • Native replay keeps a generic Predict's recorded outputs, with a test that fails on main.

Found while reviewing #260.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions