feat(v1): add Harness.transform_prompt hook#1561
Conversation
Add an overridable, default-identity Harness.transform_prompt(prompt, state), applied in Runtime.submit_model_request immediately before the model call — the single point every model request passes through (base-program, fn-mode, and host-loop). Subclasses override it to return a modified prompt; the result is what the host tokenizes, sends to the model, and records as the trajectory step. Default is identity, so existing harnesses are unaffected. Sync and async overrides are both supported. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
ApprovabilityVerdict: Needs human review This PR introduces a new feature (prepare_prompt hook) that modifies the core model request flow. Additionally, an open review comment identifies a medium-severity issue where sandbox mode may diverge from the prepared prompt, warranting human review. You can customize Macroscope's approvability policy. Learn more. |
|
@cursoragent change function name to "prepare_prompt", ensure it's triggered for base program (see comment), add to byo-harness documentation |
I ran ruff via uvx (lint clean), pushed the updates to feat/harness-transform-prompt-hook, and updated the existing PR.   |
… in Runtime.submit_model_request Co-authored-by: will brown <willccbb@users.noreply.github.com>
Co-authored-by: will brown <willccbb@users.noreply.github.com>
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
| class MyHarness(vf.Harness[MyHarnessConfig]): | ||
| def prepare_prompt(self, prompt: vf.Messages, state: vf.State) -> vf.Messages: | ||
| # Example: prepend a brief reminder to the first system message | ||
| messages = vf.normalize_messages(prompt) |
There was a problem hiding this comment.
🟢 Low docs/byo-harness.md:440
The documentation example on line 440 calls vf.normalize_messages(prompt), but normalize_messages is not exported from the verifiers.v1 package. Users following this example will hit AttributeError: module 'verifiers.v1' has no attribute 'normalize_messages'. The function exists in verifiers.utils.message_utils but is not re-exported through the vf namespace. Either re-export normalize_messages from the vf package or update the example to import from verifiers.utils.message_utils directly.
- messages = vf.normalize_messages(prompt)
+ messages = vf.utils.normalize_messages(prompt)🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @docs/byo-harness.md around line 440:
The documentation example on line 440 calls `vf.normalize_messages(prompt)`, but `normalize_messages` is not exported from the `verifiers.v1` package. Users following this example will hit `AttributeError: module 'verifiers.v1' has no attribute 'normalize_messages'`. The function exists in `verifiers.utils.message_utils` but is not re-exported through the `vf` namespace. Either re-export `normalize_messages` from the `vf` package or update the example to import from `verifiers.utils.message_utils` directly.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit dadff8e. Configure here.
| result = harness.prepare_prompt(prompt, state) | ||
| if inspect.isawaitable(result): | ||
| result = await result | ||
| return result if result is not None else prompt |
There was a problem hiding this comment.
Sandbox base ignores prepared prompt
Medium Severity
prepare_prompt runs on every submit_model_request, including sandbox base-program /vf/model calls, but only the in-process base_program loop re-reads the prepared prompt from the trajectory afterward. The sandbox runner keeps extending its local messages with the raw thread, so overrides that reshape history diverge from what the model saw and what call_user receives.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit dadff8e. Configure here.
| content = str(messages[0].content or "") | ||
| messages[0].content = "Answer concisely. " + content | ||
| return messages | ||
| ``` |
There was a problem hiding this comment.
Missing reference documentation entry
Low Severity
This PR adds the user-facing Harness.prepare_prompt hook but only documents it in byo-harness.md. Core harness customization points are also enumerated in docs/reference.md (alongside load_system_prompt); that reference list was not updated for the new hook.
Triggered by project rule: BugBot Instructions
Reviewed by Cursor Bugbot for commit dadff8e. Configure here.
3 participants
Adds a default-identity
Harness.transform_prompt(prompt, state)hook, applied inRuntime.submit_model_requestright before the model call — the single point every model request passes through (base-program, fn-mode, host-loop).Subclasses override it to reshape the prompt before each request (e.g. context compaction, redaction, reminders); the returned messages are what the host tokenizes, sends, and records as the trajectory step. Default is identity, so existing harnesses are unaffected. Sync and async both supported.
2 files, 28 insertions.
🤖 Generated with Claude Code
Note
Medium Risk
The hook runs on every model request and can change what is sent and logged; incorrect overrides could break rollouts or trajectories, though the default identity path leaves existing harnesses unchanged.
Overview
Adds a
Harness.prepare_prompt(prompt, state)hook (default: no change) so subclasses can reshape messages immediately before each model call—e.g. compaction, redaction, or injected reminders.Runtime.submit_model_requestis the single choke point: it runs the hook (sync or async) before tokenization and the client request, and the trajectory step stores the prepared prompt. That applies uniformly to base-program, fn-mode, and host-loop rollouts.The built-in base program loop now re-reads the last trajectory step’s
promptafter each turn so its in-memorymessagesstay aligned with what was actually sent when the hook modifies the prompt.docs/byo-harness.mddocuments the hook with an override example.Reviewed by Cursor Bugbot for commit dadff8e. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Add
Harness.transform_prompthook to reshape messages before each model requestprepare_prompt(prompt, state)method toHarnessthat is an identity transform by default and intended to be overridden by subclasses.Runtime.submit_model_requestnow calls an internal_prepare_prompthelper before issuing the model request, supporting both sync and async hook implementations.Harnessrealigns its workingmessagesvariable to the prepared prompt after each model request, so subsequent turns build on exactly what was sent to the model.docs/byo-harness.mdwith an example subclass that prepends a reminder to the first system message.prepare_promptwill now alter what the model sees and what is recorded in the trajectory.📊 Macroscope summarized dadff8e. 3 files reviewed, 0 issues evaluated, 0 issues filtered, 0 comments posted
🗂️ Filtered Issues
No issues evaluated.