Skip to content

Conversation

@fdubut
Copy link
Contributor

@fdubut fdubut commented Jan 30, 2026

Description

Currently the SelfAskTrueFalseScorer constructor fails if neither a path nor a question is passed. However, a lot of basic scenarios require only a simple "task achieved" scorer, which we are proposing to use as fallback/default.

The PR also includes a minor change to the "task achieved" YAML template to make it clear the scorer should return true if the model completes the task, not only when it returns instructions to complete the task.

Tests and Documentation

  • Added four tests to cover the 2x2 matrix of path provided or not x question provided or not.
  • Added an example of the default behavior in the documentation for SelfAskTrueFalseScorer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants