Skip to content

feat(evaluators): new api from runners-api#242

Open
namrataghadi-galileo wants to merge 8 commits into
mainfrom
feature/67157-refactor-api-runner-api
Open

feat(evaluators): new api from runners-api#242
namrataghadi-galileo wants to merge 8 commits into
mainfrom
feature/67157-refactor-api-runner-api

Conversation

@namrataghadi-galileo

Copy link
Copy Markdown
Contributor

Summary

  • Refactored Galileo Luna invocation so Agent Control calls runners-api directly at /api/v1/scorers/invoke.
  • Switched Luna auth to internal JWT only, signed with GALILEO_API_SECRET_KEY or GALILEO_API_SECRET.
  • Made scorer_id the required runtime identity; scorer_label is optional metadata and scorer_version_id is optional pinning.
  • Ensured requests always include config: {} and never forward Galileo-API-Key to runners-api.

Scope

  • User-facing/API changes:

    • Luna configs now require scorer_id; label-only and version-only configs are rejected.
    • Galileo Luna examples/docs now document GALILEO_RUNNERS_API_URL, GALILEO_LUNA_SCORER_ID, and internal secret auth.
  • Internal changes:

    • Replaced API /scorers/invoke and /internal/scorers/invoke client routing with runners-api routing.
    • Removed Luna auth mode selection and API URL/console URL derivation.
    • Added tests for endpoint, JWT payload, scorer ID validation, version forwarding, API-key stripping, and request body shape.
  • Out of scope:

    • Agent Control UI changes.
    • runners-api/API service implementation changes.
    • Scorer hydration, version resolution, caching, retries, or protect execution logic inside Agent Control.

Risk and Rollout

  • Risk level: medium
  • Rollback plan:
    • Revert this PR to restore the previous API-mediated Luna invocation path.
    • If rollout exposes saved controls without scorer_id, migrate those controls or temporarily roll back before retrying cutover.

Testing

  • Added or updated automated tests
  • Ran make check (targeted package validation was run instead)
  • Manually verified behavior

Targeted validation:

  • make -C evaluators/contrib/galileo test -> 86 passed
  • make -C evaluators/contrib/galileo lint -> passed
  • make -C evaluators/contrib/galileo typecheck -> passed
  • git diff --check -> clean

Checklist

  • Linked issue/spec: RFC on direct runners-api scorer invoke
  • Updated docs/examples for user-facing changes
  • Included required follow-up tasks: confirm/migrate any legacy saved Luna controls missing scorer_id

@codecov

codecov Bot commented Jun 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@namrataghadi-galileo namrataghadi-galileo changed the title feat: new api from runners-api feat(evaluators): new api from runners-api Jun 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants