Skip to content

AI feedback loop: feedback inbox, ai_instructions fix suggestions, and AI evals - #9801

Draft
nishantmonu51 wants to merge 1 commit into
mainfrom
nishant/ai-feedback-loop
Draft

AI feedback loop: feedback inbox, ai_instructions fix suggestions, and AI evals#9801
nishantmonu51 wants to merge 1 commit into
mainfrom
nishant/ai-feedback-loop

Conversation

@nishantmonu51

Copy link
Copy Markdown
Collaborator

Closes the loop between users rating AI answers and admins improving the project's AI.

Feedback capture and review

  • Chat feedback (ratings and explicit review requests) is persisted as ai_feedback rows in a new SQLite migration (0043.sql), including sentiment, categories, comment, AI-predicted attribution and review status.
  • New ListAIFeedback, GetAIFeedback and UpdateAIFeedbackStatus runtime APIs, gated on a new ManageAIFeedback permission that is granted to cloud project admins.
  • New ListProjectAIFeedback, GetProjectAIFeedback and ResolveProjectAIFeedback local APIs so Rill Developer can review feedback recorded against the project's cloud deployment. Cloud connectivity problems are returned as states (NOT_LOGGED_IN, NOT_DEPLOYED, NO_PERMISSION, ERROR) rather than errors, so the UI can render precise empty states.

Suggested fixes

  • GenerateAIFeedbackFix proposes a concrete project change for a feedback item: an ai_instructions rule on the project or a metrics view, or a new measure, along with a draft eval case capturing the exchange.

AI evals

  • New eval resource type (parser, reconciler, runner, LLM judge and structural assertions on the metrics view, measures and dimensions the agent queried). Evals never run on a schedule; they are triggered on demand through RefreshTrigger/CreateTrigger, which also supports running a subset of cases and cancelling an in-flight run.
  • GenerateAIEvalFix proposes ai_instructions changes that fix the latest run's failing cases without regressing the passing ones.

Frontend

  • Feedback inbox with the full conversation transcript and appliable fix suggestions, an eval workspace with a runner and fix panel, and an "add to eval" flow from chat.
  • Everything is behind the feedback_inbox and ai_evals feature flags, both default off.

Checklist:

  • Covered by tests
  • Ran it and it works as intended
  • Reviewed the diff before requesting a review
  • Checked for unhandled edge cases
  • Linked the issues it closes
  • Checked if the docs need to be updated. If so, create a separate Linear DOCS issue
  • Intend to cherry-pick into the release branch
  • I'm proud of this work!

…and AI evals

Closes the loop between users rating AI answers and admins improving the project's AI:

- Persist chat feedback as `ai_feedback` rows (new SQLite migration 0043) with kind,
  sentiment, categories, comment, predicted attribution and review status.
- Add `ListAIFeedback`, `GetAIFeedback` and `UpdateAIFeedbackStatus` to RuntimeService,
  gated on a new `ManageAIFeedback` permission granted to cloud project admins.
- Add `ListProjectAIFeedback`, `GetProjectAIFeedback` and `ResolveProjectAIFeedback` to
  LocalService so Rill Developer can review feedback from the project's cloud deployment,
  reporting connectivity problems as states instead of errors.
- Add `GenerateAIFeedbackFix`, which proposes an `ai_instructions` rule or measure
  addition for a feedback item, plus a draft eval case capturing the exchange.
- Add a new `eval` resource type: parser, reconciler, runner, LLM judge and structural
  assertions, triggered on demand via `RefreshTrigger`/`CreateTrigger`.
- Add `GenerateAIEvalFix`, which proposes `ai_instructions` changes that fix the latest
  run's failing cases without regressing the passing ones.
- Frontend: feedback inbox with transcript and suggested fixes, eval workspace with
  runner and fix panels, and an "add to eval" flow from chat. Both behind the
  `feedback_inbox` and `ai_evals` feature flags (default off).

Claude-Session: https://claude.ai/code/session_018c6Uo6WgWZZ2mPKKEK37XK
@nishantmonu51 nishantmonu51 added Type:Feature New feature request Size:XL Very large change: 2,000+ lines labels Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Size:XL Very large change: 2,000+ lines Type:Feature New feature request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant