Skip to content

feat: review-state — distinguish raw-LLM entities from human-verified #85

Description

@rajnavakoti

Part of #82 (calibrated honesty).

Problem

There's no way to tell a raw LLM-extracted entity (unverified, possibly wrong like "pickup-point-manager is a domain") from one a human has actually reviewed and confirmed. Consumers treat both identically. The human-in-the-loop curation loop needs a state to track.

Goal

A review_state on every entity (and ideally relationship) so both curators and consuming agents know what a human has vetted.

Proposed states

unreviewed (raw LLM output, default) → verified (human confirmed) → corrected (human edited) → rejected/deprecated.

Acceptance Criteria

  • review_state field on entities, defaulting to unreviewed on extraction
  • Curation actions (edit/confirm via viewer or API) transition the state
  • Re-extraction (source re-publish) does not silently reset a human-verified/corrected entity to unreviewed without signalling
  • Queryable: "show all unreviewed entities" (the curation worklist) and "% of KB human-verified" (a health metric)
  • Tests: state defaults to unreviewed; a confirm action sets verified

Out of scope

  • The full curation UI/queue (separate) — this is the state model + transitions
  • Propagating the state to retrieval/MCP (that issue)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    frameworkScientific frameworks

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions