Principle
CB cannot own extraction accuracy — an LLM reasoning from partial, per-document context will over-fit (e.g. "Quarkus underlies Argo for Cleveron", or mis-typing Confluence/GitHub as service). But CB must own honesty: surfacing calibrated uncertainty so a human curator AND a consuming agent know what to trust.
A KB that's 80% right and honest about which 80% beats one that's 80% right and confidently wrong about the rest.
The real bug (observed)
On a real 3-doc extraction, every entity came out at confidence 0.9+ (zero below 0.6). The LLM is systematically overconfident, and CB passes that straight through to the consuming agent via MCP with no signal to doubt it. An agent reads "pickup-point-manager is a domain," believes it, and acts on it — because CB never told it to be unsure.
Tracking — four concrete pieces
- Calibrated confidence — replace uniform 0.9 self-ratings with signals that actually discriminate.
- Provenance — track source corroboration per entity AND per relationship (single-doc = weak).
- Review-state — distinguish raw-LLM output from human-verified.
- Propagate trust signals into retrieval + MCP — so the consuming agent receives confidence/provenance/review-state and can hedge.
(Sub-issues linked below.)
Pitch implication
Position CB as "honest structured knowledge that tells you and your agents exactly what to trust" — not "perfect KB", not "just a mechanism." It's the only position that survives the pickup-point-manager example, and more defensible than promising accuracy nobody can deliver.
Out of scope
- Guaranteeing extraction accuracy (impossible; user owns ontology/prompts/source quality + unavoidable error).
Principle
CB cannot own extraction accuracy — an LLM reasoning from partial, per-document context will over-fit (e.g. "Quarkus underlies Argo for Cleveron", or mis-typing Confluence/GitHub as
service). But CB must own honesty: surfacing calibrated uncertainty so a human curator AND a consuming agent know what to trust.The real bug (observed)
On a real 3-doc extraction, every entity came out at confidence 0.9+ (zero below 0.6). The LLM is systematically overconfident, and CB passes that straight through to the consuming agent via MCP with no signal to doubt it. An agent reads "pickup-point-manager is a domain," believes it, and acts on it — because CB never told it to be unsure.
Tracking — four concrete pieces
(Sub-issues linked below.)
Pitch implication
Position CB as "honest structured knowledge that tells you and your agents exactly what to trust" — not "perfect KB", not "just a mechanism." It's the only position that survives the pickup-point-manager example, and more defensible than promising accuracy nobody can deliver.
Out of scope