Schema-grounded analysis agents, data contracts, and the checks that stop a confident wrong answer from reaching a slide.
An analysis agent produces a number. The number is wrong. It is wrong in a way that is invisible, because the query was syntactically valid, the chart rendered, and the figure is plausible.
Three weeks later it is in a board deck.
This is a materially different risk from a chatbot being wrong. When a model writes bad prose, a human reads it and notices. When a model writes a subtly incorrect join and returns a number, there is nothing to notice — the output is a number, and numbers do not look wrong. Every fluency advantage the model has works against you here.
| File | What it is |
|---|---|
agents/analysis-agent.md |
Natural-language-to-SQL that grounds in schema and refuses when the question is underspecified |
contracts/data-contract.md |
What a model needs to know about a table that the schema does not tell it |
checks/answer-verification.md |
Catching the plausible wrong number before a human anchors on it |
The schema is not the semantics. A column named revenue tells the model its type and nothing that matters: whether it is recognized or booked, gross or net, which currency, which restatements are applied, and which three rows are known bad and excluded by every analyst in the company as a matter of habit. That knowledge lives in people. Text-to-SQL fails on semantics far more often than on syntax, and adding a better model does not help.
Ambiguity is the main event. "How did we do last quarter?" has at least four defensible interpretations at any real company. A system that picks one silently is not answering the question — it is guessing, and hiding the guess. The correct behavior is to state the interpretation before the number, every time, or refuse.
Verify the answer, not just the query. Query-level review catches syntax and obvious join errors. It does not catch a correct query answering the wrong question, which is the dominant failure. Verification has to happen at the level of the claim.
Before any text-to-SQL system goes to users, collect the 20 questions people actually ask, in their own words, from Slack and email rather than from a workshop. Hand-write the correct SQL and the correct answer for each.
That set is your eval. It is also, reliably, the artifact that reveals your data model cannot answer six of them at all — which is more valuable than the agent, and considerably cheaper to discover this way than after launch.