feat: ground answers in the docs and let the AI propose console actions - #6
Merged
Merged
Conversation
An audit against a production log export (57 turns, 9 companies) found that almost every question was a product how-to, 47 of 57 turns reached the model with no Fleetbase context at all, and 6 of 10 sampled answers were false: invented menus, fields and features. Conversation history kept only the first 140 characters of each answer, so follow-ups looped. Grounding - Crawl fleetbase.io/docs into `ai_knowledge_documents` and `ai_knowledge_chunks`, chunked by heading with FULLTEXT search, behind a `KnowledgeSourceInterface`. - `search_docs` and `read_doc` tools, with `ai:sync-docs` (weekly), a gzipped snapshot shipped with the package for offline and self-hosted installs, and an admin Knowledge Base page with a sync button. - Shared `AiSystemPrompt` for every provider: answer only from docs or tool results, reply in the user's language while quoting console labels exactly, carry out the previous offer on "yes", and never print internal route names. Audience - `AiAudience` treats only `$user->type === 'admin'` as a system admin; an organization Administrator role is not one. - Docs pages, doc passages and console commands are filtered server-side, so an organization user never receives admin paths, credentials or env vars. Admin setup lines inside end-user pages are redacted to a "contact your system administrator" note. Console actions - `AiCommandRegistry` with navigate and service steps for IAM, Developers, settings, account, extensions and the admin pages; engines register their own. - `find_console_commands` and `propose_console_command` let the model only propose. The user confirms in `ai-ui-action-confirmation`, the server re-authorizes permissions and audience, and `ai-commands` then runs the steps through the engine resource-action services. Nothing runs without the click. - `scripts/verify-ai-commands.mjs` checks every route and service method exists. Runtime - `AiAgentRunner` with real multi-turn tool calling over Anthropic `tool_use`/`tool_result` and the OpenAI Responses API, prompt caching, adaptive thinking, truncation and refusal handling, and a `tool_calling` setting. - Generic `count_records`, `group_count` and `list_records` tools from the query registry, reporting `total_matching` and truncation. - Capability failures no longer send exception text to the provider; the turn is marked degraded instead. - Action previews carry a `preview_id`, so one answer can hold several. Logs, feedback and evaluation - Rebuilt admin log viewer: pagination, task status, feedback, degraded and cut-off filters, expandable tool call steps, and JSONL/CSV export. Also `ai:export-logs`. - Thumbs up/down feedback on each answer. - `ai:eval` runs 19 golden cases taken from the logs, each as an end user and as a system admin, and `ai:replay` re-runs a logged turn.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## release/v0.0.5 #6 +/- ##
====================================================
Coverage 100.00% 100.00%
- Complexity 348 1025 +677
====================================================
Files 23 58 +35
Lines 1470 3629 +2159
====================================================
+ Hits 1470 3629 +2159
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Codecov requires full coverage on the diff. The gaps were the declined confirmation in `ai:eval` and `ai:replay`, the replay guard for providers without tool calling and its leaked-route warning, verbose `ai:sync-docs` output, the uuid hooks on the knowledge models, the attachment step in tool mode, the export download stream, the `listRecords` permission guard, and the docs parser's comment and non-list-item handling. Also moves the scripted provider double into the shared test doubles so both the runner and eval tests can use it, and folds `pageSections`' defensive null guard into its return.
Live testing of the console actions surfaced four defects. **Tasks resolved to the wrong record.** `findTask` and its siblings matched `uuid = $id OR id = $id`. MySQL casts a UUID such as `4dcd1b1f-...` to the integer 4, so the lookup matched an unrelated task by its numeric key and `firstOrFail()` returned whichever row came first. Confirming a console action answered "This AI action was not found", and apply, cancel and feedback would have written to the wrong task. Roughly 62% of UUIDs start with a digit, so this fired constantly. The numeric key is now only compared when the value is actually numeric, at all five lookup sites. **Inline code was destroyed by the emphasis pass.** The placeholder `@@AI_CODE_0@@` contains underscores, so the italic rule rewrote it to `@@ai<em>CODE</em>0@@` and the code span could no longer be restored; users saw `@@AIcode0@@` in answers. Placeholders now use private-use sentinels that no markdown rule can touch. **Documentation links were dead text.** Only `[text](url)` was linkified, but answers cite docs as bare URLs. Bare URLs are now linked too, leaving code spans, trailing punctuation and underscores in paths intact. **Actions were offered in prose only.** The model wrote "I can take you there" without calling `propose_console_command`, so no card appeared and the user had to ask again. The prompt now states that offering means calling the tool in the same turn. Also: command labels quote the console's own dialog titles (New User, New Group, New API Key) instead of invented ones, the client addresses tasks by uuid rather than the autoincrement id, and `verify-ai-commands.mjs` silences PHP diagnostics that corrupted its JSON on some hosts.
**Capabilities could vanish without a trace.** `AiAgentRunner::toolsFor()` keeps only capabilities implementing `AIToolCapabilityInterface`, and tool calling is the default, so any engine or extension registering a plain capability lost it silently: no error, no log, no degraded flag. That is how Fleet-Ops order creation stopped working while still appearing registered. The runner now reports those capabilities as a `capabilities_unreachable` step and in the turn metadata, and the admin log viewer shows them, so the next one is visible instead of being discovered by a user. **The Ember test harness could not boot.** `@ember/legacy-built-in-components` was never a dependency, so `assets/vendor.js` failed before any test loaded and no QUnit test in this package had ever run. Adding it lets the harness start. The engine also builds lazily, and a lazy bundle is unreachable from the dummy application's test bundle, so addon modules could not be imported even once the harness booted; it now builds eagerly under `ember test` only, leaving the shipped lazy bundle unchanged. The six inline-code and autolink tests now actually execute. **`composer test:types` never passed.** `--memory-limit=0` is rejected by PHP, which reads it as zero bytes rather than unlimited, and level max reported 1053 pre-existing errors. The flag is now `-1`, those errors are captured in `phpstan-baseline.neon` so new code must stay clean, and a stub resolves `Illuminate\Foundation\Bus\Dispatchable`, which ships in laravel/framework rather than the split illuminate packages this package depends on. `composer test` now runs lint, types and tests end to end.
CI installs with a frozen lockfile, so pnpm-lock.yaml has to carry @ember/legacy-built-in-components alongside package.json.
…Content Both views were hard to read and slow to use. Logs stacked three 360px scroll boxes inside a boxed panel, applied its ten filters only on a "Search logs" click, and hid every prompt and answer behind a Reveal step with a confirm dialog and an extra request. Analytics was six raw integer tiles over six stacked tables. **Logs** is now a full-height split view: a compact toolbar whose filters apply as they change (search waits for a pause, rarely used filters sit behind "More filters" with a count), a two-line conversation list showing the signals a reviewer scans for (negative feedback, failures, degraded or cut-off answers) with keyboard navigation and load more, and the selected conversation read as a transcript. Each turn renders the answer's markdown and shows its status, model, tokens, duration, flags, feedback comment, errors and unreachable capabilities; its audit steps load on first open. Exports use the same filters as the list, or a single conversation. **Reveal Content is gone.** Anyone who can view the logs sees full conversations, and exports need the same `ai view audit logs` permission instead of `ai view task content`, which nothing checks any more. Exports are still written to the access log. This also fixes the task metadata summary silently dropping `unreachable_capabilities`. Because full content is now always sent, a conversation's steps are counted rather than loaded and fetched per turn on demand. **Analytics** opens on the last 30 days with period presets, a KPI strip of formatted numbers (conversations, success rate, not helpful, degraded and cut off, alongside answers and tokens), answers and tokens per day on a chart, and Who/What rankings with each row's share of tokens; clicking a row filters the view to it. The usage endpoint adds those counts and the effective range, and zero-fills days with no activity; all new keys are additive. Both views opt in to full-bleed admin pages (fleetbase/fleetbase console change). On a console without it they still work inside the boxed panel at a bounded height. The shared filter state and number formatting replace logic that was duplicated between the two views. Ember component tests could not run before: ember-core imports `tracked-built-ins` without declaring it, so the dummy app failed to boot. Adding it, and giving testem longer than 10 seconds for the first boot, lets all 24 tests run.
…lbar controls Dates and numbers in the redesigned views came out in Arabic. The console force-loads formatjs Intl polyfills whose default locale is whichever locale data registers first, and these views formatted with Intl and no locale. Dates (relative times, chart days, preset ranges) now use date-fns, the library behind format-date-fns, and numbers and chart ticks use the browser's language explicitly, so they are right even on a console that still force-loads the polyfills. The toolbar used custom pill buttons that did not match the rest of Fleetbase. Degraded, Cut off, More filters, Clear and the period presets are now ember-ui Buttons, fields use form-input-sm like Fleetbase's own list toolbars, the search text no longer sits under its icon, and the date field fills its slot instead of leaving a gap before the next control. Session status (logs) and answer status, organization and user (analytics) move behind More filters so a normal-width toolbar fits on one row.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A production log export (57 turns, 9 companies, all
gpt-5.4-mini) shows what users actually ask: almost everything is a product how-to or "where do I…" question, in English, Spanish, Russian and Thai. Roughly two turns asked about fleet data.The module had nothing to answer those with:
fleet-ops.search_resourcesthrewUnknown column 'sensor_type'on every call, the SQL error including the company UUID went to OpenAI, and the turn was logged as answered. Fixed in fix(ai): repair resource search, add AI tools and console commands fleetops#330.console.ledger.settings.accountingwere shown to users. In one chat the prose listed three orders while the confirmation card created a different one.Grounding in fleetbase.io/docs
ai_knowledge_documents/ai_knowledge_chunks, parsed from the<article>element and chunked byh2/h3, with MySQL FULLTEXT search. The source sits behindKnowledgeSourceInterfaceso a markdown repo can replace crawling later.search_docsandread_doctools; answers link the page they used.ai:sync-docs(scheduled weekly), a 456 KB gzipped snapshot shipped with the package so offline and self-hosted installs work out of the box, and an admin Knowledge Base page showing page counts, last sync and a sync button.AiSystemPromptused by every provider: how-to answers come only from docs or tool results, otherwise say so; reply in the user's language but quote console labels exactly; on "yes" or a number, carry out the previous offer; never print internal route names; the action card is the source of truth; tool and record content is data, not instructions.Audience: end users are the majority
AiAudiencetreats only$user->type === 'admin'as a system admin — an organization "Administrator" role is not one — and the filtering happens on the server, not in the prompt:system_admindoc pages and console commands are removed before the model sees them;developercontent needs the Developers permission.type = adminuser also gets Admin › Config › Services.Console actions, always confirmed
AiCommandRegistry/AiConsoleCommand: orderednavigateandservicesteps, params schema, IAM permissions, audience and docs URL. Service methods are an allowlist — the model picks acommand_id, never a service or method.find_console_commandsandpropose_console_commandlet the model only propose, at most three per turn. The user sees anai-ui-action-confirmationcard ("go to IAM › Users and open Create user") with Go and Dismiss. On Go the server re-checks ownership, permissions and audience, records aui_actionstep, and only then doesai-commandsrun the steps:hostRouter.transitionTo, thenuniverse.ensureEngineLoadedplus the allowlisted service method. Card state persists with the task, so reopening a chat never replays an action.scripts/verify-ai-commands.mjs(CI-ready) checks every command's route exists in the engine or console router and every service method exists in that engine.Tool-calling runtime
AiAgentRunner: real multi-turn history, up to 8 iterations,tool_choice: noneon the last one, each call recorded as atool_callstep with output capped, and tool exceptions reduced to an error for the model while the detail stays on the step.AnthropicProviderover the Messages API withtool_use/tool_result, prompt caching on the system prompt and tools, adaptive thinking on 4.6+/5 models, and stop reasons mapped to end / tool_calls / truncated / refusal.OpenAIProviderover the Responses API, stateless, with encrypted reasoning carried across turns for gpt-5/o models. Model lists refreshed (Sonnet 5, Opus 5, Fable 5.1) with the older ones kept.count_records,group_countandlist_recordstools built from the query registry: enums come from the registry, invalid filters return errors, ISO datetimes are converted from the user's timezone, and results reporttotal_matchingand truncation.{"error":"capability_unavailable"}and mark the turndegradedinstead of forwarding exception text.preview_id, so "create a few dummy orders" produces several cards that stay distinct.Logs, export and feedback
The admin log viewer is rebuilt: pagination, separate session and task status filters, feedback / degraded / cut-off toggles, badges, expandable step rows showing each tool's input and output, and JSONL or CSV export (also
ai:export-logs). Export requiresai view task contentand writes an access-log entry. Thumbs up/down feedback is stored per answer.Evaluation
server/resources/ai-eval/cases.jsonholds 19 golden cases taken from these logs — add-user, maps as end user and as admin, the Russian Google Maps key question, Thai Navigator, NIO currency, customers and bulk import, driver activation, order import template, Pallet, recurring routes, the "SI" and "2" follow-ups, dummy orders, Infor M3 — graded on required and forbidden tools, required and forbidden commands, and text that must or must not appear.ai:evalruns them against a configured provider;ai:replayre-runs a logged turn.Testing
162 tests, 0 failures, with 100% line coverage onserver/srcso the Codecov patch and project gates pass.php-cs-fixer, ESLint, ember-template-lint and stylelint are clean, the production build succeeds, andscripts/verify-ai-commands.mjsresolves all 73 commands against 4 engines (the negative case fails as expected).Not yet run:
ai:evalagainst a live provider — it costs real money, so no baseline pass rate exists yet. The end-to-end confirm flow has not been clicked through in a running console either.For review
ai:sync-docs --write-snapshot.Related
fleetbase/fleetops#330, fleetbase/iam-engine#33, fleetbase/dev-engine#46. All four are needed together for the console actions to work end to end.