release: Fleetbase AI v0.0.5 - #7
Merged
Merged
Conversation
An audit against a production log export (57 turns, 9 companies) found that almost every question was a product how-to, 47 of 57 turns reached the model with no Fleetbase context at all, and 6 of 10 sampled answers were false: invented menus, fields and features. Conversation history kept only the first 140 characters of each answer, so follow-ups looped. Grounding - Crawl fleetbase.io/docs into `ai_knowledge_documents` and `ai_knowledge_chunks`, chunked by heading with FULLTEXT search, behind a `KnowledgeSourceInterface`. - `search_docs` and `read_doc` tools, with `ai:sync-docs` (weekly), a gzipped snapshot shipped with the package for offline and self-hosted installs, and an admin Knowledge Base page with a sync button. - Shared `AiSystemPrompt` for every provider: answer only from docs or tool results, reply in the user's language while quoting console labels exactly, carry out the previous offer on "yes", and never print internal route names. Audience - `AiAudience` treats only `$user->type === 'admin'` as a system admin; an organization Administrator role is not one. - Docs pages, doc passages and console commands are filtered server-side, so an organization user never receives admin paths, credentials or env vars. Admin setup lines inside end-user pages are redacted to a "contact your system administrator" note. Console actions - `AiCommandRegistry` with navigate and service steps for IAM, Developers, settings, account, extensions and the admin pages; engines register their own. - `find_console_commands` and `propose_console_command` let the model only propose. The user confirms in `ai-ui-action-confirmation`, the server re-authorizes permissions and audience, and `ai-commands` then runs the steps through the engine resource-action services. Nothing runs without the click. - `scripts/verify-ai-commands.mjs` checks every route and service method exists. Runtime - `AiAgentRunner` with real multi-turn tool calling over Anthropic `tool_use`/`tool_result` and the OpenAI Responses API, prompt caching, adaptive thinking, truncation and refusal handling, and a `tool_calling` setting. - Generic `count_records`, `group_count` and `list_records` tools from the query registry, reporting `total_matching` and truncation. - Capability failures no longer send exception text to the provider; the turn is marked degraded instead. - Action previews carry a `preview_id`, so one answer can hold several. Logs, feedback and evaluation - Rebuilt admin log viewer: pagination, task status, feedback, degraded and cut-off filters, expandable tool call steps, and JSONL/CSV export. Also `ai:export-logs`. - Thumbs up/down feedback on each answer. - `ai:eval` runs 19 golden cases taken from the logs, each as an end user and as a system admin, and `ai:replay` re-runs a logged turn.
Codecov requires full coverage on the diff. The gaps were the declined confirmation in `ai:eval` and `ai:replay`, the replay guard for providers without tool calling and its leaked-route warning, verbose `ai:sync-docs` output, the uuid hooks on the knowledge models, the attachment step in tool mode, the export download stream, the `listRecords` permission guard, and the docs parser's comment and non-list-item handling. Also moves the scripted provider double into the shared test doubles so both the runner and eval tests can use it, and folds `pageSections`' defensive null guard into its return.
Live testing of the console actions surfaced four defects. **Tasks resolved to the wrong record.** `findTask` and its siblings matched `uuid = $id OR id = $id`. MySQL casts a UUID such as `4dcd1b1f-...` to the integer 4, so the lookup matched an unrelated task by its numeric key and `firstOrFail()` returned whichever row came first. Confirming a console action answered "This AI action was not found", and apply, cancel and feedback would have written to the wrong task. Roughly 62% of UUIDs start with a digit, so this fired constantly. The numeric key is now only compared when the value is actually numeric, at all five lookup sites. **Inline code was destroyed by the emphasis pass.** The placeholder `@@AI_CODE_0@@` contains underscores, so the italic rule rewrote it to `@@ai<em>CODE</em>0@@` and the code span could no longer be restored; users saw `@@AIcode0@@` in answers. Placeholders now use private-use sentinels that no markdown rule can touch. **Documentation links were dead text.** Only `[text](url)` was linkified, but answers cite docs as bare URLs. Bare URLs are now linked too, leaving code spans, trailing punctuation and underscores in paths intact. **Actions were offered in prose only.** The model wrote "I can take you there" without calling `propose_console_command`, so no card appeared and the user had to ask again. The prompt now states that offering means calling the tool in the same turn. Also: command labels quote the console's own dialog titles (New User, New Group, New API Key) instead of invented ones, the client addresses tasks by uuid rather than the autoincrement id, and `verify-ai-commands.mjs` silences PHP diagnostics that corrupted its JSON on some hosts.
**Capabilities could vanish without a trace.** `AiAgentRunner::toolsFor()` keeps only capabilities implementing `AIToolCapabilityInterface`, and tool calling is the default, so any engine or extension registering a plain capability lost it silently: no error, no log, no degraded flag. That is how Fleet-Ops order creation stopped working while still appearing registered. The runner now reports those capabilities as a `capabilities_unreachable` step and in the turn metadata, and the admin log viewer shows them, so the next one is visible instead of being discovered by a user. **The Ember test harness could not boot.** `@ember/legacy-built-in-components` was never a dependency, so `assets/vendor.js` failed before any test loaded and no QUnit test in this package had ever run. Adding it lets the harness start. The engine also builds lazily, and a lazy bundle is unreachable from the dummy application's test bundle, so addon modules could not be imported even once the harness booted; it now builds eagerly under `ember test` only, leaving the shipped lazy bundle unchanged. The six inline-code and autolink tests now actually execute. **`composer test:types` never passed.** `--memory-limit=0` is rejected by PHP, which reads it as zero bytes rather than unlimited, and level max reported 1053 pre-existing errors. The flag is now `-1`, those errors are captured in `phpstan-baseline.neon` so new code must stay clean, and a stub resolves `Illuminate\Foundation\Bus\Dispatchable`, which ships in laravel/framework rather than the split illuminate packages this package depends on. `composer test` now runs lint, types and tests end to end.
CI installs with a frozen lockfile, so pnpm-lock.yaml has to carry @ember/legacy-built-in-components alongside package.json.
…Content Both views were hard to read and slow to use. Logs stacked three 360px scroll boxes inside a boxed panel, applied its ten filters only on a "Search logs" click, and hid every prompt and answer behind a Reveal step with a confirm dialog and an extra request. Analytics was six raw integer tiles over six stacked tables. **Logs** is now a full-height split view: a compact toolbar whose filters apply as they change (search waits for a pause, rarely used filters sit behind "More filters" with a count), a two-line conversation list showing the signals a reviewer scans for (negative feedback, failures, degraded or cut-off answers) with keyboard navigation and load more, and the selected conversation read as a transcript. Each turn renders the answer's markdown and shows its status, model, tokens, duration, flags, feedback comment, errors and unreachable capabilities; its audit steps load on first open. Exports use the same filters as the list, or a single conversation. **Reveal Content is gone.** Anyone who can view the logs sees full conversations, and exports need the same `ai view audit logs` permission instead of `ai view task content`, which nothing checks any more. Exports are still written to the access log. This also fixes the task metadata summary silently dropping `unreachable_capabilities`. Because full content is now always sent, a conversation's steps are counted rather than loaded and fetched per turn on demand. **Analytics** opens on the last 30 days with period presets, a KPI strip of formatted numbers (conversations, success rate, not helpful, degraded and cut off, alongside answers and tokens), answers and tokens per day on a chart, and Who/What rankings with each row's share of tokens; clicking a row filters the view to it. The usage endpoint adds those counts and the effective range, and zero-fills days with no activity; all new keys are additive. Both views opt in to full-bleed admin pages (fleetbase/fleetbase console change). On a console without it they still work inside the boxed panel at a bounded height. The shared filter state and number formatting replace logic that was duplicated between the two views. Ember component tests could not run before: ember-core imports `tracked-built-ins` without declaring it, so the dummy app failed to boot. Adding it, and giving testem longer than 10 seconds for the first boot, lets all 24 tests run.
…lbar controls Dates and numbers in the redesigned views came out in Arabic. The console force-loads formatjs Intl polyfills whose default locale is whichever locale data registers first, and these views formatted with Intl and no locale. Dates (relative times, chart days, preset ranges) now use date-fns, the library behind format-date-fns, and numbers and chart ticks use the browser's language explicitly, so they are right even on a console that still force-loads the polyfills. The toolbar used custom pill buttons that did not match the rest of Fleetbase. Degraded, Cut off, More filters, Clear and the period presets are now ember-ui Buttons, fields use form-input-sm like Fleetbase's own list toolbars, the search text no longer sits under its icon, and the date field fills its slot instead of leaving a gap before the next control. Session status (logs) and answer status, organization and user (analytics) move behind More filters so a normal-width toolbar fits on one row.
Feature pull requests now target a release branch rather than main, but CI only ran on pull requests into main, so they merged unchecked. Match fleetops, which already runs on release/v* and the legacy dev-v* branches.
feat: ground answers in the docs and let the AI propose console actions
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #7 +/- ##
=============================================
Coverage 100.00% 100.00%
- Complexity 348 1025 +677
=============================================
Files 23 58 +35
Lines 1470 3629 +2159
=============================================
+ Hits 1470 3629 +2159
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
… latest ember-core ^0.3.24 and ember-ui ^0.4.2, the latest published versions.
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Planned release work
Depends on