1.0.0-alphax - End to end test of current docs

[Rachael-Graham]

Testing of 1.x docs have been ad-hoc during development. Need to test all 1.x docs that are currently available against the 1.0.0-alpha2 (or whatever is current) release to verify that 1) the docs are running correctly, and 2) the code is running as described. Where there is a discrepancy, either the docs must be updated to match the code, or a code bug must be filed.

The checklists below group every published 1.x page into 11 sections, ordered so that each section can be run against the cluster the previous one left behind.

What counts as tested

A page passes when all five of these hold. Anything that fails is a finding, whether the fix lands in the docs or in the code.

1. Every command runs as written and returns output that matches the documented example.
2. Every YAML applies without validation errors and produces the resource state the page describes.
3. The Before you begin section lists everything the page actually needs, at the versions it actually needs.
4. Descriptive claims about behavior match what the cluster does, not what the page assumes.
5. Cross-references resolve to the page and section they name, and that section still covers what the link says it covers.

Where the docs and the code disagree, fix the docs when the docs are wrong, and file a code bug when the code is wrong. Code bugs go to kagent-dev/kagent or agent-substrate/substrate; docs fixes come back here as PRs.

How to record results

• Check a box only when the page passes all five criteria on a live alpha1 cluster.
• Leave a box unchecked and add a comment on this issue naming the section, the page, and the specific step that failed.
• Link every filed bug and every docs PR back to this issue, so the section comment stays the single place to look.
• Mark a page N/A in a comment when it cannot be tested for lack of credentials or infrastructure, rather than leaving it silently unchecked.

Known open bugs

These are already filed and are expected to cause failures during the pass. Confirm the behavior still matches the issue rather than filing a duplicate.

• kagent#2764 — a Harness has no status subresource, so kubectl get harness shows a blank READY column.
• kagent#2759 — audit records are dropped when the Actor checkpoints, which affects Audit prompts.
• kagent#2758 — a byo Harness advertises the agent on :80 but injects no PORT, which affects Run your own agent image.
• kagent#2604 — plugin resources cannot be materialized while creating golden Actors, which affects Plugins and external skill sources.

---

1. Install and setup

Everything else depends on this section, so it runs first and blocks the rest of the pass if it fails.

• Install kagent — both the Agent Substrate and the kagent install, plus every command under Verify the installation
• About model providers
• The controller gRPC port-forward on 8083 works as documented, and the CLI reaches the same port

2. Model providers

Each provider page follows the same shape, so a failure that repeats across pages is one finding rather than many. Test the providers you hold credentials for, and mark the rest N/A.

Providers reachable with a standard API key:

• OpenAI
• Anthropic
• Gemini
• xAI
• Ollama — local, no credentials needed
• BYO OpenAI-compatible endpoint
• agentgateway — a prerequisite for the agentgateway example in section 9

Providers that need a cloud account or a deployed endpoint:

• Amazon Bedrock
• Azure OpenAI
• Azure AI Foundry
• Google Vertex AI
• SAP AI Core

Across the whole section:

• Every ModelConfig field name and provider block matches the CRD schema in alpha1
• The model names shown in each example are still served by that provider

3. Get started

These two pages are the entry point for new users and the stated prerequisite for most later sections, so they must pass on a cluster with nothing but a fresh install.

• Your first agent — leaves behind the my-first-harness Harness and AgentTemplate that later sections reuse
• Your first MCP tool
• The Harness admission label shown on these pages matches what the shipped Harness actually admits

4. Agents

These pages configure the agent itself, so each one builds on the Harness and AgentTemplate from section 3.

• Agent harness — including the model provider support matrix
• Bring your own agent
• System prompts
• Agent memory — needs an external PostgreSQL database with pgvector, which the bundled database cannot provide
• Human in the loop — needs grpcurl

5. Skills and MCP

Two of these three pages are conceptual, so the check is accuracy against the code rather than a command-by-command run.

• About tools
• Skills
• Plugins — see kagent#2604 before filing anything here
• Every CRD field named on these pages exists in alpha1 with the described meaning

6. Substrate runtime

These pages describe runtime behavior rather than walking through a procedure, so verify each claim against a running Actor and against the substrate 0.2.0-beta4 code.

• Sandboxing
• Suspend and resume
• Identity — including the claims about what the open source authorizer does and does not enforce
• Networking and egress control — confirm whether egress is enforced in alpha1, because the guidance changes if it is

7. Observability

Each page here installs its own backend, so run them in order and leave the collector in place for the audit page.

• Launch the UI — the only page with screenshots, covered by the Playwright specs in docs-site/playwright/tests/
• The UI screenshots on that page match the alpha1 UI, and the Playwright baselines are regenerated if they do not
• Tracing — installs Jaeger
• Audit prompts — installs a collector and a logging backend; see kagent#2759

8. Operations

The uninstall and upgrade pages destroy or replace an installation, so run them last within this section and on a cluster you are finished with.

• Operational considerations
• Tune Agent Substrate — needs grpcurl, controller.grpc.reflection=true, and cluster-admin
• Debug — confirm each symptom is still reachable and each remedy still works
• Upgrade from 0.x — needs a 0.10.x installation and an empty PostgreSQL database
• Uninstall — destructive, so run it against a cluster you are done with

9. Examples

Every example depends on section 3, and several add their own infrastructure, so check the prerequisites before picking one up.

• Agent Substrate — needs grpcurl and a completed agent turn
• Use agents from an MCP client — needs grpcurl and reflection enabled
• Call an agent over A2A — needs grpcurl
• Add a skill to an agent — needs Docker and a registry the cluster can reach over HTTPS
• Delegate work to another agent
• Build a documentation agent — needs Node.js 20 or later and its own embeddings API key
• Run your own agent image — needs Docker, git, and a registry; see kagent#2758
• Block PII in an agent's model requests — needs the agentgateway model routing from section 2

10. Concepts

No commands run here, so the check is whether each explanation still describes the alpha1 architecture and whether the terminology matches the CRDs and the glossary.

• What is kagent?
• Core concepts
• kagent architecture
• Agent Substrate architecture
• Every diagram renders in both light and dark mode and matches the components alpha1 actually installs

11. Reference

The API and CLI pages are generated, so the check is whether they were regenerated from the alpha1 tag rather than whether each line reads correctly.

• API reference — regenerated from the alpha1 CRDs
• Helm reference — matches values.yaml in the alpha1 chart
• Version support
• Release notes 1.0
• Tools ecosystem
• Frequently asked questions
• Glossary — every term still matches how the docs and the CRDs use it
• Community and contributing

CLI pages, regenerated from the alpha1 binary and diffed by command family:

• kagent install and kagent uninstall
• kagent apply, kagent create, kagent delete, kagent get
• kagent invoke and kagent dashboard
• kagent db migrate — up, down, goto, status, version
• kagent mcp — init, build, run, deploy, add-tool, secrets sync
• kagent version, kagent bug-report, kagent completion