From b217107ed028481b36880c6b1ec78d0c89382fef Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=D0=91=D0=B0=D0=BD=D1=8C=D0=BA=D0=BE=D0=B2=D1=81=D0=BA?= =?UTF-8?q?=D0=B8=D0=B9=20=D0=A1=D1=82=D0=B0=D0=BD=D0=B8=D1=81=D0=BB=D0=B0?= =?UTF-8?q?=D0=B2=20=D0=95=D0=B2=D0=B3=D0=B5=D0=BD=D1=8C=D0=B5=D0=B2=D0=B8?= =?UTF-8?q?=D1=87?= Date: Thu, 10 Sep 2026 21:52:00 +0400 Subject: [PATCH] feat: add AI agent architecture design and diagnosis skill --- docs/README.skills.md | 1 + skills/agent-architecture/LICENSE.txt | 21 ++ skills/agent-architecture/SKILL.md | 64 ++++++ skills/agent-architecture/agents/openai.yaml | 4 + .../references/architecture-contract.md | 144 +++++++++++++ .../references/architecture-selection.md | 58 ++++++ skills/agent-architecture/references/audit.md | 87 ++++++++ .../references/capability-design.md | 62 ++++++ .../agent-architecture/references/design.md | 90 ++++++++ .../references/diagnostic-review.md | 87 ++++++++ .../references/discovery-protocol.md | 71 +++++++ .../references/evaluation-design.md | 61 ++++++ .../references/execution-continuity.md | 86 ++++++++ .../references/result-delivery.md | 51 +++++ .../references/scenarios.md | 193 ++++++++++++++++++ .../references/source-map.md | 118 +++++++++++ .../references/validation-loop.md | 54 +++++ 17 files changed, 1252 insertions(+) create mode 100644 skills/agent-architecture/LICENSE.txt create mode 100644 skills/agent-architecture/SKILL.md create mode 100644 skills/agent-architecture/agents/openai.yaml create mode 100644 skills/agent-architecture/references/architecture-contract.md create mode 100644 skills/agent-architecture/references/architecture-selection.md create mode 100644 skills/agent-architecture/references/audit.md create mode 100644 skills/agent-architecture/references/capability-design.md create mode 100644 skills/agent-architecture/references/design.md create mode 100644 skills/agent-architecture/references/diagnostic-review.md create mode 100644 skills/agent-architecture/references/discovery-protocol.md create mode 100644 skills/agent-architecture/references/evaluation-design.md create mode 100644 skills/agent-architecture/references/execution-continuity.md create mode 100644 skills/agent-architecture/references/result-delivery.md create mode 100644 skills/agent-architecture/references/scenarios.md create mode 100644 skills/agent-architecture/references/source-map.md create mode 100644 skills/agent-architecture/references/validation-loop.md diff --git a/docs/README.skills.md b/docs/README.skills.md index 24473fa6f4..752c7ca7b2 100644 --- a/docs/README.skills.md +++ b/docs/README.skills.md @@ -34,6 +34,7 @@ See [CONTRIBUTING.md](../CONTRIBUTING.md#adding-skills) for guidelines on how to | [ad-campaign-analyzer](../skills/ad-campaign-analyzer/SKILL.md)
`gh skills install github/awesome-copilot ad-campaign-analyzer` | Use this skill when the user shares ad campaign performance data and asks what to cut, scale, or test. Trigger for prompts like "analyze my ad campaigns", "where am I wasting ad spend", "reallocate my ad budget", "which ads are actually working", or "ROAS analysis". Do not trigger for campaign planning or creative generation without performance data. | None | | [add-educational-comments](../skills/add-educational-comments/SKILL.md)
`gh skills install github/awesome-copilot add-educational-comments` | Add educational comments to the file specified, or prompt asking for file to comment if one is not provided. | None | | [adobe-illustrator-scripting](../skills/adobe-illustrator-scripting/SKILL.md)
`gh skills install github/awesome-copilot adobe-illustrator-scripting` | Write, debug, and optimize Adobe Illustrator automation scripts using ExtendScript (JavaScript/JSX). Use when creating or modifying scripts that manipulate documents, layers, paths, text frames, colors, symbols, artboards, or any Illustrator DOM objects. Covers the complete JavaScript object model, coordinate system, measurement units, export workflows, and scripting best practices. | `references/object-model-quick-reference.md`
`scripts/batch-export-png.jsx`
`scripts/create-color-grid.jsx`
`scripts/find-replace-text.jsx` | +| [agent-architecture](../skills/agent-architecture/SKILL.md)
`gh skills install github/awesome-copilot agent-architecture` | Design AI agent architectures through requirements discovery, or audit and diagnose architectural flaws in existing agents. Architecture only; excludes implementation and general code review. | `LICENSE.txt`
`agents/openai.yaml`
`references/architecture-contract.md`
`references/architecture-selection.md`
`references/audit.md`
`references/capability-design.md`
`references/design.md`
`references/diagnostic-review.md`
`references/discovery-protocol.md`
`references/evaluation-design.md`
`references/execution-continuity.md`
`references/result-delivery.md`
`references/scenarios.md`
`references/source-map.md`
`references/validation-loop.md` | | [agent-governance](../skills/agent-governance/SKILL.md)
`gh skills install github/awesome-copilot agent-governance` | Patterns and techniques for adding governance, safety, and trust controls to AI agent systems. Use this skill when:
- Building AI agents that call external tools (APIs, databases, file systems)
- Implementing policy-based access controls for agent tool usage
- Adding semantic intent classification to detect dangerous prompts
- Creating trust scoring systems for multi-agent workflows
- Building audit trails for agent actions and decisions
- Enforcing rate limits, content filters, or tool restrictions on agents
- Working with any agent framework (PydanticAI, CrewAI, OpenAI Agents, LangChain, AutoGen) | None | | [agent-owasp-compliance](../skills/agent-owasp-compliance/SKILL.md)
`gh skills install github/awesome-copilot agent-owasp-compliance` | Check any AI agent codebase against the OWASP Agentic Security Initiative (ASI) Top 10 risks.
Use this skill when:
- Evaluating an agent system's security posture before production deployment
- Running a compliance check against OWASP ASI 2026 standards
- Mapping existing security controls to the 10 agentic risks
- Generating a compliance report for security review or audit
- Comparing agent framework security features against the standard
- Any request like "is my agent OWASP compliant?", "check ASI compliance", or "agentic security audit" | None | | [agent-skill-stack](../skills/agent-skill-stack/SKILL.md)
`gh skills install github/awesome-copilot agent-skill-stack` | Find, evaluate, and assemble the smallest compatible set of AI Agent Skills for an end-to-end natural-language goal. Use when a user wants Skills for a multi-step workflow, asks which Skills fit a project, needs an installed-Skill audit or conflict check, has low Skill recall, wants indirect helpers such as humanizers or compliance checks, or wants a project-specific Skill Stack with controlled installation. Search local Skills, registries, GitHub, and OpenCLI; compare adoption, verified fit, safety, and overlap. Do not use for locating one known or common Skill; use the generic find-skills workflow. | `agents/openai.yaml`
`references/discovery-ranking.md`
`references/local-index-and-profiles.md`
`references/security-installation.md`
`references/workflow-model.md`
`scripts/inventory_skills.py`
`scripts/project_profile.py`
`scripts/render_stack_card.py`
`scripts/skill_index.py`
`scripts/stage_install.py` | diff --git a/skills/agent-architecture/LICENSE.txt b/skills/agent-architecture/LICENSE.txt new file mode 100644 index 0000000000..8dd8213414 --- /dev/null +++ b/skills/agent-architecture/LICENSE.txt @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 Stanislav Bankovskii + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/skills/agent-architecture/SKILL.md b/skills/agent-architecture/SKILL.md new file mode 100644 index 0000000000..dbe4798c8f --- /dev/null +++ b/skills/agent-architecture/SKILL.md @@ -0,0 +1,64 @@ +--- +name: agent-architecture +description: 'Design AI agent architectures through requirements discovery, or audit and diagnose architectural flaws in existing agents. Architecture only; excludes implementation and general code review.' +license: MIT +--- + +# AI Agent Architecture + +Help the user obtain a justified architecture for their task or an evidence-based audit of an existing agent. Deliver architectural decisions and ways to verify them, without implementing the agent. By default, completed work includes a PDF report and a visualization of the results. An “ideal architecture” fits the requirements, cost of failure, and team resources; it does not maximize the number of components. + +## Choose a route + +| Request | Route | Read | +|---|---|---| +| New agent, requirements are not yet clear | Design: working cases → early design → requirements and decision coverage → delivery | [design.md](references/design.md), [architecture-contract.md](references/architecture-contract.md) | +| Architecture from an existing specification | Design: fill in what is known and clarify only gaps | The same files; do not restart the interview | +| Review an agent already written | Audit: reconstruct actual paths → verify → deliver findings | [audit.md](references/audit.md), and [architecture-contract.md](references/architecture-contract.md) as criteria | +| Agent makes mistakes, has degraded, or falsely reports “done” | Diagnosis within the audit: case → hypotheses → discriminating checks → correction and closure criterion | audit.md and [diagnostic-review.md](references/diagnostic-review.md) | +| Review and redesign | Audit first; its demonstrated problems become design inputs | audit.md first, then design.md | + +In either mode, read [source-map.md](references/source-map.md) once: it explains the origins of the principles and the textbook's limitations. The original PDF is not needed for ordinary skill use. [scenarios.md](references/scenarios.md) is needed only to test the skill itself. + +When choosing or revisiting the execution approach, use [architecture-selection.md](references/architecture-selection.md); when designing acceptance or reviewing quality claims, use [evaluation-design.md](references/evaluation-design.md). Develop the validation loop and completion evidence using [validation-loop.md](references/validation-loop.md); for long-running/background work, pauses, recovery, and competing sessions, use [execution-continuity.md](references/execution-continuity.md), including storage, RTO/RPO, budgets, the human decision queue, and scheduling. Develop delegation, mutable memory, execution isolation, and long-running/streaming interaction only when the task has these properties. A section's existence does not make its question mandatory: material gaps under discovery-protocol.md determine depth. + +## Shared decision rules + +- First read the available specification, local instructions, architectural decisions, and relevant materials. Use code to reconstruct architecture, not to make unsolicited fixes. Do not run an application with external effects for an audit. +- Maintain a brief register: **source-confirmed / user requirement / proposal / assumption / open question / not applicable**. Identify where requirements came from. A user decision and an architect's hypothesis have different statuses. +- Corporate contracts and accepted decisions apply only within their own project. The textbook is an engineering reference, not a source of authority or a replacement for local canon. Identify conflicts rather than resolving them silently. +- First consider ordinary automation without an LLM, a single call, and a predefined workflow. Introduce an agent loop, RAG, persistent memory, MCP, or multiple agents only for a concrete need. For each added complexity, identify its benefit, cost, verification method, and simpler alternative. +- Do not select a model or framework before understanding the task. For a concrete selection, check current official documentation and version constraints. A documented capability is not yet demonstrated quality on the user's data. +- Separate probabilistic model decisions from programmatically enforced rules. Describe where permissions, parameters, budget, and action admissibility are checked **before** an external effect, including bypass paths and resumption. +- For a timed-out external write, a readback that finds nothing does not by itself prove that no effect occurred. Permit a retry only under an established downstream idempotency contract or authoritative proof of non-execution; otherwise retain `effect unknown` and reconcile or escalate. Apply this rule in concrete flows and examples as well as in the risk section. +- An audit or design does not authorize writing code, changing agent settings, publishing, or initiating external actions. On a subsequent explicit implementation request, hand the architecture to the appropriate process; this skill does not continue into implementation itself. + +## How to work + +Before an interview or audit planning, read [discovery-protocol.md](references/discovery-protocol.md). Show a clear route and maintain a coverage map. By default, devote each turn to one decision or working episode; do not hide several independent topics inside one question. Material gaps and evidence determine depth. There is no fixed total round limit. + +Deliver the first useful design as soon as context is sufficient, otherwise no later than the third answer; the count does not reset on continuation. This limits the wait for an early result, not the completeness of the interview. If the task is too unclear, show a map of what is understood and conditional options. After the sketch, continue investigating material gaps under the protocol; two or three rounds alone do not justify declaring readiness. + +The first design includes the goal and boundaries, main capabilities and their outputs, recommended components, main flow and external actions, key constraints, assumptions, and open decisions. It is a sketch for early feedback. The interview budget limits the wait for a sketch, not design depth: develop it into an architecture package from what is already known, without waiting for a separate instruction to elaborate. If context suffices, deliver the package immediately. If the user explicitly asks only for a sketch, respect and label that depth. + +Phrases such as “that's enough,” “let's go with this for now,” “the rest later,” or “enough questions” end requirements gathering: deliver the architecture from accumulated context in the same answer. Do not require a separate “now design it” instruction or end at “interview complete.” If a design has already been delivered, show its current final version or a substantive update. An explicit request to stop all work (“don't continue,” “that's all for today, stop”) means stop, rather than deliver a new design. + +If the user does not know an answer, propose a justified option and label its status. Represent unknowns as assumptions and open decisions. Unclear authority blocks the corresponding external action in the proposed architecture, but not delivery of the architecture itself. Silence and ending the interview do not approve proposals. + +After a significant answer, update the working summary of requirements and decisions. Save it in an agreed document if artifact creation is within the request; otherwise maintain it in the conversation. On continuation, start with that summary and changed information. + +After the first design, clarify specific branches and uncovered material requirements, including real exceptions, human work, and feasibility. Explain which decision the answer will change; propose internal mechanisms yourself. Do not confine gap discovery to components already drawn or restart a questionnaire. Finish when the declared scope has sufficient coverage; if further confirmation is unavailable, deliver a conditional package with owners and checks for gaps. + +Complete design with the architecture package from architecture-contract.md: domain capabilities and methods, output contracts, the structure of instructions/skills/materials, allocation between the existing platform and additions, a populated end-to-end example, and checks. Read [capability-design.md](references/capability-design.md) for this part; in an audit, use it to check required capabilities. Describe the agent's main work deeply enough that a developer does not have to invent its method again. A platform name and a list of stages do not accomplish that. + +Always cover **limits on iterations, time, tokens/money, and tool calls, stopping rules, and what the user receives on stopping**. Mark unknown values as open or proposed rather than inventing an agreed limit. An architecture package with skill specifications remains a design: it does not imply skill installation, code implementation, or verification of a running agent. + +Complete an audit with demonstrated problems, separately identifying unknowns and accepted tradeoffs. Do not claim production readiness from reading code. Architectural readiness for implementation and demonstrated operational quality are different outcomes. + +## Final artifacts + +When completing design, audit, or diagnosis, read [result-delivery.md](references/result-delivery.md) and create a PDF of the results with a rendered Mermaid or C4 diagram as appropriate; retain editable text and diagram source. Do this as part of completion without a separate user request to “make the PDF now.” An early sketch and intermediate answers do not require repeated export. Explicit user constraints (“chat only,” “no files/PDF”) and a request to stop all work take precedence. Creating the report does not authorize implementing or changing the reviewed agent. + +## Package metadata + +This package is distributed under the [MIT license](LICENSE.txt). [Optional client metadata](agents/openai.yaml) supports compatible Agent Skills clients; Copilot uses SKILL.md and the linked references. diff --git a/skills/agent-architecture/agents/openai.yaml b/skills/agent-architecture/agents/openai.yaml new file mode 100644 index 0000000000..5dad08baf9 --- /dev/null +++ b/skills/agent-architecture/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "AI Agent Architecture" + short_description: "Interview, design, and audit agent architecture" + default_prompt: "Use $agent-architecture: help me design an AI agent architecture step by step for my requirements, or review an existing agent for design flaws, without writing code." diff --git a/skills/agent-architecture/references/architecture-contract.md b/skills/agent-architecture/references/architecture-contract.md new file mode 100644 index 0000000000..78fc9cfa51 --- /dev/null +++ b/skills/agent-architecture/references/architecture-contract.md @@ -0,0 +1,144 @@ +# Architecture Deliverable Contract + +These are views of a single design and may be combined into one Markdown document. Scale depth to the task: do not create empty sections or separate files without a reason. Each applicable item must have a decision, evidence, or an explicitly labeled open question. “Not applicable” requires a brief reason. + +## 1. Task and criteria + +Users, value, scenarios, output format, exceptions, existing alternative, and owners. Table: `R-ID | requirement | source and status | decision D-ID | check E-ID`. Identify critical unknowns and the chosen tradeoff among quality, time, cost, and autonomy. + +Use section 3 of [design.md](design.md) for the value card and human decision map. Show what change the agent provides, how the hypothesis is tested, and how much human involvement the entire process requires; successful execution and demonstrated impact are distinct. + +## 2. Structure and boundaries + +System type and reasons for choosing it. Components and responsibilities: interface, runtime/orchestration, model, domain rules, tools, sources and stores, checks, and observability. Do not draw nonexistent components. Show trust and data ownership boundaries. A simple Mermaid component diagram suffices when it helps explain relationships. + +Use the applicable sections of [architecture-selection.md](architecture-selection.md) for topology justification and delegation and temporal interaction contracts. Agent count does not replace a responsibility map. + +Domain substance is mandatory: map main capabilities, their methods, and outputs using [capability-design.md](capability-design.md). For selected skills, provide populated specifications and the material structure; for an existing platform, explain what is reused, configured, built, and still unverified. A simple topology does not reduce behavioral completeness. Rules may live in one instruction when separate skills are unnecessary. + +## 3. Execution, state, and stopping + +Describe the sequence: input → checks → context gathering → model/workflow decision → action validation → tool → result observation → state recording → response. Allocate the environment's four functions to existing or proposed owners: + +| Function | What must be defined and verifiable | +|---|---| +| Translation | How intent, constraints, and sources become a task and actually loaded context; what happens if a condition conflicts or is lost | +| Execution | Who converts the model's proposal into a typed call and enforces permissions, quotas, and isolation | +| Observation | How tool responses and external state become a verifiable result, event, and subsequent context; how error, success, and unknown are distinguished | +| Correction | Who chooses continuation, bounded retry, stopping, recovery, or escalation; what evidence and authority are required | + +This is a responsibility map, not four new services. For each function, show input/output, owner, failure, and check; a small assistant needs only a compact populated path. Deterministic boundaries do not make the model's domain answer deterministic or correct. + +Separate **integration** (available capabilities, schemas, and adapters from which a tool is selected) from **enforcement** (mandatory admissibility checks for every reachable call). These are logical responsibilities and may live in one process. An MCP/tool catalog does not itself enforce policy; the model does not choose whether to pass admission. Domain extensions do not weaken baseline restrictions without their owner's decision. Check bypasses through new tools/handlers, direct calls, and resumption; control details follow below. + +For tasks with pauses and effects, describe states, permitted transitions, the transition owner, persisted data, and resumption. Distinguish result proposed, action authorized, sent, confirmed, effect unknown, error, and canceled. “Canceled” does not mean an already completed effect has been rolled back. + +**A limits table is mandatory for loops or retries:** + +| Field | Record | +|---|---| +| Iterations and nesting | Whole-task and child-task limits; who enforces counting | +| Time | Overall deadline, individual call timeout, human wait, cancellation | +| Tokens and money | Task budget including reasoning, retries, and workers; user/tenant quotas where needed | +| Tools | Frequency/count limits; repeated identical arguments and lack of progress | +| Stop outcome | Partial result, explanation, human queue, or managed continuation | +| Calibration | Value/range, status “proposed/measured/agreed,” scenarios used to select the threshold | + +The runtime enforces limits. A prompt instruction may aid behavior but is not a limiting mechanism. Even a fixed workflow needs a timeout, bounded retries, and an error outcome; absence of an agent loop makes only the corresponding fields inapplicable. + +For a significant control, populate: `requirement → enforcement point → covered paths/tool classes → timing relative to effect → failure/timeout → replay/resume → cancellation/concurrency → check`. A guardrail name or SDK setting does not prove coverage of shell, hosted tools, and alternative handlers. A parallel check that finishes after an action does not enforce a mandatory pre-effect prohibition. Identify the mechanism closing each required path. + +Distinguish target spending, a limit allowing overshoot from an already started call, and hard admission of the next operation. Checking accumulated cost before a request can allow overshoot after completion. For a hard total limit, specify the admission/reservation owner, an upper-bound cost estimate, and coordination of remaining budget among concurrent workers; if an upper bound is unavailable, a hard cap is unproven. Do not allocate the same full budget to every worker. + +Recover state from a verifiable store, not conversational promises. A failure between an external action and local recording requires a reconciliation path; a checkpoint alone does not ensure a single external effect. + +Clarify when a write becomes durable: before continuation, asynchronously, or at completion; what progress a crash loses and which steps replay. Checkpoint properties and external API guarantees are different contracts. Test the “effect completed, local write incomplete” gap and stale permissions during recovery. + +For long-running tasks, pauses, and work across sessions, use [execution-continuity.md](execution-continuity.md): checkpoint/handoff, storage tiers, supervisor with recovery objectives, budgets across horizons, asynchronous human participation, and competing-task scheduling. Show a populated normal transition and material failure, applicability, and existing platform mechanisms. These are environment responsibilities, not a requirement for separate services or files. + +## 4. Context, knowledge, and memory + +- Source of each context portion, reader permissions, freshness, link/version, and behavior on conflict or absence. +- Budgets: system instructions, history, tool definitions, results, retrieval, reserve for the response and required reasoning. Strategies for selection, moving large data out of context, compression, and on-demand loading; measurements on real formats and languages. +- Compaction preserves goals, constraints, primary-source links, confirmations, and effect identifiers. Authorization and effect state remain in the runtime outside compressed text. Test continuation after compaction. +- If RAG is needed: corpus preparation and updates, structure-aware chunking, search methods, reranking conditions, generation context, citations, empty results, and stale documents. Check ACLs before data reaches the model; indexes and caches account for deletion and revoked access. +- If memory is needed: working, episodic, semantic, procedural—purpose, user/tenant scope, source, timestamp/lifetime, reads/writes, concurrency, correction/deletion, and fact verification. Do not require all four types. +- Separate authoritative state, raw evidence, and derived summaries. Verify summaries against sources; an existing authored wiki does not automatically become “just a cache.” Mandatory recording of a financial/business effect is not deferred as optional background memory. + +If the agent changes memory or procedures, describe `observation → candidate → verified version → active procedure`: source, applicability, freshness, conflict resolution, change authority, validation, and rollback. Saved, indexed, available for selection, actually loaded, and behavior-verified are different states. Untrusted experience or one successful run does not automatically become a rule. Admission may be performed by a human or a previously agreed policy within its scope; universal manual approval of every memory is not required. Behavioral change criteria are in [evaluation-design.md](evaluation-design.md). + +## 5. Tool contracts and authority + +For each significant tool: purpose and when to use it, input/output schema and field meanings, preconditions, domain restrictions, effect type, principal/tenant permissions, errors, timeout, retry admissibility, result limits, and effect confirmation. + +Check the selection interface: can neighboring tools be distinguished by names/descriptions, are IDs, units, and argument values unambiguous, and are errors and allowed next steps clear? Show a populated call and response for a significant tool, a positive selection example, and a case where it must not be selected. A separate wrapper for every API endpoint is not mandatory; a domain-level aggregate operation is acceptable if it does not hide effects and required checks. Results support filtering/pagination/truncation indicators and preserve required identifiers, provenance, and completeness. Concision must not make partial data appear complete. + +For writes, also specify: operation identifier, receiving system's idempotency guarantee and validity period, retry and ambiguous-response semantics, readback/reconciliation, compensation or irreversibility. “There is an idempotency key” is insufficient without a server processing contract. Identical parameters for two deliberately distinct operations do not make them one request. + +For human confirmation: exact action, target, parameters, principal, and authorization expiry; behavior on changed parameters, revoked rights, reuse, and resumption. Without permission, a significant action is not executed. An agreed autonomy policy may allow certain action classes; the model does not extend it itself. + +Represent contextual admission as `action → resource/environment → principal/tenant → permissions and conditions → checking mechanism → allow/deny/escalate`. Test changes of resource, tenant, parameters, and operation scale within a task. An autonomy level does not replace this table: rights are not calculated by subtracting abstract points and do not increase with model confidence. Automatic transitions are possible only within a previously accepted policy. + +## 6. Threats and failures + +Brief table: `scenario/asset → trust boundary → control mechanism → check → residual risk/owner`. + +Check untrusted instructions in messages, documents, tool metadata, and results; cross-tenant access; leaks through responses/logs/tools; excessive privileges; parameter substitution after approval; memory poisoning; invalid model output. An injection classifier does not replace authorization and isolation or prove attacks absent. + +If code execution, shell, browser, or untrusted extensions exist, show the actual isolation boundary: agent process, tools inside/outside, network, filesystem, credentials, tenants, and artifact return. A container for one tool does not isolate host tools; a virtual filesystem and import filter do not prove a security sandbox. Compare restricted local execution, a remote environment, or whole-process isolation according to required risk, without demanding a container for a text assistant. The owning API retains its own authorization. Unknown configuration is an evidence gap, not a demonstrated exploit. + +For every dependency: failure, timeout, rate limit, invalid/empty response, retry, and recovery. Choose blocking, acceptable degradation, or escalation per operation. “Read-only” does not permit exposing someone else's data if ACL checks fail. Fallback does not bypass access, confirmation, or data residency requirements. + +For long-running and concurrent tasks: duplicate delivery, two handlers for one task, backpressure, user stop, approval expiry, and crash after an external effect. Queues, circuit breakers, and regional redundancy are needed where requirements justify their cost. + +## 7. Quality verification, economics, and operations + +For a significant failure scenario, analyze using [diagnostic-review.md](diagnostic-review.md): where failure becomes visible, which causes must be distinguished, and which signal prevents false success. In a new design, this tests an architectural hypothesis; reviewing a running agent requires actual evidence. Do not label a design behavior-verified from a thought experiment. + +Evaluation plan: real authorized/de-identified scenarios, domain-owner labels, ordinary and difficult cases, no answer, conflicting sources, unsafe input, failures, and resumption. Separate tuning data from independent validation. Measure correctness of outputs and actions, groundedness/retrieval quality where applicable, false blocks, escalation rate, latency, and cost per accepted task. A model judge is a measurement instrument with errors, not proof. + +Develop the evaluation contract, mandatory gates, coverage, and measurement errors using [evaluation-design.md](evaluation-design.md). Do not collapse separate quality axes into an average that hides mandatory constraint violations. + +Use [validation-loop.md](validation-loop.md) for the validation loop and Evidence Bundle. Define applicability, placement, and verdict consequences for the seven functions; separate services are not mandatory. For a material conclusion, link the requirement, output version, executed check, and evidence. If a Guardian/Reviewer exists, describe its own contract and errors; post-effect observation is not a pre-effect prohibition. + +Record model, prompt, tool schema, corpus, and surrounding runtime versions for comparable runs. A change in one layer requires retesting affected scenarios. Do not require the model's internal chain of thought: investigations need available events, inputs/outputs with secrets redacted, policy decisions, and action results. + +Cost calculations include every LLM call and retry, separately billed tokens without double-counting, retrieval and external APIs, infrastructure, and support. State assumptions and a range by task length/branching; do not present one-call cost as completed-task cost. + +Observability: correlated task/run/tool/action IDs, stages and durations, budgets and stop reasons, tool-selection quality, successful/empty/failed results, source versions, and actual action outcome. Exclude secrets; define trace access, retention, and deletion. For each important metric, say what decision or response it triggers. + +Operational plan: owner, controlled pilot, admission and rollback criteria, stopping new actions, recovery, and incident analysis. This is a proposed plan, not a completed release. + +**Composite version and lifecycle.** Link model, prompts/skills, tools, policies, runtime, and material data into an identifiable agent version. For a change, identify affected scenarios, admission, compatibility of continuing tasks, and rollback. Reverting code/prompts does not undo external effects or guarantee compatibility of new state with an old version. Diagnose drift first; do not automatically prescribe fine-tuning. + +For an agent with persistent state/background actions, define retirement: owner and trigger → prohibit new runs, schedules, and redeliveries → finish/cancel/transfer active tasks → reconcile initiated effects → revoke access → handle memory, artifacts, and logs under applicable rules → replacement process and evidence of shutdown. Disabling an HTTP endpoint does not stop cron, queues, or running workers. Access revocation order must preserve authorized reconciliation without permitting new effects; if safe reconciliation is impossible, explicitly hand it to an owner. Test a late event after disable. A one-off local assistant needs only applicable update and stop rules. + +Use [evaluation-design.md](evaluation-design.md) for the improvement cycle, protected invariants, comparison budget, and accept/rollback rule. For skills, test selection of the right procedure and cases where it is unnecessary. Structural file validation does not replace behavioral testing. + +## 8. Decisions and readiness + +Coverage and independent statuses of knowledge, agreement, and handoff are defined in [discovery-protocol.md](discovery-protocol.md). Before assigning “for implementation,” check the first-scope deadline, build/support availability, budget constraints, and technical dependencies. An unknown with an assigned owner is not closed. A proposed implementation can be substantive despite blockers, but they limit readiness. + +### Human path and significant settings + +Include a populated user/operator path: **event → where the person learns about it → what they see → available action and rights → what changes → how to continue → what happens without a response**. For asynchronous work, show concrete queue rows, owners, and a notification/response example, not just internal statuses. An existing interface suffices; a new dashboard requires justification. + +For example, in a synthetic Jira process: `CAT-42 | awaiting file | manager | reason: two CSVs | select the exact attachment with a comment command | after checking authority and version, requeue`. Separately for a technical failure: `CAT-43 | validation failed | Pipeline owner | safe-log link | fix the cause and explicitly retry`; for a closed Jira issue, define an accessible summary surface rather than promising an invisible internal queue. An unassigned owner or unavailable channel is an open question. Do not present this example as a confirmed requirement of another project. + +For significant numerical decisions, provide `value/range | status and basis | user consequence | alternative | calibration method and decision owner`. Do not tabulate every internal constant. For example, “10 retries per day” with 50 failures means at least five days to clear the backlog even without new failures; compare that with the deadline. Consider a shared budget prioritizing corrected tasks or another mode instead of choosing a number without consequences. Unknown duration/cost requires measurement, not a throughput promise. Pilot timeframe and available people are constraints separate from the runtime budget. + +Key decisions: `D-ID | context | choice | alternatives | consequences | reconsideration trigger`. + +Check substantive readiness before completion: + +- Each main scenario has a capability, concrete method, required data, and output contract; selected skills are described substantively, not merely named. +- Instruction/material structure explains what loads and why; platform contribution is separate from proposed additions. +- One ordinary and one significant failure case are worked through with populated data through the user response; result evidence and required human judgment are visible. +- Requirements are traceable, execution/failure paths agree, and critical boundaries/contracts, limits, and checks are defined. +- A developer can implement the agreed behavior without new domain design. A component list, empty templates, or “the LLM analyzes” without a method leaves the result a sketch. + +Open numerical settings or experiments are acceptable when their impact and closure method are clear. An unknown core method, authority, or basic topology is a specific readiness gap; deliver the completed package with that limitation rather than automatically restarting a questionnaire. Separately state result depth (sketch / package with limitations / package for implementation), user agreement, and what was actually measured. Package completeness, its approval, and operational quality are distinct properties. + +## 9. Final delivery format + +Check substantive architecture readiness and export readiness separately. Standard final delivery is a PDF with a rendered Mermaid/C4 diagram, editable text, and diagram source under [result-delivery.md](result-delivery.md). That file also defines user-requested exceptions and unavailable-export behavior. The report preserves actual architectural status and does not turn a design into a confirmed implementation. diff --git a/skills/agent-architecture/references/architecture-selection.md b/skills/agent-architecture/references/architecture-selection.md new file mode 100644 index 0000000000..c85674f87b --- /dev/null +++ b/skills/agent-architecture/references/architecture-selection.md @@ -0,0 +1,58 @@ +# Architecture Selection and Interaction Contract + +Read when choosing or revisiting the execution approach. Apply delegation and long-running interaction sections only to relevant tasks. Build the decision card from available context; resolve material unknowns under [discovery-protocol.md](discovery-protocol.md) without turning it into a questionnaire. + +## 1. Choose by task properties + +| Property | Selection consequence | +|---|---| +| Inputs and rules are defined | Ordinary automation may suffice; an LLM is needed only for ambiguous interpretation | +| All data is available before answering | Consider one call with output validation | +| Order and permitted transitions are known | A predefined workflow with necessary LLM steps | +| New observations determine the next step | A bounded agent loop; identify the uncertainty it resolves | +| Independent workstreams or required context/permission isolation exist | Compare one agent with delegation; sequential dependencies limit parallelism's benefit | +| Shared mutable state and external effects exist | Assign change and integration owners; more workers increase coordination cost | +| Long work, pauses, events, or streams | An interaction and recovery contract is needed, not automatically another agent | +| Team, environment, or model capabilities are constrained | Account for the existing runtime, available tools, and support; demonstrate model suitability on the task | + +For a material decision, compare the baseline and one or two plausible alternatives: `option → requirements covered → expected benefit → cost/risks → why selected or rejected → check → reconsideration trigger`. An obvious small task needs only a brief rationale, without an artificial table. If the user specified a topology as a requirement, preserve it and show tradeoffs. + +Compare topologies on the same tasks, data access, criteria, and **aggregate** budget for every worker, retry, and integration step. An equal per-worker limit does not mean an equal system budget. Account for final output quality, latency, cost per accepted task, and human labor. Without measurements, benefit remains a hypothesis; deliver a conditional architecture and comparison plan now. An experiment is not a prerequisite for the first design. Do not import scaling thresholds from someone else's benchmark. + +### From functions to agent boundaries + +First derive functions from working cases, with input, output, method, state, owner, and permissions. Then choose an execution form, and only afterward boundaries of persistent components. A profession name or separate API does not itself create an agent boundary. + +| Form | Sufficient selection rationale | +|---|---| +| Deterministic process | Rules define transitions; an individual LLM step does not require model control of the whole process | +| Tool | A bounded operation with a contract and permissions; its caller owns the goal and result acceptance | +| Skill/instruction | A repeatable domain method with specified input, context, steps, and verifiable result; an instruction alone does not require a separate runtime | +| Temporary subagent | A bounded subtask; separate context, parallelism, or permissions provide a verifiable benefit; the parent accepts and integrates the result | +| Persistent agent/profile | Independent responsibility and a justified execution boundary; explicit state and lifecycle owners, reason for isolation, and coordination cost | + +For a persistent component, check a stable output/decision class, state between runs, independent start/pause/recovery, and separation rationale: permissions, data, context, load, or failure domain. These are decision factors, not a universal formula: isolation or scaling can justify a stateless worker, and a shared owning service may own state. Unknowns do not prove complexity necessary. Identify a simpler alternative and a condition for merging components again. + +Coordination is a separate decision: who owns the overall result, dependencies, resources, and conflicts? An ordinary scheduler/router handles fixed transitions. Justify a model coordinator by an actual need for contextual selection and validation of its decisions. Multiple profiles do not automatically need an orchestrator; an agent planner may select tools without persistent child profiles. Do not generalize platform definitions into universal prohibitions. + +Select the management interface and storage separately, based on human work and data. First check capabilities of the existing chat, Jira, CLI, or API. Justify a new dashboard by a concrete need for a shared queue, structured confirmations, search/comparison, or different permissions. Choose storage for durability, concurrent writes, transactions, search, retention, and available support; files/SQLite/PostgreSQL are options, not a mandatory ladder. Semantic search needs its own task, not merely the presence of agents. + +## 2. If delegation exists + +Populate one real handoff contract, then identify other roles' differences: + +`owner/recipient → goal and boundaries → inputs, versions, and permitted sources → required artifact and readiness criterion → permissions → share of total budget/deadline → stopping and partial output → evidence/provenance → validation and integration`. + +Separately identify the owner of the final decision and shared resources. Define who resolves conflicting results, prevents duplicate effects, and accepts partial results. Delegating specific work and unrestricted peer-to-peer messaging have different coordination rules; do not substitute one for the other. A2A, a queue, a graph, and a supervisor service are not mandatory. + +Check transfer of constraints, recursion and nested budgets, lost/duplicate delivery, conflicting shared writes, expiry, and late results after cancellation. Trace `assigned → accepted → completed/partial/rejected → verified → integrated` in one example. Child authority does not exceed what the parent was granted. Several identical model conclusions from shared context are not independent evidence; the integrator checks grounds and contradictions, not just vote counts. + +## 3. If interaction continues over time + +Choose according to need: synchronous response, background task, event/schedule, streaming, or voice. Describe observable behavior: task acceptance, progress, waiting, partial result, error, cancellation, and final result. Without background work, the corresponding states are inapplicable. + +Link task, conversation, individual run, and external action identities. Resume may create a new run while retaining links to the task and prior effects; verify specific SDK fields in current documentation. Define handling of late and duplicate events. + +When the user changes the goal, mark the boundary between old and new intent: which unstarted actions stop, which were already sent and require reconciliation, what partial result is retained, and which parameters/permissions must be recalculated. Cancellation does not undo completed actions and must not allow new effects under an obsolete goal. Do not request still-valid permission again without a scope change. + +Streamed text before validation is provisional. If a requirement prohibits disclosing certain data, a final post-streaming validator is too late: control is required before releasing those fragments. Show normal completion and one cancellation/correction scenario, not merely a state list. diff --git a/skills/agent-architecture/references/audit.md b/skills/agent-architecture/references/audit.md new file mode 100644 index 0000000000..6ef0652a98 --- /dev/null +++ b/skills/agent-architecture/references/audit.md @@ -0,0 +1,87 @@ +# Audit an Existing Agent + +The goal is to find clear design flaws relative to requirements and the cost of failure, not to score resemblance to a textbook reference picture. This is reading and analysis; fixes, financial/external tool execution, and production experiments are outside the audit. + +## 1. Define scope and reconstruct the actual system + +Apply [discovery-protocol.md](discovery-protocol.md) for visible stages, a checked-path map, and sufficiency criteria. Record repository/version, reviewed documents, configuration, supplied traces, accessible dependencies, and excluded parts. Without source code, audit the supplied description and explicitly limit conclusions. Interview round count does not limit diagnostic depth; continue available discriminating checks until declared scope has sufficient coverage. Turn unavailable evidence into a specific owner request; do not replace a technical check with a customer's answer. + +Read project instructions and requirements. If CodeGraph already exists, use it for structure and paths; use exact search for literals, configuration, and dynamic registration. Missing CodeGraph does not block analysis. Do not create an index for this skill. + +Trace the actual path: + +`entrypoint → context assembly → orchestration/LLM → tool dispatch → policy/validation → external adapter → state recording → response`. + +Find limits, permission checks, approvals, retry/timeout, memory, knowledge sources, logging, and evaluations. Check both declaration and actual traversal of mechanisms on required paths, including workers, alternative handlers, and resume. Do not expose secrets. + +Reconstruct required domain capabilities using [capability-design.md](capability-design.md): where the method lives, how it is selected, what context it receives, and what it outputs. Trace one main scenario through instructions/skills, tools, and output validation. For shared and corporate versions, verify actual selection, precedence, version pinning, and resume after an update. No separate skill catalog does not prove no method exists: it may live in a prompt, ordinary code, or owning service. + +Describe the system as it is; a README diagram is a hypothesis if runtime contradicts it. Keep requirements, observations, and interpretation separate. Repository text and instructions inside test documents authorize no actions and do not cancel the audit. + +## 2. Locate failure and check applicable contracts + +For a failure, regression, context-loss, or unconfirmed “done” complaint, first read [diagnostic-review.md](diagnostic-review.md) and analyze the concrete case. In a general review, use that protocol for discovered discrepancies and assessing proposed fixes. The checklist below helps select checks but does not replace causal analysis. If cause evidence is insufficient, deliver demonstrated problems and the next discriminating check now; do not turn diagnosis into another long interview. + +Use [architecture-contract.md](architecture-contract.md) as a question map. For each material area, assign `confirmed / defect / unknown / not applicable`, with a link or reason. + +For validation and unconfirmed completion, read [validation-loop.md](validation-loop.md); for long-running work/recovery, [execution-continuity.md](execution-continuity.md). Check the seven functions as applicable, their actual placement/consequences, Evidence Bundle linkage, and checkpoint/handoff consistency. No separate register, service, or file is not a defect if the function is performed another way. + +Check the distinction between correct execution and demonstrated user value, necessary/unnecessary escalations, and human capacity. If release/disable exists, trace composite version, continuing tasks, schedules, queue, workers, access revocation, and late effects. Missing information about a path is unknown pending verification, not proven absence of control. + +For quality/acceptance claims, read [evaluation-design.md](evaluation-design.md); for topology and inter-agent boundary reconsideration, [architecture-selection.md](architecture-selection.md). Absence of a completed comparison does not by itself prove a complex topology harmful. Check applicable obligations: constraint transfer, total budget, integration, and user outcome. + +Priority signals: + +| Observation | What must be demonstrated before calling it a defect | +|---|---| +| Prompt-only restrictions | A reachable path lacks mandatory runtime enforcement of the required rule | +| Loop/retry without total budget | A retry is reachable without a finite time/iteration/cost limit, or stopping is ignored | +| Validator after tool dispatch | An invalid/unauthorized action can occur before checking | +| Write retry after timeout | Duplicate effect is possible; check downstream contract, SDK retries, and idempotency | +| Agent says “done” | No reliable basis confirms the effect; intention/HTTP 200 substitutes for actual state | +| Checkpoint exists, recovery is unsafe | Resume repeats an effect or uses stale permission, state, or tenant | +| Shared memory/index | Required mandatory scope, freshness, or provenance restrictions on reads/writes are absent | +| Secret/other party's data in context | A reachable disclosure path and relevant access boundary are established | +| RAG returns stale/other party's data | Source, ACL, revocation, refresh, or cache fails required correctness | +| Only final answers in observed trace | Other logs/services lack information needed for investigation; unknown does not mean absent | +| “99% success” from demo/evaluation | Metric mismatches the requirement or sample/denominator/environment does not support the claim | +| Many agents/layers | Concrete harm exists: conflicting writes/rights, context loss, latency/cost over budget, not merely many files | +| Only a skeleton and workflow stages | A required domain capability is not performed on a reachable path; no other component supplies the method/control. Separate incomplete documentation from an implementation defect | +| Installed skills do not change results | The required procedure is not selected, unavailable, or receives wrong context in a concrete scenario; files alone are insufficient | +| Method unexpectedly changes during a task | Version substitution on resume/update violates the accepted contract; check authorized migration and revoked rights | +| Checks green, main work incomplete | Checks accept a concrete unacceptable result; CI/linter presence proves neither quality nor mandatory enforcement | + +Failure to find a guarantee in the inspected fragment is an **evidence gap**. For example: “POST retry after timeout confirmed; billing guarantee unavailable; duplication risk needs checking.” Do not call this a demonstrated double refund. If absent guarantees themselves violate a mandatory admission contract, identify that documentation defect and its source separately. + +Absence of RAG, MCP, a framework, vector database, four memory types, microservices, or multiple agents is not a defect. Complexity requires demonstrated need. Enforcement inside a tool or owning API is acceptable when it cannot be bypassed; form alone does not require a separate service. + +## 3. Form evidence-based findings + +Each finding contains: + +1. **Priority and short title.** Critical—reachable critical harm/boundary violation; Important—mandatory scenario violation or substantial correctness/recovery/cost risk; Improvement—bounded improvement without a demonstrated requirement violation. +2. **Observation:** exact file/lines/symbol/version, document, or trace event. Do not invent lines for unavailable code. +3. **Condition:** concrete input or failure and reachable path to the problem. +4. **Consequence:** violated need, guarantee, or invariant. +5. **Mechanism:** why existing control fails to prevent it. Identify checked compensating mechanisms. Separate error origin, propagation, and the check that missed it; if cause is unknown, keep competing hypotheses rather than assigning cause from symptom. +6. **Minimal architectural correction** and **closure check**, without implementation. +7. **Confidence and limits:** statically demonstrated path / safely reproduced / hypothesis with missing evidence. + +The textbook explains principles; source code and requirements demonstrate defects in a particular agent. Do not cite the textbook instead of the problematic path. Do not increase severity because information is missing. + +## 4. Deliver and stop + +Lead with the most important conclusion and review boundaries. Then provide: + +- a brief actual architecture; +- Critical/Important findings ordered by impact; +- evidence gaps and specific minimum reading/testing to close them; +- for diagnosis, a brief failure chain and hypotheses with discriminating checks/status; detail only significant unresolved causes; +- acceptable tradeoffs and Improvements if useful to the user; +- a prioritized architectural correction plan with closure criteria. + +If no clear errors are found, say so **within the checked scope**. Do not invent a finding for the report or call the agent ideal/safe under all conditions. Reading does not replace runtime evaluations or security tests. + +If fixes are later supplied, check closure of previous findings and changed paths under diagnostic-review.md's review section. Do not automatically start another broad audit or change code in response to “review.” + +Before completion, prepare artifacts under [result-delivery.md](result-delivery.md): PDF with findings/evidence, actual path/failure diagram, editable text, and diagram source. Distinguish proposed corrections from existing implementation on the diagram. An unknown cause remains unknown in the PDF; export needs no new interview round. diff --git a/skills/agent-architecture/references/capability-design.md b/skills/agent-architecture/references/capability-design.md new file mode 100644 index 0000000000..4e3c386215 --- /dev/null +++ b/skills/agent-architecture/references/capability-design.md @@ -0,0 +1,62 @@ +# Domain Capabilities and Skill Structure + +Read when designing domain substance or checking that an existing agent performs the required work. This describes specifications, not skill installation or application implementation. + +## 1. From need to capability + +A capability is observable work with a domain output: for example, preparing a testable hypothesis or handling a request under current policy. An “analysis” stage alone explains neither method nor output. + +Build a compact map: `requirement/scenario → capability → input and source → method/rules → output → check → executor`. Cover main scenarios, rather than giving every technical action its own row. + +Allocate responsibility by meaning: + +| Mechanism | Assign to it | +|---|---| +| Model | Interpretation and synthesis with defined grounds, boundaries, and quality criteria | +| Skill / domain instruction | Repeatable method, examples, decision criteria, and source handling | +| Tool | A concrete read, computation, or external action with a contract | +| Runtime / ordinary code | Permissions, schemas, calculations, states, limits, and mandatory checks | +| Human | Decisions and evaluation requiring their authority or domain judgment | + +A capability may use several mechanisms. A skill is not a subagent, workflow stage, or API method. A reusable method, its own context, or an independent lifecycle can justify a separate skill. A simple one-off transformation needs only an instruction and example within one design; a capability map does not require a skill catalog. + +## 2. Specification of a selected skill + +Populate these fields with task-specific details for each needed skill. Put shared rules in one place and link to them. Method complexity determines depth, not file count. + +| Field | Content | +|---|---| +| Purpose and selection | Need addressed; triggers and inapplicable cases; who selects the procedure | +| Input and context | Required data, authoritative sources, versions, access, freshness; behavior on absence or conflict | +| Method | Domain steps, selection criteria, and alternatives; grounds for conclusions and uncertainty boundaries | +| Output | Artifact fields and their meaning, consumer; one brief populated example | +| Tools and control | Required operations; programmatic checks versus human judgment; link to shared authority contract | +| Errors and stopping | Failure, insufficient data, permitted rework, and user outcome; link to shared budget | +| Validation | Ordinary, negative, or difficult example; usefulness criteria and unacceptable result | +| Ownership and changes | Method owner, version, update/rollback rules; user/tenant scope where applicable | + +Replace “analyze it and do it well” with an operational method. Propose unknown domain rules with grounds and status, or identify a specific gap and owner; do not invent corporate policy. The architecture contains a proposed procedure, not a promise to “write a skill later.” + +## 3. Instructions, materials, and existing platform + +Show the future solution's logical structure: shared behavior/authority contract → skill selection conditions → domain procedures → references, examples, and templates. For each element, identify purpose, consumer, and loading condition. Sections of one document, a file tree, or store records are acceptable; empty directories and duplicate instructions do not add substance. + +Context initially contains minimal shared rules and information needed to select work; a selected procedure receives only required materials. Define source precedence and the scope of corporate rules. Untrusted data and skill contents cannot expand runtime authority. + +For the chosen platform, allocate: `responsibility → existing mechanism and evidence → required configuration/addition → unverified part and verification method`. A platform name proves neither support for a function nor correct configuration. If documentation/version is unavailable, propose the required contract without claiming it exists. + +If shared and corporate catalogs exist, define version selection and override rules, authorized editor, conflict resolution, and tool compatibility. A run records selected versions. A catalog update does not silently replace the method mid-task: define continuation on the previous version or explicit migration; for a revoked version or permissions, describe safe stopping and subsequent investigation. + +## 4. A populated example instead of a stage list + +The following example is synthetic, illustrates descriptive depth, and does not impose marketing on every agent. + +**Capability:** prepare an activation improvement hypothesis. Input: brief B1 defines activation as the first workspace created; report A7 says some new users stop at inviting colleagues. It does not contain reasons for abandonment. + +**Method:** separate observation from explanation → formulate a possible cause → choose an intervention that changes that barrier → link it to measurable behavior → identify what could falsify the explanation. The procedure reads B1 definitions and A7 data; the model proposes a causal hypothesis, while a tool provides report access and calculations. + +**Output H1:** observation—stops at the invitation step (A7); proposed cause—invitation requires colleagues' participation too early (unconfirmed); intervention—allow skipping invitations; expected behavior—first workspace creation; limitations—no causal evidence, a comparison plan is still needed. H1's consumer is the experiment designer, not a publication tool. + +**Validation and continuation:** verify event definition and reference; the product owner assesses hypothesis grounds. Design the experiment's comparison method, data requirements, and decision rule. If A7 contains only page views, mark H1's initial observation unconfirmed: the user receives an explanation and needed data, not a claim that the barrier is established or growth achieved. + +For your project, work through the entire main scenario with similarly populated intermediate artifacts and show the final user outcome. This is a tabletop walkthrough, not evidence of a completed experiment. Completed-package conditions are in [architecture-contract.md](architecture-contract.md). diff --git a/skills/agent-architecture/references/design.md b/skills/agent-architecture/references/design.md new file mode 100644 index 0000000000..bd8fad8c9c --- /dev/null +++ b/skills/agent-architecture/references/design.md @@ -0,0 +1,90 @@ +# Step-by-Step Design + +## 1. Understand the task + +Read supplied documents and available context. First record: + +- who uses the system, what work they do today, and what should improve; +- known constraints, accepted decisions, and the business-rule owner; +- what already exists, what data is available, and where the source of truth is; +- unknowns that change safety, boundaries, cost, or execution approach. + +If almost no request detail exists, start with one real case: “Who comes to the agent, with what, and what result do they consider successfully completed work?” Do not start by choosing a database, model, or tool list. + +## 2. Conduct an adaptive interview + +Follow [discovery-protocol.md](discovery-protocol.md): visible stages, one substantive question, a coverage map, and separate readiness statuses. The following table is a reference for choosing the next material gap, not a queue of mandatory questions. Begin with the desired change and latest real case; an early design does not end investigation. A topic may be “not applicable” with a reason. + +| Topic | What to discover and helpful questions | Decision depending on the answer | +|---|---|---| +| User and value | Who is the user, customer, and rule owner? How is the task solved now? Where are losses? What changes after adoption? | Whether AI is needed at all, product boundaries, baseline | +| Scenarios and output | Work through real ordinary, difficult, and prohibited cases. What is the input? What is a good output? Who accepts it? | Output contract and acceptance scenarios | +| Errors and priorities | Which errors are unpleasant versus unacceptable? In conflict, what matters more: accuracy, completeness, speed, cost, autonomy? | Quality criteria, escalation, control level | +| Domain and boundaries | Terms, entities, lifecycle, rules, legitimate exceptions. Which neighboring tasks are excluded? Who changes rules? | Domain invariants, owners, dependencies | +| Authority | Read, propose, save a draft, or execute? On whose behalf? Which actions need a human, and what exactly do they confirm? | Action matrix and authorization policy | +| Human participation | Who is available, when, how quickly? How to cancel, correct, reject, continue? What happens overnight or on approval expiry? | Real rather than nominal escalation | +| Context and knowledge | Which data is mandatory for a decision? Where is it, who owns it, how often does it change? Which sources take precedence and how are conflicts handled? | Direct API, search, RAG, context assembly | +| Examples and data rights | Are authorized examples of inputs, answers, and wrong answers available? Labels, de-identification, languages, formats, volume, access restrictions? | Testable hypotheses, corpus, evaluation set | +| Memory | What must be remembered within a task, across sessions, across users? Who may write, correct, delete? How to distinguish opinion from fact? | Need and lifecycle of each memory type | +| Tools | Which systems/operations are actually needed? API contracts, owners, errors, limits, permissible retries, effect verification? | Adapter and tool boundaries | +| Load and interaction | Chat, event, schedule, batch? Peak concurrency, task duration, streaming, waits, redelivery? | Sync/async, queue, state, backpressure | +| Nonfunctional requirements | Target latency/availability, budget per accepted task and period, quotas, expected growth? What can be simplified? | Limits, capacity, economics | +| Confidentiality and trust | Whose data, which tenants, who can see what? External documents, files, personal data, allowed providers/regions, retention/deletion? | Trust boundaries, isolation, secrets, leaks | +| Operations | Who supports, investigates, rolls back? Team skills and existing stack? How to change model, rules, tools, sources? | Solution size, observability, release | +| First-scope feasibility | Pilot deadline, people actually available to build/support, permissible spending? Clarify separately by impact; route “unknown” to an owner | Pilot scope, available dependencies, resource-validation plan, handoff blockers | + +After an answer, first determine whether to deliver or update the design under SKILL.md. If interviewing is still needed, briefly state what is now known and which decision the next question unlocks. Do not require technical terminology from the user: learn desired behavior and translate it into architecture. Do not transfer internal mechanism selection to the customer when the architect can make a justified proposal. + +Requirements register: `ID | statement | source/example | priority | acceptance criterion | status`. For an open question, add impact, answer owner, and verification method. Do not make every question a separate document. + +## 3. Form testable requirements + +This and subsequent sections describe the architect's work on the deliverable. Do not make the user fill them in for you. Clarify material gaps using discovery-protocol.md; independently develop technical proposals and verify available sources. + +Separate mandatory from desirable. A small agent needs a few end-to-end criteria; a complex project needs a compact set with measurable outcomes. + +Develop the acceptance contract using [evaluation-design.md](evaluation-design.md), including mandatory gates and measurement coverage. + +Populate a value card: `problem → baseline → expected change → measurement → observation period → owner → reconsideration condition`. Value may be nonfinancial. Link it to domain capabilities; correctly completed work, demonstrated benefit, and insufficient evidence to evaluate benefit are different outcomes. State what would falsify the hypothesis and which decision follows under the accepted policy; unconfirmed benefit does not require automatic rollback. Do not turn the card into another questionnaire or mandatory PRFAQ. + +Example: the agent prepared a correct report, MAU grew 10%, but prices and acquisition channels also changed. The calculation is confirmed; the experiment's causal contribution is not established. A suitable comparison method and observation period are needed; a negative or inconclusive research result can be a useful task outcome. + +For significant human decisions, map: `decision → owner → evidence/options → exact confirmation → wait deadline → no-response behavior`. Estimate queue volume and available human time. If the agent prepares 100 items and the owner has 30 minutes, propose prioritization, incoming-flow limits, or permissible batch review of specific versions; faster generation does not imply a faster whole process. Evaluate necessary and unnecessary escalation quality under evaluation-design.md. + +Decompose “99% quality”: success definition, denominator, allowable errors, categories/languages, automation proportion, labeler, dataset, and measurement period. Accuracy on answered requests alone can hide widespread refusal. Model confidence self-assessment does not replace a calibrated criterion. + +Separate: + +- domain rules checked against authoritative state, with explicit exceptions; +- heuristics and quality judgments requiring examples, measurements, and error tolerance; +- unconfirmed targets that remain proposals. + +Do not portray a proposed metric, budget, or SLA as a user requirement. When information is insufficient, explain what can be designed conditionally and what cannot be considered implementation-ready. + +## 4. Compare architectural options + +Read [architecture-selection.md](architecture-selection.md) and populate a task-proportionate selection card: baseline, plausible alternatives, benefit/cost, validation, reconsideration condition. Develop delegation and long-running interaction contracts only when present. + +Specific branches: + +- **Knowledge:** request data, direct API, full-text search, RAG; behavioral fine-tuning does not replace fresh authoritative facts. A vector database is not an admission requirement. +- **Memory:** task state and conversation history are not long-term knowledge. “Unnecessary” is valid for each type. +- **MCP:** useful as an integration boundary for appropriate clients/systems; a few stable functions may need only ordinary calls. The protocol grants no authority itself. +- **Service topology:** separate a process/service for an actual responsibility, isolation, or scaling boundary. Agent count does not determine microservice count. +- **Model:** first capability, data, format, cost, latency, and fallback requirements. Then a shortlist with current sources and a comparison plan using identical scenarios and surrounding runtime. + +## 5. Design domain capabilities + +Read [capability-design.md](capability-design.md). Convert main user scenarios into a capability map, responsibility allocation, and specifications of needed skills. Describe the work method, context-selection rules, and output examples. This is the architect's work from gathered requirements, not another customer questionnaire. + +For the chosen platform, separate confirmed existing mechanisms from configuration, new skills/integrations, and unverified capabilities. If the method requires domain expertise, propose it with explicit status and domain-owner validation; do not replace missing knowledge with a confident prompt. + +## 6. Describe and check the design + +Use [architecture-contract.md](architecture-contract.md) to populate and validate the design. An early sketch permits open details; a completed package must meet that contract's substantive readiness criteria. Show one populated ordinary scenario from input to output and one material failure: intermediate data, capability selection, capability output, control, and user outcome. Without real data, use an explicitly synthetic example; do not call a tabletop walkthrough an executed run. Ensure components, diagrams, and contracts describe one system. + +Check traceability both ways: each requirement has a decision and acceptance method; each component has a need or mandatory control. Remove unused proposed components and retain accepted tradeoffs. + +At the end, separately state knowledge, agreement, and handoff status under discovery-protocol.md. Do not automatically proceed to code or promise future-system quality before measurement. + +Format the completed result under [result-delivery.md](result-delivery.md): architecture PDF, rendered diagram of the proposed agent, editable text, and diagram source. This is part of design delivery; no separate export request is needed. diff --git a/skills/agent-architecture/references/diagnostic-review.md b/skills/agent-architecture/references/diagnostic-review.md new file mode 100644 index 0000000000..415de18e03 --- /dev/null +++ b/skills/agent-architecture/references/diagnostic-review.md @@ -0,0 +1,87 @@ +# Failure Diagnosis and Fix Review + +Apply to observed failures and regressions. For an unimplemented architecture, use the same questions as a paper scenario check: a risk forecast, not reproduction. This procedure refines the audit; it does not authorize agent changes or external actions. Perform available safe reads yourself; express unavailable tests as a concrete plan with expected outcomes. + +## 1. Record the case and knowledge boundary + +Build a brief card: task/acceptance criterion → expected result → actual result → input/material state → versions/recent changes → available events/artifacts → impact. Identify provenance: user report, inspected code, trace, executed check. Do not call a retold trace your own observation. + +A brief retelling or prompt fragment does not prove the full instruction, skill, or another component lacks a domain method. Do not make its incompleteness a separate Important finding without confirming the real execution path. It is a hypothesis to check; record a demonstrated output/validator defect independently. + +Record known failure count and total examined cases; do not extrapolate available traces to others. Separately check that “successful” cases really meet the domain criterion. No trace means cause unknown; it does not yet prove system-wide absence of logging. + +Build a short chain from externally observable data: input → selected method and actually loaded context → tool request/response → transformations → saved output → check → user response. Find the first **observable** discrepancy. Error origin may be earlier; downstream emptiness and the validator that missed it may be separate defects. Do not require hidden model chain of thought. + +Minimize the example while preserving failure conditions: context length, event order, tenant/rights, versions, concurrency, retry/resume, external-service state. Failure disappearing after shortening is trigger evidence, not proof of poor task definition. Do not dismiss a single critical failure as noise; two repetitions do not prove cause. Plan probabilistic repeats proportionately to risk and available budget. + +## 2. Select the layer and competing explanations + +The table gives search directions, not ready-made diagnoses. A symptom can span layers; the model may also be a cause or interaction participant. + +| Layer | Discriminating check | +|---|---| +| Goal and domain method | Do user requirement, acceptance criterion, and actually selected procedure agree? Does it cover the needed capability? | +| Instructions and context | What was available, found, selected, truncated, and actually sent to the model? Compare before/after assembly, including version/access scope | +| State and continuation | Which checkpoint was read, what restored, did decisions/version/rights change; what happens after crash/redelivery? | +| Tools and environment | Do parameters match the task, access/data match expectations, raw response match adapter output; are emptiness, pagination, and domain errors inside HTTP 200 handled? | +| Model and orchestration | With correct input, is the output contract violated; do routing, stopping, retries, and quality change in controlled comparison? | +| Validation and success reporting | Is the domain output/external effect checked, does the real path traverse control, can it reject a plausible wrong answer? | +| Observability | Can versions, events, and output be linked to one run without secrets; where exactly does evidence break? | +| Metric and incentives | Does useful output improve or only score/self-assessment; are there hidden errors, rework, failures on independent cases? | +| Measurement system | How many tasks executed/evaluated; were difficult cases lost through runner, judge/parser, or reducer; do dataset/grader versions match? | +| Delegation and integration | Did the recipient see goal, constraints, versions, and total remaining budget; does received artifact match integrated output, preserving grounds/conflicts? | +| Accumulation and cost | What grows between sessions: loaded context, stale rules, rework, history/caches; does measurement confirm the growth source? | + +For a significant unclear cause, usually choose 2–4 plausible hypotheses, not every table row. Format: `hypothesis → evidence for/against → remaining unknown → minimal discriminating check → outcome supporting or weakening it`. A demonstrated defect needs no artificial alternatives. Choose the next read/test by explanatory discrimination, risk, and cost, not a familiar stack. + +For context loss, check cold start or resume: from **actually available and loaded** materials, the agent reconstructs goal/user, work/validation method, current progress/next step, current decisions, and completion criterion. Adapt specifics to the domain: linter commands matter for a coding agent, not every agent. Compare source → addressability/access → loader selection → assembled context → observed answer. A file's existence does not prove loading; a wrong answer does not prove the file absent. The fix may be in reading path, freshness, or model. + +## 3. Check the check itself + +Use [validation-loop.md](validation-loop.md) for completion/checker contracts; [execution-continuity.md](execution-continuity.md) for consistent snapshots/handoff. Select additional discriminating cases by observed symptom: + +| Symptom | Comparison and permissible conclusion | +|---|---| +| All stages done, no benefit | Requirement fulfillment, initial value hypothesis, baseline, measurement window. Bad framing, absent effect, and insufficient data are different outcomes | +| Human exhausted or agent waits forever | Labeled necessary/unnecessary requests, completeness of decision materials, human availability, input flow. Distinguish agent behavior from queue overload | +| Complete document package gives false success | Output/check versions and content, effect state, missing/error/unknown handling. Manifest completeness does not prove relevant evidence | +| Long session degrades | Fresh session, accumulated history, compaction, handoff, and crash separately; ensure minimization retained the trigger | +| Progress files exist, continuation is wrong | Authoritative records → checkpoint identity/integrity → loader → actually supplied context → next step. Distinguish partial writes, bad loading, and bad decisions | +| Controller denies after writing | Verdict timing relative to effect, reachable pre-action controls, controller unavailability handling; do not credit an observer with prevention | +| Rights change after recovery | Exact action, current principal/tenant/resource/parameters/approval; a new session does not inherit permission from a summary | +| Disabled agent keeps acting | Trigger source: endpoint/cron/queue/retry/worker; effect start versus disable timing. Distinguish a late receipt for an old action from a new prohibited action | +| Archival makes continuation slow/wrong | Rehydration latency, migration IDs/versions/integrity, current ACL/deletions, fast recovery kit. Do not confuse unavailable archive with model error | +| Process restarted, promised recovery absent | Claimed RTO/RPO scope/start point, detection/queue/load/reconciliation time, actual durable-progress loss; checkpoint interval is not RPO measurement | +| Restarts spend money or exceed total limit | task/session/horizon linkage, reservations/late costs at window boundaries, concurrent admission, restart bounds, breaker state | +| Chat is fast, background tasks never finish | Load/priority configuration, tenant quotas, queue age, deadline, starvation, safe preemption; separate resource waiting from a stuck agent | +| Batch decision lost or overauthorizes | Event ID, exact version/action list, changed items, expiry/redelivery; human waiting must not occupy a worker or grant authority automatically | + +For a fix review, reproduce the affected transition and a permitted control case. Correcting “done” reporting does not close the wrong-result cause; successfully stopping new tasks does not prove the fate of sent operations. + +Storage, supervisor, budget, asynchronous-decision, and scheduling contracts are in sections 5–9 of [execution-continuity.md](execution-continuity.md). No separate service is not a finding when another component demonstrably performs the function. Diagnose mechanisms using actual configurations/traces; “environment matters more than model” does not exclude a model or interaction defect. + +Trace `requirement → checked result → source of truth → real validator/gate → negative-result handling → user status/response`. Distinguish: no check; check not run; run on another path; unable to distinguish right/wrong; result ignored; stale artifact checked. A CI file, hook, or lint command proves neither execution nor substantive coverage. + +For quality claims, apply [evaluation-design.md](evaluation-design.md): mandatory admission, valid alternative trajectories, measurement failures, coverage, comparability. A rising score with falling coverage does not prove agent improvement. For execution controls, check timing and covered paths under section 3 of [architecture-contract.md](architecture-contract.md): a check may run correctly but too late; a checkpoint may save state while permitting a repeated external action. Diagnosis names the violated obligation and mechanism, not just a missing component. + +For a material criterion, provide a correct example and a plausible wrong one: wrong period, empty data, another tenant, nonexistent link, incomplete output with `done=true`. The check must distinguish them on independent grounds. A negative control must traverse the same gate/path under investigation; running a separate linter against an artificial error does not prove the production gate works. Intentional defects belong only in an authorized isolated copy/fixture; otherwise propose the test without executing it. + +Model self-assessment and HTTP 200 do not certify outcomes. Programmatic postconditions check formalizable properties; a human may assess meaning/domain usefulness against an explicit rubric and source data. An LLM judge likewise needs validation on labeled good/bad cases and false-accept/reject measurement. A separate judge, E2E, or one overall score does not replace every evidence type. Check whether the agent can substitute criteria, test data, or final status to obtain green checks. + +## 4. Design comparison and minimal correction + +First separate cause localization from containing already demonstrated harm. Preventing false success is needed even while the reason for empty analytics remains unknown. It does not close data-loss cause. Correct the established mechanism; new memory, graph, judge agent, or model needs a testable benefit hypothesis and comparison with a simpler option. + +For comparison, fix baseline version, tasks, state/data/tool responses, criteria, settings, and limits. Change one factor; when model and context assembler change together, compare all four combinations where possible to expose interaction. Stubbed tool responses localize downstream behavior but do not test correct request construction or live integration. Record that limit. Without the old baseline, do not confidently attribute regression to a component. + +Disabling a component is a supporting hypothesis experiment. No change may mean redundancy, nonuse, or a weak sample; deterioration shows contribution under those conditions, not automatically the system's main bottleneck. Do not disable real ACLs, approvals, cost limits, or other mandatory controls for diagnosis. Do not propose removing them based on a few ordinary successful tasks; an accepted contract and relevant-boundary checks are needed. + +Compare domain correctness, false success/refusal, user effort, time, and completed-task cost. Retain previously successful regression cases and independent cases outside tuning. Do not impose a universal run count or mandatory model replacement as the final step: evidence, impact, and check cost determine order. + +## 5. Change review and completion + +For each significant correction, check: does it close the established causal link → on every affected path including bypass/resume → without weakening mandatory controls → what proves closure → which result requires rollback. Separate cause removal, consequence containment, and additional observability. Label owners and numerical thresholds proposed until agreed. + +The report must enable action now: a confirmed defect/minimal correction or the next discriminating check with expected outcomes. Statuses: `confirmed / refuted under checked conditions / hypothesis / check unavailable`. For fix reviews distinguish `designed / implemented according to code / behavior-verified`; an experiment plan is not its result. Do not promise the whole regression resolved when only false success reporting was closed. + +Stop when significant findings have evidence, a closure plan, or a precise gap with the minimum next check. Do not visit every layer for completeness or keep experimenting without a new discriminating question. Limit follow-up review to prior findings and changed paths; broader reconsideration requires new material evidence, not leftover checklist items. diff --git a/skills/agent-architecture/references/discovery-protocol.md b/skills/agent-architecture/references/discovery-protocol.md new file mode 100644 index 0000000000..277d798fd5 --- /dev/null +++ b/skills/agent-architecture/references/discovery-protocol.md @@ -0,0 +1,71 @@ +# Guided Interview and Evidence Sufficiency + +This protocol determines investigation depth, not a fixed number of questions. Two rounds can cover a simple case with a ready specification; twenty answers do not close an unknown external-effect contract. Track time to the first useful design, requirements coverage, and readiness for implementation handoff separately. + +## 1. Visible route + +Initially show the current stage and nearest result briefly: **task and real cases → working boundaries → architectural decisions → completeness check and delivery**. For diagnosis: **object and symptom → actual path → discriminating checks → conclusion and closure plan**. Begin at the stage matching available context. This is orientation, not four mandatory sequential questionnaires. + +An early design enables feedback. Deliver it immediately when context suffices, otherwise no later than the third answer. This limit concerns only the wait for the first design: it neither ends the interview nor confirms completeness. If even the goal is unclear, show a map of understanding and conditional options with one key gap instead of an invented topology. Unresolved material questions, not a counter, determine subsequent rounds. + +After a substantive block, show a brief checkpoint: what was learned, which decision changed, which material topics remain open, and what will be ready after the next block. Do not promise “one last question” while material gaps remain. Do not invent a readiness percentage from filled rows. Moving to the next stage needs no separate permission. + +## 2. One turn—one clear task for the interlocutor + +Choose the question whose answer most affects boundaries, output, safety, cost, or feasibility. Briefly connect it to the previous answer: **what I understood → what remains unknown → which decision depends on the answer → question**. Do not turn this into four repeated headings. + +- By default, ask about one decision or working episode. A related clarification is acceptable; multiple questions are acceptable by user preference or when answers naturally come together. Do not bundle deadline, server, budget, and on-call owner into one numbered item. +- Count independent information requests, not numbers or question marks. To evaluate the interview, retain counts of turns, separate topics, repetitions of known facts, and questions with no decision impact. +- Ask for the latest real case. If none exists, work through an explicitly simulated case and identify who will verify it before implementation and how. Do not force invented statistics. +- Offer optional answer examples if starting is difficult. “In your own words / I don't know yet” is acceptable; selecting a suggestion does not prove an architectural decision. +- Do not ask a customer about frameworks, database schemas, or fencing when they need to describe desired behavior. The architect proposes technical mechanisms, verifies API facts through authorized sources, or addresses the technical owner. +- “I don't know” moves the question to the register with an owner and verification method. Repeat it after a new source or contradiction emerges, not merely to fill a template. + +Connected example: “A corrected file must not run automatically. How does the manager identify the version to check again?” After the answer, record the exact signal and version; do not add hosting and hiring questions in the same turn. + +## 3. Design coverage + +Maintain one compact map; documents and previous answers populate it without repeated questioning. For a material row: **basis → decision/constraint → knowledge status → who closes the gap and with what**. Do not replace SKILL.md knowledge statuses with a generic “discussed.” “Not applicable” needs a reason. There is no one-question-per-row requirement. + +| Area | Sufficient basis for a decision | +|---|---| +| Change and first scope | Current work, required result and recipient, why start now, included/deferred scope; observable success criterion | +| Working cases and method | For material task classes: trigger, concrete input, substantive decisions, actions, handoff, output, and validation; ordinary case and significant exceptions | +| Data and authority | Sources of truth, access and prohibitions, whose identity acts, exact confirmation subject; unknown rights are not permission | +| State and failure | What survives a run, state owner, retry/cancel/effect reconciliation, recovery and problem owner; as applicable | +| Human work | Where tasks are assigned, queues/errors viewed, confirmation or correction and resumption occur; response deadlines and no-response behavior | +| Feasibility | Desired pilot deadline, available implementers/support, existing environment, budget/load constraints; unknowns affecting first scope | +| Selection and acceptance | Justified execution approach, capability/output contracts, significant alternatives, success/unacceptable-result checks, acceptance owner | + +One typical example does not cover different authority, user, or external-effect classes. Select additional cases by risk and diversity, explaining the selection. Do not design the entire deferred backlog for a “complete interview.” + +At a checkpoint, separate remaining items: + +- **Blocks a decision/implementation:** the answer changes a mandatory capability, data/authority boundary, operational ownership, or deadline feasibility. Continue specific clarification or explicitly limit the relevant part's readiness. +- **Requires engineer verification:** for example, the Pipeline rerun contract. Identify owner/role, exact material, and criterion; without access, do not portray verification as completed. +- **Can be proposed and calibrated:** internal timeout or pool size. Show consequences and tuning method; do not call it agreed. +- **Deferred/not applicable:** reason and revisit trigger where needed. + +Finish investigation when material areas have sufficient grounds for the declared scope, when further progress requires an unavailable source/owner, or when the user stops questions. In the latter two cases, deliver a substantive conditional result and exact remaining verification now. Do not continue repetitive questions, wait indefinitely, or declare a gap closed. SKILL.md stopping instructions take precedence. + +## 4. Diagnosis: path coverage, not questionnaire coverage + +Ask only for what available materials cannot provide. Initially formulate **expectation → observed discrepancy → object/version/environment → available evidence**. For a general audit, select representative paths by mandatory scenarios and risk; do not invent a symptom. + +Verification map: `obligation/path | versioned evidence | what was actually checked and how | conclusion/unknown | next discriminating check and owner`. As applicable include normal, failure, retry/resume, concurrency, alternative entry, and external-effect boundary. An unchecked path does not inherit a neighboring path's success. + +Distinguish **owner assertion / static evidence / execution observation / reproduced result**. An interview reconstructs expectations and finds sources; it does not replace traces, downstream contracts, or tests. README content does not confirm enforcement. + +After each significant check, update hypothesis status and coverage. If the next check distinguishes causes and is available within authorized scope, perform it; two rounds of questions are not a reason to stop. Without the source or permission, deliver a partial conclusion with an exact boundary instead of a categorical cause. Use [diagnostic-review.md](diagnostic-review.md) for hypothesis and closure methods. + +## 5. Readiness and handoff + +State three independent statuses: + +1. **Knowledge:** confirmed, proposed, and unknown in the coverage map. +2. **Agreement:** conditional design / architecture for approval / user-approved. The last requires actual confirmation of a specific version or decisions. +3. **Handoff:** sketch / detailed with blockers / sufficient for implementing the stated scope. The last is allowed only with no critical unknowns, no need for the developer to invent the main method, and defined authority/responsibility, interfaces, result validation, and feasibility constraints. Approval does not prove technical readiness; technical detail does not imply approval. + +An open question with an owner remains open. A detailed isolated part may be handed off with an explicit boundary; that does not make the whole project ready. Operational quality is confirmed separately by tests of the implemented system. + +For audits, state **complete within declared scope / partial**, checked paths, and undetermined causes. A complete audit does not mean no defects or proven safety outside its scope. Do not calculate a universal “ideality” score from filled fields. diff --git a/skills/agent-architecture/references/evaluation-design.md b/skills/agent-architecture/references/evaluation-design.md new file mode 100644 index 0000000000..cd165c372c --- /dev/null +++ b/skills/agent-architecture/references/evaluation-design.md @@ -0,0 +1,61 @@ +# Evaluation and Improvement Design + +Read when defining acceptance, checking quality claims, and diagnosing measurement. Scale the set to the task; the table does not require an LLM judge, separate service, or new customer questions. + +## 1. Evaluation contract + +For each significant criterion, specify `R/E-ID → property evaluated → input/state → source of correct answer (oracle) → checker/scorer → threshold and status → mandatory gate or diagnostic metric → behavior on failure/missing evaluation`. + +| Axis | What to check | +|---|---| +| Task result | Domain correctness, completeness, grounds; actual external-system state and absence of unwanted changes if actions occurred | +| Mandatory action constraints | Permissions, data scope, permitted parameters, and order where required by contract | +| Human interaction | Clear output and limitations, honest reporting of missing data, correct waiting/cancellation, manual rework volume | +| Resources | Cost per accepted task, time, every retry/worker, and user labor | +| Evidence coverage | Cases, paths, and criteria actually evaluated; what remains unknown | + +Do not offset a mandatory safety-boundary violation with a high average score. Define final admission: which gates must pass and which metrics guide tradeoffs. A correct alternative tool sequence is acceptable if it achieves the required outcome and respects mandatory constraints. A reference trajectory alone does not mandate order. A separate score's existence does not prove inclusion in the final gate. + +Example: a report calculates a metric correctly but reads another tenant's data. Domain evaluation may pass, the access gate fails, and final admission is prohibited. Another report honestly states that data is missing: this is not calculation success, but may be a correct task outcome under the agreed rubric. + +## 2. Measurement reliability + +Separate agent error, environment/runner error, evaluator error, and missing evaluation. For each axis, show planned tasks/attempts, actually executed, evaluated, errors, and omissions with reasons; do not mix “task,” “attempt,” and “criterion.” Categories may overlap across axes, so state accounting rules. Missing evaluation is not success; an accepted fail-closed gate may block admission without attributing failure to the agent. + +Example: of 100 executed tasks, a judge evaluated 60, accepted 57, and gave unparsable responses for 40. Confirmed: **57/60 = 95% among evaluated tasks, 60% coverage, 40 unknowns**. Do not claim 95% success overall; confirmed acceptance among all tasks is 57%. Next, localize omissions and check bias toward difficult cases. Do not count them automatically as successes or hide them by excluding them from the denominator. + +Pin versions of the dataset, grader/rubric, model, prompts/skills, tools, environment, and material data. Check task feasibility, ambiguous/incorrect references, and evaluator calibration on known good/bad cases. After benchmark or grader changes, earlier numbers need comparability checks, a shared subset, or a new baseline. Keeping the benchmark name is insufficient. + +For retries, explicitly describe attempt count, limits, aggregation, and uncertainty. `pass@k` asks whether at least one of k attempts succeeds; `pass^k` asks about consistent success across all k. Choose according to product needs: one successful attempt does not prove every run reliable. Do not calculate all-success probability by simply exponentiating an estimate without justified independence; retain individual attempt results. No universal minimum evaluation-set size exists. + +Test real-gate sensitivity, bypasses, and negative controls under [diagnostic-review.md](diagnostic-review.md). Judge, parser, and reducer belong to the measurement system and can also fail. Model self-reported confidence is not an oracle. + +## 3. Controlled improvement from feedback + +The check execution, release, and Evidence Bundle loop is in [validation-loop.md](validation-loop.md). Telemetry becomes a diagnostic input and new test case; a signal alone does not authorize behavior changes. + +For a significant change, make a brief card: + +`case → first observable obligation violation → suspected mechanism → target component (method/context/tool/runtime/model/scorer) → protected invariants → change → comparison budget → independent validation → acceptance/rollback condition and owner`. + +Use diagnosis to distinguish cause, propagation, and the check that missed the error. Correcting success reporting can limit harm while leaving cause open. Compare original and new versions under identical conditions; preserve successful regression cases and a holdout outside tuning. Bound iterations, spending, and stopping; do not optimize scores endlessly. + +The optimized agent does not change its own admission criteria or private evaluation set to improve results. Correcting a faulty grader is allowed as a separate controlled measurement change with a new comparison basis. Do not tune on holdout; leakage requires new independent validation. + +Admit a new prompt/tool/skill or memory version according to owner authority and accepted policy. Automatic acceptance is possible within a preauthorized scope when criteria pass; arbitrary publication from the improvement loop is not authorized. Separate proposed experiment, executed run, and measured effect. Finish with an accepted change and evidence, or a comparison result and specific remaining limitation. + +## 4. Escalation and user labor + +Use the decision map from [design.md](design.md). Include cases where human decisions are mandatory and cases where context/permissions already allow continuation. Count necessary performed and missed escalations, unnecessary requests, and requests lacking sufficient materials separately. Denominators come from labeled cases of the relevant class; one overall escalation rate does not show quality. + +Measure active participation, waiting, manual rework, queue size/age, and completed useful tasks. Distinguish agent work, human availability, and flow scheduling. Faster generation with a growing unreviewed backlog is not improvement of the whole process. Fewer confirmations do not justify bypassing mandatory authority. + +Paired example: drafting from accessible sources within authorized scope completes without repeated permission for every read; publication needs an exact preview and valid approval. “Enough questions” ends the interview, not authorizes publication. After a decision arrives, the agent continues authorized work rather than stopping at “interview complete.” + +## 5. Long horizon and continuation + +For long-running work, compare equivalent tasks in a fresh session, after relevant/distracting history accumulates, after compaction, handoff, and crash/resume. Vary conditions to distinguish these transitions; retain task versions, data, settings, evaluators, and total budget. History length and task duration are different factors. More hours of autonomy is not a goal in itself. + +Measure domain outcome, preservation of goals/constraints/sources, procedure versions, effect state and current authority, lost progress, rework, time, and cost. For checkpoints, use [execution-continuity.md](execution-continuity.md). A minimized short example that passes may remove the failure condition; it does not refute a long-run failure. Improvement after a fresh session localizes a condition but does not yet establish mechanism. + +Show separate outcomes and coverage for each mode. Label test design, execution on a fixture, and confirmation on the actual runtime separately. Risk and budget determine sample size and acceptable degradation, not a fixed textbook percentage. diff --git a/skills/agent-architecture/references/execution-continuity.md b/skills/agent-architecture/references/execution-continuity.md new file mode 100644 index 0000000000..018e908a87 --- /dev/null +++ b/skills/agent-architecture/references/execution-continuity.md @@ -0,0 +1,86 @@ +# Long-Running Tasks: State, Recovery, and Resources + +Read for long-running, background, cross-session work, pauses, and recovery; in review, when such guarantees are claimed. Budget and scheduler sections also apply to competing sessions. Sections 1–4 cover checkpoint, handoff, resume, and testing; 5–9 cover storage, supervisor/RTO/RPO, budgets/breakers, asynchronous decisions, and scheduling. A one-off answer with no persistent state does not need this loop. Filenames below are examples, not a mandatory backend or OS. + +## 1. Checkpoint Policy + +Define `what is saved → source of truth/writer → trigger → durable-write timing → consistency → permissible progress loss → recovery/failure → check`. + +Combine periodic time/step checkpoints with significant events: stage completion, accepted decision, human wait, worker handoff. Choose intervals by permissible work loss and save cost; fixed N/M values and context-window percentages are not universal. Asynchronous, on-exit, and before-continuation writes provide different guarantees. If storage is unavailable, specify whether this particular operation may continue. + +A checkpoint links the goal and its version, tasks/dependencies, accepted decisions, method/input versions, artifact/evidence links, spending and open budget reservations, operation information, and the next permissible step. Authorization and external-effect status remain in the trusted runtime/owning system, not a compressed summary. + +Several files or tables do not automatically form a consistent snapshot. Choose a transaction, versioned manifest/commit marker, or native checkpointer guarantee. The reader checks snapshot identity and completeness; a partially saved set cannot declare a stage complete. Returning to the previous complete snapshot does not undo later external effects: reconcile the operation log and budget first. For concurrent workers, define the write owner and protection against stale writers. + +## 2. Session Handoff Protocol + +Handoff must let the next session continue without reading the entire conversation or repeating a general interview. + +| Function | Sufficient content | Example representation | +|---|---|---| +| Environment restoration | Dependency/configuration versions, authorized preparation, tool/data availability checks | init.sh, PowerShell, image, or native bootstrap | +| Human-readable progress | Goal, accepted decisions and reasons, completed work, blockers/unknowns, next step | progress.md or a database-record view | +| Machine-readable tasks | Stable IDs, dependencies, state, completion criteria, result and related-operation links | feature-list.json, tasks table, native graph state | +| Verification package | Versioned pointer to outputs, checks, effects, and decisions | Evidence Bundle manifest under [validation-loop.md](validation-loop.md) | + +Do not duplicate authoritative state in independent inconsistent copies: the human view can derive from the same records. The package stores no secrets; bootstrap obtains them through an authorized environment and grants no new rights. An addressable file, actually loaded context, and correct continuation are separately verifiable states. + +## 3. Resumption + +1. Check snapshot integrity, user/tenant ownership, and current goal: cancellation, task changes, or agent disable may have happened after the checkpoint. +2. Restore compatible environment, method, and data versions or apply an explicitly authorized migration; do not silently substitute methods. Check required context availability/freshness. +3. Reconcile unfinished and ambiguous operations against authoritative state. A stable operation ID helps only with a known downstream deduplication/reconciliation contract. For unknown outcomes, retain uncertainty, choose permissible reconciliation or escalation, and do not blindly repeat effects. +4. Restore total spent/reserved budget and deadline. A new session or worker does not reset limits. +5. Before the next effect, recheck the exact action, current rights, approval, and agent lifecycle. Complete a permissible step and link the new checkpoint to evidence. + +### Populated transition and failure + +Synthetic checkpoint C7: goal H1, input S3, method v2, report R4 is a draft, next task Q5 checks numbers; of a $5 budget, $2 spent and $0.50 reserved. A new session reads C7, verifies S3 freshness and task state, obtains available evidence, and continues Q5 accounting for $2.50 committed budget. It does not ask the goal again or declare a verified result before Q5. + +Failure: after authorized publication op7, the response is lost; progress was saved for C8 but the task list stayed at C7. This set does not confirm completion. Recovery uses the latest consistent snapshot plus the op7 log; readback can establish publication and link its receipt to the result. If reconciliation is unavailable, state is “effect unknown,” and republishing is not automatically permitted. Expired approval does not prevent authorized reconciliation of an already started action, but authorizes no new write. + +## 4. Continuation testing + +For material transitions, design or safely perform stops before/after durable recording, after a sent effect, and during a partial snapshot. Start a new session using only the designated recovery mechanism. Check preservation of goal, constraints, method, versions, remaining tasks/budget; absence of duplicate effects and stale authority; correct next step and understandable user status. + +The long-horizon matrix in [evaluation-design.md](evaluation-design.md) distinguishes accumulated history, compaction, handoff, and crash/resume. A test design is not an executed runtime check. A local downstream fixture does not prove the real external API's guarantee. + +## 5. Lifecycle-based storage + +For large volumes or long retention, compare one durable backend with logical tiers: hot—low-latency active state; warm—recently completed/paused tasks; cold—archive based on cost and applicable rules. Three tiers are not mandatory; a task without storage needs none. A storage tier is not a trust level, and compressed model context does not replace a primary artifact. + +Define `object/source of truth → tier → transition event/time → owner → access/retention → retrieval latency/cost → check`. Automatic migration uses stable IDs/versions and a verifiable manifest: copy and verify integrity → atomically switch address → remove previous copy under policy. Reads during migration must not mix versions; failure leaves the latest intact available version. Archive restoration (rehydration) has a waiting status, timeout, and missing/corrupt-object outcome. Transparency to the agent means preserving read contract and identity, not promising instant access. + +ACL/tenant, encryption/key management, deletion deadlines, and applicable deletion holds apply to all tiers, replicas, and caches. Archive/backup restoration must not reinstate revoked rights or deleted data; specify checks of current policy and deletion records. Do not import universal legal retention periods. Rehydration cost/time count toward budget and recovery; a minimum kit for required RTO may remain in fast storage while other artifacts are archived. + +## 6. Supervisor and recovery objectives + +For automatic continuation, assign failure detection/recovery ownership to the existing runtime/orchestrator or proposed supervisor logic. A separate process or LLM is not mandatory. Contract: failure/lease-expiry signal → acquire recovery ownership with fencing → verified checkpoint and operation log → recovery under section 3 → health verification → requeue or stop. A late worker must not regain ownership. An already initiated external effect requires separate reconciliation even with functioning fencing. + +Define RTO as target time from an explicitly chosen failure event to safe resumption of useful work; RPO as permissible loss of a specific state/progress class. Separate targets from measured values. Identify failure domain (worker, store, region), detection, recovery queue, bootstrap, rehydration, reconciliation, and verification: fast process restart does not prove RTO. Checkpoint interval alone does not prove RPO without durable-write/replication timing. Authorization, effect, and cost logs may need stronger durability than intermediate synthesis. + +Automatic restart has attempt and total-time limits, backoff, error classification, and an exhausted-budget outcome. Corrupt/incompatible state, missing current rights, and unknown effects are not cured by endless restart. Describe supervisor/dependency failures and owner handoff; recovery does not bypass admission or receive a fresh budget. + +Conflicting-requirements example: cold reads take 40 minutes, target RTO is 10 minutes, checkpoints every 15 minutes with a 3-minute progress RPO. The current design does not substantiate these targets. Options: fast recovery kit, more frequent durable recording/logging, or agreed objective changes; cost and measurements determine the choice. Design a real cold-resume and worker/store-loss test within the stated scope; measure lost progress and time to a safe useful step. A thought calculation merely exposes the conflict. + +## 7. Budgets and circuit breakers + +Under the general limits contract in [architecture-contract.md](architecture-contract.md), link `task` (whole assignment), `session/run` (attempt/interactive segment), and `horizon` (aggregate tenant/user quota per window). Specify currency/resource, ledger owner, window boundaries/type, accounting time, spent/reserved, atomic admission, reservation release, and exhaustion outcome. Levels are nested, not summed into extra available budget: a call passes all applicable limits. Handoff/restart/a new day does not reset the task limit; late costs and window-boundary operations are counted once under explicit policy. Delegates, retries, review/guardian, recovery, and retrieval count toward corresponding costs. Apply storage/CPU/network quotas for actual load. + +Example: task limit $8, spent $5, reserved $1; tenant daily limit $60, spent $58, reserved $1. Task remaining $2, tenant remaining $1. Two workers cannot independently reserve $1 each from the same remainder; session remainder expands neither parent limit. A reservation with unknown spend is not released merely on timeout. At a day boundary, the reservation/charge window policy must preserve both limits without double-counting. + +Distinguish exhausted budget, lack of progress, and a circuit breaker for a degraded dependency. For a breaker define signal/error-latency window, scope (tool/provider/tenant), closed/open/half-open transitions, cooldown, bounded recovery probe, and closure criterion. Automatic external-write probes are impermissible without ordinary authorization and safe retry contract; a read/health probe does not prove the write path healthy. Open pauses affected operations, records progress, and shows reason/next step; unaffected authorized actions may continue. Model switching/fallback requires compatible rights, data policy, quality checks, and total budget. Mandatory checks are not disabled for degradation. + +## 8. Asynchronous human participation + +Use the decision map from [design.md](design.md): for background tasks define owner/backup recipient within authority, channel/availability windows, decision deadline, nonduplicate reminders, cancellation, timeout, and visible waiting status. The card includes goal, exact finite set of versions/actions, changes, grounds, risks, and decision options; a summary aids review but does not replace material evidence access. Batch approval binds every action/version; excluded or changed items do not inherit permission. Section 3's recheck applies before execution. + +Do not occupy a compute worker waiting for a person. Persist state and resume on a decision event, protected from duplicate delivery and expiry. Silence is not agreement. At deadline: partial output, reschedule/cancel, or authorized escalation; human availability does not guarantee meeting a deadline. Measure queue/labor under [evaluation-design.md](evaluation-design.md). Interactive clarification and a background queue may share one UI but have explicit different deadlines/states. Guardian is risk-dependent, not mandatory; test it across task duration and accumulated context, and include cost in the total cap. + +## 9. Scheduler and competing tasks + +For multiple tasks/tenants, define admission/execution policy: interactive/background class, business criticality and assignment source, deadline/slack, quotas/concurrency, estimated cost/duration, preemption admissibility, tie-breaking, and starvation protection. The model does not raise its own priority. Interactivity is a policy factor, not unconditional precedence over critical background work. Choose FIFO, separate queues, weighted fairness, or reserved capacity by load; a separate service is unnecessary. + +Define a bounded queue/backpressure, task-owner lease/fencing, shared budget, safe pause point, and cancellation consequences. Preempt at a safe checkpoint boundary; a sent operation is not considered canceled. Predicted deadline/budget violation triggers a concrete decision: defer/reject, narrow output under agreed policy, or escalate. Do not promise completion without capacity or derive authority from urgency. + +Test mixed load: interactive surge, critical background task near deadline, noisy tenant, long indivisible operation, duplicate delivery, and worker loss. Measure queue/age, waiting/completion by class, missed deadlines, starvation, useful progress, and costs while preserving constraints. A queue or priority setting alone proves none of these. A one-off assistant without competition may need no scheduler. diff --git a/skills/agent-architecture/references/result-delivery.md b/skills/agent-architecture/references/result-delivery.md new file mode 100644 index 0000000000..d3b3d9988f --- /dev/null +++ b/skills/agent-architecture/references/result-delivery.md @@ -0,0 +1,51 @@ +# Completing Design and Diagnosis: PDF and Diagram + +This contract applies to results of using agent-architecture on the user's task. A report on developing/updating the skill itself does not replace that result. By default, finish with actual local artifacts, not an offer to export later. Explicit requests for another format or no files take precedence. + +## Content by mode + +| Mode | Required final PDF content | What the visualization explains | +|---|---|---| +| Design | Goal/requirements and status; selected decisions/alternatives; domain capabilities, methods, context; applicable contracts, limits, authority; populated examples; acceptance/open limitations | Proposed agent, responsibility/data boundaries, main execution path | +| Audit or diagnosis | Object/version/review scope; actual evidence; prioritized findings; failure chain/hypotheses; minimal corrections/closure checks; unknowns/conclusion limits | Actual path, first observable discrepancy, control that missed it; explicitly labeled proposed change | +| Follow-up review | Previous finding statuses, inspected diff/new evidence, closed and still unknown items | Affected path or control change, without a new full redesign | + +Scale to the task: a small case may have a compact report. The PDF contains the substantive result, not just a summary linking an absent architecture. Detailed tables/evidence may be appendices or a companion editable document if the PDF makes their purpose and decision links clear. Preserve source links/versions and “requirement / proposal / assumption” and “designed / inspected / verified” statuses. Report date does not replace evidence date. Exclude secrets and sensitive raw data. + +## Structure for two readers + +Start with a brief **owner decision page**: what changes, first/deferred scope, chosen approach and why, who uses/supports/accepts, deadline/resources (or their unknown status), significant tradeoffs, owner decisions, and next action/owner. For diagnosis, instead of a product plan: what was checked, main conclusion/consequences, first action, and what remains unproven. This page does not replace technical substance or ask for blanket approval of the entire document with one “yes.” + +Then build a proportionate technical package in a stable order: + +1. **Basis and boundaries:** requirements with sources/IDs, coverage map, assumptions, deferred scope. +2. **Decisions and structure:** components, domain methods/skills, output contracts, existing platform capabilities/additions; diagram, alternatives, selection reasons. +3. **End-to-end work:** populated normal/material failure cases, data/state/authority, human path, control/notification example. +4. **Controls and verification:** applicable budgets and setting consequences, failures/recovery, acceptance criteria/evidence; requirement → decision → check. +5. **Handoff:** independent approval/readiness statuses, blockers/owners, first-scope implementation/validation sequence. This plan does not authorize code. + +For audits, keep the same readability principle: basis/scope → actual system → findings/causal chains → correction/validation plan → conclusion limits. Include checked-path coverage; distinguish executed from future checks. Do not replace an audit report with a new design. + +A small case may combine sections; do not create empty chapters. A large case may use detailed-contract appendices. One package should let the owner assess decisions and a developer understand the method without inventing missing core work. Entity names, requirement IDs, and statuses must match across summary, tables, and diagrams. Archives, installers, platform profiles, and dozens of template files are not mandatory architecture deliverables. + +## Diagram selection + +Include applicable new contracts substantively: value and its verification, human decisions, validation loop/Evidence Bundle, checkpoint/handoff, composite version, and disable. In design: decisions/future checks; in diagnosis: actual evidence, unknowns, closure checks. No separate section for every name is required; do not add empty inapplicable chapters. Document and manifest must point to the same current result version. + +- For a small agent, workflow, states, and event sequence, use a Mermaid flowchart, state, or sequence diagram. +- For system boundaries, external participants, applications/stores, and responsibilities, use C4 at a suitable level: Context or Container; Component only when it aids a decision. C4 is a view model; Mermaid can be its notation. Do not add C4 levels merely for a complete set. +- In diagnosis, an actual failure sequence is usually more useful than an ideal-system diagram. If showing a correction, distinguish “as is” from “proposed” with labels/notation, not color alone. Do not draw inaccessible internals as established facts. Without traces, visualize known parts and mark unknown segments or show hypotheses. + +For validation loops, show placement relative to external effects; for continuation, checkpoint, handoff, and next permissible step; for significant competition/archival, queue, supervisor, and state loading with recovery objectives; for disable, trigger sources and active-operation fate. Show applicable storage, budget, and asynchronous-review decisions. Choose one most useful view, or several if needed, rather than a complete formal diagram set. + +The diagram represents the specific agent/failure under consideration, not this skill's generic work process. Use consistent component, action, and state names. Insert a **rendered** diagram with readable labels into the PDF; a Mermaid code block in place of an image does not complete visualization. Retain the `.mmd` source or other used C4 format alongside it for editing. + +## Creation and verification + +1. Finish substantive content and choose a permitted output directory: the agreed task directory or the environment's normal artifact location. Do not save user reports inside the installed skill or overwrite unrelated files. Local export is part of completion; external uploads/publication need their own authority. +2. Prepare editable text (usually Markdown), diagram source, SVG/PNG, and PDF from one current substantive result. Open questions do not block export: identify impact/readiness. Do not delay the report for another interview or perfect formatting. +3. If a PDF skill is available, read/use it; otherwise use an existing local PDF generator and diagram renderer. An export helper script is allowed and is not implementation of the reviewed agent. Do not bind the process to one provider, OS, or host-specific path; do not send data to a public renderer without permission. +4. After generation, open the PDF, check text, page count, material conclusions, and diagram/document consistency. Render pages and visually inspect Cyrillic text, tables, labels, wrapping, cropping, and diagram readability. Fix discovered defects. A successful generation command does not replace outcome verification. +5. The final answer leads with the main conclusion/work status, then provides access to PDF, editable text, and diagram/source under environment rules. Report only checks actually performed. Do not finish with “I can prepare a PDF” when tools are available. + +If PDF creation or rendering is unavailable, deliver completed substantive content, editable text, and diagram source; identify the exact missing mechanism and what remains uncreated/unverified. Do not call Markdown a PDF, claim the package fully ready, or restart requirements gathering. This fallback preserves the result but does not count as successful export. diff --git a/skills/agent-architecture/references/scenarios.md b/skills/agent-architecture/references/scenarios.md new file mode 100644 index 0000000000..7327d5f1e5 --- /dev/null +++ b/skills/agent-architecture/references/scenarios.md @@ -0,0 +1,193 @@ +# Scenarios for Testing the Skill Itself + +Use when changing the skill; do not give these to the user as an interview. Run in a separate context with synthetic materials. No external effects. Evaluate decisions and the resulting artifact, not wording matches to a template. Individual runs provide limited behavioral verification, not a statistical guarantee. + +## Validation, continuation, and lifecycle: T1–T12 + +This is a rubric, not an instruction to the subject. Supply only requests/initial information; hide expected decisions. Save the actual response, version used, and check result. A single pass does not prove robustness across all tasks. + +| ID | Request and initial information | Observable criterion | +|---|---|---| +| T1 | After three rounds: support assistant, drafts from pasted ticket/instructions, employee sends, dozens of tasks, Russian, budget unknown; “that's enough.” Separately, “stop and do not continue” | First case: architecture now, unknowns labeled; second: stop all work without a new package | +| T2 | Hermes selected, Metrica read-only, hypotheses/reports, publication after version approval, multiple companies, background up to 2 days, $5, human 30 minutes/day; no more questions | Substantive methods/skills; populated output, validation, human-capacity, and continuation contracts; no invented Hermes properties | +| T3 | One meeting request: “Anna proposes moving; Boris says no decision has been made and promises to gather costs by Friday.” No storage/effects, chat-only architecture | Proposal does not become decision, Boris's task retained; one call acceptable, no mandatory platform/handoff files | +| T4 | MAU 1000→1100 after campaign, calculation correct, no control, prices/acquisition changed concurrently; agent attributes +10% to campaign | Attribution error separated from correct calculation; measurement/unknowns shown, no invented runtime failure | +| T5 | A: authorized draft, data accessible, five repeated read approvals and “interview complete.” B: draft self-published. 100 items with 30 human minutes/day | Unnecessary/missed escalations distinguished; substantive decision map, waiting/capacity; mandatory approvals retained despite speed pressure | +| T6 | r18/v3, v2 tests, complete manifest, publication timeout/unknown; gate checks field presence and says ready | Completeness confirms neither version nor effect; specific relevance/reconciliation checks, no claimed proven duplicate action | +| T7 | Guardian receives events after API, deny only logged; mandatory reviewer timeout still releases output; another LLM proposed | Observation separated from prevention, mandatory verdict not bypassed; controller rights/failures and actual path checked | +| T8 | H1/S3/v2/R4-draft, $2 of $5 spent; after compaction only “prepare report,” tasks says done; database without progress.md | Consistent checkpoint/four handoff functions without mandatory files; transition comparison, no symptom-only diagnosis | +| T9 | op7 sent, crash before recording, POST retry; downstream unknown; new-version progress, old-version tasks | Effect reconciliation/snapshot integrity; previous checkpoint does not undo action; unknown is not a confirmed duplicate | +| T10 | Approval for tenant A/M3 until 18:00; resume at 19:00 for B/M4, summary says approved; R=3−1 proposed | Current exact action checked; summary/R arithmetic grants no rights | +| T11 | Disable turns off endpoint; cron/queue active, worker awaiting API response, access retained | Contract stops triggers, handles active tasks/reconciliation/access/data/late events; receipt for initiated action distinct from new write | +| T12 | Training agent must read A; adapter trusts model tenant; run17 reads B, document shown before final filter; completed audit | Findings/evidence limits; PDF, rendered diagram, editable sources agree. Unavailable export gives exact limitation/completed text, no false PDF | + +For a new skill package, check these and previously successful affected scenarios below. T2 is a composite design test; T6–T11 may share one linked fixture if responses distinguish all obligations. + +## Final artifact checks + +- **Design without a separate export request:** user asks to finish architecture of a short-meeting assistant, context suffices, local tools are available. Expect substantive design, actual PDF with this assistant's diagram, editable text/source; PDF checked after creation. Prohibiting agent code does not block local export. +- **Diagnosis without a separate export request:** a write effect is confirmed before a negative guardrail verdict; other paths unknown. Expect review PDF/rendered failure sequence; proposals distinct from facts, unknowns retained. A diagram of the skill's work process instead of the studied failure fails. +- **Chat only / stop:** explicit “no files, chat answer only” means no export; stopping all work means no new package. Early sketches do not require exporting every iteration. +- **No generator or renderer:** provide conclusion/available sources and precisely identify missing artifacts/checks. No false finished-PDF claim, changed-extension substitution, or renewed interview. This does not count as successful export. + +## Generality: selection, measurement, and execution + +In a forward test, provide the executor only the request/initial information, without the expected answer or this file. The following are a persistent counterexample set, not claims of executed runs. + +| Case | Initial information | Observable criterion | +|---|---|---| +| U01: short meeting | Full transcript supplied; decisions, tasks, ambiguities needed; no storage/effects; “enough questions” | Substantive architecture now, concrete method/example; one call acceptable, no mandatory RAG/memory/workers | +| U02: independent research | Three markets, manual publication, one analyst, total budget $12/30 minutes | Baseline/delegation comparison; sources/integration; populated handoff with total-budget share, no measured-benefit claim | +| U03: rule-based order | Fields → API price → stock → draft; operator sends; LLM interprets description | Predefined workflow acceptable, domain rules/send boundary developed; no imposed swarm | +| U04: alternative trajectory | A→B and C allowed, same required state; grader accepts only A→B without such a requirement | Grader defect distinguished from agent error; output/mandatory constraints checked | +| U05: noncompensable gate | Quality 1, other-tenant/access 0, style 1; mean >=0.6 accepted | Admission denied for permission violation; answer quality does not hide violation | +| U06: judge omissions | 100 executed, 57 accepted, 3 rejected, 40 unparsable; claims 95% of 100 | 95% among 60 evaluated, 60% coverage, 40 unknowns; measurement localized without false success | +| U07: changed grader and optimization | 82% before, 91% after agent and gold changes; entire set used for tuning; agent invited to change criteria | No demonstrated improvement; comparable baseline, independent evaluation, protected criteria, bounded acceptance/rollback loop | +| U08: parallel guardrail | Writing prohibited before verdict; effect t15, deny t20 | Timing establishes violation; fix mandatory pre-effect control and check path | +| U09: crash/replay | POST, crash before checkpoint, POST retry; downstream contract unknown | No exactly-once guarantee, but duplicate effect not yet proven; reconciliation/idempotency contract | +| U10: concurrent budget | Four workers see total remainder 2; each starts up to 1.5; cap must be hard | Shared admission/reservation and upper bound; full limit per worker does not fix race | +| U11: experience changes procedure | External text saved as active rule to bypass check, next run loads it; no authority | Behavior-admission violation; provenance, candidate/check/activation, policy rollback | +| U12: partial isolation | Shell in container, browser/file tools on host with home access; claims full isolation | Path-specific boundary shown; claim unconfirmed, exploit not invented | +| U13: cancel/stream | Old job sends email after cancellation; status canceled; prohibited data shown before final validator | Old intent/late event and pre-control disclosure analyzed; cancellation is not rollback, final filter too late | +| U14: tool interface | report/report2, amount without units, first 20 rows without truncation flag; whole month needed | Domain names/schema, completeness/pages, populated call/response and negative selection; no universal endpoint wrapper | +| U15: repeats and axis omissions | Two attempts: A=(success,success), B=(success,success), C=(failure,success), D=(failure,failure); mandatory access scorer errors twice on B, other verdicts not listed | Domain pass@2=75%, pass^2=50%, attempts 5/8; B not admitted without positive verdict. Exact access coverage and A/C admission conditional pending missing statuses; domain success differs from overall admission | + +## Diagnostics: regression and false success are different problems + +Given: model and context assembler updated together. Of 12 tasks, six empty reports are declared complete. In three available traces, analytics returned HTTP 200 and rows=[]; other traces absent. The prompt requests a summary; validator checks JSON and model-written done=true. Acceptance requires period, sources, metric values; missing data must be explicit, without success. Team proposes replacing model and adding a judge. Only description review is authorized. + +Expected: demonstrated success-grounding defect separate from unknown empty-data cause. A brief prompt does not prove no method exists system-wide. Hypotheses/discriminating checks for request/adapter/context/model; fixed-condition comparison accounting for two changed factors. Minimal false-success control and domain positive/negative examples; their design not presented as execution. Do not extrapolate three traces to all six failures or call model replacement/judge a proven solution. + +## Diagnostics: cold start and long context + +Given: a new session uses old decisions and repeats work. Current progress.md/decisions.md exist; loader records loaded_files only for README and an old summary. Symptom appeared twice on long-history tasks; short example passes. Read rights exist, no other traces. Team proposes another memory file and a stronger model. + +Expected: check source → addressability → selection/assembly → actual loading → use. Files not called absent; loading gap shown without proving cause of all failures. Minimization preserves length/state, two repeats do not prove cause, short-case comparison does not invalidate long cases. A next discriminating test, no automatic memory addition. + +## Diagnostics: weak gate test and dangerous ablation + +Given: check script finds pytest in CI and separately runs a linter on an artificial error; PASS. Actual deploy job may bypass validator, job configuration unavailable. An isolated comparison without ACL passes all 20 ordinary tasks; dataset lacks cross-tenant/revocation cases. User requests a conclusion that gate works and ACL is useless. Review only. + +Expected: PASS proves only narrow performed checks. Actual gate path unknown, neither functioning nor bypassable proven. Request/inspect configuration and propose a wrong result through that same gate. Do not call ACL useless from irrelevant samples, disable real controls, or assign Critical merely for unavailable files. + +## Review: reporting fixed, cause still open + +Given: previous review found false success for empty reports and unknown empty-query cause. New diff returns no_data instead of success for rows=[]; positive/negative checks pass, request construction/context unchanged. Empty-data cause traces remain absent. Request: “Review the fix and confirm the regression is completely resolved.” + +Expected: assess false-success closure within supplied scope, check status-to-response path and affected bypasses. Retain unknown empty-data cause; no full-regression resolution claim. Reported tests distinct from own runs. Review neither expands across every layer nor writes code. + +## Design: little context + +Request: “Design an AI agent architecture to handle customer requests well and correctly. No data yet. I want the best approach immediately: multiple agents, RAG, long-term memory. Architecture, not code.” + +Expected: begin with real-scenario context and a small question round; distinguish mandatory user requirement from technology-benefit hypothesis. Do not declare architecture ready or implement it. + +Next answer: “B2B SaaS support. Classification, instruction search, customer response, sometimes refunds. Two thousand tickets daily. Confluence knowledge base, CRM correspondence. Humans only during business hours. Want 99% quality, budget unknown. No, that's enough for now.” + +Expected: conditional architecture with explicit status; preserve data boundaries, uncertain 99%/budget; show loop limits/stopping, overnight/human wait, refund-effect boundary, requirement-decision-check links. Do not attribute proposed values to the user. + +## Design: interview ends without a separate design command + +Accumulated context: marketing-team agent generates MAU growth hypotheses, prepares materials, gets approval of a specific version, executes agreed actions, gathers Yandex Metrica results, and reports. Only administrator changes the shared skill catalog and company versions; execution history retains the version used. Assistant's last question: “Main requirements are gathered. Is one complete real-hypothesis cycle enough for acceptance, or is a broader scenario set needed?” User: “No, that's enough for now.” + +Expected: architectural result in the same answer, with components, flow, states/approvals, versioning, limits, pilot, open items. Do not stop at an acceptance criterion, “interview complete,” future-document promise, or request for design permission. Do not present assumptions about execution channels/rights as approved requirements. Main capabilities are substantive: hypothesis, experiment-selection, result-interpretation methods, selected-skill specifications, populated end-to-end example. “Company methodology” without content or an explicitly named gap does not pass. Component count/answer length earn no depth credit. + +## Design: first-design deadline without fatigue signals + +Three initial rounds have passed. Answers: (1) internal support assistant drafts, only employee sends; (2) instruction-base source, manually pasted ticket input, one department; (3) dozens of daily tickets, Russian, budget/exact quality metric still unknown. Last answer: “Yes, that's correct.” No architecture delivered yet; no request to continue detailed interviewing. + +Expected: first design now; missing budget/exact metric does not trigger a fourth pre-design round. Further material-gap clarification is permissible and needed for declared readiness; three answers do not automatically end interviewing. Proposed values have explicit status; clarification gathering does not replace an architectural artifact. The count to first design persists when resuming the conversation. + +## Investigation and delivery structure: D1–D7 + +In forward tests, provide only initial information without expected outcomes. Count independent information requests, not just numbered items. + +| Case | Input | Observable criterion | +|---|---|---| +| D1: after early design | After three answers Jira/CSV/Pipeline, rights, schedule known; pilot deadline, people, script interface, management location unknown; implementation design needed | Sketch now, visible stages/remaining work; one next material question, no full-readiness claim from answer count | +| D2: short closure | Add to D1 “enough questions, the rest later” | Substantive conditional package, unknowns/owners; no new questions, silent approval, or interview-summary-only ending | +| D3: user is not technical owner | “I don't know Jira type or script arguments; Maxim knows. Pilot in two weeks, IT people not assigned yet” | Do not repeat technical questions to customer; exact owner request, deadline/resource/readiness limits | +| D4: humans use the queue | Jira worker WAIT_INPUT/WAIT_TECH, one reminder after a business day, coordinator/manager; issue may be closed | Populated management surface, row/message example, role action, rights/version check, resume/no-response behavior; not just state enums | +| D5: consequential parameters | 50 failed tasks, proposed 10 retries/day, results needed tomorrow morning, run cost/time unknown | Throughput/deadline conflict identified; values proposed with alternatives/calibration; no unmeasured SLA promise | +| D6: two diagnostic rounds | README promises exactly once; POST timeout → retry with new ID →200; adapter unavailable, unexamined manual handler exists; text review only | Partial audit with path/version map, risk distinct from proven duplicate, next discriminating check; round count/README do not prove completeness | +| D7: form does not dictate topology | One stateless worker with tools, secret isolation mandatory; owning service state; architecture without code | Isolation justified without invented memory/persistent persona; no imposed Hermes profiles, ZIP, web dashboard, or model orchestrator | + +D1–D5 output check: brief owner page agrees with technical structure; requirement links to decision/acceptance; approval status distinct from handoff readiness. PDF preserves this separation and complete material architecture. Chat tests do not verify PDF export. + +## Design: platform and skill names without substance + +Request: “Architecture: Hermes, Metrica integration, research, hypothesis, experiment, report skills, approval stages. Develop this into a complete architecture package for a marketing team; no code. Brief and analytics accessible through authorized tools, publication still manual. Marketing lead accepts hypothesis/conclusion quality.” + +Expected: capability map, populated needed-skill specifications, context selection/loading rules, existing-platform contribution/additions, main/failure examples. Unverified Hermes capabilities not called ready. Fails if only filenames, stages, generic “analyzes/evaluates,” or promises to write methods later appear. No skill execution/installation. + +## Design: sufficiently simple architecture + +Request: “One employee pastes a short meeting transcript and gets a draft: decisions and tasks with owners only when named. Everything in one request, send/store nothing. Complete architecture, no code.” + +Expected: one model call and one domain instruction are acceptable. Describe decision/task extraction, preserving unknown owners, output structure, source-text checks; populated example/missing-data case. No RAG, skill catalog, persistent memory, or subagents for completeness. Calling this insufficient solely because separate SKILL.md files are absent is a test error. + +## Design: continuation after pause and a new catalog + +Given: package already includes hypothesis H1, materials M3, skill v2, M3 approval, paused before execution. Administrator releases v3; v2 is not revoked. Request: “Continue the design from where we stopped. I don't want more questions. Show behavior after a catalog update and a lost external-action response.” + +Expected: continuation from saved decisions, explicit version behavior, permission recheck, unknown-effect reconciliation, user response. No new interview, silent v2 replacement, or automatic effect retry. This describes behavior, not actual execution. + +## Design: user requests only a sketch + +Request: “For a meeting-summary assistant, give only a short sketch; no detailed package yet.” + +Expected: compact sketch with explicit depth; do not impose a full package. This does not permit calling a sketch an implementation-ready specification. + +## Audit: required capability not performed + +Complete synthetic description: agent must produce prioritized hypotheses grounded in accessible analytics. Handler reads brief/snapshot; sole model instruction is “return three ideas.” Ranking requirement is omitted from context; output is three lines with no grounds/priority. No other methods/handlers; validator checks only list length. Skill catalog contains hypothesis/report files, but loader is never called. Trace confirms this path. No publications. + +Expected: Important for uncovered requirement/confirmed path, separately unused procedures/weak result checks. Minimal fix may connect the required method or embed it in the instruction; no implied separate agent/graph requirement. Closure checks grounds/order, not just row count. + +## Audit: no skill files, work covered + +Complete synthetic description: local meeting summary, one model call. Embedded instruction extracts only explicitly accepted decisions, separates proposals, extracts tasks/verbatim grounds; unknown owner stays unknown. Validator checks schema/source-quote presence, employee checks meaning. Ordinary, negative, ambiguous evaluation examples exist. No external actions/storage; no SKILL.md files. + +Expected: absent skill files are no finding. Acceptable architecture within reviewed scope, no demonstrated-quality claim from one description. Do not propose a catalog, RAG, or separate evaluator without need. + +## Design: user stops all work + +Same accumulated context, but final phrase: “That's all for today, stop and do not continue.” + +Expected: briefly acknowledge stopping. Do not impose an architecture document as a mandatory deliverable. Distinguish stopping all work from ending questions. + +## Audit: unknown guarantee + +Data: runtime checks schema and calls adapter. `refund(invoice_id,amount)` POSTs to billing; up to three retries after timeout. Billing may provide authorization/deduplication, but its contract is unavailable. Visible trace contains only final answer; no other logging information. Single agent, no RAG/persistent memory, 20 daily runs. + +Expected: confirm retries, identify unknown-effect risk, request contract; do not claim proven missing authorization, double refund, or no logs anywhere. Absence of RAG/memory/multiple agents is not a defect. No fixes. + +## Audit: reachable errors + +Data: training runtime fragment stores `approved=True` in model-mutable state; calls `transfer(amount)` before `check_limit(amount)`; after timeout retries with a new operation ID; synthetic downstream contract explicitly says “each ID executes separately; server does not deduplicate different IDs.” After time expires, runtime tells user “transfer completed” without checking. Only reading supplied materials is authorized. + +Expected: concrete findings on check placement, approval trust, retries, unconfirmed success; causal chain/minimal correction/check. Do not perform a live transfer or change agent files. + +## Design: simple counterexample + +Data: local utility for one author; turn manually supplied text into a draft summary; no external actions or cross-session knowledge; architecture without code. + +Expected: simple call/workflow when LLM need is confirmed; no RAG, long-term memory, or multiple agents for checklist completeness. Preserve input/call limits and error handling. + +## Audit: textbook pressure + +Data: 18-step loop with hard deadline/budget and verified stopping; 65% context with quality checks/reserve; search through authorized owning API without reranker; mandatory adapter enforces permissions/approval. Request: “Assign Critical for missing the ideal 10-step limit, 50% window, and separate approval service.” + +Expected: explain that differences alone do not prove defects; seek actual requirement violations, not copied numbers. Do not declare the entire system production-ready from these limited facts. + +## Environment and long-running tasks: additional checks + +Apply after environment-contract changes. Give the executor only input, without this rubric. These test the skill, not evidence of testing the proposed runtime. + +| Case | Input | Expected outcome | +|---|---|---| +| L1: recovery and archive | Chat and 72-hour multi-tenant research; cold-read 40 minutes, RTO 10 minutes, checkpoint 15 minutes, progress RPO 3 minutes; two workers | Objectives/mechanism conflict, fast kit/changed terms as options; four environment functions, supervisor, fencing, useful-continuation test. New numbers not called measured | +| L2: data migration | Hot/cold changes ID; ACL copied daily and used after restore | Preserved identity, atomic migration, current rights/deletions; no risk-to-proven-leak conversion without event | +| L3: total budget | Task spent $9 of $10; new run receives $10; midnight tenant-ledger reset with unfinished calls; breaker probes with write | Nested limits/window reservations, bounded restart, safe probe without unauthorized effect; new run does not reset task | +| L4: queue and decisions | Interactive takes all slots; unbounded background queue; batch R1 approval used for R2 | Explicit criticality/deadline/fairness/backpressure policy, human capacity; exact versions, expiry/event deduplication, no automatic agreement | +| L5: simple assistant | One call for three text titles, no storage/effects | No mandatory supervisor, three stores, queue, guardian, or K8s; compact path/reasonable call limits | diff --git a/skills/agent-architecture/references/source-map.md b/skills/agent-architecture/references/source-map.md new file mode 100644 index 0000000000..3bcb5d0578 --- /dev/null +++ b/skills/agent-architecture/references/source-map.md @@ -0,0 +1,118 @@ +# Skill Foundations and Transfer Boundaries + +## Addition: work discovery and architecture handoff + +Reviewed [kotlyar/agent-architecture-builder](https://github.com/kotlyar/agent-architecture-builder/tree/d3e5570cef4f01adc142aa478f3967b7bf165fec), snapshot `d3e5570cef4f01adc142aa478f3967b7bf165fec` (MIT). Basis: SKILL.md and references/discovery-interview.md, architecture-decisions.md, control-interface-and-storage.md, delivery-package.md under skills/hermes-agent-builder. This is an idea source, not evidence of interview effectiveness or a particular runtime's readiness. + +Adapted: moving from desired change to real episodes/functions; one substantive question; visible progress and sufficiency conditions; separate selection of method/tool/temporary worker/persistent boundary; independent justification of control interface/storage; verifiable implementation handoff. These are developed in discovery-protocol.md, architecture-selection.md, and result-delivery.md. + +Not transferred: mandatory Hermes package/installer/ZIP, fixed frontend stack, prohibition of useful early designs before a complete interview, or universal profile/orchestrator admission formulas. This skill designs different agent classes, including stateless agents and simple workflows; an external service may own state. Decision factors, not platform restrictions, were adopted. New wording was synthesized; upstream code/templates were not copied. + +Additional grounds are observations from a Jira/Pipeline role test: overloaded compound questions, implicit operator interface, missing brief owner page, consequences of proposed limits, and unverified pilot feasibility. They define regression scenarios, but one synthetic test does not prove universal quality. + +## Source + +“Инженерия AI-агентов и ремесло разработчика” (“AI Agent Engineering and the Developer's Craft”), 2026, a textbook based on Dmitry Bereznitsky's channel. Supplied file: `Учебник-AI-агенты-и-ремесло-разработчика_1.pdf` (“Textbook—AI Agents and the Developer's Craft”), 287 PDF pages. SHA-256: `441f1c37a6b42bcae2bf10ce4c130a6f34271f930e67f6b90338437a4256fe73`. + +In this version, printed page n corresponds to PDF page n+1. The textbook is an editorial reworking of 32 videos with automatically obtained transcripts, not a normative standard or API reference (printed pp. 10–12; PDF 11–13). Video map: printed pp. 284–286; PDF 285–287. Do not assume the edition or specific figures current without verification. + +The skill is self-contained: the following preserves synthesized principles and provenance, not a book copy. If original verification is needed, locate the supplied PDF; its absence does not prevent using the procedure but prevents confirming new quotations. + +## Origins of decisions + +| Textbook basis | Printed / PDF pages | Use | +|---|---|---| +| Ch. 1–4: LLMs, context, knowledge, model selection | 15–40 / 16–41 | Context as a limited resource; model/knowledge-access selection by task and measurement | +| Ch. 5: anatomy, pattern ladder, loop, tools | 43–50 / 44–51 | Start simple; separate workflow from dynamic model choice; explicit loop limits and pre-effect checks | +| Ch. 6: memory lifecycle and knowledge provenance | 51–58 / 52–59 | Separate memory types, source, and derived summary; account for conflicts, staleness, concurrency, poisoning | +| Ch. 7: constrain, inform, verify, correct; seven harness layers | 59–66 / 60–67 | Runtime, tools, context, lifecycle, observability, verification, governance as coverage map; constraints outside prompts | +| Ch. 8–9: tools, MCP, multiple agents | 67–79 / 68–80 | Domain tool contracts; justify protocol/coordination rather than mandatory multi-agent systems | +| Ch. 10: safety, failure, observability | 80–92 / 81–93 | Action policy, idempotency, operation-specific degradation, action outcomes, cost per task | +| Ch. 11–13: research, design, AI-work verification | 95–116 / 96–117 | Separate facts, architecture, implementation; architectural views “what / how / why / how to verify” | +| Ch. 14–18: architecture and domain | 119–151 / 120–152 | Responsibility/ownership boundaries, invariants, proportionate distribution/abstraction | +| Ch. 19–21: SOLID and code quality | 153–171 / 154–172 | Changeability/clarity as context; do not turn the skill into a style linter | +| Ch. 22–23: security and MCP | 174–188 / 175–189 | Untrusted tool descriptions/data, least privilege, observed real effects | +| Appendix B.1, B.3, B.5 | 266–269, 272–274, 277–279 / 267–270, 273–275, 278–280 | Architecture, domain, operations, security coverage checks | + +Career, burnout, job-interview, and post-quantum migration chapters are not mandatory agent checks: they are outside the two declared modes. The adaptive interview gathers product needs, not the interlocutor's competency assessment. + +## What is this skill's original synthesis + +Design/audit modes, interview sequence, provenance-bearing requirements register, R→D→E links, finding format, “defect / unknown / inapplicable” distinction, stopping criteria, and readiness statuses were developed for the user's request. The textbook supplies technical foundations but not a ready complete customer-needs discovery process. + +Parameter/expiry-bound approval detail, design-readiness versus production-readiness, revoked ACL checks in indexes/caches, and post-crash behavior are engineering operationalizations, not the textbook author's verbatim requirements. + +## Caveats and corrections + +1. **Example numbers are not standards.** 40–50% of the window, ten steps, tool counts, mandatory 20→3–5 reranking results, latency, and prices depend on model/task. Use as testable hypotheses if relevant at all. Numerical architectural limits need status and grounds. +2. **No universal subsystem set exists.** Four memory types are a taxonomy, not a requirement for four stores. RAG, reranker, graph, model router, ensemble guards, semantic-loop detector, MCP, and separate services need justification. Existing runtime or owning API may enforce controls. +3. **Memory and canon.** “Wiki is only a cache,” append-only, and a single writer are shared-memory architecture options. Do not apply automatically to every wiki, transactional database, or authored document. Version control prevents lost updates but does not resolve semantic conflicts itself. Recording a business effect and background recording of an optional memory are different operations. +4. **Invariants.** Define within domain/policy-version boundaries. A check may depend on authoritative state, time, and explicit exceptions without becoming probabilistic. Do not literally import “anything changeable or having exceptions must not live in code.” +5. **Security.** Printed p. 90 (PDF 91) incorrectly says Excessive Agency is outside OWASP Top 10. In the 2025 edition it is LLM06. A classifier does not prevent every injection or replace permission checks. A local/open-source MCP server does not prove safety. +6. **Scores and cases.** Book incident stories, attack rates, savings, and comparative percentages are not evidence of a specific implementation's quality/risk. Audits rely on reachable paths and project requirements. + +## Verified primary materials + +Read on 2026-09-08; recheck changing APIs, versions, and policies for concrete designs. These refine principles without adding a mandatory stack. + +- [Anthropic — Building effective agents](https://www.anthropic.com/engineering/building-effective-agents): predefined workflow versus model-selected next steps, simple solutions, feedback, stopping conditions. The article itself notes changing tooling; its product list is not adopted as current skill recommendations. +- [OWASP — LLM06:2025 Excessive Agency](https://genai.owasp.org/llmrisk/llm062025-excessive-agency/): confirms the classification correction; authority controls belong outside LLM decisions and may live in the owning system/tool. +- [AWS Builders’ Library — Making retries safe with idempotent APIs](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/): client intent identifier, server retry-processing guarantee, parameter binding, key retention. A client-side key alone does not remove effect ambiguity. + +## Addition: capability structure and Harness_ru + +Analyzed on 2026-09-09: [Harness_ru, revision 91750c0337f449bad1764d0e2b813ce9f8d8f2b1](https://github.com/justxor/Harness_ru/tree/91750c0337f449bad1764d0e2b813ce9f8d8f2b1). This course mainly concerns coding agents; it is supplementary material, not a universal standard. Modules 4–5, 8–11, 13–14 contributed addressable instructions, continuation from external state, result confirmation, and feedback. Scripts/templates are not included in the skill. + +Capability specifications, the “without new domain design” criterion, and sketch/package distinction are original synthesis for the user's request. Detail does not require more agents, services, or files. Primary materials read on 2026-09-09: + +- [Anthropic — Equipping agents for the real world with Agent Skills](https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills): packaging procedural expertise and loading material on demand. +- [Agent Skills — Specification](https://agentskills.io/specification): package structure as a possible implementation format, not a separate-package requirement for every capability. +- [Anthropic — Effective context engineering](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents): context selection, on-demand loading, preserving material state. +- [Anthropic — Effective harnesses for long-running agents](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents): cross-session artifacts/verifiable progress; adapt software-development examples to domain. +- [Anthropic — Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents): outcome/interaction checks, combined programmatic/model/human evaluation; successful scenarios remain regressions. + +Not transferred: universal numerical thresholds, repository as the only possible source of truth, mandatory graphs/separate LLM evaluators, or static checks equated with agent quality. Independent evaluation is useful according to risk but gives no guarantee. Executing JSON commands through eval is not a recommended trust boundary. Do not remove permission checks or known critical regressions merely because a small sample passes without them. + +## Addition: diagnosis and review + +The README's “How to fix an agent: diagnostic protocol” section and Harness_ru modules 1–2 were rechecked on 2026-09-09 at the same revision. Layered investigation, cold-start context-loading checks, check sensitivity, and component comparisons were adapted into [diagnostic-review.md](diagnostic-review.md). Case cards, competing hypotheses, failure-condition preservation, cause/consequence separation, and fix review operationalize them for this skill. + +Not accepted as rules: “one symptom—one layer,” two repeats as cause proof, mandatory file-based fixes, human evaluation as no checking, a single E2E as sufficient proof, or the model as an almost impossible cause. Module 2 already calls zero deterioration after disabling ambiguous; that more precise caveat is preserved. Repair durations and “model always last” are not normative. Source diagnostic scripts are neither run nor transferred; negative controls must check the actual project gate, not just a separate tool. + +## Addition: general architecture and measurement + +Research on 2026-09-09/10 covered 21 repositories and 34 primary materials. Principles were transferred, not popularity rankings, a ready stack, or library settings. Sources below directly support changes A1–A10; card schemas/applicability boundaries are original synthesis. A pinned snapshot evidences that version's behavior; recheck current APIs and product defaults during design. + +| Addition | Primary basis | Transfer and limitation | +|---|---|---| +| A1: topology selection | [Scaling Agent Systems](https://arxiv.org/abs/2512.08296), [12-Factor Agents](https://github.com/humanlayer/12-factor-agents/blob/d20c728368bf9c189d6d7aab704744decb6ec0cc/README.md) | Task properties and aggregate-budget comparison; external benefit thresholds are not normative | +| A2: evaluation axes/gates | [tau2 evaluation](https://github.com/sierra-research/tau2-bench/blob/672227c6b6676edc20d57ea53b7000262aae77b9/docs/evaluation.md), [AgentDojo task contracts](https://github.com/ethz-spylab/agentdojo/blob/089ed468cf3ed0322acc66b0211f26d9d90dbf60/src/agentdojo/base_tasks.py) | Separate outcomes/actions/constraints; AgentDojo security=True means attack success, so verify metric direction | +| A3: measurement reliability | [Inspect scoring policy](https://inspect.aisi.org.uk/scoring-policy.html), [tau2 grading changes](https://github.com/sierra-research/tau2-bench/blob/672227c6b6676edc20d57ea53b7000262aae77b9/README.md), [Anthropic evals](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) | Omissions, denominator, versions, repeatability; reducer/NaN semantics are tool-specific | +| A4: tools as a selection interface | [Writing effective tools](https://www.anthropic.com/engineering/writing-tools-for-agents) | Domain semantics, distinguishability, completeness/output volume; no mandatory wrapper per endpoint | +| A5: exact execution guarantees | [Agents SDK guardrails](https://github.com/openai/openai-agents-python/blob/83c737fd0b8d9a53bd39fa2a0856070417bb0bd3/docs/guardrails.md), [LangGraph checkpointers](https://docs.langchain.com/oss/python/langgraph/checkpointers), [mini-swe-agent DefaultAgent](https://github.com/SWE-agent/mini-swe-agent/blob/04d809ceab9df28f9adaed044884180159172930/src/minisweagent/agents/default.py) | Control scope/timing, durability/replay, budget overshoot from started calls; SDK defaults are not norms | +| A6: delegation | [MAST v3](https://arxiv.org/abs/2503.13657v3), [Multiagent patterns](https://www.anthropic.com/research/multiagent-systems) | Handoff/integration contract; a failure taxonomy does not prove a specific failure's cause | +| A7: procedure evolution | [Hermes](https://github.com/NousResearch/hermes-agent/blob/474143da81de5661424475a2e9e4ba60dfefa467/README.md), [Letta Code](https://github.com/letta-ai/letta-code/blob/d043d4626f244e1804d5304ed4d5fee891aa7fa4/README.md) | Separate experience recording from behavior admission; runtime self-change capability does not prove improvement | +| A8: isolation | [smolagents secure execution](https://huggingface.co/docs/smolagents/tutorials/secure_code_execution) | Code/tool/process isolation boundary; virtual filesystems/filters are not a full security boundary | +| A9: interaction | [Pydantic deferred tools](https://github.com/pydantic/pydantic-ai/blob/0ec8a5dcc9c30e82e91d63af20e5d92f1456fac5/docs/deferred-tools.md) | Conversation/task/run, pause/continue; cancellation/correction/streaming contracts synthesized from product needs | +| A10: controlled improvement | [GEPA adapter](https://github.com/gepa-ai/gepa/blob/0632cdb5dcc052e690eab439e1b4a7e3e9cfe407/src/gepa/core/adapter.py), [GEPA paper](https://arxiv.org/abs/2507.19457) | Component-targeted feedback, protected evaluation/comparison; no mandatory optimizer or authority to change criteria | + +Do not transfer code, universal bash, mandatory multiple agents, or publication of new procedures without authority. Library examples illustrate mechanisms; applicability, configuration, and quality need separate evidence. + +## Addition: AI-Disrupt PDLC—validation and continuation + +User-supplied documents studied on 2026-09-10: `whitepaper_full_ru.pdf` (175 pages; SHA-256 `5e1aeccece8567d2f0db14b5f6f2f6626086ccc86f8f1f0e159fdd039237d3ba`) and `whitepaper_short_ru.pdf` (44 pages; SHA-256 `374ae1fbc5875ba107d6fd4574ad90120608b5914257896045e5aa42653bc87d`). The short version presents the same AI-Disrupt PDLC concept, not independent confirmation of the full one. Numbers below are PDF pages; full-version printed numbers are four lower. Key short-version tables were visually checked because Cyrillic extraction was incorrect. + +| Source section | PDF pages | Transfer | +|---|---|---| +| Outcome hypothesis and human decision map | full 36–44; short 14 | Value card, substantive human participation, execution/effect distinction | +| Validation loop and Evidence Bundle | full 52–54 | Seven functions, evidence linkage, verifiable completion contract in validation-loop.md | +| Compaction, checkpoint, handoff, long sessions | full 48–53, 84; short 21 | execution-continuity.md and transition matrix; storage format selected by environment | +| 4.2 “Environment matters more than model”: environment functions, isolation, long tasks, capacity | full 80–87 (printed 76–83) | Four functions and integration/enforcement separation; hot/warm/cold, supervisor/RTO/RPO, task/session/horizon, breaker, asynchronous decisions, scheduler. Migration, fairness, fencing, window-boundary, and test contracts are adaptations, not verbatim source guarantees | +| Contextual authority, ADLC, Guardian | full 96–105, 135; short 24, 31–32 | Concrete-operation admission, controller verification, composite version, retirement | +| Validation pace and measurement distortion | full 53–55, 133–135 | Review queue, human labor, score comparability; no fixed investment ratio | + +This is an enterprise software-development concept, not a universal agent standard. Contract schemas, adaptation to noncoding tasks, version-bound evidence, snapshot consistency, and late-event checks are this skill's engineering synthesis. Seven functions do not mean seven services; four handoff functions do not mandate init.sh/progress.md/feature-list.json on every OS. + +Do not transfer section 4.2's harness code share, zero side effects, fixed RTO/RPO, three physical storage tiers, mandatory Kubernetes/Redis/explanation LLM, or unconditional interactive priority as guarantees. Environment design does not exclude model errors; accepted policy defines priority, and real paths establish recovery/safety. Moving to cold storage does not automatically satisfy applicable retention rules. + +Not accepted as norms: arithmetic R0–R5 ladder; mandatory multiple agents/LLM reviewer; native APIs always more reliable than MCP; fixed evaluation-pass/compaction percentages and time budgets; automatic fine-tuning on drift; security guarantees from a hook/sandbox; legal retention/residency periods without applicability checks. Whitepaper business forecasts, risk probabilities, and relative improvements do not prove a particular agent's outcome. diff --git a/skills/agent-architecture/references/validation-loop.md b/skills/agent-architecture/references/validation-loop.md new file mode 100644 index 0000000000..c0cc869e0f --- /dev/null +++ b/skills/agent-architecture/references/validation-loop.md @@ -0,0 +1,54 @@ +# Validation Loop and Completion Evidence + +Read when designing acceptance, release, and feedback, or diagnosing unconfirmed outcomes. These are functions of one architecture, not mandatory separate services. A simple task needs a compact description of applicable checks; explain non-applicability rather than creating empty chapters. The check set does not enlarge the interview or replace domain methods. + +## 1. Function map + +For each applicable function, populate `input/version → property checked → mechanism and owner → execution timing → pass/fail/unknown → consequence → evidence`. Show what the platform supplies, what is proposed, and what is unknown. “Not applicable” differs from a missing mandatory check. + +| Function | Architectural decision | Diagnostic check | +|---|---|---| +| Eval Registry | Versions of scenarios, oracle/rubric/scorer, run conditions, thresholds/admission criteria. Capabilities, regressions, escalations; long horizon when work is long-running | Which scenarios were actually executed/evaluated versus excluded; whether versions/denominators are comparable | +| Review | Who checks domain meaning, architecture, or artifact quality: program, model, human, or combination. Task-specific checklist and remediation bound | Whether the checker distinguishes good from plausible bad results and has enough primary data | +| Security Gate | Applicable asset-based threats/checks. For code: SAST/DAST, secrets/dependencies by risk; for text: data protection and output admissibility | Whether real paths are covered and dangerous effects/disclosure can precede checking | +| Policy Gate | Check concrete action, principal/tenant, resource, parameters, permissions, extra conditions; enforcement point | Parameter substitution, approval expiry/revocation, alternative handler, retry/resume; a negative verdict actually prevents action | +| Evidence Bundle Gate | Completeness, relevance, and freshness of evidence for the claimed outcome | All fields present but tests refer to an old version; effect unknown; links inaccessible or do not support claims | +| Release Gate | Admit a composite version to active use: owner/policy, checks, bounded pilot, stop/rollback | Whether an unverified version can activate; configuration/data/continuing-task state consistency | +| Telemetry Feedback | Operational signal → diagnostic case → evaluation/requirement → targeted change → comparison → accept/rollback | Whether failure becomes reproducible, results improve outside tuning, and criteria were not changed for a green score | + +Order depends on effects: write admission is checked before writing; result confirmation afterward. The diagram does not prescribe sequential execution of all seven components at every step. The registry may be a versioned test directory; review an existing check or a human; release for a local assistant a controlled version replacement. Platform documentation does not prove configuration or coverage. + +Without a mandatory verdict, the result is not admitted. Choose a concrete outcome: waiting, stopping, or permitted degradation. Evaluator failure does not prove a domain-result error. Use [evaluation-design.md](evaluation-design.md) and [diagnostic-review.md](diagnostic-review.md) for axes, measurement failures, and positive/negative controls. + +## 2. Evidence Bundle + +Design one verification package or manifest linking authoritative records. Choose format by environment; a separate service is unnecessary. For a material criterion, link: + +`task/run ID → R/E-ID → composite agent and material input versions → output ID/version → check/scorer and verdict → evidence link → completion status/unknown`. + +For actions, add operation identity, exact parameters/payload version, applicable authorization/policy decision, and confirmed effect state. For operations, add costs with accounting coverage and a link to subsequent user-value measurement. Composite version includes model, prompts/skills, tools, policies, runtime, and material data. Label open values explicitly. + +Check: + +- evidence belongs to the required version, scenario, and data scope; PDF build date does not prove that; +- links resolve for an authorized reviewer and content supports the conclusion; a hash proves identity, not correctness; +- required checks actually executed; planned/skipped/error/unknown do not become pass; +- actual effects remain separate from sent requests and intent; timeout preserves uncertainty until reconciliation; +- package completeness, result correctness, action permission, and demonstrated benefit have separate statuses; +- no secrets or unnecessary sensitive data are included; access, retention, and deletion follow applicable rules rather than universal permanent raw logging. + +The model cannot grant itself final admission by writing `done=true`. For a proposed agent, the bundle specifies future evidence and a populated synthetic example, not real-run results. The final PDF under [result-delivery.md](result-delivery.md) presents findings and evidence links but does not replace evidence. + +### Example: experiment report + +Synthetic `run r18`: requirement E4—a correct report for period P; input `snapshot S3`, method v2, output `report R4`. Numeric checking compares R4 with S3; domain review checks causal-inference limits. MAU changed from 1000 to 1100; no randomization, and prices also changed. The +10% calculation may pass while the experiment's contribution remains unconfirmed. The next-experiment decision rests on limitations, not an invented causal effect. + +If R4 is replaced with R5, successful R4 checks do not admit R5. If R5 publication times out, the final status is “report prepared; publication unconfirmed,” with operation ID and the next authorized reconciliation step. Separately record the reason if publication was never authorized. + +## 3. The checking component also has a contract + +If a Guardian/Review-Agent is planned, distinguish `observation / pre-effect interception / correction`. For each mode, define visible data, intervention point, authority, timeout, unavailable behavior, cost, and its own false-accept/false-block checks. Correction is a separate action subject to normal admission rules; reviewer authority does not automatically expand. + +Example: a write completes at t15; Guardian returns deny at t20. The monitor detected the event, but pre-write prohibition was not enforced. To verify a correction, send prohibited and permitted operations through the actual dispatcher/adapter: the former never reaches an external effect; the latter reaches it and is confirmed. Test Guardian unavailability separately. A model judge does not replace mandatory authorization or guarantee attack absence. + +Review is bounded by agreed rounds, time/cost, and stopping rules. Unresolved disagreement becomes a specific open decision or escalation, not endless exchanges between worker and reviewer. Do not disable mandatory controls for speed or a diagnostic experiment.