From dada6192e933d4ac342534f70cf7565c8bf60219 Mon Sep 17 00:00:00 2001 From: Codex Date: Sat, 12 Sep 2026 21:53:00 -0300 Subject: [PATCH] feat(methodologies): integrate task, error, interaction, and variability analysis --- ...k-error-interaction-variability-methods.md | 22 +++++ skills/select-methodologies/SKILL.md | 8 +- .../references/00-taxonomy-selection.md | 4 + .../06-testing-reliability-safety-security.md | 24 +++++ .../references/10-composition-patterns.md | 16 ++++ .../references/11-source-registry.md | 5 + .../18-learning-training-education.md | 2 +- .../20-design-experience-communication.md | 48 ++++++++-- ...35-human-factors-health-medical-devices.md | 91 ++++++++++++++++++- tests/scenarios/selection-invariants.md | 24 +++++ 10 files changed, 231 insertions(+), 13 deletions(-) create mode 100644 docs/adr/0005-integrate-task-error-interaction-variability-methods.md diff --git a/docs/adr/0005-integrate-task-error-interaction-variability-methods.md b/docs/adr/0005-integrate-task-error-interaction-variability-methods.md new file mode 100644 index 0000000..6046944 --- /dev/null +++ b/docs/adr/0005-integrate-task-error-interaction-variability-methods.md @@ -0,0 +1,22 @@ +# ADR 0005: Integrate task, error, interaction, and variability analysis + +- **Date:** 2026-09-12 +- **Status:** accepted + +## Context + +Task and cognitive analysis appeared as generic catalog entries. Users needed actionable selection boundaries and procedures for HTA and CTA, followed by credible task-error analysis, practiced interaction-time comparison, and analysis of everyday sociotechnical variability. + +## Decision + +Keep HTA, CTA, and SHERPA in human factors (`35`), GOMS/KLM in interaction design (`20`), and FRAM in safety and systems (`06`). Add canonical cards with prerequisites, procedures, outputs, sources, and evidence limits. Route training and UX task-analysis uses to the human-factors owner. Record ownership in the taxonomy and expose optional compositions without requiring every method. + +HTA supplies goals, operations, and execution plans; CTA elicits cognitive demands and expert cues. SHERPA identifies credible task errors and recovery. KLM estimates execution time for a specified practiced, error-free method. FRAM distinguishes a potential-coupling model from scenario/event instantiations. No model or worksheet alone establishes achieved performance, calibrated error probabilities, causal proof, or safety. + +## Alternatives and consequences + +Separate domain-specific copies or new skills would duplicate definitions and weaken routing. A generic task-analysis entry would leave these different outputs and assumptions hidden. The selected owners preserve one definition per method while allowing downstream tailoring. + +Source consultation is recorded in the [registry](../../skills/select-methodologies/references/11-source-registry.md). Where original full protocols were unavailable, the cards state that limitation. Exact protocol replication and effectiveness claims require further evidence. + +Repository validation checks structure and links. The [selection scenarios](../../tests/scenarios/selection-invariants.md) define future behavioral evaluation expectations; they are not reported as executed model benchmarks. diff --git a/skills/select-methodologies/SKILL.md b/skills/select-methodologies/SKILL.md index 461c679..5d6d161 100644 --- a/skills/select-methodologies/SKILL.md +++ b/skills/select-methodologies/SKILL.md @@ -72,7 +72,7 @@ Read only the files needed: | Process discovery/design, adaptive flow, operations, service management, quality, or continuous improvement | `references/03-process-operations-quality.md` | | Software lifecycle, architecture, APIs, distributed systems, or delivery | `references/04-software-systems-architecture.md` | | Requirements, planning, formal specification, contracts, or state reasoning | `references/05-requirements-formal-planning.md` | -| Verification, testing, reliability, safety, security, or assurance | `references/06-testing-reliability-safety-security.md` | +| Verification, testing, reliability, safety, FRAM, sociotechnical performance variability, security, or assurance | `references/06-testing-reliability-safety-security.md` | | Data/information governance, metadata, semantics, knowledge management, data quality, lineage, or provenance | `references/07-data-information-knowledge.md` | | Decision analysis, optimization, uncertainty, or causal inference | `references/08-decision-optimization-causality.md` | | Probability, statistics, sampling, estimation, prediction, or statistical inference | `references/37-probability-statistics-inference.md` | @@ -102,7 +102,7 @@ Read only the files needed: | Crisis leadership, organizational resilience, business continuity, BIA, incident/emergency management, disaster recovery, exercises, or recovery | `references/32-crisis-continuity-emergency-management.md` | | GIS, spatial-data quality, geocoding, spatial statistics/econometrics, remote sensing, service areas, suitability, or location decisions | `references/33-geospatial-location-analysis.md` | | Asset management, criticality, maintenance strategy, RCM, RBI, condition monitoring, spares, obsolescence, or lifecycle renewal | `references/34-asset-maintenance-reliability.md` | -| Human factors, ergonomics, human-systems integration, workload, medical devices, clinical evidence, usability engineering, or medical QMS | `references/35-human-factors-health-medical-devices.md` | +| Human factors, ergonomics, task/subgoal analysis, HTA, CTA, expert decisions/cues, human-systems integration, workload, medical devices, clinical evidence, usability engineering, or medical QMS | `references/35-human-factors-health-medical-devices.md` | | Construction, infrastructure, BIM, ISO 19650, constructability, Last Planner, commissioning, or built-asset handover | `references/36-construction-infrastructure-bim.md` | | Urban, territorial, regional, land-use, mobility, public-space, or place-based planning | `references/38-urban-territorial-planning.md` | @@ -110,6 +110,10 @@ For a cross-domain problem, begin with the primary decision catalog and at most When a prompt says only “uncertainty,” identify the required output before routing: probability/statistical inference, decision choice, optimization, causal effect, forecast, risk treatment, or evidence confidence. Route statistical/probabilistic inference to `37`; route choices, optimization, causal identification, and decision robustness to `08`. Use both only when the statistical result is an input to a distinct decision or causal design. +For task analysis, route goal/subgoal decomposition and execution plans to Hierarchical Task Analysis (HTA); route expert cues, judgments, strategies, and cognitive demands to Cognitive Task Analysis (CTA). Both are owned by `35`; use training (`18`) or UX (`20`) only for a distinct downstream design decision. Clarify ambiguous `HTA`: Health Technology Assessment is a different method in the same catalog. HTA is not Hierarchical Task Network (HTN) planning, and a cognitive walkthrough is not a substitute for eliciting expert cognition. + +For credible task errors and recovery, select SHERPA in `35` with an HTA and practitioner evidence. For modeled execution time of practiced, error-free interaction, select GOMS/KLM in `20` with a specified interface and user method. For variability and couplings in everyday sociotechnical work, select FRAM in `06` with work evidence and scenario instantiations. These outputs answer different questions; do not load or compose all three by default. + ### 3. Build a role-aware longlist Search by problem signal, deliverable, and failure mode—not only by familiar names. Include candidates from distinct roles where warranted: diff --git a/skills/select-methodologies/references/00-taxonomy-selection.md b/skills/select-methodologies/references/00-taxonomy-selection.md index 1ae1483..28e79cc 100644 --- a/skills/select-methodologies/references/00-taxonomy-selection.md +++ b/skills/select-methodologies/references/00-taxonomy-selection.md @@ -132,6 +132,10 @@ When the same family appears in multiple catalogs, retrieve its generic definiti | Product realization and manufacturing readiness | `14-physical-engineering-manufacturing.md` | process scale-up in `15` | | Asset management, maintenance, inspection, condition monitoring, spares, and renewal | `34-asset-maintenance-reliability.md` | manufacturing equipment in `14`; built-asset handover in `36` | | Human factors, ergonomics, medical devices, and clinical-device evidence | `35-human-factors-health-medical-devices.md` | UX in `20`; product realization in `14`; health regulation in `13` | +| Hierarchical Task Analysis (HTA), Cognitive Task Analysis (CTA), and expert task/decision elicitation | `35-human-factors-health-medical-devices.md` | training objectives/scenarios in `18`; interface/information requirements in `20`; workforce application in `25` | +| SHERPA: task-based human-error prediction and recovery | `35-human-factors-health-medical-devices.md` | broader hazard analysis and assurance in `06`; interaction redesign in `20` | +| GOMS family and Keystroke-Level Model (KLM): skilled interaction modeling | `20-design-experience-communication.md` | HTA task structure in `35`; measured usability/performance evidence remains distinct | +| Functional Resonance Analysis Method (FRAM): performance variability and functional couplings | `06-testing-reliability-safety-security.md` | work-as-done evidence in `35`; process interventions in `03` | | Construction, infrastructure, BIM, constructability, and commissioning | `36-construction-infrastructure-bim.md` | project controls in `30`; asset operations in `34`; physical engineering in `14` | | Urban, territorial, regional, land-use, mobility, public-space, and place-based planning | `38-urban-territorial-planning.md` | geospatial data/models in `33`; construction/BIM in `36`; participation in `24`; climate resilience in `15`/`21`/`32` | | LCA, environmental claims, TEA, industrial processes | `15-sustainability-circular-energy.md` | agricultural sustainability in `17` | diff --git a/skills/select-methodologies/references/06-testing-reliability-safety-security.md b/skills/select-methodologies/references/06-testing-reliability-safety-security.md index b1d6fa3..9e063de 100644 --- a/skills/select-methodologies/references/06-testing-reliability-safety-security.md +++ b/skills/select-methodologies/references/06-testing-reliability-safety-security.md @@ -79,11 +79,35 @@ Use this catalog to choose how confidence will be earned, not merely asserted. S | Event Tree Analysis (ETA) | forward consequence analysis | initiating events branch through barrier success/failure to outcomes | state dependencies violate simple branch assumptions | | HAZOP | structured deviation analysis | process parameters and guidewords can reveal hazardous deviations | software/control intent lacks meaningful process variables without adaptation | | STPA | systems-theoretic hazard analysis | unsafe control actions, inadequate constraints, software/human interaction, and emergent behavior matter | low-risk simple component defect analysis | +| [Functional Resonance Analysis Method — FRAM](#functional-resonance-analysis-method--fram) | systemic functional analysis; variability and couplings in work-as-done | everyday adaptations and interactions may combine into successful or harmful outcomes | a static diagram is expected to prove causality, predict event probabilities, or satisfy safety assurance by itself | | Bow-Tie Analysis | barrier-risk visualization | threats, top event, consequences, preventive and mitigative barriers need one view | quantitative proof or complex feedback dynamics | | Safety Integrity Levels / assurance levels | risk-classification/control scheme family | domain standards require rigor proportionate to hazardous failure | invent generic SILs outside the applicable sector standard | | Independent V&V | governance/assurance practice | consequences and conflicts of interest justify organizational independence | independence exists only on an org chart without technical authority/evidence access | | Safety Case | assurance case for safety | regulator/stakeholders need explicit claims, argument, evidence, context, and defeaters | document created after design solely for approval | +### Functional Resonance Analysis Method — FRAM + +- **Type / domain / lifecycle role / stack role:** systemic functional analysis method / sociotechnical systems, safety, and resilience / design exploration, operational learning, and event analysis / explanation and intervention design. +- **Purpose and output:** understand how ordinary performance variability combines across functions. Produce a bounded function model, evidence-backed variability descriptions, scenario/event instantiations showing relevant couplings, and monitored intervention proposals. The general model describes potential couplings; an instantiation identifies those relevant to a particular situation. +- **Use when:** everyday adjustments, coordination, resources, and timing may explain successful or harmful outcomes, and practitioners/work evidence can support a functional account beyond a linear failure chain. +- **Do not use when:** only a simple component failure needs analysis, evidence cannot distinguish modeled assumptions from actual work, or a quantified risk estimate, proven event cause, or compliance claim is expected from a FRAM diagram alone. +- **Preconditions and required capability:** bounded question/system, access to work-as-done evidence and practitioners, explicit scenario/context, and facilitation able to characterize functions without treating adaptation as operator failure. Proposed future work must be labeled as such. +- **Typical procedure:** + 1. Frame the decision and boundary; gather everyday work evidence, including successful operation and adaptations. + 2. Identify functions needed for activity, starting broadly before refining detail. Describe relevant aspects: Input, Output, Preconditions, Resources, Control, and Time; not every function needs all six filled. + 3. Trace output couplings to other functions' input, preconditions, resources, control, or time. Account for relevant external sources/receivers with explicit boundary/background functions; justify what is fixed for this analysis. + 4. Characterize evidence-supported potential variability and context, often including timing and precision. Separate observation from analyst hypothesis. + 5. Instantiate the model for representative scenarios or a documented event; examine how coupled variability may be amplified or attenuated. Do not infer actual couplings solely from potential ones in the general model. + 6. Design controls, buffers, monitoring, or resource/coordination changes that preserve necessary adaptation; review with practitioners and evaluate effects under representative conditions. +- **Complements:** [human-factors work analysis](35-human-factors-health-medical-devices.md#human-factors-and-ergonomics) supplies work evidence; process improvement implements interventions; independent hazard analysis and assurance establish additional safety claims. +- **Alternatives or variants:** STPA analyzes unsafe control actions/constraints; FMEA analyzes failure effects; HTA represents goal/task hierarchy; BPMN represents process flow. These are different outputs. FRAM Model Visualiser is optional tooling; quantitative simulation extensions require their own assumptions and validation. +- **Failure modes and gaming risks:** rename process boxes as functions without aspects; treat every coupling as observed cause; fill all six aspects mechanically; equate variability with error; invent probabilities; freeze adaptive work indiscriminately; label a model itself as validated risk reduction. +- **Adoption cost:** medium for bounded qualitative modeling; high for multiple actors/scenarios, evidence collection, and intervention evaluation. +- **Maturity:** established method; no universal normative edition or assurance certification implied. +- **Canonical research anchors:** author/community guidance on [building a FRAM model](https://functionalresonance.com/how-to-build-a-fram-model/) and [basic principles](https://functionalresonance.com/basic-principles/). +- **Current version/status checked on:** 2026-09-12. Official procedure and principles consulted; tool releases are not method editions. +- **Evidence and unresolved questions:** the four principles concern equivalence of success/failure, approximate adjustments, emergence, and functional resonance. Modeling relies on evidence quality, boundary choices, and analyst judgment; the guidance establishes method structure, not universal causal validity or intervention effectiveness. + ## Reliability engineering | Candidate | Type and output | Use when | Avoid when | diff --git a/skills/select-methodologies/references/10-composition-patterns.md b/skills/select-methodologies/references/10-composition-patterns.md index 0ba6fb2..7ae3651 100644 --- a/skills/select-methodologies/references/10-composition-patterns.md +++ b/skills/select-methodologies/references/10-composition-patterns.md @@ -148,6 +148,22 @@ Use bioremediation only for contaminant treatment. It is not a substitute name f Satisfaction is not learning, learning is not workplace transfer, and transfer is not causal proof of operational impact. +### Understand work before designing procedures, training, or decision support + +`bounded goal + representative work evidence → HTA when goals/operations/plans need structure → CTA on judgment-intensive decisions when expert cues/strategies need elicitation → practitioner review + evidence triangulation → procedure/interface/job aid or training requirements → representative-task evaluation` + +Use the canonical [HTA and CTA cards](35-human-factors-health-medical-devices.md#task-analysis-canonical-cards-and-selection). Select either method alone when its output is sufficient; their order is a possible composition, not a compulsory pipeline. Without practitioner/work evidence, stop at a provisional model and collection plan. HTA does not establish error probabilities, and CTA testimony does not establish improved outcomes or safety. + +### Analyze task errors, skilled interaction, or functional variability + +Choose only the output needed; these are separate optional compositions: + +- `validated HTA + work evidence → SHERPA credible errors/consequences/recovery → owned design and organizational remedies → representative control/recovery evaluation`. Use the [SHERPA card](35-human-factors-health-medical-devices.md#sherpa); ordinal judgments are not calibrated probabilities. +- `specified interface + practiced user method → GOMS/KLM operator sequence + sourced durations and assumptions → equivalent-method comparison → user validation of decisive predictions`. Use the [GOMS/KLM card](20-design-experience-communication.md#goms-and-klm); modeled execution time does not establish learning, accessibility, error rates, or achieved savings. +- `observed everyday work → FRAM function/aspect model → scenario instantiations → evidence-backed variability/coupling analysis → monitored interventions`. Use the [FRAM card](06-testing-reliability-safety-security.md#functional-resonance-analysis-method--fram); potential couplings do not prove actual causes or safety. + +If prerequisite evidence is missing, deliver a provisional model and collection plan rather than a definitive risk or performance conclusion. + ### Launch and grow a market offer `decision + market evidence → STP/ICP/buying committee → positioning + claim–proof → experience/offer/price → GTM + route-to-market → launch-readiness gate → instrumented beachhead → acquisition + activation + retention → experiments/MMM → staged expansion` diff --git a/skills/select-methodologies/references/11-source-registry.md b/skills/select-methodologies/references/11-source-registry.md index a480885..cfa00f9 100644 --- a/skills/select-methodologies/references/11-source-registry.md +++ b/skills/select-methodologies/references/11-source-registry.md @@ -142,6 +142,11 @@ Use this registry to begin research. Open the owner page and verify current stab | Asset management | `iso.org/standard/83053.html` and `iso.org/standard/83054.html` | ISO 55000:2024 and ISO 55001:2024 current | | Reliability-centred and risk-based maintenance | SAE JA1011/JA1012; API RP 580/581 | use only in their applicable asset/process context and with competent multidisciplinary analysis | | Human-centred design and ergonomics | ISO 9241-210, ISO 6385, and ISO 26800 owner pages | framework standards; validate with representative users, tasks, and environments | +| Hierarchical Task Analysis — task HTA | [Stanton 2006](https://doi.org/10.1016/j.apergo.2005.06.003); [UXPA HTA guidance](https://www.usabilitybok.org/hierarchical-task-analysis) | checked 2026-09-12: established technique, no universal normative version; primary abstract and professional guidance consulted; distinguish Health Technology Assessment | +| Cognitive Task Analysis — CTA; ACTA and CDM | [UXPA CTA guidance](https://www.usabilitybok.org/cognitive-task-analysis); [Militello–Hutton 1998](https://doi.org/10.1080/001401398186108); [Klein et al. 1989](https://doi.org/10.1109/21.31053) | checked 2026-09-12: CTA is a family; ACTA/CDM differ in elicitation structure. Original publication identities verified; full original protocols require further consultation for exact replication | +| SHERPA | [open maritime SAR study (2024)](https://doi.org/10.1016/j.heliyon.2024.e32043) | checked 2026-09-12: full eight-step methods and taxonomy consulted; established task-based technique. Ordinal likelihood/criticality and domain-specific adaptations do not establish calibrated probabilities or general effectiveness | +| GOMS family and KLM | [Card et al. 1980](https://doi.org/10.1145/358886.358895); [Kieras full KLM guide, 2001](https://web.eecs.umich.edu/~kieras/docs/GOMS/KLM.pdf); [John–Kieras 1996](https://doi.org/10.1145/235833.236050); UXPA [GOMS](https://www.usabilitybok.org/goms) and [KLM](https://www.usabilitybok.org/klm-goms) | checked 2026-09-12: author guide/professional guidance read; original publication identities verified. KLM models specified practiced, error-free execution; other GOMS variants require their own protocols | +| Functional Resonance Analysis Method — FRAM | official [procedure](https://functionalresonance.com/how-to-build-a-fram-model/) and [principles](https://functionalresonance.com/basic-principles/) | checked 2026-09-12: full author/community guidance consulted; distinguish potential-coupling model from scenario instantiation. Six aspects are relevant descriptors, not six compulsory entries; tooling is optional and a model does not prove risk reduction | | Medical-device quality and risk | ISO 13485, ISO 14971, IEC 62366-1, ISO 10993-1, ISO 14155, IEC 62304; jurisdictional regulator | regulatory recognition and transition vary by device, jurisdiction, and date; ISO 14155:2026 and ISO 10993-1:2025 current at check date | | BIM information management and interoperability | ISO 19650 family; ISO 7817-1:2024; ISO 16739-1:2024; buildingSMART IFC/IDS/BCF | information exchange conformity does not prove engineering correctness | | Building commissioning | ISO 24359-1:2026; ASHRAE Standard 202-2024 | building-oriented anchors; tailor for other infrastructure sectors | diff --git a/skills/select-methodologies/references/18-learning-training-education.md b/skills/select-methodologies/references/18-learning-training-education.md index 72bec59..a144f83 100644 --- a/skills/select-methodologies/references/18-learning-training-education.md +++ b/skills/select-methodologies/references/18-learning-training-education.md @@ -19,7 +19,7 @@ Use this catalog when the intended outcome is learning, skill transfer, behavior | --- | --- | --- | --- | | Performance gap analysis | diagnostic method; desired vs actual performance, causes, and candidate interventions | a business/mission outcome is poor and training is only one possible remedy | assume every gap is lack of knowledge rather than incentives, tools, process, authority, workload, or design | | Training Needs Analysis — TNA | needs-assessment process at organization, task, and person levels | determine who needs what capability and under which work conditions | begin from a requested course title without validating the performance need | -| Job / Cognitive Task Analysis | elicitation and modeling family; tasks, decisions, cues, knowledge, errors, and conditions | expert performance or safety-critical judgment must become teachable and assessable | decompose adaptive expertise into rote steps or rely on expert self-report alone | +| Task / Cognitive Task Analysis for training | linked analysis family; inherit [HTA and CTA](35-human-factors-health-medical-devices.md#task-analysis-canonical-cards-and-selection), then derive observable objectives, practice scenarios, and assessment evidence | task structure or expert decisions/cues must become teachable and assessable; choose HTA for goals/plans and CTA for cognitive demands | decompose adaptive expertise into rote steps, rely on expert self-report alone, or treat a task model as proof that training transfers | | Competency modeling | representation method; observable capabilities, proficiency levels, and evidence | roles, development paths, and assessments need a shared capability model | create vague traits, overlapping competencies, or levels without observable evidence | | Learner and context analysis | discovery method; prior knowledge, goals, access, language, constraints, motivation, and environment | design decisions depend on heterogeneous learners and delivery conditions | use demographic stereotypes or collect personal data with no instructional decision attached | diff --git a/skills/select-methodologies/references/20-design-experience-communication.md b/skills/select-methodologies/references/20-design-experience-communication.md index 11cbb2a..fea7fc4 100644 --- a/skills/select-methodologies/references/20-design-experience-communication.md +++ b/skills/select-methodologies/references/20-design-experience-communication.md @@ -6,14 +6,15 @@ Use this catalog when the decision concerns understanding people and contexts, s 1. Design framing and lifecycle 2. Design and user research -3. Synthesis, information architecture, and interaction -4. Prototyping and usability evaluation -5. Service and experience design -6. Visual, information, and communication design -7. Inclusive design and accessibility -8. Design systems and design operations -9. Boundaries and composition patterns -10. Research anchors and status +3. Predictive modeling of skilled interaction +4. Synthesis, information architecture, and interaction +5. Prototyping and usability evaluation +6. Service and experience design +7. Visual, information, and communication design +8. Inclusive design and accessibility +9. Design systems and design operations +10. Boundaries and composition patterns +11. Research anchors and status ## Design framing and lifecycle @@ -45,7 +46,7 @@ Use this catalog when the decision concerns understanding people and contexts, s | Focus group | facilitated group-research method | social language, norms, reactions, and concept diversity are relevant | sensitive individual behavior, usability, prevalence, consensus, or dominant voices distort the question | | Generative / co-creation session | exploratory method | participants can externalize needs, experiences, metaphors, or future possibilities through making | creative artifacts are treated as prioritized requirements or realistic adoption evidence | | Critical incident technique | structured recall method | unusually successful/failed episodes can reveal triggers, actions, context, and consequences | ordinary frequency/base rates or prospective behavior is inferred from memorable incidents alone | -| Task analysis | analysis method; goals, actions, conditions, information, dependencies, and error opportunities | workflows must become design requirements or evaluation tasks | adaptive/knowledge work is forced into one ideal linear sequence; cognitive specialization in `35` | +| Task analysis for UX | linked analysis method; inherit [HTA and CTA](35-human-factors-health-medical-devices.md#task-analysis-canonical-cards-and-selection), then derive interaction/information requirements and evaluation tasks | goals/plans or expert cues/decisions must inform interface, content, or recovery design | adaptive work is forced into one ideal linear sequence, a hierarchy predicts usability, or a cognitive walkthrough is treated as elicitation of expert cognition | | Mental-model elicitation | research/synthesis family | users' concepts, causal beliefs, categories, and expectations affect comprehension and navigation | analyst diagram is called the user's model or misconceptions are preserved despite safety/domain truth | | Survey research for design | quantitative self-report specialization | prevalence, attitudes, satisfaction, segmentation, or tracking requires a defensible instrument and sample | convenience responses prove behavior, causality, usability, or unmet need severity; generic survey design and evidence ownership belong to `31-research-evidence-investigation.md` | | Concept testing | evaluative research family | alternative value propositions, concepts, comprehension, relevance, and concerns need early evidence | stated liking predicts purchase, use, feasibility, or safety | @@ -53,6 +54,35 @@ Use this catalog when the decision concerns understanding people and contexts, s Recruit participants by the target population and relevant capability/context—not convenient demographics alone. Separate user, purchaser, operator, administrator, supporter, bystander, and person bearing risk. Include nonusers, abandonment, assistive-technology use, low literacy, adverse cases, and edge conditions when material. +## Predictive modeling of skilled interaction + +| Candidate | Type and output | Use when | Avoid when | +| --- | --- | --- | --- | +| [GOMS family / KLM](#goms-and-klm) | analytical interaction-model family; specified methods and modeled execution time | detailed alternatives for practiced tasks must be compared with declared operators and assumptions | learning, errors, open-ended problem solving, accessibility, or observed outcomes are inferred from a routine execution-time estimate | + +### GOMS and KLM + +- **Type / domain / lifecycle role / stack role:** Goals, Operators, Methods, and Selection rules (GOMS) / interaction design and human-computer interaction / detailed design and comparison / predictive analysis. Keystroke-Level Model (KLM) is a simplified model for execution time within this ecosystem. +- **Purpose and output:** describe how specified goals are achieved by methods/operators and, where alternatives exist, selection rules. For KLM, produce a scenario/method-specific operator sequence, sourced durations, assumptions, waiting treatment, and modeled expert execution time; compare equivalent task outcomes. +- **Use when:** interfaces and a practiced user method are specified sufficiently to enumerate operations, and an estimate or comparison of routine execution time will inform a design decision. +- **Do not use when:** the needed outcome is novice learning, error probability, complex diagnosis, user satisfaction, accessibility, or measured real-world performance; do not model unconstrained decisions as a fixed mental-preparation cost. +- **Preconditions and required capability:** representative scenarios, specified interface/input device, experienced performer assumptions, explicit chosen method, applicable operator definitions/durations, and an analyst able to justify mental-operator placement. Record excluded interruptions, errors, and learning. +- **Typical procedure:** + 1. Define equivalent task goals, scenario, start/end state, interface alternatives, and assumed performer expertise. + 2. Choose the GOMS technique by output and concurrency/learning assumptions; specify methods and selection rules where relevant. For KLM, assume a known practiced method and error-free execution. + 3. Enumerate physical operators using the chosen protocol: K for keystrokes, P for pointing, B for button press/release, H for homing in Kieras's guide. Document notation differences across sources. + 4. Insert M operators for routine mental preparation using the selected placement/chunking rules. Model only waiting that blocks the user; Kieras uses W(t), while other descriptions use response-time notation. Avoid counting overlapping work and latency twice. + 5. Source or calibrate durations for the relevant device, task, and population; document assumptions and sensitivity rather than silently reuse universal constants. + 6. Sum the KLM sequence under those assumptions, compare equivalent methods, and check decisive predictions with representative users. Keep measured values and modeled estimates separate. +- **Complements:** [HTA](35-human-factors-health-medical-devices.md#hierarchical-task-analysis--hta) establishes goal/task structure; usability observation tests actual behavior; accessibility and error/recovery evaluation answer distinct quality questions. +- **Alternatives or variants:** CMN-GOMS, NGOMSL, and CPM-GOMS differ in representation and modeling capability. NGOMSL supports specific learning-time analyses; CPM-GOMS handles overlapping perceptual, cognitive, and motor activity. Consult their own protocols; these capabilities are not properties of every GOMS model or of simple KLM. Empirical task timing is preferable when directly observable performance is the required evidence. +- **Failure modes and gaming risks:** omit selection rules or assume the designer's fastest method is used; ignore errors/interruptions; place M before every click; charge total backend latency despite overlap; transplant unsupported timing constants; report predicted savings as achieved outcomes. +- **Adoption cost:** medium for bounded KLM comparison; higher for elaborate GOMS variants, model calibration, and concurrent interaction. +- **Maturity:** established analytical family; applicable model/protocol matters more than a generic version label. +- **Canonical research anchors:** [Card, Moran, and Newell (1980)](https://doi.org/10.1145/358886.358895); [Kieras's full KLM guide (2001)](https://web.eecs.umich.edu/~kieras/docs/GOMS/KLM.pdf); [John and Kieras (1996)](https://doi.org/10.1145/235833.236050); UXPA [GOMS](https://www.usabilitybok.org/goms) and [KLM](https://www.usabilitybok.org/klm-goms). +- **Current version/status checked on:** 2026-09-12. Author's KLM guide and professional guidance consulted; original paper identities verified. +- **Evidence and unresolved questions:** estimates are conditional on method, operator timing, expertise, and model assumptions. The original ACM full texts were not consulted; no universal timing accuracy, learning benefit, or design effectiveness is claimed. + ## Synthesis, information architecture, and interaction | Candidate | Type and output | Use when | Avoid when | diff --git a/skills/select-methodologies/references/35-human-factors-health-medical-devices.md b/skills/select-methodologies/references/35-human-factors-health-medical-devices.md index 8cde5b2..c355899 100644 --- a/skills/select-methodologies/references/35-human-factors-health-medical-devices.md +++ b/skills/select-methodologies/references/35-human-factors-health-medical-devices.md @@ -9,14 +9,103 @@ Use this catalog when human–system performance, physical/cognitive ergonomics, | Ergonomic design of work systems — ISO 6385/26800 | human-factors design framework | tasks, organization, tools, environment, physical/cognitive demand, and worker variability must be designed jointly | “train the operator” compensates for a preventable design hazard | | Human Systems Integration — HSI | program integration framework | manpower, personnel, training, human factors, safety, survivability, habitability, and maintainability interact across acquisition | add a usability review at the end or optimize headcount independently of workload/safety | | Human-centred design — ISO 9241-210 | iterative design lifecycle specialized for human–system engineering | interactive physical/digital systems require explicit users, context, requirements, design, and evaluation | UI preference testing substitutes for system safety, domain correctness, or accessibility obligations; UX/service execution belongs to `20-design-experience-communication.md` | -| Work / task / cognitive task analysis | analysis family | work-as-done, goals, decisions, information, variability, coordination, and error opportunities must be understood | document only work-as-imagined or decompose adaptive work into misleading linear steps | +| Work / task analysis | analysis family | work-as-done, goals, variability, coordination, and error opportunities must be understood; select HTA or CTA below by required output | document only work-as-imagined or assume one decomposition captures every aspect of work | +| [Hierarchical Task Analysis — HTA](#hierarchical-task-analysis--hta) | established analysis method; goal/subgoal hierarchy plus execution plans | observed or explicitly proposed tasks must become procedures, interface requirements, job aids, or a basis for error analysis | a hierarchy alone is expected to reveal tacit expertise, predict task time/error probability, or prove safety | +| [Cognitive Task Analysis — CTA](#cognitive-task-analysis--cta) | elicitation and analysis family; cognitive demands, cues, strategies, and decision requirements | expert judgment, diagnosis, uncertainty, or novice/expert differences must inform training, interfaces, or decision support | domain experts/representative work evidence are unavailable, only physical actions need description, or one self-report is treated as verified cognition | | Anthropometric accommodation | physical-design method | reach, clearance, strength, posture, fit, egress, and population coverage determine geometry | design to an “average person,” mix incompatible percentiles, or ignore clothing/PPE/dynamics | | Human-in-the-loop evaluation | iterative assurance method | representative users must perform representative tasks under realistic conditions before release | experts substitute for target users or scripted happy paths hide workload and recovery | | NASA-TLX / workload measures | subjective workload measurement technique | compare workload across tasks/designs with a validated protocol and complementary performance evidence | one score diagnoses the causal source or replaces errors, physiology, observation, and context | | Revised NIOSH Lifting Equation | specialist ergonomic assessment | specified two-handed lifting conditions fit the model's scope | apply outside assumptions, to pushing/pulling/carrying or complex unstable loads as a universal safe limit | | Human Reliability Analysis — HRA | risk-analysis family | human actions, dependencies, context, recovery, and performance-shaping factors affect safety/reliability | assign generic “human error probabilities,” blame operators, or ignore system design | +| [SHERPA](#sherpa) | task-based human-error prediction method; credible errors, consequences, recovery, and remedies | a validated HTA and practitioner evidence can support systematic analysis of operational or interface errors | invented probabilities, operator blame, or a completed worksheet is treated as proof of safety | | Cognitive Work Analysis — CWA | constraint-based analysis framework | work is complex/adaptive and must remain safe under unanticipated conditions | a stable routine task only needs direct task analysis or analysts lack domain access | +## Task analysis: canonical cards and selection + +Both methods support human factors across operational, physical, digital, and training contexts. HTA describes goal-directed task structure; CTA investigates the knowledge and cognitive demands that make performance possible. They are complements when both outputs feed a real decision, not two mandatory stages for every task. Preserve the distinction between observed current work and a proposed future task model. + +`HTA` also means Health Technology Assessment in the health section below. Resolve the acronym from the decision/output before selecting it. HTN is an AI planning formalism, not HTA; CWA analyzes work-system constraints, and cognitive walkthrough evaluates learnability rather than eliciting expert cognition. + +### Hierarchical Task Analysis — HTA + +- **Type / domain / lifecycle role / stack role:** method/technique; human factors and ergonomics; frame/design/improve; specialist, producing a task representation. +- **Purpose and output:** explain how a top-level goal is achieved through subordinate goals and operations. Produce a numbered hierarchy with execution plans specifying order, conditions, alternatives, repetition, and coordination where relevant. A tree or numbered checklist without plans is incomplete HTA. +- **Use when:** a procedure, interface, job aid, allocation of function, training analysis, or error-analysis input needs explicit task structure and representative work can be observed or elicited. Label future-design models as proposals for evaluation. +- **Do not use when:** the required output is tacit decision expertise, a performance-time prediction, a quantified human-error rate, or an independent safety demonstration. Add the appropriate method rather than infer these from the hierarchy. Do not force variable/adaptive work into one supposedly universal sequence. +- **Preconditions and required capability:** explicit goal, actors, context, system boundary, analysis purpose, and access to representative practitioners and task evidence; an analyst able to distinguish goals from interface features. Agree on numbering and decomposition grain. If evidence is unavailable, deliver a provisional model and an evidence-collection plan. +- **Typical procedure:** + 1. Define the goal, scope, context, current versus proposed work, and decisions the analysis will support. + 2. Observe or elicit representative task execution, including relevant variations; record evidence and unresolved assumptions. + 3. Decompose goals into subordinate goals/operations, with stable IDs and traceable parent relationships. Avoid a fixed number of levels or decomposition into every click by default. + 4. Write plans for each decomposed goal: sequence, guards, branches, repetition, and concurrent or coordinated work where observed or proposed. State completion conditions and recovery/escalation when material to the task. + 5. Stop decomposing when the chosen grain supports the downstream decision and further detail adds no material clarity or risk control; deepen uncertain or consequential operations. The historical probability × consequence stopping principle is a tailoring aid, not permission to invent numerical probabilities or acceptance thresholds. + 6. Walk through the model with representative practitioners and task evidence; check plan coverage, alternatives, completion, and recovery. Revise disputed branches before using the model to derive requirements or procedures. +- **Complements:** CTA for selected judgment-intensive operations; [SHERPA](#sherpa) for credible task errors and recovery; HRA or hazard analysis for use-related risk; UX evaluation, training design, or job-aid evaluation for the artifact derived from the model. +- **Alternatives or variants:** generic task analysis for a lightweight description; GOMS/KLM when their assumptions fit performance prediction; BPMN for process/handoff representation; HTN for executable AI planning. These differ in output and are not equivalent HTA variants. Graphical and tabular HTA are alternative representations of the same model. +- **Failure modes and gaming risks:** work-as-imagined, arbitrary decomposition, missing plans, one ideal user/path, stale numbering, inconsistent diagram/table, and treating fewer steps as proof of better performance. Validate usability, performance, and safety separately. +- **Adoption cost:** low for a bounded sketch; medium or high for a validated model with many roles/branches. Evidence access, analyst training, and maintenance dominate; software is optional. +- **Maturity:** established, research-grounded practice; no single normative owner or universal release version. +- **Canonical research anchors:** [Stanton (2006)](https://doi.org/10.1016/j.apergo.2005.06.003), [UXPA HTA guidance](https://www.usabilitybok.org/hierarchical-task-analysis), and its Annett–Duncan origin reference. [Hornsby (2010)](https://www.uxmatters.com/mt/archives/2010/02/hierarchical-task-analysis.php) provides a UX worked example rather than universal validation evidence. +- **Current version/status checked on:** 2026-09-12; an established technique, not a versioned normative specification. +- **Evidence and unresolved questions:** the primary review abstract identifies broad applications; professional guidance documents plans, stopping logic, training needs, and analyst variability. The full Stanton manuscript was unavailable during this research. Validate task coverage and downstream effects in the target context; the method itself proves neither time savings nor safety. + +**Suggested output fields:** goal ID and parent; actor/context; operation or subordinate goal; evidence reference; plan and completion condition; material exception/recovery; decomposition-stop rationale. Treat these as a compact local reporting shape, not an external mandatory standard. + +### Cognitive Task Analysis — CTA + +- **Type / domain / lifecycle role / stack role:** elicitation and analysis family; human factors/cognitive ergonomics; frame/design/improve; specialist, producing evidence-based decision and cognitive-demand representations. +- **Purpose and output:** identify knowledge, cues, situation assessment, goals, strategies, uncertainty, and difficult judgments required for proficient task performance. Produce traceable cognitive demands and implications for training, information/interface design, or decision support; descriptions of actions alone are insufficient. +- **Use when:** diagnosis, anomaly detection, prioritization, expert decisions, or adaptive performance depends on knowledge not captured by a procedure, and domain practitioners and representative episodes/tasks are accessible. +- **Do not use when:** only simple observable actions require description; the needed experts/task evidence are inaccessible; or outputs will be treated as a direct readout of cognition, an exhaustive expert rulebook, or an independently validated safety claim. Mark evidence-poor reconstructions as hypotheses. +- **Preconditions and required capability:** a bounded task and downstream decision, domain-appropriate practitioners, representative routine and difficult work, trained interviewing/analysis, and consent/confidentiality where required. Preserve differences between expertise levels and contexts; do not prescribe a universal sample size. +- **Typical procedure:** + 1. Define the task, population/context, cognitive question, and the artifact or decision that will consume findings. + 2. Select a CTA technique by scope and evidence: ACTA for an applied overview of cognitive demands; CDM for probing specific challenging episodes. Use observation/protocol analysis when appropriate to the task and record any limits introduced by that choice. + 3. Elicit concrete examples: what was noticed, what it meant at that time, what goals/constraints mattered, which options were considered, what could go wrong, and what a less experienced performer might miss. Separate contemporaneous knowledge from hindsight. + 4. Analyze demands, cues, strategies, uncertainty, and errors with links to their episode/source. Preserve divergent strategies and context instead of merging them into one invented universal rule. + 5. Validate interpretations with practitioners and, where available, observation, records, contrasting episodes, or other participants. Distinguish testimony, corroborated findings, and unresolved analyst inference. + 6. Translate supported findings into information requirements, practice scenarios, job aids, or decision-support requirements; evaluate those products on representative tasks. Claims of improved performance need separate outcome evidence. +- **Complements:** HTA for task/goal structure where useful; learning and assessment in `18`; UX/information design in `20`; workload and human-reliability methods for distinct measurement/risk questions. +- **Alternatives or variants:** CTA is the family. ACTA and CDM have different interview structures and evidence focus; choose and document the technique instead of using the family name as a complete protocol. CWA is a complementary constraint-based framework, not an interchangeable interview method. +- **Failure modes and gaming risks:** retrospective/hindsight bias, leading questions, expert omission of automated knowledge, unsupported generalization across roles/sites, conflating confident explanation with observed performance, and ignoring physical/social/resource constraints. +- **Adoption cost:** medium for bounded applied elicitation, high for multi-context studies and triangulated analysis. ACTA streamlines some CTA work but does not eliminate practitioner access or evidence-analysis cost. +- **Maturity:** established research/practice family; no single owner or universal normative release version. +- **Canonical research anchors:** [UXPA CTA guidance](https://www.usabilitybok.org/cognitive-task-analysis); [Militello and Hutton (1998), ACTA](https://doi.org/10.1080/001401398186108); [Klein, Calderwood, and MacGregor (1989), CDM](https://doi.org/10.1109/21.31053). +- **Current version/status checked on:** 2026-09-12. Original publication identities and ACTA abstract were verified; technique choice and local tailoring remain context-specific. +- **Evidence and unresolved questions:** the ACTA abstract reports an evaluation of usability/usefulness, not universal performance benefits. An [open 2023 coaching study](https://doi.org/10.3389/fpsyg.2023.1154168) documents application, protocol adaptation, and retrospective/context-transfer limitations. Original ACTA/CDM full texts were not available during this research; consult them before claiming exact replication of either protocol. + +#### Choosing and representing CTA techniques + +| Technique | Evidence focus and procedure shape | Output / boundary | +| --- | --- | --- | +| Applied Cognitive Task Analysis — ACTA | task-diagram interview identifies demanding parts; knowledge audit probes expertise; simulation interview probes assessment/actions/cues and potential errors in a scenario | synthesize a cognitive-demands table and product implications; the three interviews are elicitation methods, while the table is the synthesis artifact. A task diagram is not automatically a complete HTA. Document omitted/adapted components rather than claim full protocol replication | +| Critical Decision Method — CDM | reconstruct and probe a concrete challenging incident and its decision points, cues, goals, judgments, and alternatives | incident/decision account with source-linked knowledge requirements; retrospective evidence needs checking for hindsight and transfer limits. Use the original/authoritative protocol for its exact interview sequence | + +**Suggested output fields:** task/decision or episode ID; context and performer expertise; cognitive demand; cue/information source; interpretation and goal; strategy/alternatives; uncertainty and potential novice error; evidence and validation status; design/training implication. Adapt the fields to the selected technique; this is a local reporting aid, not a claim that all CTA protocols share one template. + +### SHERPA + +- **Type / domain / lifecycle role / stack role:** Systematic Human Error Reduction and Prediction Approach; established human-error identification method / human factors across operational and digital work / design, review, and improvement / task-risk analysis. +- **Purpose and output:** identify credible errors in HTA operations, their consequences and recovery opportunities, and remedies. Produce a traceable worksheet linking task ID, behavior category, error mode/description, consequence, recovery, supported ordinal likelihood, criticality, remedy, evidence, and validation status. +- **Use when:** operational or interface errors need systematic prediction and redesign, and a bounded HTA can be reviewed with practitioners who know the real task/context. +- **Do not use when:** task structure or practitioner evidence is unavailable, the question is chiefly tacit judgment rather than task errors, or quantitative human-error probabilities or demonstrated safety are required from this method alone. +- **Preconditions and required capability:** current HTA with operations/plans, credible work observations or incident evidence, trained analyst, practitioner review, and explicit system boundaries. Distinguish proposed tasks from observed work. +- **Typical procedure:** + 1. Establish and validate the HTA; retain operation IDs and context. + 2. Classify lowest-level operations as action, checking, retrieval, information communication, or selection. + 3. Apply the relevant SHERPA error taxonomy and retain only credible modes with a task-specific explanation; consult the chosen source for exact codes. + 4. Describe consequences and identify later checks, actors, or system functions that permit recovery; state when recovery is absent or only assumed. + 5. Assess ordinal likelihood and consequence criticality separately using declared scales and supporting evidence. Mark unsupported judgments unknown; do not turn ordinal classes into numerical probabilities. + 6. Propose remedies in equipment/interface, procedures, training, and organization; assign ownership and test the control and recovery under representative conditions. +- **Complements:** [HTA](#hierarchical-task-analysis--hta) supplies task structure; CTA investigates difficult judgments; [safety/hazard analysis](06-testing-reliability-safety-security.md#safety-and-hazard-analysis) covers wider system mechanisms; usability and recovery evaluation test resulting changes. +- **Alternatives or variants:** broader HRA methods for context/dependency or quantitative questions when their data assumptions hold; FMEA for component/process failure effects; STPA for unsafe control actions. Fuzzy risk scoring is an extension, not a required part of base SHERPA. +- **Failure modes and gaming risks:** apply every taxonomy code mechanically; omit recovery or context; equate possible-error counts with observed frequencies; rank unsupported likelihood; blame individuals or prescribe training for interface/resource defects; report proposed remedies as validated risk reduction. +- **Adoption cost:** medium for a bounded task, high for large HTAs, multiple contexts, and independent validation. +- **Maturity:** established technique; no universal normative edition or certification implied. +- **Canonical research anchors:** [maritime SAR application, full methods (2024)](https://doi.org/10.1016/j.heliyon.2024.e32043), which describes the eight-step procedure and taxonomy and cites Embrey's origin work. +- **Current version/status checked on:** 2026-09-12. Full open article consulted; local reporting fields and validation gate are explicit implementation aids. +- **Evidence and unresolved questions:** the application supports understanding of procedure, not universal effectiveness. Its ordinal labels and domain-specific criticality scheme should not be transplanted as calibrated probabilities or a generic severity scale. Exact replication requires the selected original taxonomy/protocol and domain validation. + ## Health and medical devices | Candidate | Type and output | Use when | Avoid when | diff --git a/tests/scenarios/selection-invariants.md b/tests/scenarios/selection-invariants.md index 5f40ebb..dfdd8aa 100644 --- a/tests/scenarios/selection-invariants.md +++ b/tests/scenarios/selection-invariants.md @@ -44,6 +44,30 @@ These scenarios define behavioral expectations for future agent evaluations. The **Expected invariants:** Every selected fragment must produce a distinct consumed output. The response should remove any element whose absence causes no material loss and preserve verified, rejected, and possible evidence states. +## Task structure and expert cognition + +**Prompt:** Choose methods to document an operational task and understand expert diagnostic judgments for training. + +**Expected invariants:** Route HTA and CTA to human factors. HTA requires goals, operations, and execution plans; CTA requires practitioner/work evidence for cues and cognitive demands. Use one or both only when their distinct outputs are consumed. Distinguish task HTA from Health Technology Assessment and do not claim exact ACTA/CDM replication without the selected protocol. + +## Credible task errors and recovery + +**Prompt:** We have a practitioner-reviewed HTA of an equipment setup task. Identify likely mistakes and recovery opportunities. + +**Expected invariants:** Select SHERPA in human factors; connect each credible mode to task context, consequences, recovery, evidence, and owned remedies. Keep ordinal likelihood separate from severity. Do not invent numerical probabilities or treat a worksheet or remedy proposal as proof of improved safety. + +## Practiced interaction timing and its boundary + +**Prompt:** Compare two fully specified interfaces for a routine task by experienced users, then estimate how quickly novices learn complex troubleshooting. + +**Expected invariants:** KLM can support the routine execution comparison with an explicit method, sourced timings, justified mental operators, and only blocking waiting. It cannot establish novice learning or complex diagnostic time. Distinguish other GOMS variants from simple KLM and modeled savings from measured outcomes. + +## Functional variability and instantiation + +**Prompt:** Successful everyday coordination sometimes produces harmful delays across a sociotechnical operation. Investigate how adaptations interact. + +**Expected invariants:** Consider FRAM in safety/systems using work-as-done evidence, relevant function aspects, and scenario instantiations. Account for couplings to preconditions, resources, control, and time as well as input. Do not fill all six aspects mechanically or infer actual event cause or probabilities from potential couplings alone. + ## Repository model organization **Prompt:** Add or revise a method that could be relevant to more than one domain catalog.