diff --git a/skills/mockapi/SKILL.md b/skills/mockapi/SKILL.md index bc57de7..732d1a1 100644 --- a/skills/mockapi/SKILL.md +++ b/skills/mockapi/SKILL.md @@ -88,9 +88,9 @@ run unbundled scripts. behavior from the matching `behavior.md` anchor, not from operation shape. - Use `reference/sidecars.md` for sidecar shape and validation expectations. - Use `reference/generated-server.md`, `reference/mock-server-structure.md`, - `reference/mock-server-examples.md`, `reference/seed-data.md`, and - `reference/mock-server-quality.md` when inspecting or finishing generated - server code. + `reference/mock-server-examples.md`, `reference/seed-data.md`, + `reference/testing.md`, and `reference/mock-server-quality.md` when + inspecting or finishing generated server code. ## Bundled Scripts diff --git a/skills/mockapi/reference/generate.md b/skills/mockapi/reference/generate.md index 687edcf..a90e20a 100644 --- a/skills/mockapi/reference/generate.md +++ b/skills/mockapi/reference/generate.md @@ -128,8 +128,9 @@ nvm-managed Node installations and corepack for pnpm/Yarn. 7. After codegen succeeds, read `reference/mock-server-structure.md`, `reference/mock-server-examples.md`, `reference/seed-data.md`, - `reference/mock-server-quality.md`, `.mockapi/behavior.md`, generated - `src/generated/**`, `src/controllers.ts`, and `src/features/**`. Create + `reference/testing.md`, `reference/mock-server-quality.md`, + `.mockapi/behavior.md`, generated `src/generated/**`, + `src/controllers.ts`, and `src/features/**`. Create feature-local `service.ts` when behavior needs orchestration. Create feature-local `repository.ts` for every completed feature that owns non-infrastructure state slices; `idCounters` alone does not need a @@ -160,6 +161,12 @@ nvm-managed Node installations and corepack for pnpm/Yarn. explicit behavior opt-in. Allocate IDs once per transaction and pass the allocator into helpers that need additional IDs. Do not infer behavior from operation shape when the anchor has open questions. + Add adjacent Vitest unit tests for completed LLM-owned feature behavior: + `service.test.ts` next to `service.ts`, `repository.test.ts` next to + `repository.ts`, and `.test.ts` next to feature-local domain helpers. + Keep generated operation controller adapter tests out of scope unless an + adapter contains non-trivial normalization. Keep HTTP workflow smoke tests in + `src/app.test.ts`. ### Final Verification @@ -167,8 +174,8 @@ nvm-managed Node installations and corepack for pnpm/Yarn. `packageManager.commands.check` from the generator JSON in `packageRoot`. Then run the generated package `test` script, if present. Generated mock server templates include Vitest by default; put HTTP smoke tests in - `src/app.test.ts`, config/env tests in `src/config.test.ts`, and only split - large feature workflows into `src/features//.test.ts`. + `src/app.test.ts`, config/env tests in `src/config.test.ts`, and unit tests + for feature behavior beside the source files they cover. Then run the final quality checker: ```bash diff --git a/skills/mockapi/reference/mock-server-quality.md b/skills/mockapi/reference/mock-server-quality.md index 0b7e4f9..4b43de4 100644 --- a/skills/mockapi/reference/mock-server-quality.md +++ b/skills/mockapi/reference/mock-server-quality.md @@ -1,10 +1,10 @@ # Mock Server Quality Gate Use this reference after `generate.py --run-codegen` and after completing -LLM-owned `src/features/**`, `src/controllers.ts`, seed data, and smoke tests. -This is a final handoff gate, not a scaffold scope probe. A freshly generated -scaffold is expected to fail because operation adapters start as TODOs and -feature seed stubs start empty. +LLM-owned `src/features/**`, `src/controllers.ts`, seed data, adjacent feature +unit tests, and smoke tests. This is a final handoff gate, not a scaffold scope +probe. A freshly generated scaffold is expected to fail because operation +adapters start as TODOs and feature seed stubs start empty. ## Required Checks @@ -94,12 +94,19 @@ when appropriate. The quality checker reports `quality.seed.emptyProductSeed` for empty product seed literals while seed data is enabled. -## Minimum Smoke Coverage +## Test Coverage Generated packages use Vitest. Keep HTTP smoke tests in `src/app.test.ts` by -default. Use `src/config.test.ts` only for config/env resolution, and use -`src/features//.test.ts` only when a feature workflow is too -large for the app-level smoke file. +default. Use `src/config.test.ts` only for config/env resolution. Add adjacent +unit tests for completed LLM-owned feature behavior: + +- `src/features//service.test.ts` for `service.ts` +- `src/features//repository.test.ts` for `repository.ts` +- `src/features//.test.ts` for feature-local helpers + +Do not unit-test generated thin operation controller adapters by default. Add an +adjacent controller test only when the adapter performs non-trivial request +normalization or branching not covered by service or app smoke tests. Before handoff, add `src/app.test.ts` coverage for: diff --git a/skills/mockapi/reference/testing.md b/skills/mockapi/reference/testing.md new file mode 100644 index 0000000..79c9fd0 --- /dev/null +++ b/skills/mockapi/reference/testing.md @@ -0,0 +1,43 @@ +# Mock Server Testing + +Generated packages use Vitest. Add tests during the LLM-owned behavior +implementation phase, after `generate.py --run-codegen` and after generated +runtime types exist. + +## Feature Unit Tests + +Write unit tests beside the feature source file they cover: + +- `src/features//service.test.ts` for `service.ts` +- `src/features//repository.test.ts` for `repository.ts` +- `src/features//.test.ts` for feature-local domain helpers + +Unit-test completed LLM-owned behavior, not deterministic generated scaffold. +Do not add tests for generated TODO adapters or thin operation controllers by +default. Add an adjacent controller test only when the controller performs +non-trivial request normalization or branching that is not covered by the +service or app smoke tests. + +Prefer narrow tests over broad snapshots. Exercise repository selectors and +mutations, service orchestration, ID allocation behavior, soft-delete flows, +validation branches, and feature-local helper rules that came from +`.mockapi/behavior.md`. + +## HTTP Smoke Tests + +Keep HTTP workflow smoke tests in `src/app.test.ts` by default. Cover: + +- custom `basePath` mounting +- one representative workflow for each non-trivial feature +- create operations that return generated counter IDs +- admin state reset or inspection when admin endpoints are generated + +Use `src/config.test.ts` only for config/env resolution tests. Split a workflow +out of `src/app.test.ts` only when the app-level file becomes hard to maintain; +even then, keep source-adjacent unit tests for feature internals. + +## Verification + +Run the generated package check command first, then the generated package +`test` script, then `check_generated_quality.py`. Fix quality errors before +handoff. Fix obvious test warnings, or report them as residual risk. diff --git a/skills/mockapi/scripts/mockapi_runtime/quality.py b/skills/mockapi/scripts/mockapi_runtime/quality.py index f13658d..c9b961a 100644 --- a/skills/mockapi/scripts/mockapi_runtime/quality.py +++ b/skills/mockapi/scripts/mockapi_runtime/quality.py @@ -191,6 +191,37 @@ def _is_feature_seed(relative: str) -> bool: return relative.startswith("src/features/") and relative.endswith("/seed.ts") +def _is_test_file(relative: str) -> bool: + return TEST_FILE_PATTERN.search(Path(relative).name) is not None + + +def _is_feature_unit_source(relative: str) -> bool: + if not relative.startswith("src/features/"): + return False + if _is_test_file(relative) or _is_feature_seed(relative) or "/controllers/" in relative: + return False + if relative.endswith(".d.ts"): + return False + if Path(relative).name in {"index.ts", "types.ts"}: + return False + return True + + +def _adjacent_unit_test_candidates(relative: str) -> list[str]: + source_path = Path(relative) + extensions = [source_path.suffix] + extensions.extend( + extension + for extension in (".ts", ".tsx", ".mts", ".cts") + if extension != source_path.suffix + ) + return [ + (source_path.parent / f"{source_path.stem}{kind}{extension}").as_posix() + for kind in (".test", ".spec") + for extension in extensions + ] + + def _feature_has_completed_behavior(source_files: list[ScannedFile], feature: str) -> bool: service_relative = f"src/features/{feature}/service.ts" if any(scanned_file.relative == service_relative for scanned_file in source_files): @@ -487,6 +518,31 @@ def _check_smoke_tests(context: QualityScanContext, warnings: list[Diagnostic]) ) +def _check_feature_unit_tests(context: QualityScanContext, warnings: list[Diagnostic]) -> None: + test_relatives = {scanned_file.relative for scanned_file in context.test_files} + + for scanned_file in context.source_files: + if not _is_feature_unit_source(scanned_file.relative): + continue + if TODO_PATTERN.search(scanned_file.text): + continue + + candidates = _adjacent_unit_test_candidates(scanned_file.relative) + if any(candidate in test_relatives for candidate in candidates): + continue + + warnings.append( + _diagnostic( + "quality.tests.missingFeatureUnit", + ( + "Completed LLM-owned feature source has no adjacent unit test; " + f"add {candidates[0]} or report the uncovered behavior as residual risk." + ), + path=candidates[0], + ) + ) + + def check_generated_quality( fs: FileSystem, package_root: Path, @@ -509,6 +565,7 @@ def check_generated_quality( _check_snapshot_set_all(context, warnings) _check_unsafe_casts(context, warnings) _check_smoke_tests(context, warnings) + _check_feature_unit_tests(context, warnings) return QualityCheckResult(errors=errors, warnings=warnings) diff --git a/tests/test_quality.py b/tests/test_quality.py index 37634c6..eaaf62b 100644 --- a/tests/test_quality.py +++ b/tests/test_quality.py @@ -2,19 +2,29 @@ import unittest from pathlib import Path +from typing import Iterable from tests.support import MemoryFileSystem, PROJECT_ROOT, profile_toml +from mockapi_runtime.diagnostics import Diagnostic from mockapi_runtime.quality import check_generated_quality, format_quality_result PACKAGE_ROOT = PROJECT_ROOT / "mock-server" -def diagnostic_ids(diagnostics: list[dict[str, str]]) -> set[str]: +def diagnostic_ids(diagnostics: Iterable[Diagnostic]) -> set[str]: return {diagnostic["id"] for diagnostic in diagnostics} +def diagnostic_paths(diagnostics: Iterable[Diagnostic], diagnostic_id: str) -> set[str]: + paths: set[str] = set() + for diagnostic in diagnostics: + if diagnostic["id"] == diagnostic_id and "path" in diagnostic: + paths.add(diagnostic["path"]) + return paths + + class GeneratedQualityTests(unittest.TestCase): def setUp(self) -> None: self.fs = MemoryFileSystem() @@ -362,6 +372,83 @@ def test_accepts_base_path_smoke_coverage(self) -> None: self.assertNotIn("quality.tests.missingBasePathSmoke", diagnostic_ids(result.warnings)) + def test_warns_when_feature_source_has_no_adjacent_unit_test(self) -> None: + self.add_package_file("src/app.test.ts", "test('custom basePath', () => {})\n") + self.add_package_file("src/features/workspaces/service.ts", "export class WorkspaceService {}\n") + + result = self.check() + + warning = next( + diagnostic + for diagnostic in result.warnings + if diagnostic["id"] == "quality.tests.missingFeatureUnit" + ) + self.assertEqual(warning["path"], "src/features/workspaces/service.test.ts") + self.assertIn("adjacent unit test", warning["message"]) + + def test_accepts_adjacent_feature_unit_test(self) -> None: + self.add_package_file("src/app.test.ts", "test('custom basePath', () => {})\n") + self.add_package_file("src/features/workspaces/service.ts", "export class WorkspaceService {}\n") + self.add_package_file("src/features/workspaces/service.test.ts", "test('service behavior', () => {})\n") + + result = self.check() + + self.assertNotIn("quality.tests.missingFeatureUnit", diagnostic_ids(result.warnings)) + + def test_accepts_adjacent_feature_spec_test(self) -> None: + self.add_package_file("src/app.test.ts", "test('custom basePath', () => {})\n") + self.add_package_file("src/features/workspaces/service.ts", "export class WorkspaceService {}\n") + self.add_package_file("src/features/workspaces/service.spec.ts", "test('service behavior', () => {})\n") + + result = self.check() + + self.assertNotIn("quality.tests.missingFeatureUnit", diagnostic_ids(result.warnings)) + + def test_warns_when_feature_repository_has_no_adjacent_unit_test(self) -> None: + self.add_package_file("src/app.test.ts", "test('custom basePath', () => {})\n") + self.add_package_file("src/features/workspaces/repository.ts", "export class WorkspacesRepository {}\n") + + result = self.check() + + self.assertIn( + "src/features/workspaces/repository.test.ts", + diagnostic_paths(result.warnings, "quality.tests.missingFeatureUnit"), + ) + + def test_warns_when_feature_helper_has_no_adjacent_unit_test(self) -> None: + self.add_package_file("src/app.test.ts", "test('custom basePath', () => {})\n") + self.add_package_file("src/features/workspaces/sortRules.ts", "export const byName = () => 0\n") + + result = self.check() + + self.assertIn( + "src/features/workspaces/sortRules.test.ts", + diagnostic_paths(result.warnings, "quality.tests.missingFeatureUnit"), + ) + + def test_feature_unit_warning_ignores_generated_adapters_and_seed(self) -> None: + self.add_package_file("src/app.test.ts", "test('custom basePath', () => {})\n") + self.add_package_file("src/features/workspaces/seed.ts", "export const seedWorkspaces = () => ({})\n") + self.add_package_file("src/features/workspaces/controllers/listWorkspaces.ts", "export const listWorkspaces = () => ({})\n") + + result = self.check() + + self.assertNotIn("quality.tests.missingFeatureUnit", diagnostic_ids(result.warnings)) + + def test_feature_unit_warning_ignores_metadata_and_incomplete_sources(self) -> None: + self.add_package_file("src/app.test.ts", "test('custom basePath', () => {})\n") + self.add_package_file("src/features/workspaces/index.ts", "export * from './service.ts'\n") + self.add_package_file("src/features/workspaces/types.ts", "export type WorkspaceId = string\n") + self.add_package_file("src/features/workspaces/model.d.ts", "export type Workspace = { id: string }\n") + self.add_package_file( + "src/features/workspaces/service.ts", + "throw new Error('TODO mockapi: implement service')\n", + ) + + result = self.check() + + self.assertNotIn("quality.tests.missingFeatureUnit", diagnostic_ids(result.warnings)) + def test_formats_every_error_and_warning(self) -> None: self.add_package_file( "src/app.ts",