Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions skills/mockapi/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,9 +88,9 @@ run unbundled scripts.
behavior from the matching `behavior.md` anchor, not from operation shape.
- Use `reference/sidecars.md` for sidecar shape and validation expectations.
- Use `reference/generated-server.md`, `reference/mock-server-structure.md`,
`reference/mock-server-examples.md`, `reference/seed-data.md`, and
`reference/mock-server-quality.md` when inspecting or finishing generated
server code.
`reference/mock-server-examples.md`, `reference/seed-data.md`,
`reference/testing.md`, and `reference/mock-server-quality.md` when
inspecting or finishing generated server code.

## Bundled Scripts

Expand Down
15 changes: 11 additions & 4 deletions skills/mockapi/reference/generate.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,8 +128,9 @@ nvm-managed Node installations and corepack for pnpm/Yarn.

7. After codegen succeeds, read `reference/mock-server-structure.md`,
`reference/mock-server-examples.md`, `reference/seed-data.md`,
`reference/mock-server-quality.md`, `.mockapi/behavior.md`, generated
`src/generated/**`, `src/controllers.ts`, and `src/features/**`. Create
`reference/testing.md`, `reference/mock-server-quality.md`,
`.mockapi/behavior.md`, generated `src/generated/**`,
`src/controllers.ts`, and `src/features/**`. Create
feature-local `service.ts` when behavior needs orchestration. Create
feature-local `repository.ts` for every completed feature that owns
non-infrastructure state slices; `idCounters` alone does not need a
Expand Down Expand Up @@ -160,15 +161,21 @@ nvm-managed Node installations and corepack for pnpm/Yarn.
explicit behavior opt-in. Allocate IDs once per transaction and pass the
allocator into helpers that need additional IDs. Do not infer behavior from
operation shape when the anchor has open questions.
Add adjacent Vitest unit tests for completed LLM-owned feature behavior:
`service.test.ts` next to `service.ts`, `repository.test.ts` next to
`repository.ts`, and `<helper>.test.ts` next to feature-local domain helpers.
Keep generated operation controller adapter tests out of scope unless an
adapter contains non-trivial normalization. Keep HTTP workflow smoke tests in
`src/app.test.ts`.

### Final Verification

8. After LLM-owned implementation edits, run the returned
`packageManager.commands.check` from the generator JSON in `packageRoot`.
Then run the generated package `test` script, if present. Generated mock
server templates include Vitest by default; put HTTP smoke tests in
`src/app.test.ts`, config/env tests in `src/config.test.ts`, and only split
large feature workflows into `src/features/<feature>/<feature>.test.ts`.
`src/app.test.ts`, config/env tests in `src/config.test.ts`, and unit tests
for feature behavior beside the source files they cover.
Then run the final quality checker:

```bash
Expand Down
23 changes: 15 additions & 8 deletions skills/mockapi/reference/mock-server-quality.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# Mock Server Quality Gate

Use this reference after `generate.py --run-codegen` and after completing
LLM-owned `src/features/**`, `src/controllers.ts`, seed data, and smoke tests.
This is a final handoff gate, not a scaffold scope probe. A freshly generated
scaffold is expected to fail because operation adapters start as TODOs and
feature seed stubs start empty.
LLM-owned `src/features/**`, `src/controllers.ts`, seed data, adjacent feature
unit tests, and smoke tests. This is a final handoff gate, not a scaffold scope
probe. A freshly generated scaffold is expected to fail because operation
adapters start as TODOs and feature seed stubs start empty.

## Required Checks

Expand Down Expand Up @@ -94,12 +94,19 @@ when appropriate.
The quality checker reports `quality.seed.emptyProductSeed` for empty product
seed literals while seed data is enabled.

## Minimum Smoke Coverage
## Test Coverage

Generated packages use Vitest. Keep HTTP smoke tests in `src/app.test.ts` by
default. Use `src/config.test.ts` only for config/env resolution, and use
`src/features/<feature>/<feature>.test.ts` only when a feature workflow is too
large for the app-level smoke file.
default. Use `src/config.test.ts` only for config/env resolution. Add adjacent
unit tests for completed LLM-owned feature behavior:

- `src/features/<feature>/service.test.ts` for `service.ts`
- `src/features/<feature>/repository.test.ts` for `repository.ts`
- `src/features/<feature>/<helper>.test.ts` for feature-local helpers

Do not unit-test generated thin operation controller adapters by default. Add an
adjacent controller test only when the adapter performs non-trivial request
normalization or branching not covered by service or app smoke tests.

Before handoff, add `src/app.test.ts` coverage for:

Expand Down
43 changes: 43 additions & 0 deletions skills/mockapi/reference/testing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# Mock Server Testing

Generated packages use Vitest. Add tests during the LLM-owned behavior
implementation phase, after `generate.py --run-codegen` and after generated
runtime types exist.

## Feature Unit Tests

Write unit tests beside the feature source file they cover:

- `src/features/<feature>/service.test.ts` for `service.ts`
- `src/features/<feature>/repository.test.ts` for `repository.ts`
- `src/features/<feature>/<helper>.test.ts` for feature-local domain helpers

Unit-test completed LLM-owned behavior, not deterministic generated scaffold.
Do not add tests for generated TODO adapters or thin operation controllers by
default. Add an adjacent controller test only when the controller performs
non-trivial request normalization or branching that is not covered by the
service or app smoke tests.

Prefer narrow tests over broad snapshots. Exercise repository selectors and
mutations, service orchestration, ID allocation behavior, soft-delete flows,
validation branches, and feature-local helper rules that came from
`.mockapi/behavior.md`.

## HTTP Smoke Tests

Keep HTTP workflow smoke tests in `src/app.test.ts` by default. Cover:

- custom `basePath` mounting
- one representative workflow for each non-trivial feature
- create operations that return generated counter IDs
- admin state reset or inspection when admin endpoints are generated

Use `src/config.test.ts` only for config/env resolution tests. Split a workflow
out of `src/app.test.ts` only when the app-level file becomes hard to maintain;
even then, keep source-adjacent unit tests for feature internals.

## Verification

Run the generated package check command first, then the generated package
`test` script, then `check_generated_quality.py`. Fix quality errors before
handoff. Fix obvious test warnings, or report them as residual risk.
57 changes: 57 additions & 0 deletions skills/mockapi/scripts/mockapi_runtime/quality.py
Original file line number Diff line number Diff line change
Expand Up @@ -191,6 +191,37 @@ def _is_feature_seed(relative: str) -> bool:
return relative.startswith("src/features/") and relative.endswith("/seed.ts")


def _is_test_file(relative: str) -> bool:
return TEST_FILE_PATTERN.search(Path(relative).name) is not None


def _is_feature_unit_source(relative: str) -> bool:
if not relative.startswith("src/features/"):
return False
if _is_test_file(relative) or _is_feature_seed(relative) or "/controllers/" in relative:
return False
if relative.endswith(".d.ts"):
return False
if Path(relative).name in {"index.ts", "types.ts"}:
return False
return True


def _adjacent_unit_test_candidates(relative: str) -> list[str]:
source_path = Path(relative)
extensions = [source_path.suffix]
extensions.extend(
extension
for extension in (".ts", ".tsx", ".mts", ".cts")
if extension != source_path.suffix
)
return [
(source_path.parent / f"{source_path.stem}{kind}{extension}").as_posix()
for kind in (".test", ".spec")
for extension in extensions
]


def _feature_has_completed_behavior(source_files: list[ScannedFile], feature: str) -> bool:
service_relative = f"src/features/{feature}/service.ts"
if any(scanned_file.relative == service_relative for scanned_file in source_files):
Expand Down Expand Up @@ -487,6 +518,31 @@ def _check_smoke_tests(context: QualityScanContext, warnings: list[Diagnostic])
)


def _check_feature_unit_tests(context: QualityScanContext, warnings: list[Diagnostic]) -> None:
test_relatives = {scanned_file.relative for scanned_file in context.test_files}

for scanned_file in context.source_files:
if not _is_feature_unit_source(scanned_file.relative):
continue
if TODO_PATTERN.search(scanned_file.text):
continue

candidates = _adjacent_unit_test_candidates(scanned_file.relative)
if any(candidate in test_relatives for candidate in candidates):
continue

warnings.append(
_diagnostic(
"quality.tests.missingFeatureUnit",
(
"Completed LLM-owned feature source has no adjacent unit test; "
f"add {candidates[0]} or report the uncovered behavior as residual risk."
),
path=candidates[0],
)
)


def check_generated_quality(
fs: FileSystem,
package_root: Path,
Expand All @@ -509,6 +565,7 @@ def check_generated_quality(
_check_snapshot_set_all(context, warnings)
_check_unsafe_casts(context, warnings)
_check_smoke_tests(context, warnings)
_check_feature_unit_tests(context, warnings)

return QualityCheckResult(errors=errors, warnings=warnings)

Expand Down
89 changes: 88 additions & 1 deletion tests/test_quality.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,29 @@

import unittest
from pathlib import Path
from typing import Iterable

from tests.support import MemoryFileSystem, PROJECT_ROOT, profile_toml

from mockapi_runtime.diagnostics import Diagnostic
from mockapi_runtime.quality import check_generated_quality, format_quality_result


PACKAGE_ROOT = PROJECT_ROOT / "mock-server"


def diagnostic_ids(diagnostics: list[dict[str, str]]) -> set[str]:
def diagnostic_ids(diagnostics: Iterable[Diagnostic]) -> set[str]:
return {diagnostic["id"] for diagnostic in diagnostics}


def diagnostic_paths(diagnostics: Iterable[Diagnostic], diagnostic_id: str) -> set[str]:
paths: set[str] = set()
for diagnostic in diagnostics:
if diagnostic["id"] == diagnostic_id and "path" in diagnostic:
paths.add(diagnostic["path"])
return paths


class GeneratedQualityTests(unittest.TestCase):
def setUp(self) -> None:
self.fs = MemoryFileSystem()
Expand Down Expand Up @@ -362,6 +372,83 @@ def test_accepts_base_path_smoke_coverage(self) -> None:

self.assertNotIn("quality.tests.missingBasePathSmoke", diagnostic_ids(result.warnings))

def test_warns_when_feature_source_has_no_adjacent_unit_test(self) -> None:
self.add_package_file("src/app.test.ts", "test('custom basePath', () => {})\n")
self.add_package_file("src/features/workspaces/service.ts", "export class WorkspaceService {}\n")

result = self.check()

warning = next(
diagnostic
for diagnostic in result.warnings
if diagnostic["id"] == "quality.tests.missingFeatureUnit"
)
self.assertEqual(warning["path"], "src/features/workspaces/service.test.ts")
self.assertIn("adjacent unit test", warning["message"])

def test_accepts_adjacent_feature_unit_test(self) -> None:
self.add_package_file("src/app.test.ts", "test('custom basePath', () => {})\n")
self.add_package_file("src/features/workspaces/service.ts", "export class WorkspaceService {}\n")
self.add_package_file("src/features/workspaces/service.test.ts", "test('service behavior', () => {})\n")

result = self.check()

self.assertNotIn("quality.tests.missingFeatureUnit", diagnostic_ids(result.warnings))

def test_accepts_adjacent_feature_spec_test(self) -> None:
self.add_package_file("src/app.test.ts", "test('custom basePath', () => {})\n")
self.add_package_file("src/features/workspaces/service.ts", "export class WorkspaceService {}\n")
self.add_package_file("src/features/workspaces/service.spec.ts", "test('service behavior', () => {})\n")

result = self.check()

self.assertNotIn("quality.tests.missingFeatureUnit", diagnostic_ids(result.warnings))

def test_warns_when_feature_repository_has_no_adjacent_unit_test(self) -> None:
self.add_package_file("src/app.test.ts", "test('custom basePath', () => {})\n")
self.add_package_file("src/features/workspaces/repository.ts", "export class WorkspacesRepository {}\n")

result = self.check()

self.assertIn(
"src/features/workspaces/repository.test.ts",
diagnostic_paths(result.warnings, "quality.tests.missingFeatureUnit"),
)

def test_warns_when_feature_helper_has_no_adjacent_unit_test(self) -> None:
self.add_package_file("src/app.test.ts", "test('custom basePath', () => {})\n")
self.add_package_file("src/features/workspaces/sortRules.ts", "export const byName = () => 0\n")

result = self.check()

self.assertIn(
"src/features/workspaces/sortRules.test.ts",
diagnostic_paths(result.warnings, "quality.tests.missingFeatureUnit"),
)

def test_feature_unit_warning_ignores_generated_adapters_and_seed(self) -> None:
self.add_package_file("src/app.test.ts", "test('custom basePath', () => {})\n")
self.add_package_file("src/features/workspaces/seed.ts", "export const seedWorkspaces = () => ({})\n")
self.add_package_file("src/features/workspaces/controllers/listWorkspaces.ts", "export const listWorkspaces = () => ({})\n")

result = self.check()

self.assertNotIn("quality.tests.missingFeatureUnit", diagnostic_ids(result.warnings))

def test_feature_unit_warning_ignores_metadata_and_incomplete_sources(self) -> None:
self.add_package_file("src/app.test.ts", "test('custom basePath', () => {})\n")
self.add_package_file("src/features/workspaces/index.ts", "export * from './service.ts'\n")
self.add_package_file("src/features/workspaces/types.ts", "export type WorkspaceId = string\n")
self.add_package_file("src/features/workspaces/model.d.ts", "export type Workspace = { id: string }\n")
self.add_package_file(
"src/features/workspaces/service.ts",
"throw new Error('TODO mockapi: implement service')\n",
)

result = self.check()

self.assertNotIn("quality.tests.missingFeatureUnit", diagnostic_ids(result.warnings))

def test_formats_every_error_and_warning(self) -> None:
self.add_package_file(
"src/app.ts",
Expand Down