Skip to content

feat(stores): pluggable per-artifact backends via typing.Protocol - #396

Open
mlieberman85 wants to merge 2 commits into
darnitdevorg:mainfrom
mlieberman85:033-pluggable-stores
Open

feat(stores): pluggable per-artifact backends via typing.Protocol#396
mlieberman85 wants to merge 2 commits into
darnitdevorg:mainfrom
mlieberman85:033-pluggable-stores

Conversation

@mlieberman85

Copy link
Copy Markdown
Contributor

Summary

Introduces darnit.stores -- four per-artifact typing.Protocols
(ProjectStateStore, AttestationStore, ReportStore,
AuditCacheStore) plus filesystem-default backends shipped in
darnit-core. Third-party packages register alternatives via
importlib.metadata entry points under
darnit.stores.{project,attestation,report,cache}.

Motivation (see spec.md User
Stories 1-4): darnit-core stays filesystem-only per constitution I,
but downstream users need to point specific artifact classes at
alternative targets (project state to shared Postgres, attestations to
S3, ...) without forking. This lays the seam without touching
zero-config behavior.

Key mechanics

  • resolve_stores validates selections eagerly (SC-007) via a
    class-shape Protocol check but defers construction to first access
    (SC-004). A backend whose kind the run never touches is never
    instantiated. close_all() runs at audit-boundary teardown, only
    for stores actually accessed.
  • Discovery is per-process cached. Name collisions raise
    StoreNameCollision; broken entry-points are logged and skipped.
  • .baseline.toml composes over the framework TOML using
    per-kind replacement (mirrors feature 031's mcp_servers).
    $VAR substitution helper factored to darnit.core.env_subst and
    reused by feature 025 exec / 031 MCP paths.
  • No new runtime dependencies. No product-package modifications
    outside darnit-core, darnit-baseline, darnit-testchecks.

Migrations

  • Attestation generator gains optional attestation_store kwarg;
    legacy output_path filesystem path is the default.
  • DotProjectReader + DotProjectMapper accept an optional
    ProjectStateStore; unset means pre-feature filesystem read
    (backward-compat locked by test_backward_compat.py).
  • Audit driver calls resolve_stores + close_all around each
    audit run.

Deferred (documented in tasks.md)

  • T026: audit-cache module migration -- location semantics
    (system-tempdir + repo hash) need a separate design conversation.
  • T022: DotProjectWriter refactor -- read seam is what US1 needs;
    write seam lands with T029's control-side integration.
  • T028: bulk rewrite of existing reader/writer tests -- backward
    compat means they still pass unchanged.
  • T040 / T041 / T044: fault-injection tests for project-store and
    report-store -- blocked on control-side integration / darnit audit cannot emit SARIF or Markdown from the CLI #341.

All follow the surface established in this PR.

Test plan

  • 30 protocol / discovery / selection tests
  • 7 US1 tests (mapper equivalence, isolation, lazy instantiation
    -- SC-002 + SC-004)
  • 6 US2 tests (zero-config invariance + backward-compat -- SC-003)
  • 5 US3 tests (entry-point discovery + TOML selection + error
    shapes)
  • 6 US4 fault-injection tests (FR-011 + FR-012)
  • 2 import-isolation guards (SC-008 + FR-017)
  • Full workspace sweep: 2928 pass, 22 skip, 0 fail
  • uv run ruff check . clean on touched files
  • uv run python scripts/validate_sync.py --verbose clean
  • git diff main..HEAD -- pyproject.toml packages/*/pyproject.toml
    adds no new [project.dependencies] entry (FR-014)

Plugin author guide at docs/plugin-authoring/stores.md includes a
worked S3-backed AttestationStore example.

Adds `darnit.stores` sub-package with four @runtime_checkable Protocols
(ProjectStateStore, AttestationStore, ReportStore, AuditCacheStore),
filesystem-default backends shipped in darnit-core, and
entry-point-based discovery so third-party packages can distribute
alternative backends without patching darnit.

Motivation: darnit-core is filesystem-only by design (constitution I),
but downstream users need to point specific artifact classes at other
targets (project state to a shared Postgres, attestations to S3, etc.)
without forking. This lays the seam without changing zero-config
behavior: with no [stores.*] block in .baseline.toml, every artifact
class continues to land on the same filesystem paths as before.

Key mechanics:

* `resolve_stores` validates selections eagerly (SC-007) via a
  class-shape Protocol check but defers construction to first access
  (SC-004). A backend whose kind the run never touches is never
  instantiated.
* Backends are discovered via importlib.metadata under four groups:
  darnit.stores.{project,attestation,report,cache}. Discovery is
  cached per-process.
* `.baseline.toml` composes over the framework TOML using the same
  per-kind replacement rule as feature 031's mcp_servers.
* No new runtime dependencies. No product-package modifications
  outside darnit-core / darnit-baseline / darnit-testchecks.

Migrations:

* Attestation generator gains an optional `attestation_store` kwarg;
  legacy `output_path` filesystem path stays default for backward
  compat.
* DotProjectReader + DotProjectMapper accept an optional
  ProjectStateStore; unset -> pre-feature filesystem read.
* Audit driver calls resolve_stores + close_all around each run.

Deferred to follow-up features (documented in tasks.md): audit-cache
module migration (T026), DotProjectWriter refactor (T022), and the
report-store integration (blocked on darnitdevorg#341's report consumer). All
follow the surface established here.

Tests: 30 protocol/discovery/selection tests, 7 US1 (equivalence +
isolation + lazy), 6 US2 (zero-config + backward-compat), 5 US3
(discovery + selection + errors), 6 US4 (fault injection), 2
import-isolation guards (SC-008 + FR-017). Full workspace sweep:
2928 pass, 0 fail.

Includes plugin-author docs at docs/plugin-authoring/stores.md.
Was created after the per-file ruff pass and slipped through. No
behavior change.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant