feat(stores): pluggable per-artifact backends via typing.Protocol

.specify/feature.json

@@ -1 +1 @@
-{"feature_directory": "specs/032-ruleset-branch-protection"}
+{"feature_directory": "specs/033-pluggable-stores"}

CLAUDE.md

@@ -381,5 +381,5 @@ else:
 <!-- SPECKIT START -->
 For additional context about technologies to be used, project structure,
 shell commands, and other important information, read the current plan:
-[`specs/032-ruleset-branch-protection/plan.md`](specs/032-ruleset-branch-protection/plan.md)
+[`specs/033-pluggable-stores/plan.md`](specs/033-pluggable-stores/plan.md)
 <!-- SPECKIT END -->

docs/architecture/framework-design.md

@@ -1120,6 +1120,10 @@ Implementation-registered sieve handlers (non-exhaustive): `github_branch_protec
 
 A handler used in a phase different from its registered affinity SHALL trigger a warning but still execute.
 
+## 13. Persistence Extension Surface
+
+The framework SHALL provide four per-artifact persistence Protocols under `darnit.stores`, alongside the existing extension surfaces `darnit.frameworks` (compliance implementations) and `darnit.question_resolvers` (feature 027). Third-party persistence backends register under Python entry-point groups `darnit.stores.project`, `darnit.stores.attestation`, `darnit.stores.report`, `darnit.stores.cache`. Filesystem-backed default implementations ship in `darnit-core` and reproduce the pre-feature on-disk layout exactly (feature 033); alternative backends are opt-in via `.baseline.toml` `[stores.<kind>] backend = "..."` blocks. See `specs/033-pluggable-stores/contracts/` for per-Protocol contracts.
+
 ## Appendix C: Removed Requirements
 
 The following requirements have been superseded by the handler dispatch architecture.

docs/plugin-authoring/stores.md

@@ -0,0 +1,172 @@
+# Authoring a Pluggable Store Backend
+
+Feature 033 exposes four per-artifact Protocols that alternative
+backends can satisfy. This document explains how to distribute a
+backend as a Python package that darnit discovers automatically.
+
+## What you can back with a plugin
+
+| Kind         | Protocol            | Discovery group             |
+|--------------|---------------------|-----------------------------|
+| project      | `ProjectStateStore` | `darnit.stores.project`     |
+| attestation  | `AttestationStore`  | `darnit.stores.attestation` |
+| report       | `ReportStore`       | `darnit.stores.report`      |
+| audit cache  | `AuditCacheStore`   | `darnit.stores.cache`       |
+
+Each Protocol lives in `darnit.stores.protocols`. All four inherit from
+a `Store` base carrying a single `close()` method.
+
+## Minimal example: an S3-backed `AttestationStore`
+
+### Package layout
+
+```
+my-s3-store/
+  pyproject.toml
+  src/my_s3_store/
+    __init__.py
+    backend.py
+```
+
+### `pyproject.toml`
+
+```toml
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "my-s3-store"
+version = "0.1.0"
+requires-python = ">=3.10"
+dependencies = ["boto3"]
+
+[project.entry-points."darnit.stores.attestation"]
+s3 = "my_s3_store.backend:S3AttestationStore"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/my_s3_store"]
+```
+
+### `src/my_s3_store/backend.py`
+
+```python
+from __future__ import annotations
+
+import boto3
+
+
+class S3AttestationStore:
+    """Writes attestation bundles to an S3 bucket."""
+
+    def __init__(self, *, bucket: str, prefix: str = "", **_) -> None:
+        self._bucket = bucket
+        self._prefix = prefix.rstrip("/") + "/" if prefix else ""
+        self._s3 = boto3.client("s3")
+
+    def write(self, bundle_id: str, bundle_bytes: bytes, content_type: str) -> None:
+        # content_type maps to filename extension per feature 033 R-004:
+        #   application/vnd.in-toto+json          -> .intoto.json
+        #   application/vnd.dev.sigstore.bundle+json -> .sigstore.json
+        ext = ".sigstore.json" if "sigstore" in content_type else ".intoto.json"
+        key = f"{self._prefix}{bundle_id}{ext}"
+        self._s3.put_object(
+            Bucket=self._bucket,
+            Key=key,
+            Body=bundle_bytes,
+            ContentType=content_type,
+        )
+
+    def close(self) -> None:
+        # boto3 clients hold no persistent connection; nothing to release.
+        return None
+```
+
+### Selecting the backend in `.baseline.toml`
+
+```toml
+[stores.attestation]
+backend = "s3"
+bucket = "my-attestations"
+prefix = "openssf-baseline"
+```
+
+Any keys beyond `backend` are passed as keyword arguments to the
+backend's constructor.
+
+Environment variables can be interpolated with `$VAR`:
+
+```toml
+[stores.attestation]
+backend = "s3"
+bucket = "$ATTESTATION_BUCKET"
+```
+
+## Protocol contracts (must-read)
+
+Every backend must uphold these invariants:
+
+* **`close()` idempotence (FR-019).** `store.close(); store.close()` must
+  not raise. The audit driver's `close_all()` calls it exactly once per
+  instantiated store, but tests may double-close.
+* **`AuditCacheStore.write` must not raise (FR-011).** Cache writes are
+  best-effort; the audit run must not fail because a cache backend
+  hiccuped. Swallow backend exceptions internally.
+* **`AuditCacheStore.read` returns `None` on miss/corruption.** Do not
+  raise on a missing key or malformed payload; return `None` and let
+  the caller run a fresh audit.
+* **Lazy construction (SC-004).** Backends whose artifact class the
+  current audit never touches are never constructed. Keep `__init__`
+  cheap; do not open connections until first `read`/`write`.
+* **`ProjectStateStore.read_project()` returns `None` when there is no
+  seeded project state.** The reader treats `None` as "no `.project/`".
+* **Backends may accept a `repo_path` kwarg.** The selector passes it
+  when the backend's `__init__` signature accepts it and falls back
+  transparently when it doesn't.
+
+## Testing your backend
+
+darnit ships `darnit-testchecks` with in-memory reference
+implementations for all four kinds -- use them as a behavior baseline.
+Then write an integration test that pip-installs your package, calls
+`resolve_stores(StoresConfig(attestation=StoreBlock(backend="s3", ...)),
+repo_path=tmp_path)`, and asserts on your backend's observable state
+after an audit run.
+
+Reference: `tests/darnit/stores/fixtures/example_store_plugin_pkg/`
+plus `tests/darnit/stores/test_us3_plugin_*.py`.
+
+## Error surfaces you may see
+
+| Exception                | When                                            |
+|--------------------------|-------------------------------------------------|
+| `StoreNotInstalled`      | TOML names a backend not registered             |
+| `StoreProtocolMismatch`  | Registered class missing a required method     |
+| `StoreNameCollision`    | Two packages register the same name/group     |
+| `StoreOperationError`    | Backend raised during read/write (except cache) |
+
+`StoreNotInstalled` and `StoreProtocolMismatch` fire at
+`resolve_stores` time, before any control runs -- so a misconfigured
+backend never wastes an audit.
+
+## Performance notes
+
+Two costs, both small:
+
+* **Entry-point discovery** is a one-time per-process cost paid at
+  framework load. `importlib.metadata.entry_points(group=...)` scans
+  installed dist-info metadata; on a fresh venv with ~50 installed
+  packages we measure single-digit milliseconds. The `discover_stores`
+  wrapper caches the result per group so subsequent audits in the same
+  process pay zero.
+* **Lazy instantiation** adds one dict lookup per audit run per
+  artifact class. A store whose kind the run never touches is never
+  constructed at all -- ideal for expensive-to-open backends (network
+  clients, DB connections). The bundle's per-property accessor memoizes,
+  so second access is a straight attribute read.
+
+Practical implication: your `__init__` cost is charged to the first
+audit in a process that actually uses the artifact class you back --
+never to zero-config runs, never to audits that skip your kind. Keep
+`__init__` cheap and open connections on first `write` if the backend
+is expensive to establish.

packages/darnit-baseline/src/darnit_baseline/attestation/generator.py

@@ -54,16 +54,22 @@ def generate_attestation_from_results(
     staging: bool = False,
     output_path: str | None = None,
     output_dir: str | None = None,
-    storage_config: dict | None = None
+    storage_config: dict | None = None,
+    attestation_store: Any = None,
 ) -> str:
     """Generate attestation from audit results.
 
     Args:
         audit_result: The audit result containing check results and metadata
         sign: Whether to sign with Sigstore (default True)
         staging: Use Sigstore staging environment for testing
-        output_path: Explicit path for the attestation file
-        output_dir: Directory to save attestation (default: repository directory)
+        output_path: Explicit path for the attestation file (legacy)
+        output_dir: Directory to save attestation (legacy; default: repo dir)
+        attestation_store: Feature 033 pluggable AttestationStore. When
+            provided, the bundle is written via ``store.write(bundle_id,
+            bytes, content_type)`` instead of the legacy filesystem
+            path. When None, ``output_path`` / ``output_dir`` control
+            the on-disk write (unchanged pre-feature behavior).
 
     Returns:
         JSON string with attestation or error message
@@ -125,24 +131,45 @@ def generate_attestation_from_results(
         )
         output = json.dumps(unsigned, indent=2)
 
-    # Determine output file path
-    if not output_path:
-        extension = ".sigstore.json" if sign else ".intoto.json"
-        filename = f"{audit_result.repo}-baseline-attestation{extension}"
-        save_dir = output_dir if output_dir else audit_result.local_path
-        output_path = os.path.join(save_dir, filename)
-
-    # Save the attestation
-    try:
-        os.makedirs(os.path.dirname(output_path) or ".", exist_ok=True)
-        with open(output_path, 'w', encoding="utf-8") as f:
-            f.write(output)
-        logger.info(f"Attestation saved to: {output_path}")
-    except OSError as e:
-        return json.dumps({
-            "error": f"Failed to write to {output_path}: {e}",
-            "attestation": json.loads(output)
-        }, indent=2)
+    # Feature 033 T027: prefer the pluggable AttestationStore when
+    # supplied. The legacy `output_path` / `output_dir` remain the
+    # zero-config path so pre-feature callers keep the same on-disk
+    # semantics without change.
+    if attestation_store is not None:
+        content_type = (
+            "application/vnd.dev.sigstore.bundle+json"
+            if sign
+            else "application/vnd.in-toto+json"
+        )
+        bundle_id = f"{audit_result.repo}-baseline-attestation"
+        try:
+            attestation_store.write(
+                bundle_id, output.encode("utf-8"), content_type
+            )
+            logger.info(f"Attestation written via store: bundle_id={bundle_id}")
+        except Exception as e:  # noqa: BLE001
+            return json.dumps({
+                "error": f"AttestationStore write failed: {e}",
+                "attestation": json.loads(output),
+            }, indent=2)
+    else:
+        # Determine output file path (pre-feature filesystem path).
+        if not output_path:
+            extension = ".sigstore.json" if sign else ".intoto.json"
+            filename = f"{audit_result.repo}-baseline-attestation{extension}"
+            save_dir = output_dir if output_dir else audit_result.local_path
+            output_path = os.path.join(save_dir, filename)
+
+        try:
+            os.makedirs(os.path.dirname(output_path) or ".", exist_ok=True)
+            with open(output_path, 'w', encoding="utf-8") as f:
+                f.write(output)
+            logger.info(f"Attestation saved to: {output_path}")
+        except OSError as e:
+            return json.dumps({
+                "error": f"Failed to write to {output_path}: {e}",
+                "attestation": json.loads(output)
+            }, indent=2)
 
     # Store via pluggable storage backend if configured
     if storage_config is not None:

packages/darnit-testchecks/pyproject.toml

@@ -57,6 +57,19 @@ testchecks = "darnit_testchecks.adapters.builtin:TrivialRemediationAdapter"
 [project.entry-points."darnit.adapters"]
 testchecks = "darnit_testchecks.adapters.builtin:get_test_check_adapter"
 
+# In-memory reference store backends (feature 033 T020)
+[project.entry-points."darnit.stores.project"]
+in-memory = "darnit_testchecks.stores:InMemoryProjectStateStore"
+
+[project.entry-points."darnit.stores.attestation"]
+in-memory = "darnit_testchecks.stores:InMemoryAttestationStore"
+
+[project.entry-points."darnit.stores.report"]
+in-memory = "darnit_testchecks.stores:InMemoryReportStore"
+
+[project.entry-points."darnit.stores.cache"]
+in-memory = "darnit_testchecks.stores:InMemoryAuditCacheStore"
+
 [tool.hatch.build.targets.wheel]
 packages = ["src/darnit_testchecks"]
 

packages/darnit-testchecks/src/darnit_testchecks/stores/__init__.py

@@ -0,0 +1,26 @@
+"""In-memory reference store backends for testing.
+
+Feature 033 T020. These backends are shipped by darnit-testchecks so
+they are only available in dev/test environments (they are NOT a
+runtime dependency of darnit-core). They exist to:
+
+* Prove the pluggable-stores machinery works end-to-end (US1 equivalence
+  test) without needing a real filesystem-free storage backend.
+* Give internal + external tests a zero-dependency, easily-inspectable
+  reference implementation to seed and assert against.
+
+Each backend exposes a ``_state`` attribute tests can read for
+assertions.
+"""
+
+from darnit_testchecks.stores.in_memory_attestation import InMemoryAttestationStore
+from darnit_testchecks.stores.in_memory_cache import InMemoryAuditCacheStore
+from darnit_testchecks.stores.in_memory_project import InMemoryProjectStateStore
+from darnit_testchecks.stores.in_memory_report import InMemoryReportStore
+
+__all__ = [
+    "InMemoryAttestationStore",
+    "InMemoryAuditCacheStore",
+    "InMemoryProjectStateStore",
+    "InMemoryReportStore",
+]

packages/darnit-testchecks/src/darnit_testchecks/stores/in_memory_attestation.py

@@ -0,0 +1,23 @@
+"""In-memory AttestationStore reference backend (feature 033 T020)."""
+
+from __future__ import annotations
+
+
+class InMemoryAttestationStore:
+    """Dict-backed AttestationStore for tests.
+
+    Records every write as (bundle_id, bundle_bytes, content_type) in
+    ``self._state`` (a list). Tests assert on it directly.
+    """
+
+    def __init__(self, **kwargs) -> None:
+        self._state: list[tuple[str, bytes, str]] = []
+
+    def write(self, bundle_id: str, bundle_bytes: bytes, content_type: str) -> None:
+        self._state.append((bundle_id, bundle_bytes, content_type))
+
+    def close(self) -> None:
+        return None
+
+
+__all__ = ["InMemoryAttestationStore"]

packages/darnit-testchecks/src/darnit_testchecks/stores/in_memory_cache.py

@@ -0,0 +1,32 @@
+"""In-memory AuditCacheStore reference backend (feature 033 T020)."""
+
+from __future__ import annotations
+
+from typing import Any
+
+
+class InMemoryAuditCacheStore:
+    """Dict-backed AuditCacheStore for tests.
+
+    ``self._state`` is a dict keyed by cache_key. write() must NOT raise
+    (FR-011); read() returns None on miss.
+    """
+
+    def __init__(self, **kwargs) -> None:
+        self._state: dict[str, dict[str, Any]] = {}
+
+    def read(self, cache_key: str) -> "dict[str, Any] | None":
+        return self._state.get(cache_key)
+
+    def write(self, cache_key: str, payload: "dict[str, Any]") -> None:
+        # FR-011: cache write is best-effort; swallow all errors.
+        try:
+            self._state[cache_key] = dict(payload)
+        except Exception:
+            pass
+
+    def close(self) -> None:
+        return None
+
+
+__all__ = ["InMemoryAuditCacheStore"]