jmjava · jmjava · Aug 2, 2026 · Apr 6, 2026 · Apr 6, 2026 · Jul 5, 2026
diff --git a/.gitignore b/.gitignore
@@ -38,6 +38,7 @@ embabel-agent-api/src/main/resources/mcp/**
 # Ignore all personal profile files except the checked-in example
 scripts/user-config/application-*.yml
 !scripts/user-config/application-*.yml.example
+scripts/user-config/ingestion-git-revisions.json
 
 # Temporary files
 *.tmp

diff --git a/codegen-gradle/build.gradle.kts b/codegen-gradle/build.gradle.kts
@@ -19,7 +19,7 @@ plugins {
 group = "com.embabel.guide"
 version = "0.1.0-SNAPSHOT"
 
-val drivineVersion = "0.0.28"
+val drivineVersion = "0.0.45"
 
 repositories {
     mavenCentral()

diff --git a/docs/spdd-branch-changes.md b/docs/spdd-branch-changes.md
@@ -0,0 +1,99 @@
+# Branch change summary for Guide developers
+
+Branch: `cursor/spike-spdd-dice-projection-17f4` (tracks upstream `main`).
+Audience: developers who work on Guide and want to understand what this branch adds,
+why, and what the blast radius is.
+
+**One paragraph:** the branch turns Guide into an optional *hybrid context backend* for
+an SDLC workflow: alongside the existing RAG chunk store, an opt-in projection writes
+**typed domain entities** (`__Entity__` nodes: `WorkId`, `Canvas`, `Area`, `Decision`,
+`Pitfall`, `Pattern`) into the same Neo4j and exposes typed-edge retrieval over HTTP and
+MCP. Everything is behind `guide.spdd-projection.enabled` (default **false**); with the
+flag off, runtime behavior matches upstream except for the additions listed under
+"Cross-cutting" below.
+
+## 1. New package `com.embabel.guide.spdd` (all opt-in)
+
+| Class | Role |
+|-------|------|
+| `SpddEntityDictionary` | `DataDictionary.fromClasses` over `NamedEntity` domain types in `spdd.domain` — schema + label validation |
+| `SpddMarkdownProjectionService` | Parses structured markdown (`spdd/canvas/*.md`, `agent-context/memory/context-index.md`) and persists via `NamedEntityDataRepository.save` + `mergeRelationship` (merge-by-id, idempotent). Read side: `subgraphForWorkId`, `lessonsForArea`, `listByLabel` |
+| `SpddProjectionController` | Operator HTTP under `/api/v1/data/spdd-projection` (`load`, `stats`, `work/{workId}`, `area?name=`) with explicit status mapping (400 validation / 404 not found / 409 disabled) |
+| `SpddDomainTools` | `@LlmTool` methods exported to MCP as `spdd_*` via `McpToolExport.fromToolObject(ToolObject(...).withPrefix("spdd_"))`; failures return `{"error": …}` JSON |
+| `SpddProjectionConfiguration` | Beans: `DrivineNamedEntityDataRepository` wired with the SPDD dictionary + the MCP export. `@ConditionalOnProperty` on the enable flag |
+
+Graph model: `WorkId —canvas→ Canvas`, `WorkId —area→ Area`,
+`WorkId —decision/pitfall/pattern→ lesson`, `lesson —about→ Area`. The `about` edge is
+what makes lessons retrievable **across** work items by code area.
+
+Hardening baked in: per-request `rootPath` overrides must resolve under
+`default-root-path` or configured `allowed-roots` (the load endpoint is on the
+permit-all list, so arbitrary filesystem roots are rejected); a malformed source file is
+skipped and counted (`skippedFiles`) instead of failing the load; list reads accept only
+schema labels and are capped (50 default / 200 max).
+
+Uses only public library APIs (`NamedEntityData`, `NamedEntityDataRepository`,
+`RelationshipDirection`, `DataDictionary`). It does **not** touch the DICE proposition
+extraction pipeline, and no consumer project names are hardcoded — the coupling is to
+the SPDD directory conventions only.
+
+## 2. RAG ingest: git-incremental directories
+
+- `GitIncrementalDirectorySupport` + `GitIngestionRevisionStore` — when
+  `guide.git-ingestion.enabled=true`, directory ingest diffs against the last recorded
+  git revision and reprocesses only changed files. Subdirectory entries (e.g.
+  `repo/spdd/canvas`) are supported: Guide walks up to the `.git` root and scopes the
+  diff.
+- `DataManager` — hooks the incremental path into `loadReferences()`.
+- `RagContentMaintenanceService`, `RagMaintenanceController`,
+  `RagMaintenanceExceptionHandler` — operator endpoints for content-element purge
+  preview/purge and git revision reset.
+
+## 3. Cross-cutting changes (active regardless of the flag)
+
+- **`GuideProperties`** — new `spddProjection` block (`enabled`, `defaultRootPath`,
+  `allowedRoots`) and git-ingestion settings; `application.yml` documents both, defaults
+  off.
+- **`SecurityConfig`** — permit-all additions: POST `…/spdd-projection/load`,
+  `…/git-ingestion/revision/reset`, `…/content-elements/purge{,-preview}`; GET
+  `…/spdd-projection/stats`, `…/work/*`, `…/area`. Same local-operator posture as the
+  existing `…/data/load-references`.
+- **`PersonaSeedingService`** — startup resilience: fails fast with an actionable
+  message if the Drivine KSP query DSL is missing from the classpath, and persona
+  seeding failures no longer abort startup (RAG/MCP stay available).
+- **Build (`pom.xml`)** — `embabel-agent-rag-neo-drivine` pinned to timestamp
+  `0.1.2-20260224.010659-19` (newer snapshots require agent 0.4.0; Guide is on
+  0.3.5-SNAPSHOT); maven-enforcer rule fails the build early when the KSP-generated
+  DSL files are missing instead of dying at runtime.
+- **Scripts** — `append-ingest.sh` gains env-driven Neo4j/profile handling;
+  `run-mcp-guide-against-hub.sh` added; `application-*-spdd-projection.yml.example`
+  profile example under `scripts/user-config/`.
+
+## 4. Tests
+
+- `SpddMarkdownProjectionServiceTest` (16) — projection, idempotent reload, lesson/about
+  edges, root allowlist enforcement, blank/unknown input validation, list caps.
+- `SpddProjectionControllerTest` (8) — standalone MockMvc against the real service +
+  in-memory repository; verifies the 200/400/404 mapping.
+- `SpddDomainToolsTest` (8) — MCP JSON contract, `{"error": …}` on bad input.
+- `GitIncrementalDirectorySupportTest`, `RagMaintenanceControllerWebMvcTest`,
+  `DataManagerControllerWebMvcTest` — incremental ingest + maintenance endpoints.
+- Fixture: `src/test/resources/spdd-fixture/` (minimal canvas + context index).
+
+## 5. How to review / try it
+
+1. `git diff upstream/main...HEAD` — ~30 files; the spdd package and rag ingest classes
+   are the substance.
+2. Operator walkthrough: `docs/spdd-projection-ingest.md` (enable, persist, retrieve,
+   MCP tools).
+3. Quick sanity: enable the flag against a project following the SPDD layout, then
+   `POST …/spdd-projection/load` and `GET …/work/{workId}`.
+
+## 6. Known limitations
+
+- Schema uses `DynamicType` + `SimpleNamedEntityData` (spike speed); a permanent home
+  would promote to typed Kotlin `NamedEntity` classes.
+- Entity→chunk join (`findChunksForEntity`) exists at the store level but is not exposed
+  on the projection API yet.
+- `Operation`, session, and domain-keyword entities are declared in the schema roadmap
+  but not projected yet.
diff --git a/docs/spdd-projection-ingest.md b/docs/spdd-projection-ingest.md
@@ -0,0 +1,112 @@
+# SPDD leg 3 entity projection (SPIKE-001) — DICE persist/retrieve contract
+
+Projects structured SPDD markdown into Neo4j `__Entity__` nodes via
+`NamedEntityDataRepository`. **Coexists** with leg 2 RAG chunk ingest (`guide.directories`).
+
+Does **not** use the DICE proposition extraction pipeline (conversation → propositions).
+
+Markdown remains **source of truth**. Projection is the write path into domain memory;
+domain-graph walk is the preferred read path for auditable context selection.
+
+## Enable
+
+```yaml
+guide:
+  spdd-projection:
+    enabled: true
+    default-root-path: /home/ubuntu/github/jmjava/sdlc-spdd-orchestrator
+    # Optional. Roots a per-request rootPath override may resolve under, in addition
+    # to default-root-path. Anything else → HTTP 400 (load is on the permit-all list).
+    allowed-roots: []
+```
+
+Or per-request `rootPath` on load (subject to the allowed-roots guard).
+
+Build note: Guide stays on Embabel agent **0.3.5-SNAPSHOT**. Pin
+`embabel-agent-rag-neo-drivine` to a pre-`EmbeddingAware` timestamp
+(`0.1.2-20260224.010659-19`); newer `0.1.2-SNAPSHOT` jars require agent 0.4.0.
+
+## Persist (write)
+
+```bash
+# Project entities from orchestrator (or retrieval-fixture) root
+curl -s -X POST http://localhost:21337/api/v1/data/spdd-projection/load \
+  -H 'Content-Type: application/json' \
+  -d '{"rootPath":"/home/ubuntu/github/jmjava/sdlc-spdd-orchestrator"}' | jq .
+```
+
+Idempotent: entity `id` values are stable; re-load merges via `save` / `mergeRelationship`.
+Malformed source files are skipped and counted in the response's `skippedFiles`; canvases
+are processed in sorted order.
+
+Sources:
+
+| Path under root | Entities | Relationships |
+|-----------------|----------|---------------|
+| `spdd/canvas/*.md` | WorkId, Canvas | WorkId —`canvas`→ Canvas |
+| `agent-context/memory/context-index.md` | Area, Decision, Pitfall, Pattern | WorkId —`area`→ Area; WorkId —`decision`/`pitfall`/`pattern`→ lesson; lesson —`about`→ Area |
+
+The `about` edges make lessons queryable by code area **across Work IDs** (cross-run
+lessons-learned lookup).
+
+## Retrieve (read)
+
+```bash
+# Counts by label
+curl -s http://localhost:21337/api/v1/data/spdd-projection/stats | jq .
+
+# WorkId subgraph (typed edges — not cosine)
+curl -s http://localhost:21337/api/v1/data/spdd-projection/work/SPIKE-001-guide-rag-context-backend | jq .
+```
+
+| Endpoint | Role |
+|----------|------|
+| `GET /api/v1/data/spdd-projection/stats` | Label counts (`WorkId`, `Canvas`, `Area`, `Decision`, `Pitfall`, `Pattern`) |
+| `GET /api/v1/data/spdd-projection/work/{workId}` | Domain subgraph via `findRelated` (canvas, area, decision, pitfall, pattern) |
+| `GET /api/v1/data/spdd-projection/area?name={area}` | Cross-run lessons for a code area via incoming `about`/`area` edges |
+
+Status mapping: validation failures (bad root, blank workId/area, unknown label) → 400
+with `{"error": …}`; unknown workId/area → 404; feature disabled → 409.
+
+**MCP (leg 3):** when `guide.spdd-projection.enabled=true`, Guide SSE also exports:
+
+| Tool | Role |
+|------|------|
+| `spdd_workSubgraph` | Same as `GET …/work/{workId}` |
+| `spdd_projectionStats` | Same as `GET …/stats` |
+| `spdd_findByLabel` | List `__Entity__` nodes by label (schema labels only; capped at 200) |
+| `spdd_areaLessons` | Same as `GET …/area?name=…` — prior lessons before touching an area |
+
+Tool failures return `{"error": …}` JSON instead of protocol errors.
+
+Implemented in `SpddDomainTools` (`@LlmTool`) + `McpToolExport` in `SpddProjectionConfiguration`.
+Complements `docs_*` chunk tools. After adding tools, reload the Cursor `embabel-dev` MCP
+server so the client refreshes its tool list.
+
+**Chunk join:** store-level `findChunksForEntity` can link entity → RAG chunks; not yet on
+this controller.
+
+## Typical flow (both legs)
+
+1. **Leg 2** — `./scripts/append-ingest.sh` (menke-5 profile) → RAG chunks  
+2. **Leg 3** — `POST /api/v1/data/spdd-projection/load` → WorkId, Canvas, Area, …  
+3. **Verify** — stats + `GET …/work/{workId}`
+
+Re-run leg 3 after markdown index/canvas changes. Leg 2 git incremental handles chunk
+updates separately.
+
+## Implementation package
+
+`com.embabel.guide.spdd`:
+
+- `domain/SpddDomain.kt` — first-class `NamedEntity` types (`WorkId`, `Canvas`, …) with `@Semantics`
+- `SpddEntityDictionary` — `DataDictionary.fromClasses("sdlc-spdd", …)` (Embabel-standard; not `DynamicType`)
+- `SpddMarkdownProjectionService` — parse + persist + `subgraphForWorkId`
+- `SpddProjectionController` — operator HTTP
+- `SpddDomainTools` — MCP `spdd_*` retrieve tools (`@LlmTool`)
+- `SpddProjectionConfiguration` — `DrivineNamedEntityDataRepository` + MCP export beans
+
+## Branch
+
+Developed on `cursor/spike-spdd-dice-projection-17f4` (pair with orchestrator
+`cursor/spike-guide-ingest-agent-context-17f4`).
diff --git a/pom.xml b/pom.xml
@@ -35,6 +35,10 @@
             <version>${embabel-agent.version}</version>
         </dependency>
 
+        <!--
+          Pin timestamp: newer 0.1.2-SNAPSHOT (20260428+) extends EmbeddingAwareChunkingContentElementRepository
+          and requires embabel-agent 0.4.0. Guide stays on 0.3.5-SNAPSHOT for this spike.
+        -->
         <dependency>
             <groupId>com.embabel.agent</groupId>
             <artifactId>embabel-agent-rag-graph</artifactId>
@@ -233,6 +237,33 @@
                 </executions>
             </plugin>
 
+            <!-- Fail the build if KSP did not emit PersonaViewQueryDsl (prevents ClassNotFoundException at ready) -->
+            <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
+                <artifactId>maven-enforcer-plugin</artifactId>
+                <version>3.5.0</version>
+                <executions>
+                    <execution>
+                        <id>require-ksp-persona-dsl</id>
+                        <phase>process-sources</phase>
+                        <goals>
+                            <goal>enforce</goal>
+                        </goals>
+                        <configuration>
+                            <rules>
+                                <requireFilesExist>
+                                    <files>
+                                        <file>${project.basedir}/codegen-gradle/build/generated/ksp/main/kotlin/com/embabel/guide/domain/PersonaViewQueryDsl.kt</file>
+                                        <file>${project.basedir}/codegen-gradle/build/generated/ksp/main/kotlin/com/embabel/guide/domain/GuideUserQueryDsl.kt</file>
+                                    </files>
+                                    <message>Drivine KSP DSL missing. From repo root: cd codegen-gradle &amp;&amp; ./gradlew kspKotlin — then rebuild. Do not run concurrent ./mvnw spring-boot:run builds.</message>
+                                </requireFilesExist>
+                            </rules>
+                        </configuration>
+                    </execution>
+                </executions>
+            </plugin>
+
             <!-- Include Gradle-generated sources in Maven build -->
             <plugin>
                 <groupId>org.codehaus.mojo</groupId>

diff --git a/scripts/README.md b/scripts/README.md
@@ -4,9 +4,12 @@
 |---|---|
 | `fresh-ingest.sh` | Wipes Neo4j RAG data and re-ingests everything from scratch. Use for first-time setup or when you want a clean slate. |
 | `append-ingest.sh` | Re-ingests without clearing existing data. Use when you've added new URLs or directories. Comment out already-ingested items in your profile to avoid re-processing them. |
+| `run-mcp-guide-against-hub.sh` | Run Guide on **`$GUIDE_PORT`** (default **1337**) for MCP at **`/sse`**, using the **same external Bolt defaults** as **`USE_EMBABEL_HUB_NEO4J=1`** in `append-ingest.sh` (self-contained script; no extra `lib/`). |
 | `shell.sh` | Runs the application in interactive shell mode. |
 
-Both ingestion scripts start Neo4j in Docker, load your personal profile, and print an **INGESTION COMPLETE** banner when done.
+Both ingestion scripts load your personal profile and run Guide with reload-on-startup. Watch application logs for ingestion progress. If **`ANTHROPIC_API_KEY`** is not set, `append-ingest.sh` exports a **`dummy-key`** placeholder so Spring starts (Anthropic autoconfigure requires the variable; ingestion embeddings use local ONNX). Put a real key in `.env` when you use Claude.
+
+By default they start **guide** Compose Neo4j (`embabel-neo4j`, Bolt `localhost:7687`). For **any other** Bolt endpoint (different host/port/password), set **`SKIP_COMPOSE_NEO4J=1`** and configure **`NEO4J_URI`**, **`NEO4J_PASSWORD`**, **`NEO4J_PORT`**, etc. The **`append-ingest.sh`** header documents a **`USE_EMBABEL_HUB_NEO4J=1`** shortcut for one common Docker layout; adjust env vars if yours differs. Do **not** point **`fresh-ingest.sh`** at a shared production-style graph (it deletes `ContentElement` nodes).
 
 ## Personal profiles
 
@@ -39,6 +42,30 @@ guide:
 
 Then run `./scripts/append-ingest.sh`. The new content is added alongside existing data in Neo4j.
 
+## Operator API (shared Neo4j)
+
+When Guide is running, these **`/api/v1/data`** endpoints are open for local/dev use (same security posture as **`load-references`** — do not expose Guide to untrusted networks without a proxy).
+
+| Method | Path | Purpose |
+|--------|------|--------|
+| `POST` | `/api/v1/data/content-elements/purge-preview` | JSON body: `{ "uriPrefix": "..." }` **or** `{ "directory": "~/path/to/repo" }` (not both). Returns match count + sample `uri`s. Prefix must be ≥ 8 characters. |
+| `POST` | `/api/v1/data/content-elements/purge` | Same prefix fields plus `"confirm": true` — **deletes** matching `ContentElement` nodes (`uri STARTS WITH` resolved prefix). |
+| `POST` | `/api/v1/data/git-ingestion/revision/reset` | JSON `{ "directory": "~/path" }` — removes that repo’s entry from the git-ingestion revision file (requires `guide.git-ingestion.enabled`). Next ingest does a full tree for that directory. |
+
+**Directory → `file:` URI:** `directory` is resolved like `guide.directories` (`~` expanded). Chunks use that `file:` prefix in Neo4j.
+
+**Examples:**
+
+```bash
+curl -s -X POST http://localhost:1337/api/v1/data/content-elements/purge-preview \
+  -H 'Content-Type: application/json' \
+  -d '{"directory":"~/github/jmjava/dice"}' | jq .
+
+curl -s -X POST http://localhost:1337/api/v1/data/git-ingestion/revision/reset \
+  -H 'Content-Type: application/json' \
+  -d '{"directory":"~/github/jmjava/dice"}' | jq .
+```
+
 ## Tips
 
 - **If ingestion seems stuck** on a URL: the thread is blocked on fetch -> parse -> embed. Try lowering `embedding-batch-size` to 20, or temporarily remove the slow URL.