Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 22 additions & 2 deletions okf/SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -630,10 +630,29 @@ saga is the cautionary tale). The rules are frozen:
actual size, before any op for that file is appended; sibling files in
the same run are unaffected. Apply's `oversize-op` reject stays
untouched — validate at the boundary, defend in depth.
- **Path-scoped one-shot ingest (ADR-001).** The one-shot scan set is the
whole bundle, restricted to the argument set when trailing
bundle-relative paths are given (concurrent-writer politeness: one
agent's ingest never sweeps up another's mid-write pending file). A
named path that does not exist, or that resolves outside the bundle
root, is an error — exit 1, path named, zero ops appended. The
zero-path form is byte-identical in behavior to the full scan.
- Debounce, write-settle, in-flight-lock coordination with materialize.
- `index.md`/`log.md` are reserved: edits to them are ignored (regenerated
faces), with a single narrated log entry.

**Per-file state vocabulary (ADR-002)** — the states implicit in the
materialize/ingest rules above are named contract, reported read-only by
the CLI `status` command (appends no ops, writes no files, touches no
manifest): `in-log` (on-disk bytes match the log's canonical projection —
hash match), `pending-ingest` (hand-authored, not yet in the log; ingest
would establish a concept), `dirty` (in the log but on-disk bytes diverge
from the projection — a hand edit ingest folds in as diffs, or a log that
moved past the last materialize), `absent` (in the log, missing from disk;
materialize would restore), `foreign` (not in the log, not
manifest-tracked, not establishable by ingest — e.g. another writer's
mid-write WIP).

## 10. Compaction: generation cycling (the CouchDB homage, part two)

Byte reclamation is an **operator act via the hydrated face**, never an op:
Expand Down Expand Up @@ -751,8 +770,9 @@ okf/store vault lifecycle on autobee, metadata pinning (incl. the
swarm replication, backpressure option
okf/faces materialize (bundle + index.md + log.md), ingest watcher,
wanted index, generation-cycle compact command
okf/cli init, join, add-writer, remove-writer, materialize, ingest,
watch, wanted, flags, resolve, undelete, compact
okf/cli init, join, add-writer, remove-writer, materialize, ingest
(optionally path-scoped, ADR-001), watch, status (ADR-002),
wanted, flags, resolve, undelete, compact
okf/root the `auto-okf` root placeholder package (published as
auto-okf@0.0.1 alongside the scope — intent D7)
okf/test unit + invariant suite (I1..I11 each ≥ 1 test)
Expand Down
30 changes: 27 additions & 3 deletions okf/cli/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,8 +68,9 @@ pnpm add @auto-okf/cli # or run the repo-local bin: okf/cli/bin/auto-okf.js
| `add-writer <dir> <writer-key> [--indexer]` | append an `AddWriter` governance op; reports whether it took effect (apply rejects issuers that are not confirmed indexers, §8) |
| `remove-writer <dir> <writer-key>` | append a `RemoveWriter` op; revocation is causal, never positional (§8) |
| `materialize <dir> <bundle-dir>` | project the OKF v0.1 bundle (concepts + per-dir `index.md` + root `log.md`); prints written/unchanged/removed counts; pending-ingest and stale files are `notice:` lines on stderr, unsafe paths `error:` lines. Exits 3 when everything owned was projected but human-authored file(s) are pending ingest (N3); an unsafe path is a real failure, exit 1. `--force-mirror` restores unconditional mirroring |
| `ingest <dir> <bundle-dir>` | one-shot: fold on-disk edits under `<bundle-dir>` back into the log as minimal ops (§9); `--confirm-deletes` lets a file deleted on disk emit `DeleteConcept` (off by default) |
| `ingest <dir> <bundle-dir> [paths…]` | one-shot: fold on-disk edits under `<bundle-dir>` back into the log as minimal ops (§9). Trailing bundle-relative paths restrict the scan to exactly those concept files (ADR-001 — concurrent-writer politeness; the shell expands globs): a named path that does not exist, escapes the bundle root, or is not a `.md` file exits 1 naming the path, with zero ops appended. Zero paths = full scan, unchanged. `--confirm-deletes` lets a file deleted on disk emit `DeleteConcept` (off by default) |
| `watch <dir> <bundle-dir> [--swarm]` | foreground ingest watcher: human edits re-enter as ops as files change; `--swarm` also joins the vault's hyperswarm topic for replication; `--confirm-deletes`, `--debounce <ms>`. No daemon — stop with Ctrl-C |
| `status <dir> <bundle-dir>` | read-only per-file state porcelain (ADR-002): one `<state>\t<path>` line per file, sorted by path, states `in-log`, `pending-ingest`, `dirty`, `absent`, `foreign`; `--json` for the stable `{ path, state }` rows. Appends no ops, writes no files, touches no manifest |
| `wanted <dir>` | print the ranked demand index (`wanted/`): `<count>\t<path>\t<- <referrers>`, most-referenced first; `--json` for the structured rows |
| `flags <dir> [family]` | list/inspect the flag queue (`flag/`), optionally one family (`conflict`, `clobber`, `pathcollision`, `staletag`, `writeafterdelete`, `undelete`, `rejected`, `squat`, `copydetect`, …); `--json`. Nothing is ever lost silently (I4) |
| `compact <dir> <out-dir> [--force]` | generation-cycle compaction (§10): materialize the live bundle, init a fresh vault at `<out-dir>`, and ingest with `_id` adoption. Live state and identity survive; history is shed. Refuses while any `flag/conflict`/`flag/pathcollision` is open unless `--force`. The hydrated face is written to `<out-dir>.face` (override with `--face <dir>`). The predecessor is never deleted |
Expand All @@ -79,9 +80,31 @@ pnpm add @auto-okf/cli # or run the repo-local bin: okf/cli/bin/auto-okf.js
There is no content-write command: content flows in through `ingest`/`watch`
(edit the markdown) or through the `@auto-okf/store` API (agents).

## Per-file state: `status`

`auto-okf status ./vault ./bundle` answers the per-file state question
*before* you run anything — one `<state>\t<path>` line per file, sorted by
path, strictly read-only (it appends no ops, writes no files, touches no
manifest; ADR-002). Five states:

- `in-log` — on-disk bytes match the log's canonical projection (hash match)
- `pending-ingest` — hand-authored, not yet in the log; `ingest` would fold
it in as a new concept
- `dirty` — in the log but on-disk bytes diverge from the projection (a hand
edit `ingest` folds in as diffs, or a log that moved past the last
`materialize`)
- `absent` — in the log, missing from disk (`materialize` would restore it)
- `foreign` — not in the log, not manifest-tracked, and not (yet) ingestable
(e.g. another writer's mid-write WIP without a parseable `type`)

`--json` emits the same rows as a stable array of `{ path, state }` objects —
byte-identical across runs with no intervening ops (the schema is contract).

## Verifying content is in the log

A vault has no privileged bundle directory; the emission manifest
`status` is the quick check: `in-log` means the on-disk bytes hash-match the
log's projection. For proof *from* the log, materialize to a scratch
directory — a vault has no privileged bundle directory; the emission manifest
(`.okf-manifest.json`) lives in each target directory, so materializing to
a scratch directory never disturbs another bundle:

Expand Down Expand Up @@ -127,7 +150,8 @@ per command) and exits 0.
From the repo root: `pnpm test` (or `pnpm test:okf`). The CLI smoke suite
(`okf/test/cli.test.js`) spawns the real `auto-okf` binary against real
vaults on disk: init→materialize produces a bundle; a hand-authored file
ingests to the expected ops; add-writer/remove-writer round trip; `wanted`
ingests to the expected ops; path-scoped ingest folds only the named files;
`status` reports the five per-file states read-only; add-writer/remove-writer round trip; `wanted`
lists an unresolved link; `flags` lists a seeded conflict; `compact` refuses
on an open conflict and succeeds after `resolve`; `undelete` restores a
tombstone; `watch` ingests a foreground edit and exits cleanly on SIGINT.
Expand Down
92 changes: 82 additions & 10 deletions okf/cli/lib/run.js
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
// auto-okf CLI (okf SPEC §13): a thin veneer over @auto-okf/store (vault
// lifecycle + §8 governance ops) and @auto-okf/faces (materialize, ingest
// watcher, wanted index, generation-cycle compact). The CLI never writes the view
// or the face itself — every effect flows through the packages, so everything
// it does is exactly what a library caller does.
// watcher, per-file status, wanted index, generation-cycle compact). The
// CLI never writes the view or the face itself — every effect flows through
// the packages, so everything it does is exactly what a library caller does.
//
// The command surface is declared once as a paparam command tree (paparam is
// Holepunch's own parser — pear's CLI runs on it): a root command composing
// the 12 subcommands from command()/flag()/arg(), with --help|-h and all help
// the 13 subcommands from command()/flag()/arg(), with --help|-h and all help
// text generated from the declarations.
//
// Exit codes: 0 success, 1 operational error, 2 usage error, 3 materialize
Expand All @@ -16,13 +16,13 @@
// flag/command, extra arguments, bad flag values, and (via per-command
// validate()) missing required arguments — to 2, never letting paparam pick
// the code. Unknown flags are rejected strictly, as before.
import { readFile, access } from 'node:fs/promises'
import { join } from 'node:path'
import { readFile, access, stat } from 'node:fs/promises'
import { join, relative, resolve, sep } from 'node:path'
import b4a from 'b4a'
import { command, flag, arg, summary, header, footer, bail, validate } from 'paparam'
import { command, flag, arg, rest, summary, header, footer, bail, validate } from 'paparam'
import { keyspace as K } from '@auto-okf/core'
import { Bundle, METADATA_FILE } from '@auto-okf/store'
import { materialize, createIngestWatcher, wanted, compact } from '@auto-okf/faces'
import { materialize, createIngestWatcher, wanted, compact, status } from '@auto-okf/faces'

const TITLE = 'auto-okf — multi-writer OKF bundles on autobee (okf SPEC §13)'
const FOOTER = 'exit codes: 0 ok, 1 error, 2 usage, 3 pending-ingest only (materialize)'
Expand Down Expand Up @@ -159,6 +159,7 @@ function buildCli() {
'fold on-disk edits back in as ops (one-shot)',
arg('<dir>', 'vault storage directory'),
arg('<bundle-dir>', 'OKF bundle directory'),
rest('[...paths]', 'bundle-relative concept files restricting the scan set (ADR-001)'),
flag('--confirm-deletes', 'let a file deleted on disk emit DeleteConcept'),
requiredArgs('<dir>', '<bundle-dir>')
),
Expand All @@ -172,6 +173,14 @@ function buildCli() {
flag('--debounce <ms>', 'ingest debounce in milliseconds'),
requiredArgs('<dir>', '<bundle-dir>')
),
sub(
'status',
'per-file vault/bundle state, read-only (in-log, pending-ingest, dirty, absent, foreign)',
arg('<dir>', 'vault storage directory'),
arg('<bundle-dir>', 'OKF bundle directory'),
flag('--json', 'print the structured rows'),
requiredArgs('<dir>', '<bundle-dir>')
),
sub(
'wanted',
'print the ranked demand index (wanted/)',
Expand Down Expand Up @@ -290,6 +299,7 @@ const HANDLERS = {
materialize: cmdMaterialize,
ingest: cmdIngest,
watch: cmdWatch,
status: cmdStatus,
wanted: cmdWanted,
flags: cmdFlags,
compact: cmdCompact,
Expand Down Expand Up @@ -471,8 +481,52 @@ async function cmdMaterialize({ args, flags }) {
})
}

async function cmdIngest({ args, flags }) {
/**
* ADR-001: resolve the trailing path arguments to bundle-relative concept
* files, validating BEFORE any op is appended. Literal paths only — the
* shell expands globs; a path that does not exist, resolves outside the
* bundle root, or is not a concept `.md` file is an error naming it.
* Returns null for the zero-path form (full scan, unchanged).
*/
async function scopedIngestPaths(bundleDir, given) {
if (given.length === 0) return null
const root = resolve(bundleDir)
const rels = new Set()
for (const p of given) {
const abs = resolve(root, p)
if (abs !== root && !abs.startsWith(root + sep)) {
throw new Error(`path resolves outside the bundle root: ${p}`)
}
let st = null
try {
st = await stat(abs)
} catch (err) {
if (err.code !== 'ENOENT') throw err
}
if (st === null) throw new Error(`no such file in the bundle: ${p}`)
if (!st.isFile() || !abs.endsWith('.md')) throw new Error(`not a concept file (.md): ${p}`)
rels.add(relative(root, abs).split(sep).join('/'))
}
return [...rels].sort()
}

async function cmdIngest(parsed) {
const { args, flags } = parsed
const { dir, bundleDir } = args
// paparam's rest is greedy: once both positionals are filled, everything
// remaining — flags included — lands in rest. Re-recognize ingest's own
// flags so the pre-ADR-001 `ingest <dir> <bundle> --confirm-deletes` form
// keeps working; the remainder is the ADR-001 path set.
const trailing = []
for (const t of parsed.rest || []) {
if (t === '--confirm-deletes') flags.confirmDeletes = true
else if (t === '--help' || t === '-h') {
process.stdout.write(parsed.help())
return 0
} else if (t.startsWith('-')) throw new UsageError(`unknown flag ${t}`)
else trailing.push(t)
}
const paths = await scopedIngestPaths(bundleDir, trailing)
return withVault(dir, {}, async (vault) => {
const watcher = await createIngestWatcher(vault, bundleDir, {
watch: false,
Expand Down Expand Up @@ -505,7 +559,8 @@ async function cmdIngest({ args, flags }) {
process.stderr.write(`delete skipped (no --confirm-deletes): ${path}\n`)
)
try {
await watcher.scan()
if (paths === null) await watcher.scan()
else for (const rel of paths) await watcher.ingestFile(rel)
await vault.update()
// The skip clause is omitted (never "0") when no skips occurred.
const skips = upToDate > 0 ? `; ${upToDate} file(s) already up to date` : ''
Expand Down Expand Up @@ -564,6 +619,23 @@ async function cmdWatch({ args, flags }) {
}
}

async function cmdStatus({ args, flags }) {
const { dir, bundleDir } = args
return withVault(dir, {}, async (vault) => {
// ADR-002: strictly read-only — status appends no ops, writes no files,
// touches no manifest. The five-state vocabulary and the --json
// [{ path, state }] rows are contract.
const rows = await status(vault, bundleDir)
if (flags.json) {
console.log(JSON.stringify(rows, null, 2))
} else {
for (const { path, state } of rows) console.log(`${state}\t${path}`)
process.stderr.write(`${rows.length} file(s)\n`)
}
return 0
})
}

async function cmdWanted({ args, flags }) {
return withVault(args.dir, {}, async (vault) => {
await vault.update()
Expand Down
9 changes: 7 additions & 2 deletions okf/faces/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@ multi-writer op log and a plain
bundle on disk. `materialize` projects a converged view into a
byte-deterministic directory of markdown files; the ingest watcher folds
human edits back into the log as *minimal* ops; `wanted` reports what the
graph is asking to have written. The log stays the source of truth — a face
graph is asking to have written; `status` reports each file's state against
the log, read-only (ADR-002). The log stays the source of truth — a face
can always be regenerated, byte-for-byte, and human edits always re-enter as
ops (SPEC §2, §9).

Expand Down Expand Up @@ -100,7 +101,11 @@ mirroring for operator use.

### Verifying content is in the log

A vault has no privileged bundle directory; the emission manifest
`status(vault, dir)` is the quick check: one `{ path, state }` row per file,
with `state` one of `in-log`, `pending-ingest`, `dirty`, `absent`, `foreign`
(ADR-002) — strictly read-only (no ops, no writes, no manifest touch). For
proof *from* the log, use a scratch materialize: a vault has no privileged
bundle directory; the emission manifest
(`.okf-manifest.json`) lives in each target directory, so materializing to
a scratch directory never disturbs another bundle. That makes verification
trivial: materialize to an empty scratch directory — an empty dir that
Expand Down
3 changes: 3 additions & 0 deletions okf/faces/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@
// ops (per-key frontmatter diff, body diff citing _rev, the §6 _id
// rules, conflict-copy awareness, guarded deletes).
// - wanted(vault): the ranked demand index from wanted/ + wantedrank/.
// - status(vault, dir): read-only per-file bundle state porcelain
// (in-log / pending-ingest / dirty / absent / foreign, ADR-002).
//
// The bundle format is normative to the OKF v0.1 SPEC; the canonical YAML
// emitter is hand-rolled for byte-determinism (I9). Nothing here imports
Expand All @@ -16,6 +18,7 @@ export { materialize, renderIndex, renderLog } from './lib/materialize.js'
export { createIngestWatcher, IngestWatcher } from './lib/ingest.js'
export { compact } from './lib/compact.js'
export { wanted } from './lib/wanted.js'
export { status } from './lib/status.js'
export { readView, readLog, memViewReader } from './lib/view.js'
export {
renderNote,
Expand Down
28 changes: 21 additions & 7 deletions okf/faces/lib/materialize.js
Original file line number Diff line number Diff line change
Expand Up @@ -105,15 +105,14 @@ function conflictHash(id) {
}

/**
* Walk the view and emit the bundle. `opts.forceMirror` restores
* unconditional mirroring (overwrite/delete regardless of manifest).
* Returns a report:
* { root, ok, written, unchanged, pendingIngest, removed, stale, unsafe }
* The full byte plan for a converged view (materialize steps 1–4: concept
* files, per-directory index.md, root log.md) WITHOUT touching disk.
* Returns Map<rel, Buffer>. Shared with status() (ADR-002) so the
* porcelain's states are judged against the exact bytes materialize would
* write.
*/
export async function materialize(vault, dir, opts = {}) {
export async function planBundle(vault) {
if (typeof vault.update === 'function') await vault.update()
const { realRoot, registry } = await rootRegistry(dir)
const forceMirror = opts.forceMirror === true

const { concepts, collisions } = await readView(vault)

Expand Down Expand Up @@ -187,6 +186,21 @@ export async function materialize(vault, dir, opts = {}) {
// ---- 4. root log.md (OKF §7) ----
planned.set('log.md', Buffer.from(await renderLog(vault, concepts), 'utf8'))

return planned
}

/**
* Walk the view and emit the bundle. `opts.forceMirror` restores
* unconditional mirroring (overwrite/delete regardless of manifest).
* Returns a report:
* { root, ok, written, unchanged, pendingIngest, removed, stale, unsafe }
*/
export async function materialize(vault, dir, opts = {}) {
const { realRoot, registry } = await rootRegistry(dir)
const forceMirror = opts.forceMirror === true

const planned = await planBundle(vault)

// ---- 5. reconcile to disk through the manifest (§9) ----
const report = {
root: realRoot,
Expand Down
Loading