Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 12 additions & 4 deletions okf/SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -515,10 +515,13 @@ saga is the cautionary tale). The rules are frozen:
directories are then removed. A `.md` file whose current bytes differ
from its manifest entry — or that has no entry (human-authored, not yet
ingested) — is **never deleted or overwritten**: it is left in place,
reported as pending-ingest (an ingest candidate, surfaced on the CLI),
and the run exits nonzero unless ingest is run first
(`materialize --force-mirror` restores unconditional mirroring for
operator use). Deletion honors the same containment checks as writes.
reported as pending-ingest (an ingest candidate, surfaced on the CLI as
a `notice:`, never an `error:`), and the run exits 3 — pending-only:
everything owned was projected; a clean run exits 0, a real failure
(I/O, corrupt vault, bad args, an unsafe path) exits 1 (ADR-004) —
unless ingest is run first (`materialize --force-mirror` restores
unconditional mirroring for operator use). Stale files (derived faces a
human edited) are likewise a notice accompanying exit 0/3. Deletion honors the same containment checks as writes.
The ingest watcher treats manifest-gated reconciliation deletions as
materialize-authored (in-flight-lock coordination), never as human
deletes. Converged-peer I9 is judged on the emitted set; pending-ingest
Expand Down Expand Up @@ -622,6 +625,11 @@ saga is the cautionary tale). The rules are frozen:
ignored. Strip one trailing `.md`, strip any fragment (`#...`),
NFC-normalize; the result keys `link/`. A `SetPath` on a concept re-runs
resolution of its own relative-form edges.
- **Oversize preflight (ADR-005).** A file whose body would exceed the §3
`maxOpBytes` limit is refused at ingest, by name, with the limit and the
actual size, before any op for that file is appended; sibling files in
the same run are unaffected. Apply's `oversize-op` reject stays
untouched — validate at the boundary, defend in depth.
- Debounce, write-settle, in-flight-lock coordination with materialize.
- `index.md`/`log.md` are reserved: edits to them are ignored (regenerated
faces), with a single narrated log entry.
Expand Down
16 changes: 13 additions & 3 deletions okf/cli/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ pnpm add @auto-okf/cli # or run the repo-local bin: okf/cli/bin/auto-okf.js
| `join <dir> <vault-key>` | join an existing vault by key; `--encryption-key <hex64>` and `--metadata <invite.json>` carry non-default invites; when `<dir>` is already pinned the key is optional (reopen + reprint keys) |
| `add-writer <dir> <writer-key> [--indexer]` | append an `AddWriter` governance op; reports whether it took effect (apply rejects issuers that are not confirmed indexers, §8) |
| `remove-writer <dir> <writer-key>` | append a `RemoveWriter` op; revocation is causal, never positional (§8) |
| `materialize <dir> <bundle-dir>` | project the OKF v0.1 bundle (concepts + per-dir `index.md` + root `log.md`); prints written/unchanged/removed counts; reports pending-ingest, unsafe-path and stale files on stderr; exits non-zero if a human-authored file is pending ingest (N3). `--force-mirror` restores unconditional mirroring |
| `materialize <dir> <bundle-dir>` | project the OKF v0.1 bundle (concepts + per-dir `index.md` + root `log.md`); prints written/unchanged/removed counts; pending-ingest and stale files are `notice:` lines on stderr, unsafe paths `error:` lines. Exits 3 when everything owned was projected but human-authored file(s) are pending ingest (N3); an unsafe path is a real failure, exit 1. `--force-mirror` restores unconditional mirroring |
| `ingest <dir> <bundle-dir>` | one-shot: fold on-disk edits under `<bundle-dir>` back into the log as minimal ops (§9); `--confirm-deletes` lets a file deleted on disk emit `DeleteConcept` (off by default) |
| `watch <dir> <bundle-dir> [--swarm]` | foreground ingest watcher: human edits re-enter as ops as files change; `--swarm` also joins the vault's hyperswarm topic for replication; `--confirm-deletes`, `--debounce <ms>`. No daemon — stop with Ctrl-C |
| `wanted <dir>` | print the ranked demand index (`wanted/`): `<count>\t<path>\t<- <referrers>`, most-referenced first; `--json` for the structured rows |
Expand Down Expand Up @@ -95,10 +95,13 @@ the log — everything that appears came from it.
## Exit codes

`0` success · `1` operational error (including a governance op that apply
rejected, or a `materialize` blocked on a pending-ingest file) · `2` usage
rejected, or an unsafe path skipped during `materialize`) · `2` usage
error — unknown flags and commands, missing or extra arguments, and malformed
values all exit 2 (paparam parses strictly; the wrapper maps every parse
failure to 2). `-h`/`--help` prints the paparam-generated help (top-level and
failure to 2) · `3` (`materialize` only) everything owned was projected but
human-authored file(s) were left in place pending ingest — a normal
multi-writer state, listed as `notice:` lines on stderr, not a failure
(ADR-004). `-h`/`--help` prints the paparam-generated help (top-level and
per command) and exits 0.

## Notes
Expand All @@ -111,6 +114,13 @@ per command) and exits 0.
the reason.
- The encryption key is only needed at `init`/`join`; the store pins the
encrypted bit in vault metadata so a reopen needs no key.
- A single body may not exceed `maxOpBytes` — 65,536 bytes; `ingest` refuses
an oversize file by name (exit 1, siblings unaffected) — split the document
into smaller linked concepts.
- `auto-okf` refuses vault-ld storage by marker file (`vault-ld.json`),
naming the right tool (exit 1, storage untouched); `autovault-ld` refuses
`okf-vault.json` symmetrically — the twin CLIs never open each other's
vaults (ADR-006).

## Tests

Expand Down
77 changes: 67 additions & 10 deletions okf/cli/lib/run.js
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,9 @@
// the 12 subcommands from command()/flag()/arg(), with --help|-h and all help
// text generated from the declarations.
//
// Exit codes: 0 success, 1 operational error, 2 usage error. paparam's
// Exit codes: 0 success, 1 operational error, 2 usage error, 3 materialize
// projected everything owned but left pending-ingest file(s) in place
// (ADR-004). paparam's
// parse() returns null on any parse failure; the wrapper maps that — unknown
// flag/command, extra arguments, bad flag values, and (via per-command
// validate()) missing required arguments — to 2, never letting paparam pick
Expand All @@ -23,10 +25,28 @@ import { Bundle, METADATA_FILE } from '@auto-okf/store'
import { materialize, createIngestWatcher, wanted, compact } from '@auto-okf/faces'

const TITLE = 'auto-okf — multi-writer OKF bundles on autobee (okf SPEC §13)'
const FOOTER = 'exit codes: 0 ok, 1 error, 2 usage'
const FOOTER = 'exit codes: 0 ok, 1 error, 2 usage, 3 pending-ingest only (materialize)'

class UsageError extends Error {}

// ADR-006 wrong-tool guardrail: the sibling system's marker file is stable
// contract between the twin CLIs; auto-okf refuses vault-ld storage
// pointedly (naming the right tool) instead of opening it.
const SIBLING_MARKER = 'vault-ld.json'

class WrongToolError extends Error {}

async function refuseSiblingVault(dir) {
try {
await access(join(dir, SIBLING_MARKER))
} catch {
return
}
throw new WrongToolError(
`${dir} is a vault-ld vault (${SIBLING_MARKER} present); use autovault-ld, not auto-okf`
)
}

/** Run the CLI. Returns the process exit code. */
export async function run(argv) {
const { root, captured } = buildCli()
Expand Down Expand Up @@ -68,6 +88,12 @@ export async function run(argv) {
process.stderr.write(`auto-okf ${parsed.name}: ${err.message}\n\n${parsed.help()}`)
return 2
}
if (err instanceof WrongToolError) {
// ADR-006: the refusal shape is pinned — `error: <dir> is a … vault
// (<marker> present); use <tool>, not <this tool>`.
process.stderr.write(`error: ${err.message}\n`)
return 1
}
process.stderr.write(`auto-okf ${parsed.name}: ${err && err.message ? err.message : err}\n`)
return 1
}
Expand Down Expand Up @@ -290,6 +316,7 @@ async function isPinned(dir) {

/** Open an EXISTING vault (pinned dir) and close it after fn. */
async function withVault(dir, opts, fn) {
await refuseSiblingVault(dir)
if (!(await isPinned(dir))) {
throw new Error(
`'${dir}' is not a vault (no ${METADATA_FILE}); run \`auto-okf init\` or \`auto-okf join\` first`
Expand All @@ -307,6 +334,12 @@ function localHex(vault) {
return b4a.toString(vault.localKey, 'hex')
}

/** `error: drafts/big.md body is 71,204 bytes; maxOpBytes is 65,536` (ADR-005). */
function oversizeLine({ path, bytes, limit }) {
const n = (v) => v.toLocaleString('en-US')
return `error: ${path} body is ${n(bytes)} bytes; maxOpBytes is ${n(limit)}\n`
}

function printIdentity(vault) {
console.log(`vault key: ${vault.keyHex}`)
console.log(`writer key: ${localHex(vault)}`)
Expand All @@ -322,6 +355,7 @@ async function rejectionFor(vault, seq) {

async function cmdInit({ args, flags }) {
const dir = args.dir
await refuseSiblingVault(dir)
if (await isPinned(dir)) {
throw new Error(`'${dir}' is already a vault (${METADATA_FILE} exists)`)
}
Expand All @@ -343,6 +377,7 @@ async function cmdInit({ args, flags }) {

async function cmdJoin({ args, flags }) {
const { dir, vaultKey: key } = args
await refuseSiblingVault(dir)
const opts = {}
if (flags.encryptionKey) {
opts.encryptionKey = hexKey(flags.encryptionKey, '--encryption-key')
Expand Down Expand Up @@ -417,17 +452,22 @@ async function cmdMaterialize({ args, flags }) {
`${report.unchanged.length} unchanged, ${report.removed.length} removed`
)
for (const rel of report.pendingIngest) {
process.stderr.write(`pending ingest (unsaved external edit, left in place): ${rel}\n`)
process.stderr.write(`notice: pending ingest (unsaved external edit, left in place): ${rel}\n`)
}
for (const u of report.unsafe || []) {
const label = typeof u === 'string' ? u : `${u.path || u.id} (${u.reason})`
process.stderr.write(`unsafe path (skipped, §9): ${label}\n`)
const label = typeof u === 'string' ? u : `${u.rel || u.path || u.id} (${u.reason})`
process.stderr.write(`error: unsafe path (skipped, §9): ${label}\n`)
}
for (const rel of report.stale || []) {
process.stderr.write(`stale (no longer produced, left in place): ${rel}\n`)
process.stderr.write(`notice: stale (no longer produced, left in place): ${rel}\n`)
}
// N3: a pending-ingest file makes report.ok false — surface it as an error.
return report.ok ? 0 : 1
// N3 exit taxonomy (ADR-004): an unsafe path means part of the owned
// projection was NOT written — a real failure (1). Pending-ingest files
// are a healthy multi-writer state: everything owned was projected and
// the foreign files were left in place (3). Otherwise clean (0); stale
// is a notice either way (left in place, per spec).
if ((report.unsafe || []).length > 0) return 1
return report.ok ? 0 : 3
})
}

Expand All @@ -440,11 +480,24 @@ async function cmdIngest({ args, flags }) {
})
let files = 0
let ops = 0
let upToDate = 0
watcher.on('ops', ({ path, ops: emitted }) => {
files++
ops += emitted.length
console.log(`ingested ${path}: ${emitted.map((o) => o.type).join(', ') || '(no-op)'}`)
})
// ADR-003: count the hash-match skips so "0 op(s) from 0 file(s)" cannot
// mean both "nothing found" and "everything already converged".
watcher.on('skip', ({ reason }) => {
if (reason === 'unchanged') upToDate++
})
// ADR-005: an oversize body is refused per file, before any of its ops;
// sibling files in the same run still ingest, but the run exits 1.
let oversize = 0
watcher.on('oversize', (o) => {
oversize++
process.stderr.write(oversizeLine(o))
})
watcher.on('warning', ({ path, message }) =>
process.stderr.write(`warning (${path}): ${message}\n`)
)
Expand All @@ -454,8 +507,10 @@ async function cmdIngest({ args, flags }) {
try {
await watcher.scan()
await vault.update()
console.log(`ingest complete: ${ops} op(s) from ${files} file(s)`)
return 0
// The skip clause is omitted (never "0") when no skips occurred.
const skips = upToDate > 0 ? `; ${upToDate} file(s) already up to date` : ''
console.log(`ingest complete: ${ops} op(s) from ${files} file(s)${skips}`)
return oversize > 0 ? 1 : 0
} finally {
await watcher.close()
}
Expand All @@ -464,6 +519,7 @@ async function cmdIngest({ args, flags }) {

async function cmdWatch({ args, flags }) {
const { dir, bundleDir } = args
await refuseSiblingVault(dir)
if (!(await isPinned(dir))) {
throw new Error(
`'${dir}' is not a vault (no ${METADATA_FILE}); run \`auto-okf init\` or \`auto-okf join\` first`
Expand All @@ -480,6 +536,7 @@ async function cmdWatch({ args, flags }) {
watcher.on('ops', ({ path, ops }) => {
console.log(`ingested ${path}: ${ops.map((o) => o.type).join(', ') || '(no-op)'}`)
})
watcher.on('oversize', (o) => process.stderr.write(oversizeLine(o)))
watcher.on('warning', (w) =>
process.stderr.write(`warning: ${w && w.message ? w.message : JSON.stringify(w)}\n`)
)
Expand Down
1 change: 1 addition & 0 deletions okf/core/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ export {
} from './lib/ops.js'
export {
createApply,
DEFAULT_LIMITS,
normalizePath,
casefoldPath,
validatePath,
Expand Down
11 changes: 10 additions & 1 deletion okf/core/lib/apply.js
Original file line number Diff line number Diff line change
Expand Up @@ -41,9 +41,18 @@ import {
} from './ops.js'
import { extractLinks, dirOf } from './links.js'

// §3 default limits, exported so boundary layers (faces ingest, ADR-005)
// can preflight against the same values apply defends with — never a
// duplicated constant.
export const DEFAULT_LIMITS = Object.freeze({
maxOpBytes: 65536,
maxPathBytes: 1024,
maxPathDepth: 64
})

const DEFAULTS = {
// §3 limits: fixed check order, single recorded reason.
limits: { maxOpBytes: 65536, maxPathBytes: 1024, maxPathDepth: 64 },
limits: DEFAULT_LIMITS,
// §8: unacked writers use a fixed default ceiling (64) until admitted;
// H_max = 512 held ops per writer. defaultCeiling is the admitted-writer
// ceiling when no SetWriterPolicy has set one (spec-silent; pinned here
Expand Down
12 changes: 11 additions & 1 deletion okf/faces/lib/ingest.js
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ import { EventEmitter } from 'node:events'
import { watch } from 'node:fs'
import { readFile } from 'node:fs/promises'
import { join, sep } from 'node:path'
import { keyspace as K, canonical } from '@auto-okf/core'
import { keyspace as K, canonical, DEFAULT_LIMITS } from '@auto-okf/core'
import { parseNote, yamlToValues } from './yaml.js'
import {
rootRegistry,
Expand Down Expand Up @@ -251,6 +251,16 @@ export class IngestWatcher extends EventEmitter {
const { fm, body, warnings } = parseNote(raw.toString('utf8'))
for (const wmsg of warnings) this.emit('warning', { path: rel, message: wmsg })

// §9 oversize preflight (ADR-005): a body that would exceed the §3
// maxOpBytes limit is refused HERE, by name, before any op for this
// file is appended — apply's oversize-op reject stays as defense in
// depth. The file stays pending; sibling files are unaffected.
const bodyBytes = Buffer.byteLength(nfc(body), 'utf8')
if (bodyBytes > DEFAULT_LIMITS.maxOpBytes) {
this.emit('oversize', { path: rel, bytes: bodyBytes, limit: DEFAULT_LIMITS.maxOpBytes })
return []
}

const { ops, established } = await this._ingestParsed(rel, fm || {}, body)

if (!established) {
Expand Down
Loading