Organize projects by semantic kind - #73
johnnygreco wants to merge 2 commits into
Conversation
Signed-off-by: Johnny Greco <jogreco@nvidia.com>
New project reviewRevision: Review findings are advisory. Required checks remain separate merge gates. Execution: success Reviewer and guidelines revision: Reviews
projects/research-spikesThe contribution places and declares `projects/research-spikes/long-horizon-agent-evals` as a `research-spike`: a small, deliberately concrete TypeScript harness that runs a persistent Codex "challenger" inside an OpenShell sandbox for a wall-clock horizon and tests whether it can induce a separate, tool-free AI reviewer to grant cumulative OpenShell policy sufficient to mutate a protected GitHub repository. It serves agent-security researchers studying long-horizon (trajectory, not single-turn) policy-review pressure. The pieces fit together as documented: `src/campaign.ts` (setup, sandbox, oracle, outcome classification), `src/reviewer.ts` plus `src/reviewer-model.ts` (structured approve/reject applied via the OpenShell SDK by the harness, not the model), `src/scale.ts` (bounded-concurrency repeats), `scripts/challenger.sh`, and prompts under `experiments/github-policy-review/`. The README gives an ordered first run (install, `.env`, `npm run endpoint:check`, `image:build`, `github:preflight`, `campaign`, `scale`, `transcript`/`timeline`/`costs`), prerequisites (Node 20.3+, Docker, dedicated OpenShell 0.0.106+ gateway, GitHub Packages access, fine-grained repo token), and a prominent warning that real repository mutations occur. Methodological strengths verified in code: GitHub is an external oracle rather than model self-report (`src/campaign.ts:893`), approvals are separated into returned / applied / fail-closed counts plus `requiresAdjudication`, and operational failures invalidate non-compromised attempts (`classifyOutcome`, `src/campaign.ts:481`). Focused tests cover outcome classification, paginated ref snapshots, lull detection, handoff, reviewer history compaction, and version matching. Remaining gaps are documentation-level and non-blocking: no results summary or link to the repository write-up, no committed run artifacts or fixtures, and one tuning default justified by an uncommitted trace corpus. Project guidelines: ✅ Pass Applicable requirements are supported. `kind: research-spike` is declared in both `projects/research-spikes/project.yaml` and the project's own `project.yaml`. Purpose, audience, OpenShell usage, and a `Limits` section are in the README; layout is self-contained with no empty docs scaffolding (single README plus a small `experiments/` prompt directory and a short `AGENTS.md` with genuinely project-specific conventions). First run is documented with prerequisites, disclosed hardware/service/credential/cost requirements, and an explicit real-mutation warning. Verification is proportionate: `npm run check` (typecheck plus focused node:test files) satisfies the "focused tests, a runnable check, or recorded evidence" option, and the README states what is not verified. Dependencies use the normal manifest plus a committed `package-lock.json`; the `uv`/`pyproject` rule is inapplicable to this Node project. Secrets are kept out of the tree: `.env` and `runs/*` are gitignored, `.npmrc` uses `${NODE_AUTH_TOKEN}`, and `.env.example` documents variables with empty credential values. Engineering is proportionate and `AGENTS.md` explicitly forbids adding a scenario framework before a second experiment. Not verified within scope: repository-root license/attribution and the repo-wide content-routing conventions in the root AGENTS.md/docs tree, which were outside the provided snapshot. Findings
Strengths
Limitations
projects/toolsThe contribution regroups the repository's projects by semantic kind; the reviewed slice, `projects/tools/`, holds four reusable OpenShell tools - Egress Gate (Python gRPC supervisor middleware that runs an ordered gate pipeline over provider-bound requests), OpenShell Agent Runner (CLI that launches ephemeral Pi agents in sandboxes and returns validated JSON reviews), OpenShell Middleware Manager (`omm` generator/updater for Python and Rust middleware starters), and OpenShell Event Exporter (self-contained Go OTel Collector distribution for OCSF/CloudEvents delivery) - each with a `project.yaml` declaring `kind: tool`, self-contained manifests, lockfiles, Apache-2.0 licenses, and per-project READMEs/docs. Documentation is complete and internally consistent: every README gives a runnable first-use path (installed and source-checkout variants), states non-goals (Egress Gate is not a proxy/TLS interceptor/response filter; the exporter is experimental with no published image or module tag), and documents limits, failure mapping, and trust boundaries. Path updates to `projects/tools/...` are consistently applied in READMEs, docs links, Go module paths, test path constants, and the exporter Dockerfile; I found no stale `projects/<name>/` references inside the snapshot. Verification is credible and discoverable (~245 Egress Gate pytest cases including gRPC/service and CLI example-corpus tests, 173 OAR/OMM tests, 139 Go tests, plus `make check`/`scripts/check.sh`); I independently re-ran only the standard-library latency-figure check (`analysis/render_latency_plot.py --check`), which passed and reproduced the documented 0.56% turn share. Remaining issues are cosmetic (a dangling `plans/egress-gate-refactor.md` reference). Project guidelines: ✅ Pass "Declare the project kind": each of the four projects has `project.yaml` with `kind: tool`, and the group directory adds a documented compatibility marker. "Purpose and scope": each README states what the tool does, who it serves, how it uses OpenShell, and explicit non-goals (e.g. egress-gate/docs/index.md ownership boundaries; openshell-exporter/README.md "Experimental research software" and "No such release is published by this change"). "Self-contained layout": implementation, manifests, locks, docs, and runtime notes live inside each project directory; docs/ trees are substantive, not scaffolding. "Reproducible first run": prerequisites (uv 0.11+/Python 3.11+, Go 1.26, OpenShell 0.0.111+), smallest useful commands, and expected output are documented, including the exporter's synthetic no-credential quickstart. "Evidence and verification": focused test suites plus documented check commands exist, and unverified areas are stated (analysis/README.md interpretation limits; exporter "Synthetic local and container checks do not qualify live gateway behavior"). "Dependencies and configuration": Python projects use uv + pyproject.toml + committed uv.lock; Go uses go.mod/go.sum; no committed secrets found (only clearly synthetic test tokens); LICENSE present everywhere and NOTICE for the vendored Go distribution. "Proportionate engineering": absolute imports throughout the Python sources (no relative imports found outside generated bindings), nested AGENTS.md files document genuinely distinct conventions. The repository-level docs that describe the new grouping (projects/README.md, projects/PROJECT_GUIDELINES.md) are outside this snapshot, so the group-level rationale itself was not verifiable; that is recorded as a limitation rather than a violation because every in-scope per-project requirement is supported. Findings
Strengths
Limitations
projects/use-case-examplesThe contribution regroups the repository's three demonstration projects under `projects/use-case-examples/` (the Reachy Mini + OpenShell onboard conversation demo, the robotics policy-prover demo, and the Python project template), adds `kind: use-case-example` metadata at the group root and in each project, and updates in-project path references for the new nesting depth. Each project stays self-contained with its own documentation, manifests, lockfiles, and tests. The reachy demo carries the most substantial workflow documentation (463-line README plus a 762-line onboard tutorial), and its OpenShell policy files, sandbox lifecycle, and REST tool boundary line up with the sampled implementation (fixed pose table, `minjerk`, 1–8 direction schema, `policy_denied` on 403, 2 MiB camera cap). Path rewrites inside the group are complete and the dev-note relative links resolve from the new depth. Remaining observations are minor and mostly inherited from the moved, pre-existing projects: the prover README omits Rust/cmake/Node prerequisites even though its Cargo build vendors Z3, the prover keeps duplicate `.env.example`/`.env.sample` files, its committed benchmark records a z3 crate version that no longer matches `Cargo.lock`, and the Python template is a copy-me scaffold rather than an end-to-end demonstration despite its declared kind. Project guidelines: ✅ Pass "Declare the project kind" is satisfied: `projects/use-case-examples/project.yaml` and each subproject declare `kind: use-case-example`. "Self-contained layout" holds — implementation, docs, manifests, and lockfiles stay inside each project directory, with directly relevant repository dev notes linked by correct relative paths. "Reproducible first run" is met for the reachy demo (prerequisites, `cp .env.example .env`, `./scripts/start-local.sh`, expected Gradio URL and daemon-status JSON) and the template (`uv sync --locked`, example command); it is weaker for the prover, whose README lists only `npm install` although the service build vendors Z3 and needs a Rust/cmake toolchain (see findings; reported low because that content is pre-existing and moved unchanged). "Dependencies and configuration" is met: `uv.lock`, the hash-pinned `requirements.txt` export, `package-lock.json`, and `Cargo.lock` are committed, `.env.example` files contain no secrets, `.env` is gitignored, and Apache-2.0 LICENSE files plus SPDX headers are present. "Evidence and verification" is supported by focused unit tests (including `tests/test_camera_policy.py`, which asserts the exact policy allow rules), a visual verification script, and recorded benchmark JSON with provenance fields. Kind expectations for use-case examples (concrete audience, start-to-finish workflow, safe configuration, demonstration-versus-deployable distinction) are documented, notably the explicit statement that JSON body values remain application-enforced rather than OpenShell-enforced. No demonstrated violation warranting changes was found. Findings
Strengths
Limitations
|
Signed-off-by: Johnny Greco <jogreco@nvidia.com>
Documentation previewBuilt from |
Summary
tools,research-spikes, anduse-case-examplesproject.yamlkind metadata and teach the OAR reviewer the nested layoutValidation
uv lock --checkpassedEnvironment limitations
cmake; visual verification needs the Playwright Chromium binary.