Skip to content

Host the Hardened Secret and Env Readers in the Hub and Carry Them as a Tree - #3509

Merged
ptr727 merged 17 commits into
developfrom
feature/hub-secret-readers
Oct 26, 2026
Merged

ptr727 merged 17 commits into
developfrom
feature/hub-secret-readers

Conversation

@ptr727

@ptr727 ptr727 commented Oct 26, 2026 •

Copy link
Copy Markdown
Owner

Implements #3505, with the design points the maintainer settled on the issue before the first edit.

What Changed

  • lib/sh/secrets.sh is the hub's runtime library: unresolved_link_reason, secret_state, the shape, content, set, and full verdicts, require_secret, and read_env_value. Each function body is byte-identical to the source implementation, compared with declare -f. Its comments now describe the mechanism rather than the adopter. The installer path helpers, the push interlock, and write_state_file stay in the source repository. The source has no owner check, so the issue's owner-exemption parameter had nothing to parameterize.
  • host-secrets is a new project type. spec/project-types.json gives it detect and one intent check, spec/scope-model.md lists its token, spec/type-model.md gains "Runtime Libraries", and the conformance matrix has its row.
  • spec/files.json carries lib/sh to lib/fleet/sh as a verbatim-tree for appliesTo: ["host-secrets"], so carry.py, the audit, and the canonical sweep pick it up with no code of their own.
  • repo_gate.py carried-trees compares each carried tree a scanned repository tracks with the hub tree beside the gate, the pinned action's in CI. It loads the hub's own scripts/carry.py and reuses its tree functions, including the new target_patterns and compare_declaration that carry.py's own check and post-apply comparison now share. A difference is a note: plus a ::warning annotation and never a finding. A repository tracking spec/files.json is a hub and compares nothing. A tree that cannot be compared, or a carry.py that will not load or run, says so in a note.
  • GOVERNANCE.md: "Hub-Hosted Tooling" states why code a repository's own runtime executes is carried rather than reached. "Repo-Scoped Secrets" states the reader rule the type's check judges, and "Repository Layout" lists lib/.
  • Tests: tests/test_secrets_sh.py drives every verdict and the env read against constructed files. tests/test_repo_gate.py covers the new check and gate_hub_root. tests/test_cli_usage.py lists the library as sourced rather than run.

Verification

  • The full unit suite under tests/requirements.txt passes: 3023 tests, 7 skipped.
  • Mutating the library, by dropping -u BASH_ENV or the unset, reading $2 at the end, dropping /dev/null, reordering the set verdict, or sourcing in process, fails the case written for each fault.
  • spec/validate.py, build_dist.py --check, the prose gate with comments allowed, every repo_gate.py check, and the Docker lint wrapper's seven linters are clean.
  • carried-trees was run against two downstream checkouts. Blog's .github/skills matches the hub. One checkout's copy differs in 37 files and warns as designed. The check also runs under Python 3.12, the runner's system interpreter.

Local Review

Each push's content went through recorded local-strict-review passes at the frontier tier, and each push spent its two-round edit budget. The seven pushes' passes raised 11, 15, and 4 findings, then 6 and 4, then 4, 4, and 3, then 5 and 3, then 2 and 2, then 3 and 1, then 3 and 1, and the eighth push's pass raised none. The seventh push also fixes the Windows unit test job, which failed on an unresolved 8.3 temp path in the new carry.run case. Every introduced finding is answered.

Declined in the sixth push

  • The import-time uv python find probe in tests/host_capability.py follows the file's existing eager can_symlink() probe.

Pre-existing findings, not filed

  • pre-existing: tests/test_repo_gate.py, whose write-verb scan reads scripts/repo_gate.py, a run_path shim, rather than the gate under .github/actions/repo-gate/.

Declined, with evidence

  • The library's own hardening (the positional-parameter guard, grep -D skip, the readarray -d / split): the bodies move byte-identical by the issue's design, so a change to them is its own issue in the authoring repository.
  • The note not naming the hub commit: the pinned action's tree is an archive with no .git, and the run log already names the action's pin.
  • Untracked files in a local hub checkout's source tree: carry.py's own inventory reads the source the same way, so the gate and a carry agree.
  • CRLF working-tree bytes on a Windows checkout: every registry entry declares lineEndings: lf.
  • The spec/files.json hub test catching an unpruned template copy: the note says "is a hub", so the result is visible rather than silent.
  • The issue's point that the same rule gives hub_fetch_run.py a declared home: that loader is not a spec/files.json tree, and the "Hub-Hosted Tooling" paragraph beginning "A loader is outside this section" already governs it.
  • The try around the manifest read: load_json belongs to a carry.py of whatever hub version the pin holds, so the reason for the other broad catches covers it too.
  • Folding the gate test helpers TestCarriedTrees.write and TestGateHubRoot.place into one: a preference between defensible forms, which "Review the Change" says to omit.

Dropped findings

  • The module docstring's table framing, and long sentences in comments and docstrings, which an advisory check reports and no gate enforces.
  • spec/audit.py check_tree_verbatim keeps its own blob-based statement of the tree ownership rule. That file is untouched here, and it reads blobs rather than inventories.

The pull request carries the comments label. The added comments are each function's caller contract and the measured reasons behind each check.

🤖 Generated with Claude Code

ptr727 and others added 3 commits October 26, 2026 12:51
… a Tree

The hub now hosts lib/sh/secrets.sh, the secret and env family that a
fleet repository runs on its own hosts: secret_state, the shape, content,
and set verdicts, require_secret, and read_env_value. A repository that
declares the new host-secrets type carries it byte-for-byte at
lib/fleet/sh through a spec/files.json tree. The hub tests it, since a
consumer's copy is the same bytes.

repo_gate.py gains a carried-trees check. It compares each carried tree
with the hub tree beside the gate, which in CI is the pinned action's,
and reports a difference as a note and an Actions warning. A difference
never fails the gate, so a hub fix does not break every downstream pull
request. GOVERNANCE.md "Hub-Hosted Tooling" states why a runtime library
is carried rather than reached.

Closes #3505

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The carried-trees check now reuses carry.py inventory, compare, and
included, so include and prune read as a carry reads them, and it knows
a hub by its tracked spec/files.json. The Hub-Hosted Tooling paragraph
no longer claims the audit refreshes a copy or that hub_fetch_run.py is
carried by its reasoning. Repo-Scoped Secrets states the reader rule the
host-secrets check judges.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The carried-trees check reads a pruned tree as owning every file under
its target, as a carry and the audit do, and a tree it cannot compare
becomes a note rather than a traceback. The reader rule names the
credential files it binds, under .secrets/ or ~/.secrets/, and which
verdict takes the shared mode. The env read isolation test now calls the
reader in the outer shell.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings October 26, 2026 20:28
@coderabbitai

coderabbitai Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@ptr727 ptr727 added the comments Permits the comment lines the pull request adds or edits, which the prose gate otherwise refuses label Oct 26, 2026
@ptr727

ptr727 commented Oct 26, 2026

Copy link
Copy Markdown
Owner Author

A recorded local strict-review pass covers head 927219b79b5b494c229947f0af1edb9e58132fd9, the content this pull request carries at that commit against develop, and it recorded 4 findings.

carry.py gains target_patterns and compare_declaration, which its own
check and the carried-trees check both call, and its docstring states
what the gate imports under plain python3. A carry.py that will not load
or a manifest that will not read is now a note, the closing count names
only compared trees, and the host-secrets check names which verdict
takes the shared mode, as the governing rule does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.54%. Comparing base (d65fb3b) to head (85b2c4a).

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #3509      +/-   ##
===========================================
+ Coverage    81.11%   81.54%   +0.42%     
===========================================
  Files           26       26              
  Lines         8914     8918       +4     
===========================================
+ Hits          7231     7272      +41     
+ Misses        1683     1646      -37     
Flag Coverage Δ
python-3.13 81.54% <100.00%> (+0.42%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The gate's hub read can stop every other check, the closing note miscounts trees, and the secret reader intent check does not match the governance rule.

5 open issues
What changed in this PR

This PR moves the hardened shell secret and env readers into the hub as lib/sh/secrets.sh. Repositories that run scripts reading credentials from disk can carry that library byte-for-byte through spec/files.json under a new host-secrets project type. It follows the synced-copy design in issue #3505, which avoids runtime fetches and submodules for code that runs on maintainer hosts.

Changes:

  • Adds the shell library with verdicts for secret files, a set check, require_secret, and read_env_value, plus tests for each verdict.
  • Adds the host-secrets type, its governance rule, and a carried-trees repo gate check. The check reports a stale carried copy as a note and warning, never a failure.
  • Updates the manifest, scope and type docs, the conformance matrix, the audit classification, and the repo gate's action description to cover the new type and check.
File Description
tests/​test_secrets_sh.py Tests each library verdict and the env read against constructed files.
tests/​test_repo_gate.py Tests the carried-trees check and gate_hub_root.
tests/​test_cli_usage.py Lists the library as sourced, not a command-line entry point.
spec/​type-model.md Adds a "Runtime Libraries" section describing the host-secrets type.
spec/​scope-model.md Adds the host-secrets token to the project type namespace.
spec/​project-types.json Declares the host-secrets type with its detect entry and reader intent check.
spec/​files.json Carries lib/sh to lib/fleet/sh for host-secrets repositories.
spec/​fidelity-model.md Drops "generated" from the verbatim-tree wording, since hub-owned trees also use it.
scripts/​README.md Documents the carried-trees check.
reports/​conformance-matrix.md Adds the host-secrets overlay row.
lib/​sh/​secrets.sh New hub-hosted secret and env reader library, meant to be sourced.
GOVERNANCE.md Adds the runtime library rule, the secret reader rule, and lib/ in the layout list.
AUDIT.md Classifies a repo that reads credential files as host-secrets.
.github/​workflows/​validate-task.yml Updates the exclude-globs input description to mention carried-trees.
.github/​actions/​repo-gate/​repo_gate.py Adds the carried-trees check that compares carried trees with the hub tree.
.github/​actions/​repo-gate/​action.yml Updates the action description to cover carried tree currency.

Review effort: Lite

Comment thread .github/actions/repo-gate/repo_gate.py Outdated
Comment thread .github/actions/repo-gate/repo_gate.py Outdated
Comment thread spec/files.json
Comment thread .github/actions/repo-gate/repo_gate.py Outdated
Comment thread spec/project-types.json Outdated
The carried-trees check reports any failure to load or run the hub's
carry.py as a note, since it runs under whatever python3 the host has,
and carry.py states that contract in a comment rather than in its help
text. carry.py's post-apply comparison reads a declaration's ownership
as its check does. The rule text says the hub owns the carried directory
whole, so a repository extends the library outside it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ptr727

ptr727 commented Oct 26, 2026

Copy link
Copy Markdown
Owner Author

A recorded local strict-review pass covers head 0f40ab05feac2957012a95ccfa5f60491d785a14, the content this pull request carries at that commit against develop, and it recorded 4 findings.

ptr727 and others added 3 commits October 26, 2026 13:55
A case runs the carried-trees check under a 3.12 interpreter, since a CI
runner's system python3 is 3.12 and hub CI otherwise runs 3.13 alone. A
case drives carry.py apply through its post-apply comparison for a tree
that does not prune. The gate cases that need a colon in a file name or
a symlink no longer fail on Windows, and the README gives carried-trees
its own reason for a note.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The case that loads carry.py under Python 3.12 now skips where uv finds
no 3.12 already installed, and never downloads one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A tree that does not prune shares its target with the repository's own
files, so those alone no longer make the gate compare it and report every
hub file missing. The 3.12 skip moves to host_capability beside the
suite's other host checks, and the README names the sha-pin note as the
subject of its last sentences.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ptr727

ptr727 commented Oct 26, 2026

Copy link
Copy Markdown
Owner Author

A recorded local strict-review pass covers head 185a41e79cfa144815989dfa4b0c4e9bef95aeca, the content this pull request carries at that commit against develop, and it recorded 3 findings.

ptr727 and others added 2 commits October 26, 2026 14:25
The gate keeps a tracked link or non-file out of the inventory it hands
carry.py and classifies it itself, changed where the hub carries that
path and extra under a pruned tree where it does not. The 3.12 probe
catches a uv that fails or hangs, as the probe beside it does, and the
README says which tracked files make a tree compared.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A target, or an ancestor of it, that Git tracks as a link or a file now
names itself in a note instead of reading as a tree the repository does
not carry. A tracked file deleted from the working tree reads as missing
rather than changed. The gate cases pin their full note strings and
cover a link under a tree that does not prune.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ptr727

ptr727 commented Oct 26, 2026

Copy link
Copy Markdown
Owner Author

A recorded local strict-review pass covers head e1b2fa164307628e617aaebe0926a25bfedf6c56, the content this pull request carries at that commit against develop, and it recorded 3 findings.

ptr727 and others added 2 commits October 26, 2026 14:49
The cases that expect no difference now assert the whole note list, so a
tree the check skips no longer passes for one it compared. The gate keeps
only tracked links apart, since a checkout produces no other non-file at
a tracked path, and the README documents the note for a target Git
tracks as a link or a file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
carry.py refuses a symlink in a tree, so the gate now reports a tracked
link under a carried target as a tree it could not compare, naming the
link, instead of classifying it by a prune rule of its own. The
ownership rule then lives in compare_declaration alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ptr727

ptr727 commented Oct 26, 2026

Copy link
Copy Markdown
Owner Author

A recorded local strict-review pass covers head 1b59d2ffde2f96e0043e902fd26bd007bf922ea7, the content this pull request carries at that commit against develop, and it recorded 2 findings.

ptr727 and others added 2 commits October 26, 2026 15:09
The gate refuses a link anywhere under a carried target, as
carry.inventory does, not only among the files the tree owns, and shows
the refusal on the pull request as a warning beside its note.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The link refusal case gets back the symlink guard the previous commit
moved onto its new neighbor, and the README says an exclude pattern
covering part of a carried target reads the files it hides as missing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ptr727

ptr727 commented Oct 26, 2026

Copy link
Copy Markdown
Owner Author

A recorded local strict-review pass covers head f22ccc6a3ed831fe6b83ec8d63ee2c5f07a5607f, the content this pull request carries at that commit against develop, and it recorded 1 finding.

ptr727 and others added 2 commits October 26, 2026 15:26
A carried target that Git tracks as a link or a file now warns on the
pull request as well as naming itself in a note, since a carry refuses
it. An ancestor tracked that way stays a note, because a tracked file at
a generic path says nothing about a carry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The carry.py apply case resolves its temp root as main resolves a target,
since a Windows temp path can be an 8.3 short name, which failed the
Windows unit test job. A carried target or a non-top-level ancestor
tracked as a link or a file now warns as well as notes, and a generic
top-level path tracked that way is passed over like an uncarried tree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ptr727

ptr727 commented Oct 26, 2026

Copy link
Copy Markdown
Owner Author

A recorded local strict-review pass covers head 15f6363406092b2423a2995c453b89cec1ee947a, the content this pull request carries at that commit against develop, and it recorded 1 finding.

The generic top-level exemption now applies to an ancestor alone, so a
tree whose target is top-level still warns when Git tracks that target
as a link or a file, and a case pins it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ptr727

ptr727 commented Oct 26, 2026

Copy link
Copy Markdown
Owner Author

A recorded local strict-review pass covers head 85b2c4adb01fae676f9df876b33b3e2dc3b0e85d, the content this pull request carries at that commit against develop, and it recorded 0 findings.

@ptr727
ptr727 merged commit 1ab4c89 into develop Oct 26, 2026
13 checks passed
@ptr727
ptr727 deleted the feature/hub-secret-readers branch October 26, 2026 23:06
ptr727 added a commit that referenced this pull request Oct 27, 2026
## Summary

Promotes the pull request merged into develop since release 2.0.1196.

- [#3509](#3509): the hub
now hosts `lib/sh/secrets.sh`, the hardened secret and env readers a
fleet repository runs on its own hosts. A repository declaring the new
`host-secrets` type carries it byte-for-byte at `lib/fleet/sh` through a
`spec/files.json` tree. `repo_gate.py` gains a `carried-trees` check,
which compares each carried tree with the hub tree beside the gate and
reports a difference as a warning that never fails the gate. `carry.py`
gains the shared `target_patterns` and `compare_declaration`, which fix
its post-apply comparison for a tree that does not prune.
`GOVERNANCE.md` "Hub-Hosted Tooling" states why a runtime library is
carried rather than reached, and "Repo-Scoped Secrets" states the reader
rule.

Closes #3505

## Constituent coverage

| Pull request | Merged head | Coverage |
| --- | --- | --- |
| [#3509](#3509) |
`85b2c4ad` | attested local pass at the merged head, base `develop`,
recording 0 findings, and a Copilot round on an earlier head whose five
findings were fixed or declined with evidence and resolved |

🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comments Permits the comment lines the pull request adds or edits, which the prose gate otherwise refuses

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants