Skip to content

refactor(analysis): discover dependencies through abnegate-vcs - #168

Open
abnegate wants to merge 2 commits into
mainfrom
migrate/clh
Open

abnegate wants to merge 2 commits into
mainfrom
migrate/clh

Conversation

@abnegate

@abnegate abnegate commented Oct 5, 2026 •

Copy link
Copy Markdown
Member

User-visible changes

  • ~ in auto_discover_paths now works for dependency discovery. claudear repos index and claudear repos sync scanned the documented auto_discover_paths = ["~/Local"] as the literal relative path ~/Local. They indexed the repositories, because build_repo_index expands ~, but found no dependencies. repos discover behaved the same with no --paths, or with a quoted --paths '~/Local'. All three now expand ~ the way the index does. The daemon was not affected, because it scans the absolute paths of indexed repositories.
  • Those commands now record more dependency rows. With ~/Local resolved, repos index and repos sync read every checkout one level below it. Forks and third-party projects that depend on a known organisation become downstream rows as well. This does no harm: a cascade skips a downstream it cannot resolve to an indexed repository, and logs it.
  • ~user/... is no longer expanded to $HOME/user/.... Only ~ and ~/... are expanded. This affects build_repo_index as well.
  • Wording. repos index and repos sync print "No dependencies found under auto_discover_paths." (was "No dependencies found between indexed repos."). repos discover --save now reports how many dependencies it saved rather than how many it found.
  • No config keys, environment variables or CLI flags change. auto_discover_paths is still a list of strings.

Commits

  1. refactor(analysis): discover dependencies through abnegate-vcs moves to the crate and keeps the old literal-path behaviour.
  2. fix(cli): expand ~ in auto_discover_paths before scanning for dependencies adds the expansion and its regression test. The test is red without the fix, as described under Tests.

What changed

  • Deleted crates/claudear-analysis/src/repo/discovery.rs. Dependency discovery now comes from the published abnegate-vcs = "0.1.3" (default features, no github). claudear donated this code to the crate, which has hardened it since.
  • repo/mod.rs re-exports DependencyDiscovery, DiscoveredDependency and Manifest from abnegate_vcs, so claudear::DependencyDiscovery, claudear::DiscoveredDependency and claudear::repo::* resolve as before.
  • New repo/dependencies.rs holds the repo-domain functions that replace three copies of one scan, warn and count loop:
    • discover_configured(known_orgs, paths) scans configured path strings, with ~ expanded (commit 2).
    • save_discovered(tracker, dependencies) -> usize stores each dependency, and warns on and skips any it cannot save.
    • repos index, repos sync and repos discover scan through discover_configured. repos index, repos sync and Watcher::discover_dependencies save through save_discovered.
    • repos discover --save keeps its own loop, because it also records each repository's path and stops at the first storage error.
  • New relationships::dependency_type(Manifest) -> Option<DependencyType> decides what is stored, and the stored string is now DependencyType::as_str(). For Composer and npm that is the same "composer"/"npm" as before. Manifest is #[non_exhaustive], so a kind claudear does not cascade through is logged and skipped wherever it would be saved.
  • repo/index.rs: expand_path delegates to expand_path_in(path, home) (commit 2), so tests can inject the home directory.
  • Field renames at the call sites:
    • repo → repository
    • dep_type: String → manifest: Manifest
    • repo_path: String → repository_path: PathBuf
  • Scans return a Vec rather than a Result that could never be an error, so the unreachable "dependency discovery failed" branches are gone.
  • House-rule fixes in the touched lines:
    • one import per statement in index.rs, relationships.rs and the repos index arm;
    • repo_paths → repository_paths, dep_pb → dependency_progress;
    • db_tracker → tracker across the repos command block.
  • Cargo.lock:
    • Adds abnegate-vcs and abnegate-secret 0.1.3.
    • Moves tokio 1.50 → 1.53.2, with mio, socket2, libc and zeroize, because abnegate-vcs requires tokio ≥ 1.53.
    • Holds tokio-macros at 2.7.0, because 2.7.2 pulls in syn 3.
    • New second majors: dirs 7, base64 0.23, and abnegate-secret's aes-gcm stack (aes-gcm 0.11, aes 0.9, aead 0.6, cipher 0.5, ctr 0.10, ghash 0.6, polyval 0.7, universal-hash 0.6, inout 0.2).
    • New: nix 0.31.

Why the config field stays Vec<String>

The plan proposed turning auto_discover_paths into Vec<PathBuf>. This PR converts only where the crate needs a path, inside discover_configured. The field holds paths as a user writes them, ~ included. Changing its type would have moved the expansion into claudear-config at load time. It would also have changed build_repo_index, build_repo_index_with_fallback, RepoInferrer::with_discovery, the e2e builder and the config tests, and none of that gains anything.

Behaviour differences

Area Before (claudear's copy) After
~ in configured or --paths paths Scanned literally, so nothing was found ~ and ~/... expanded before the scan (commit 2)
~user/... Expanded as $HOME/user/... (index only) Left as written
Duplicates A package in both require and require-dev, or in both dependencies and devDependencies, or written as both @org/x and org/x, was reported twice Reported once per manifest. Stored rows were already deduplicated by ON CONFLICT. The repos discover count and listing shrink
Ordering Within a manifest, HashMap order. Composer before npm Within a manifest, sorted by package name. Composer before npm. Directories still in read_dir order
"require": [] (PHP's empty object) The whole composer.json failed to parse, losing require-dev and name The empty list requires nothing, and the rest still counts
Manifest that is a symlink Followed Skipped, with a warn log
Unreadable or malformed manifest Ignored silently Ignored, with a warn log
Leading @ Stripped from npm package names only Also stripped from Composer package names (real ones never start with @)
Stored type string The crate-side dep_type string DependencyType::as_str() via dependency_type. Same strings for Composer and npm
Depth The path itself, or one level below it when the path declares nothing Same
Ignored directories None (hidden directories are scanned too) Same
Manifests read composer.json (require, require-dev), package.json (dependencies, devDependencies) Same
Repository name Composer name, then the package.json name without @, then the directory name Same
Org matching Exact and case-sensitive on the part before the first / Same
Missing path Skipped Same

Tests

These are claudear-level tests through claudear's own functions. Tests that duplicated the crate's own suite were not ported: private helpers, struct construction, the PHP empty-list section, and empty or missing paths.

  • relationships::test_every_manifest_maps_to_a_dependency_type_stored_as_it_parses_back: the string contract between saving and the cascade loader.
  • dependencies::test_a_configured_directory_of_checkouts_is_read_one_level_down: an ~/Local-style root covering:
    • composer require and require-dev, with a duplicate collapsed;
    • npm sections, with the scope stripped;
    • a manifest without a name, which falls back to the directory name;
    • a malformed manifest, which is skipped;
    • exact repository_path values.
  • dependencies::test_a_configured_repository_that_declares_dependencies_is_not_read_one_level_down
  • dependencies::test_a_configured_home_relative_path_is_scanned_in_the_home_directory (commit 2): scans a configured ~/Local under a TempDir home, so nothing is written under the real home. With the expansion inside discover_configured_in reverted to a literal PathBuf::from, this test fails (0 dependencies found instead of 1). Re-checked after the rebase.
  • index::test_expand_path_home, test_expand_path_leaves_another_users_home_as_written and test_expand_path_without_a_home_leaves_the_path_as_written (commit 2).
  • claudear-engine, watcher::test_discover_dependencies_stores_what_the_cascade_loads: runs Watcher::discover_dependencies over an indexed checkout. It checks the stored row's upstream and downstream, and that its dep_type parses as DependencyType::Composer. The duplicate is stored once.

How it was verified

After rebasing onto main at 95fab34 (#154, #160 and #166 merged), each commit was checked on its own. The runs were local on macOS from the migrate/clh worktree, with --features sqlite for the member crates. CI uses --all-features, which adds cuda. The environment was FASTEMBED_CACHE_DIR/HF_HOME/CLAUDEAR_EMBEDDING_CACHE_DIR pointed at a local copy of the nomic model, and CLAUDEAR_VECTORLITE_PATH=/opt/homebrew/lib/vectorlite.dylib, so no test was skipped.

Check Commit 1 Commit 2 (head)
cargo fmt --all -- --check pass pass
cargo check --workspace --all-targets pass pass
cargo clippy --workspace --all-targets -- -D warnings -A clippy::double_must_use not re-run after the rebase pass
cargo test -p claudear-analysis --features sqlite not re-run after the rebase 2868 passed
cargo test -p claudear-engine --features sqlite not re-run after the rebase 1365 passed
cargo test -p claudear --features sqlite not re-run after the rebase lib 315, bin 11, e2e_real_repo 2, retries 1, shutdown 13 passed

Before the rebase, clippy and the three test suites also passed on commit 1 on its own. The regression test was re-checked at head by reverting the expansion to a literal PathBuf::from: it fails, and passes again with the fix.

Cargo.lock was rebuilt from main's lock with cargo metadata, then cargo update -p tokio-macros --precise 2.7.0. The result is identical to the auto-merged lock: registry sources only, and no syn 3.

Not verified

Overlapping open PRs (by file)

Rebased on #154, #160 and #166. Overlaps are the intersection of this PR's files with each open PR's files.

🤖 Generated with Claude Code

@hansi-codes

hansi-codes Bot commented Oct 5, 2026 •

Copy link
Copy Markdown

🟢 Tier S · Ready to merge

No concrete regressions were found in the changed code or its repository callers.

Moves dependency discovery to abnegate-vcs and centralizes configured-path scanning and persistence. Configured paths now expand ~ consistently with repository indexing, and manifest kinds are mapped to the dependency types used by cascades. Adds regression coverage for discovery, home-relative paths, and stored dependency rows.

Verdict New comments Fixed Still open
✅ Approved 0 0 0
📂 Walkthrough · 7
File Change
Cargo.toml; crates/claudear-analysis/Cargo.toml; Cargo.lock Adds abnegate-vcs and updates the resolved dependency graph.
crates/claudear-analysis/src/repo/discovery.rs; crates/claudear-analysis/src/repo/mod.rs Replaces the local discovery implementation with crate re-exports.
crates/claudear-analysis/src/repo/dependencies.rs Adds shared configured-path scanning, best-effort persistence, and discovery tests.
crates/claudear-analysis/src/repo/index.rs Makes home expansion injectable for tests and leaves other users' tilde paths unchanged.
crates/claudear-analysis/src/repo/relationships.rs Maps supported manifests to cascade dependency types and tests the storage contract.
crates/claudear-engine/src/watcher.rs Uses PathBuf scan inputs and shared persistence, with a database integration test.
src/main.rs Routes repository CLI discovery through shared helpers and updates saved-count reporting.

Reviewed a260030 · Details · Comment @hansi-codes review to re-run, or mention @hansi-codes with a question.

@hansi-codes hansi-codes Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Tier S · Looks good to merge. Summary

abnegate and others added 2 commits October 5, 2026 20:41
abnegate-vcs's DependencyDiscovery was donated from claudear's
repo/discovery.rs and has been hardened in the crate since, so claudear
now uses the published crate (0.1.3, default features, no `github`) and
drops its own copy. repo/mod.rs re-exports DependencyDiscovery,
DiscoveredDependency and Manifest, so `claudear::DependencyDiscovery`
and `claudear::repo::*` keep working.

The crate's DiscoveredDependency names its fields repository,
depends_on, manifest (an enum) and repository_path (a PathBuf), and its
scans return the dependencies directly instead of a Result that could
never be an error, so the unreachable "discovery failed" branches are
gone.

`repos index`, `repos sync` and the daemon's Watcher each carried the
same scan, warn and count loop. It now lives in the repo domain:
discover_configured scans the configured paths (still literally, as
before) and save_discovered stores each dependency, warning on and
skipping one that fails. `repos index`, `repos sync` and
`repos discover` scan through discover_configured, and `repos index`,
`repos sync` and the Watcher save through save_discovered.
`repos discover --save` keeps its own loop because it also records each
repository's path and stops at the first storage error.

What is stored is now chosen by relationships::dependency_type, which
maps a Manifest to the DependencyType a cascade loads back, so the
string written is DependencyType::as_str() rather than the crate's
naming. Manifest is non-exhaustive: a manifest kind claudear does not
cascade through is logged and skipped where it would be saved.

Behaviour that changes with the crate:
- A package named in both require and require-dev (or dependencies and
  devDependencies, or as both @org/x and org/x) is reported once per
  manifest, and each manifest's dependencies come back sorted.
- A Composer section PHP encoded as an empty list (`"require": []`) no
  longer makes the whole manifest unreadable, so require-dev and the
  manifest's name still count.
- A manifest that is a symlink is skipped, and an unreadable manifest
  is logged as a warning instead of being ignored silently.
- A leading `@` is stripped from Composer package names too, not only
  npm ones.
- `repos discover --save` reports how many dependencies it saved
  rather than how many it found.
- `repos index` and `repos sync` say "No dependencies found under
  auto_discover_paths." when the scan finds nothing, since they scan
  those paths rather than the indexed repositories.

Tests pin what claudear relies on: every Manifest maps to a
DependencyType whose string parses back; a configured directory of
checkouts is read one level down, with names from composer.json, then
package.json without its scope, then the directory; a configured
repository that declares dependencies is not descended into; and the
Watcher stores a discovered dependency as a row that loads back as a
Composer DependencyType.

Cargo.lock gains abnegate-vcs and abnegate-secret 0.1.3 and moves tokio
to 1.53 (with mio, socket2, libc and zeroize), the minimum abnegate-vcs
requires. tokio-macros is held at 2.7.0, because 2.7.2 pulls in syn 3.
This adds second majors of dirs (7), base64 (0.23), and abnegate-secret's
aes-gcm stack (aes-gcm 0.11, aes 0.9, aead 0.6, cipher 0.5, ctr 0.10,
ghash 0.6, polyval 0.7, universal-hash 0.6, inout 0.2), plus nix 0.31.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ncies

The dependency scanner takes paths literally, and so did claudear's own
copy before it, while build_repo_index expands `~`. With the documented
`auto_discover_paths = ["~/Local"]`, `repos index` and `repos sync`
indexed the repositories under the home directory but scanned a relative
`~/Local` that does not exist, and found no dependencies. The same was
true of `repos discover` with no `--paths`, or with a quoted
`--paths '~/Local'` that the shell leaves alone. The daemon was not
affected, because it scans the absolute paths of the repositories it
indexed.

discover_configured now resolves `~` the way the index does, through
the shared expand_path_in, before it scans. With `~/Local` resolved,
those commands now read every checkout one level below it. Forks and
third-party projects that depend on a known organisation therefore
become downstream rows too. That does no harm: a cascade skips a
downstream it cannot resolve to a repository, and logs it.

expand_path used to treat any leading `~` as the current user's home, so
`~other/projects` became `$HOME/other/projects`. It now expands only `~`
and `~/...` and leaves another user's home as written. `~//x` no longer
joins as the absolute path `/x`. The home directory can be injected, so
the tests no longer read or write under the real home.

The regression test scans a configured `~/Local` under a temporary home.
Scanning the paths literally again inside discover_configured turns it
red.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant