Tecnix eval caching

[joshheinrichs-shopify]

robo summary 🤖

The high-level goal is threefold: speed up evaluation in WCB,
support dependency graph generation for the merge queue, and decouple
Tecnix from Tectonix's zone business logic. WCB primarily needs fast target
discovery and selective target evaluation. The merge queue needs to know which
targets are affected by which source changes so it can safely handle concurrent
merges without serializing everything or rebuilding the world. Tecnix should
provide evaluator primitives for these workflows without embedding
Tectonix-specific knowledge about zones, modules, or target construction.

The interface is split around those caller needs. builtins.tecnixTargetNames
supports WCB's discovery step: it returns a flat list of fully-qualified target
names such as //areas/tools/dev:__devEnv without forcing target values.
builtins.tecnixTargets supports WCB's evaluation step: given explicit target
names, it calls the resolver and returns one value per target in input order.
builtins.tecnixDependencies supports merge-queue dependency analysis: it
evaluates selected targets under tracking and returns the repo-relative source
inputs that influenced each target.

The resolver contract is the boundary between Tecnix and Tectonix. Tecnix
imports a repo-relative resolver such as system/tectonix/resolve.nix, calls
it with { system = ...; }, and expects something like:

{
  resolve = target: ...;
  allTargetNames = [ "//areas/tools/dev:__devEnv" ... ];
}

Everything zone-specific stays behind this contract. Tecnix does not need to
know how zones are discovered internally, how zone.nix is interpreted, or how
Tectonix modules construct target definitions. This is what lets Tecnix stay
focused on evaluation, caching, tracking, and source access.

For WCB, the normal flow is: discover names, select targets, evaluate those
targets. Discovery must be cheap because it happens at world scale. Evaluation
must be selective because constructing every target is too expensive. This is
why tecnixTargetNames intentionally avoids resolving target bodies, and why
tecnixTargets takes an explicit targets list instead of implying "all
targets."

For the merge queue, the important artifact is the dependency graph. Example:

{
  "//areas/tools/dev:__devEnv" = [
    "system/tectonix/resolve.nix"
    "system/tectonix/lib/resolve.nix"
    "areas/tools/dev/zone.nix"
    "areas/tools/dev"
  ];
}

Given a merge that changes areas/tools/dev/zone.nix, the queue can identify
affected targets. Given a merge that touches unrelated files, it can avoid
unnecessary evaluation/builds. This is what makes concurrent merges tractable:
the queue can compare changed paths against target dependency closures instead
of serializing all merges or rebuilding everything.

Dependency tracking happens at the source-accessor boundary, not by statically
analyzing Nix. During tecnixDependencies, Tecnix installs a per-target
tracking context. When evaluation imports files, reads source paths, or
fingerprints source directories, the accessor records repo-relative paths. This
captures resolver imports, Tectonix library imports, zone files, and source
directories in the same format as Git changed paths.

Dirty worktrees are first-class source state. Tecnix evaluates against a clean
Git tree plus an optional checkout overlay. Clean paths fingerprint as Git
blob/tree SHAs. Dirty checkout paths are read from disk, and their content is
folded into the fingerprint. This means local development behaves like committed
source from the dependency tracker's point of view: dirty resolver files, dirty
zone.nix files, or dirty children under tracked source directories all change
the source closure and prevent stale cache reuse. It also means tecnixTargets
sees local edits without requiring a commit.

Directory inputs are tracked coarsely. For a pattern like src = ./., Tecnix
records the directory path and fingerprints it by Git tree SHA rather than
recording every child file. That keeps dependency lists compact while still
invalidating correctly when any child file changes, because the directory tree
SHA changes. With a dirty checkout overlay, a dirty child under that tracked
directory augments the directory fingerprint, so we still avoid listing every
child while remaining correct for local edits.

The cache is source-closure based so it works across commits. It does not care
whether the overall commit SHA changed; it cares whether the tracked inputs for
a target changed. Conceptually, a cached source closure looks like:

{
  "target": "//areas/tools/dev:__devEnv",
  "resolver": "system/tectonix",
  "system": "aarch64-darwin",
  "pathFps": {
    "system/tectonix/resolve.nix": "git:4b825dc642cb6eb9a060e54bf8d69288fbee4904",
    "system/tectonix/lib/resolve.nix": "git:9af3b7d7c2a53477e8b99d3a51a78e1c6d12abcd",
    "areas/tools/dev/zone.nix": "dirty:sha256:abc123...",
    "areas/tools/dev": "git:3e7f91ac7b13f7c1e61b0e6e8f2a0c1b22222222;dirty=sha256:def456..."
  }
}

The public dependency list is the set of paths in that source closure. For
files, the fingerprint is the Git blob SHA when clean, or a dirty content
fingerprint when overlaid from the checkout. For directories, the fingerprint
is the Git tree SHA plus dirty child state when relevant. If two commits have
the same fingerprints for the tracked paths, the dependency result can be
reused across those commits.

When multiple source closures are retained for a target, they can be organized
as a trie to reduce Git ODB lookups. Each edge is a path plus expected
fingerprint. Lookup fingerprints only enough paths to distinguish candidate
closures:

root
└── system/tectonix/resolve.nix = git:4b825...
    └── areas/tools/dev/zone.nix = dirty:sha256:abc123...
        └── areas/tools/dev = git:3e7f...;dirty=sha256:def456...
            └── cached dependency list

If an early fingerprint does not match, Tecnix rejects that candidate without
checking the rest of the closure. This matters when there are many retained
closures for the same target across different source states.

The current limitations are mostly around interface and storage.
builtins.tecnixDependencies is useful for explicit target sets, but
whole-world dependency graph generation may be cleaner as a Tecnix-specific
CLI/evaluator side output because making "the graph of all targets" itself a
target creates self-reference problems. The cache is also still stored in Nix's
generic SQLite fetcher cache as JSON-ish values, and bounded retention like "50
closures per target" is a crude scaling knob. Long term, this should become a
real dependency index with explicit schema, path-prefix lookup, GC/versioning,
and first-class support for answering: "given these changed paths, which
targets are affected?"

Future work: the dependency graph can support developer tooling beyond the
merge queue. For example, a shell hook could compare the current checkout
against the dependency closure of the active dev environment and warn: "your dev
environment is stale; please run dev up." This should be treated as a
consumer of the graph, not the core reason for the evaluator changes.