Multi-language, trait-based code-analysis metamodel (FamixNG-style), implemented in TypeScript. Language-specific extractors (Java first, via Spoon) emit a JSON interchange file; a TypeScript analyzer validates it, builds the dependency graph, and runs analyses (imports, coupling, cycles, architecture).
Source of truth for design decisions: the August 2026 exploration document (traits, not hierarchy; provenance on every edge; imports as the first-class comparable layer; containment ≠ attachment; stubs for external types).
┌─────────────────────┐ ┌──────────────────────┐ ┌─────────────────────┐
│ Extractors │ │ JSON interchange │ │ TypeScript analyzer │
│ (per language, │ ──▶ │ contract │ ──▶ │ validate → graph → │
│ native tooling) │ │ model.json │ │ queries → exports │
│ │ │ (versioned schema) │ │ │
│ Java: Spoon (JVM) │ └──────────────────────┘ └─────────────────────┘
│ Clojure: clj-kondo │ ▲
│ TS/JS: ts compiler │ │ JSON Schema published from the
│ … (later) │ │ TS core package → extractors in any
└─────────────────────┘ │ language can self-validate output
Key principle: extractors are federated behind one contract. Each extractor uses the best native tool for its language and only has to produce conforming JSON. All intelligence about the metamodel (traits, profiles, validation, derived indexes, analyses) lives once, in TypeScript.
codegraph/
├── PLAN.md
├── package.json # pnpm workspace root
├── pnpm-workspace.yaml
├── tsconfig.base.json
├── packages/
│ ├── core/ # @codegraph/core — metamodel: traits, kinds,
│ │ # entities, edges, profiles, validation,
│ │ # JSON Schema export
│ ├── analyzer/ # @codegraph/analyzer — graph construction,
│ │ # derived indexes, queries, metrics, exports
│ ├── cli/ # @codegraph/cli — `codegraph` command
│ └── viz/ # (future) Three.js "code city" 3D visualization
│ # of the model — the only package allowed
│ # to depend on three
├── extractors/
│ └── java/ # Maven/Gradle project, Spoon-based, emits model.json
│ └── src/main/java/...
├── schemas/ # generated JSON Schema files (committed,
│ # versioned — the cross-language contract)
└── fixtures/
└── java/ # small reference corpus + expected model.json
Tooling: Node ≥ 22, pnpm, TypeScript strict, Zod v4 (runtime validation + static types + native JSON Schema export), Vitest + fast-check (property-based tests), tsup for builds.
Why Zod: one definition per trait yields (a) the inferred TS type, (b) the
runtime validator, (c) z.toJSONSchema() output for non-TS extractors — the
exact role Malli plays in the original Clojure design.
- pnpm workspace,
tsconfig.base.json(strict,"module": "NodeNext"), ESLint, Vitest. - Empty
core,analyzer,clipackages wired together; CI script (pnpm -r build && pnpm -r test). - Rewrite README.md (currently GitLab template) with project pitch + this plan's summary.
// Opaque entity id: "<lang>:<module>/<symbol>[#<disambiguator>]"
// disambiguator = signature (overloads, receivers) or "file:startLine" for anonymous entities.
type EntityId = string; // opaque — never parsed by the analyzer, only compared
const SourceAnchor = z.object({
file: z.string(),
span: z.tuple([z.number().int(), z.number().int()]), // [startLine, endLine]
});
const Provenance = z.enum(["declared", "derived", "dynamic-candidate", "generated"]);Each trait declares only the keys it contributes:
| Trait | Keys contributed |
|---|---|
TNamed |
name |
TSourceAnchor |
anchor: SourceAnchor |
TComment |
comments: string[] |
TWithChildren |
children: EntityId[] (lexical containment) |
TChildOf |
parent: EntityId |
TAttachedTo |
attachedTo: EntityId (semantic attachment — Go receivers, Rust impl blocks, C# extension methods, Clojure extend-type) |
TModule |
definedIn: string[] (CodeFile paths; 1-1, 1-N or N-N per language), isStub: boolean |
TType |
isStub: boolean |
TWithInheritances |
(marker — edges carry the data) |
TWithImplements |
(marker — edges carry the data) |
TTypedEntity |
declaredType?: EntityId |
TInvocable |
signature: string |
TWithParameters |
parameters: EntityId[] |
TWithLocalVariables |
localVariables: EntityId[] |
TWithInvocations |
(marker — outgoing Invocation edges) |
TStructural |
(marker — value holder) |
TWithAccesses |
(marker — outgoing Access edges) |
Design rules carried over verbatim:
- Outgoing only — incoming indexes are always derived by the analyzer, never serialized (consistency + no serialization cycles).
- Containment ≠ attachment —
TChildOf/TWithChildren(where it's written) vsTAttachedTo(what it semantically belongs to) are distinct. space?: ("type" | "value")[]on entities for TypeScript profiles (interface = type-space only; class = both) — optional field, only meaningful in TS/… profiles.
Implementation: TRAITS: Record<TraitName, ZodObject> + a TraitName string
literal union. An Entity is:
const EntityBase = z.object({
id: z.string(),
kind: z.string(), // constrained per profile, not globally
traits: z.array(TraitName),
}).passthrough(); // trait keys validated per declared traitAll edges carry from, to, provenance, anchor (evidence), optional
sourceFile (C# partial classes), optional candidates: EntityId[]
(uncertain dispatch).
Edge kinds: import, inheritance, interfaceImplementation, invocation,
access (+ isRead/isWrite), reference, embedding (Go), traitUsage
(PHP), fileInclude (PHP). Discriminated union on edge field.
interface Profile {
lang: string;
kinds: Record<string, { required: TraitName[]; optional: TraitName[] }>;
edges: EdgeKind[]; // which associations this language can emit
space?: Record<string, Space[]>; // per kind, the declaration spaces it MAY occupy
notes?: string[]; // documented static-analysis blind spots
}Decision (M1 audit, added after the first draft of this section): space?
belongs on the Profile, not only on the Entity. METAMODEL §1.4 says the type/value
split is "only meaningful in profiles that declare it"; without a profile-side
declaration that sentence is unenforceable and space: ["type"] on a Java entity
would validate. A profile that omits space licenses none, which is exactly the
statement "this language has no type/value split". Only the TypeScript profile
declares it. The field is optional and additive: every profile literal predating
the decision still compiles unchanged.
Decision (was open in the design doc): validation uses
required ⊆ traits ⊆ required ∪ optional — strict equality is too brittle for
optional traits like TComment/TSourceAnchor; unrestricted subset would hide
extractor bugs. This gives both.
- Define all 9 profiles (Java, C#, Go, Clojure, JS, TS, Python, PHP, Rust) as data files — specifiable without being implemented (robustness test). Only Java's needs to be exercised in Phase 2.
Decision (M1 review) — lang ids are frozen as declared:
clj |
csharp |
go |
java |
js |
php |
python |
rust |
ts |
|---|
Abbreviated where the abbreviation is the idiomatic name of the language
(clj, js, ts), spelled out otherwise. The mixed style is deliberate and
not to be "tidied": the lang id is the EntityId prefix, so it is embedded in
every id an extractor emits and in every edge endpoint referencing one. Renaming
one after an extractor ships invalidates that extractor's whole id space and any
model.json already produced. Frozen before M2 for exactly that reason.
validateEntity(profile, entity):
1. entity.kind ∈ profile.kinds
2. required(kind) ⊆ entity.traits ⊆ required(kind) ∪ optional(kind)
3. ∀ t ∈ entity.traits: TRAITS[t].parse(entity) succeeds
validateModel(model):
+ every edge's provenance is set
+ edge kinds allowed by the profile
(graph closure — every from/to/parent/children id resolves — is checked in
the analyzer, since stubs are legitimate targets)Reference example (the doc's §9 EDN example, translated):
{
"id": "java:com.acme.order/OrderService.bill(com.acme.order.Order)",
"kind": "method",
"traits": ["TNamed", "TInvocable", "TWithParameters", "TWithLocalVariables",
"TWithInvocations", "TWithAccesses", "TTypedEntity", "TChildOf",
"TSourceAnchor"],
"name": "bill",
"signature": "bill(com.acme.order.Order)",
"declaredType": "java:com.acme.billing/Invoice",
"parent": "java:com.acme.order/OrderService",
"anchor": { "file": "OrderService.java", "span": [15, 22] }
}{
"edge": "invocation",
"from": "java:com.acme.order/OrderService.bill(com.acme.order.Order)",
"to": "java:com.acme.order/TaxCalculator.apply(double)",
"candidates": ["java:com.acme.order/TaxCalculator.apply(double)"],
"provenance": "declared",
"anchor": { "file": "OrderService.java", "span": [19, 19] }
}Id parameter types are erased FQNs, not simple names (M2, verified against
Spoon). archive(java.util.List) and archive(com.acme.order.legacy.List) are
legal overloads whose simple names are identical; rendered as archive(List)
they collapse into one id and one method disappears silently. Ids must be
unique per model, so the FQN form is normative — here, in METAMODEL.md §10, and
in the extractor. The fixture corpus pins the collision as a regression test.
-
pnpm run gen:schemas→z.toJSONSchema()→schemas/model.schema.json(committed; the Java extractor validates against it in its own tests).
Decision (M1 audit): the published schema must be sufficient on its own. Zod
refinements do not survive z.toJSONSchema(), so the trait-key rule of §4.5 step 3
is re-stated in the emitted schema as one if/then conditional per key-contributing
trait, generated from the same TRAITS table. Without it the contract accepted
{traits: ["TNamed"]} with no name, and "an extractor in any language can
self-validate using only the published schema" was only partly true. What remains
outside the schema — deliberately — is everything profile-aware: which kinds exist
and which trait compositions each kind licenses. That is validateEntity's job and
requires the profile data, so an extractor self-validates structure + vocabulary +
trait keys against the schema, and the analyzer adds the profile pass on load.
Deliverable Phase 1: @codegraph/core published locally; 9 profile data
files; JSON Schema generated; unit tests incl. the canonical hierarchy-breaking
case (a Clojure fn-var validating as TNamed + TStructural + TInvocable).
Separate Maven project in extractors/java/, JDK 17+, Spoon
in noClasspath mode (legacy, non-compilable corpora). Output: model.json
via Jackson. CLI: java -jar codegraph-java.jar --src <dir> --out model.json.
| Java construct | kind | traits |
|---|---|---|
| package | package |
TNamed, TModule, TWithChildren (TModule brings definedIn and isStub) |
| class / interface / enum / record / annotation | class/interface/… |
TNamed, TType, TWithInheritances, TWithImplements, TWithChildren, TChildOf, TSourceAnchor, (TComment) |
| method | method |
TNamed, TInvocable, TWithParameters, TWithLocalVariables, TWithInvocations, TWithAccesses, TTypedEntity, TChildOf, TSourceAnchor |
| constructor | constructor |
TInvocable, TWithParameters, … — no TNamed, no TTypedEntity |
| lambda / anonymous class | lambda |
TInvocable without TNamed; disambiguator = (file, startLine) |
| field | attribute |
TNamed, TStructural, TTypedEntity, TChildOf, TSourceAnchor |
| parameter / local var | parameter/localVariable |
TNamed, TStructural, TTypedEntity, TChildOf |
Edges: import (package/type imports → module-level), inheritance,
interfaceImplementation, invocation (via getReferencedTypes()/executable
references), access (field reads/writes), reference (type usages).
Lombok-generated members, if visible: provenance: "generated".
Spoon in noClasspath mode invents plausible FQNs. Rule: build a whitelist
of corpus-declared types first (pass 1); anything referenced but not in the
whitelist becomes { kind: "class", traits: ["TNamed","TType"], isStub: true }.
Never filter by package prefix. Edges to stubs are kept; internal-only
analysis = analyzer-side filter(!isStub).
Decision (M2): isStub is contributed by TModule as well as TType. The
import graph is module-level (§4.6, METAMODEL §9), so import java.util.List
yields an edge to java:java.util — a package no corpus file declares and, with
isStub on TType alone, one nothing could represent. The choices were a class
stub named util (a fabrication), a permanently dangling endpoint (breaking
closure, invariant 10), or making a degraded module expressible. An external
package is now { kind: "package", traits: ["TNamed","TModule","TWithChildren"], definedIn: [], isStub: true } — definedIn: [] is precisely what makes it
external — so "internal view = filter stubs" holds for the import layer too.
Only TType and TModule contribute isStub; a dangling member id is
still refused and reported, because a method stub would need a degraded-member
concept core does not have.
Primitives, void and <nulltype> are not entities. EntityIds.erasedTypeName
keeps primitives as-is by design, so pass 2 omits declaredType for them rather
than letting java:<unnamed>/int reach pass 4 and be fabricated into a stub
class named int. TTypedEntity's value is optional even when the trait is
declared, so omitting is legal and lossless — and a phantom class with a fan-in
of hundreds would distort every coupling metric the analyzer computes.
-
Fixture corpus in
fixtures/java/(overloads, lambdas, inner classes, constructors, static imports, an unresolvable external lib) + snapshotfixtures/java/expected/model.json, pretty-printed and sorted so it is reviewable in a diff.SnapshotTestreproduces it;-Dcodegraph.updateSnapshot=trueregenerates it deliberately. -
Extractor test validates its output against
schemas/model.schema.json(ModelSchemaValidationTest, networknt). -
Profile conformance across the language boundary: a JSON Schema cannot express per-kind trait rules, so
packages/core/test/fixtures-java.test.tsloads the snapshot,parseModels it and runsvalidateModelagainstjavaProfileexpecting zero issues. This is the M2 acceptance gate — the first check that runs both halves of the system against each other. -
CLOSED (M7) — the lambda id disambiguator was too coarse.
Type#file:startLinecould not separate two nameless entities that START ON THE SAME LINE (chain(() -> a, () -> b)), so they shared an id and the first in AST order won: one real entity was absent from the model and nothing distinguished that from an entity never written.**It was worse than a missing entity.** A nameless entity written INSIDE another on the same line took the outer one's id as its `parent`, so the model contained an entity that was ITS OWN PARENT — a containment cycle no analysis walking `parent` survives, and one `validate` could not see. Found on apache/fineract at `json -> gson.fromJson(json, new TypeToken<HashSet<JobParameterDTO>>() {})`: a lambda and an anonymous class, both starting on line 79. The id now carries the COLUMN — `Type#file:line:column`. A column is a source FACT; an ordinal would have made the id depend on the order the extractor happens to walk the AST, so an unrelated change could renumber a corpus. What the corpora recovered: | corpus | entities | edges | |---|---|---| | fixtures/java | 166 → 167 | 173 → 175 | | apache/commons-lang | 15 338 → 15 362 | 24 631 → 24 648 | | apache/fineract | 240 929 → 241 101 | 782 032 → 782 046 | Both corpora still validate clean, and fineract now has zero containment cycles. `StubDisciplineTest.everyNamelessEntityOnASharedLineGetsItsOwnIdentity` replaced the pinned-loss test, as that test's own doc instructed, and `noEntityIsItsOwnParent` guards the consequence. -
Graph closure and self-reference asserted on the snapshot from both sides (
StubDisciplineTestin Java,unknownReferences/selfReferencesin TS). -
Measure resolution rate in noClasspath on a real corpus; report unresolved counts in the extractor's stderr summary. Measured (M2 audit): 100.0% on apache/commons-lang — 263 files, 133 478 type references, 0 unresolved, 15 338 entities (261 stubs), 24 631 edges, 6.6 s wall clock — and 77.8% on spring-petclinic (30 files, 2 189 references, 487 unresolved).
Categorising the failures is what makes those numbers readable, and it first corrected the metric: **every one of commons-lang's 1 466 originally "unresolved" references was `<nulltype>`**, Spoon's static type for the `null` literal — a pseudo-type that can never have a declaration, which the extractor already refuses to make an entity. It is now excluded from the denominator alongside type variables, for the same stated reason. The old 98.9% was measuring how many `return null;` statements commons-lang contains. What is left is a clean statement: the rate measures the corpus's **dependency surface**, not the extractor. commons-lang is self-contained and depends on nothing but the JDK, which resolves against the runner's own classpath — so it resolves perfectly and the 85% target is **vacuous** there. petclinic's 487 failures are Spring and Jakarta types (`jakarta.persistence` 86, `org.springframework.web.bind.annotation` 82, `org.springframework.data.domain` 48, …) whose jars are simply absent; a jar that is not there cannot be resolved by any extractor, so 77.8% is a structural floor for that corpus, not a defect to fix. This is exactly the legacy/non-compilable case M2 exists for, and it is why the stub discipline of §5.2 — not the resolution rate — is the property worth asserting. `ResolutionRateTest` therefore keeps a deliberately low floor plus `unresolved > 0` rather than pinning a number that would pressure someone into deleting the unresolvable half of the corpus. -
Two fabrication bugs found only under real load (M2 audit), both fixed and both now in the fixture corpus (
Batch.java) andArrayAndAnonymousIdentityTest. The fixtures had arrays and an anonymous class, but never readarray.lengthand never touched an anonymous class's own members, so neither was visible: (a) an array type ownslengthandT[]::new; folding those up to the component type produced 8 stub CLASSES namedint,boolean… with a fan-in of 44–88, plus 624 access edges claiming dependencies the source never wrote — the exact phantom §5.2 forbids by name, reaching the model through the one path that never asked the primitive question; (b) Spoon'sOuter$Nname for an anonymous class rendered asjava:pkg/Outer.N, giving one declared class two ids and laundering the second into a stub inside the corpus's own package (invariant 6 failing from the inside). Edges into such a class now name its members precisely.
Input: one or more model.json files (multi-language later — union of models).
- Load & validate against core (profile-aware). Hard fail on schema
errors, collected warnings on profile violations (
loadModels,isClean). - Graph construction: entity map by id; derived inverse indexes
(incomingInvocations, incomingAccesses, subtypes, importers…) computed in
memory, never persisted.
test/no-inverse-index-serialized.test.tsscans every export for an inverse-index key rather than trusting the convention. - Closure check: every edge endpoint / parent / child resolves to a known
id or a stub — asserted as a property, using core's
unknownReferences. - Queries / analyses:
importGraph(module→module, the cross-language layer),typeDependencyGraph,dependenciesOf/dependentsOf,neighboursOf(METAMODEL §9 in one record),coupling(fan-in/fan-out, Ca/Ce, instability) withtopByFanIn/topByFanOut,cycles(Tarjan SCC) at module and type level — each SCC carrying its minimum feedback set and tangle metric (weighted ELS-GR + minimality pass,metrics/tangle.ts; pinned intangle.test.tsand the property suite) — and the view stack (internalOnly,declaredOnly,provenanceOnly,composeViews). - Exports: DOT/Graphviz, CSV (folded graph, coupling, cycles) and JSON artefacts (GraphML/Mermaid later).
Folding aggregates, and keeps what it aggregated. A FoldedEdge carries
count, kinds and provenances; collapsing parallel edges to a bare pair
would discard exactly what makes the result auditable. Self-loops created BY
folding (a method calling a sibling method of its own class) are kept and
flagged — they are cohesion, not the forbidden self-edge of METAMODEL §4.
Self-loops are excluded from coupling by default, on all four counters
together. fanIn/fanOut AND outgoingEdgeCount/incomingEdgeCount obey
includeSelfLoops as a unit, so a row can never read "fanOut 0, outgoing weight
6". Counting a self-loop would give every cohesive class Ce ≥ 1 and Ca ≥ 1 and
instability could never reach 0 or 1. The conservation law follows per mode:
with includeSelfLoops: true the row sums equal diagnostics.foldedEdges
exactly; by default they equal the non-self folded weight. Both are pinned.
Tarjan is iterative, with an explicit frame stack. The recursive formulation exhausts V8's stack at the ~15 000 nodes commons-lang folds to, and it fails only on real corpora because every hand-built test graph is shallow. Guarded by a 20 000-node chain and a 20 000-node ring.
A stub is its own container at every fold level. A stub has no parent, so a
stub CLASS becomes its own module node. Consequence, measured on the fixture:
24 of the 27 module-level nodes are external classes. Anything user-facing
should default to internalOnly at module level. See the M4 note below.
An analysis number without its view is not a fact. FoldedGraph,
CouplingTable and CycleReport all carry level and view, and every CSV
row repeats them as columns (a # comment line is data to an RFC 4180 parser).
- KNOWN GAP carried into M4 — stub classes do not fold into their stub
package. The extractor emits stub packages (
java:java.util) and stub classes (java:java.util/List) as unrelated roots, so the unfiltered module graph lists external CLASSES as modules. The import layer is unaffected (the extractor already writes imports module→module:nonModuleEndpointsis 0 on the fixture, gson and commons-lang), but the full type-fold at module level is noisier than it should be. The fix is aparenton stub classes in the extractor, not a special case in the fold.
✅ Shipped in M4. Every command takes one or more model paths and loads them as a union (decision 5).
codegraph validate <model.json...> [--json]
codegraph analyze <model.json...> --report deps|cycles|coupling [--level module|type]
[--internal-only] [--declared-only] [--json] [--top N]
codegraph export <model.json...> --format dot|json|csv|plantuml [--level module|type]
[--internal-only] [--declared-only] [--out FILE]
codegraph profiles [--lang java] [--json] # print a profile spec
-
validate— runs the analyzer'scheckConformancegate over the union. -
analyze— deps / cycles / coupling, at module or type level. -
export— DOT, JSON, CSV and PlantUML (class diagram) of the folded graph. The PlantUML rendering keeps the DOT encoding: solid arrow = all-declared, dashed = contains an inference, label = folded count,<<stub>>= external. -
profiles— prints core's profile data; synthesizes nothing. -
--helpper command,--version, and a usage error naming the valid values for a bad flag.
Locked behaviour, all covered by the end-to-end binary suite:
- Exit codes:
0ok ·1internal bug ·2usage ·3findings. 1 and 3 are never conflated, so a CI job can gate on model quality alone. - Streams: stdout is the artifact ONLY; every warning, summary and fold diagnostic is stderr. Verified by real shell redirection, not in-process.
- No ANSI, ever. Deterministic: identical inputs give byte-identical stdout.
--jsonon every reporting command, always a self-describing object with akindstamp — the same facts as the text form, differently printed.
✅ Shipped in M4 as checkConformance (in the analyzer, so the CLI, the property
suite and CI all ask the same question), plus a fast-check property suite.
Invariants from the design doc, run against every extractor output:
- Closure: no edge to an unknown id (stubs count as known).
- No self-reference:
from !== toon every edge. - Provenance always set;
candidatesnon-empty when present, and present only ondynamic-candidateedges. - Profile validity: every entity passes
validateEntity. - Anchors: every edge anchored; spans 1-based, ordered.
- Duplicate ids: identical redeclaration is legal (§1.1); only a disagreement on kind or trait set is an error.
- Determinism: two runs on the same corpus produce identical output.
- Generative side: arbitrary entities from a profile always round-trip JSON → validate → JSON.
Not asserted, deliberately: that to appears in its own candidates list.
§4 calls to the "best candidate", but measured against real Spoon output
(commons-lang: 361 dynamic-candidate edges) 119 correctly EXCLUDE it — those
resolve to an interface or abstract method, which cannot itself run, so the
candidates are the concrete overriders. The model records no abstractness, so
nothing can distinguish a correct exclusion from a mistaken one; re-instating
the rule requires an abstractness fact in the metamodel first.
Cross-validation strategy (later, when a 2nd Java extractor exists, e.g. Tree-sitter): Spoon output is the oracle; property = Tree-sitter edge set ⊆ Spoon edge set; any gap = a missed resolution case.
Motivation (M4 aftermath): extracting apache/fineract produced a 559.5MB
model.json — over Node's string ceiling (0x1fffffe8 ≈ 512MB), so the
analyzer cannot read it at all. Profiling showed ≈76% of the bytes are
repeated strings (edge endpoint ids 186MB, anchor paths 117MB, entity ids +
parents 88MB) with tiny value sets. Design docs, which this phase executes:
docs/model-metamodel.md (MM-1…MM-5, format-independent) and
docs/model-encoding.md (JSONL contract + SQLite cache). Clean break
in place: schemaVersion stays "1.0.0" — the format is entirely
internal for now, there is no external consumer to negotiate with, so the
contract changes under the same version and old files are simply
regenerated. No old-format reader (decision below).
The three milestones are sequenced so every one lands green: M5 adds v2 concepts to core while v1 keeps building; M6 is the breaking change; M7 is additive on top of M6.
✅ Shipped. The wire format is untouched (schemas/ regenerated to a byte-identical
file); what changed is what the vocabulary MEANS, so M6 has something to encode.
- METAMODEL.md amendments: §1.1 restated — identity is the structured key
(lang, module, symbol, disambiguator?), rendered ids display-only (MM-1); §4 extended to nameparent/children(MM-2); §5.1 vocabularies as closed referential sets (MM-3); §5.2 profile validity as a function of(kind, trait set)(MM-4); §8 split into 8a model vs. 8b encodings (MM-5). CLAUDE.md invariants 4 and 7 restated to match. -
core:packages/core/src/identity.ts—NaturalKey(type + Zod schema),renderId,naturalKeyIndex,naturalKeysEqual,compareNaturalKeys/sortByNaturalKey(canonical order),duplicateNaturalKeys,naturalKeyIssues. -
core:validateEntitysplit into a memoized(kind, trait set, isStub)verdict and the per-entity value checks (MM-4). - Property suite: natural-key uniqueness generatively, plus rendering injectivity and canonical order as a total order.
- DoD met: METAMODEL.md v2 merged; core exports the identity vocabulary;
the entire v1 suite still green (1 089 TS tests,
pnpm -r buildclean,./mvnw packageclean,schemas/unchanged).
Decision (M5) — separators are reserved so rendering is injective. A key
whose module contained /, or whose symbol contained #, would render as
some other key's id, and the two entities would silently merge — the M2
archive(List) collision (§4.6) one level up, where it again costs a whole
entity with no error. renderId therefore validates and throws rather than
rendering a lossy id. Pinned by an exhaustive test over an alphabet built from
the separators themselves plus their concatenations: an alphabet of unrelated
words let a deliberately broken renderer pass, which is why the generated
pieces are a, b, a.b, a/b, ab.
Decision (M5) — a module names ITSELF in the module component, with an
empty symbol, rather than referencing its parent module as the design doc's
MM-1 first drafted. The parent form renders java:com.acme.order as
java:com.acme/order, breaking the frozen id scheme, and needs a fabricated
java module to place the stub package java:java.util whose parent no corpus
declares — the fabrication METAMODEL §6 exists to prevent. Consequence for M6:
a package record's m surrogate points at its own row.
Note — MM-4's memo key carries isStub and the profile. Validity is a
function of (kind, trait set) only within one profile and one side of the
stub exemption (§6 waives the lower bound for stubs). Both bounds are pinned by
tests that fail if the key is narrowed.
✅ Shipped.
-
core: record schemasHeaderRec/FileRec/EntityRec/EdgeRec/EofRecdiscriminated ont(wire.ts); streaming codec (jsonl.ts) and file I/O (jsonl-file.ts);childrenkey removed fromEntity(TWithChildrenstays a marker trait); traits ride as int arrays into the header'sdict.traits; everyEntityId-typed field is a surrogate int, driven byENTITY_REFERENCE_KEYSso a new referencing trait cannot be forgotten. -
gen:schemas: one JSON Schema per record type + a GENERATED container contract (schemas/README.md) for what a single line cannot express — section order, dict resolution, trait keys, closure, canonical order. Its trait/key table is printed fromWIRE_TRAITS, so the published rules cannot drift from the code that enforces them. -
extractors/java:NaturalKey+ canonical sort → surrogate assignment → streamingJsonlWriter(one Jackson generator, one line at a time);eoftrailer carries counts. -
Property suite reformulated over surrogates: closure as
ref < entities countchecked as references resolve; truncation detection (every proper prefix must be refused); determinism as byte-identical output whatever order the extractor emitted. -
Fixtures regenerated to
.jsonl; the v1 document path andschemas/model.schema.jsondeleted. -
CLI reads
.jsonlthrough a CHUNKED SYNC reader — the ceiling is one JavaScript string, not sync I/O, so the CLI stays synchronous end to end instead of every command becoming async to buy nothing. -
DoD met, measured:
corpus v1 v2 commands fixtures/java 155KB model.json41KB model.jsonl(3.8×)all apache/commons-lang 17.1MB 6.9MB (2.5×) all, < 1 s each apache/fineract 559.5MB — unreadable 127.4MB (4.4×) all, 11–12 s, ≤ 1.6GB RSS Fineract extracts in 94 s (240 929 entities / 782 032 edges, 0 edges dropped) and
validatereports a clean bill of health. The 127MB is short of the ~90MB the encoding doc estimated — the estimate assumed more of the file was id strings than it is — but the number that mattered was never the ratio: v1 could not be READ at any size, and now it can.Regression check on commons-lang: the v2 model has the same 15 338 entity ids and the same 24 631 edges as the v1 extraction, byte-for-byte identical content modulo the two intended metamodel changes below. The format changed; what the extractor claims did not.
Decision (M6) — closure is enforced at WRITE time, not reported after. A reference is a surrogate into the file's own entity section, so "points at nothing" is not expressible. Consequences, all deliberate:
- the extractor gained a pass 4.5 that DROPS an edge whose endpoint nothing declares, and counts it on stderr (0 on the fixtures, commons-lang and fineract) — v1 wrote such an edge and left the analyzer to report it;
- a dangling reference in a FILE is now a malformed file (a schema error),
not a finding about a valid one.
LoadDiagnostics.danglingReferencessurvives for models built in memory, and the CLI still exits 3 either way; - cross-model references are gone too: each model is closed on its own, and a reference to another language's entity is a STUB that the union merges by natural key. That is the stub discipline of METAMODEL §6, applied to the multi-language case.
Decision (M6) — a stub's MODULE is materialized even when its PARENT is refused. Identity names a module (MM-1), so an entity whose module is not declared cannot be written at all; containment is a separate claim, and §6.1's two refusals (a corpus-declared package, the unnamed package) still stand. The two questions were conflated before because nothing forced them apart.
Bug found by fineract, fixed — the corpus whitelist claimed types the corpus
never wrote. CorpusWhitelist walked every CtType Spoon exposed, including
the SHADOW types noClasspath materializes for unresolvable references
(jakarta.ws.rs.core.MediaType, java.math.BigDecimal, …). The entity pass
correctly refused them — no source position, so no evidence — and the stub pass
then refused to degrade them ("declared by the corpus but never emitted"),
leaving every reference to them dangling. Under v1 those became dangling
references; under v2 the model could not be written. The whitelist now asks the
same "is it written here?" question the entity pass asks, which is what its own
contract already demanded.
Bug found by fineract, fixed — entities.push(...model.entities). A spread
passes one ARGUMENT per element, so the union overflowed the call stack at
240 929 entities and validate reported an internal error. It fails long before
memory does, and no test-sized graph reaches it; packages/analyzer/test/scale.test.ts
now unions a 200 000-entity model.
Bug found by MM-2, fixed — executables never declared themselves containers.
A method's parameters and locals carry parent, but method/constructor/
lambda carried no TWithChildren and no children, so METAMODEL §3.2's "both
stored, must agree" was false for every method with a parameter. v1 could not
see it: the check only looked at entities that HAD a children key. The Java
profile now licenses TWithChildren on the three executable kinds and the
extractor emits it (4 795 entities on commons-lang).
Work in progress. Design settled by a scout-and-judge pass over the code
(4 readers → 3 independent proposals → one synthesis); the recommendation is
cache the parse, not the graph — model.db replaces the JSONL decoder and
nothing above it, so byte-identity is arithmetic rather than vigilance.
Measured on apache/fineract, which is why:
| stage | time | |
|---|---|---|
| decode JSONL | 7.1 s | the dominant cost |
parseModel (Zod over the whole Model) |
2.6 s | redundant — the decoder already validated every record |
validateModel (profile) |
1.3 s | real work |
unknownReferences (closure) |
0.8 s | redundant since M6 — a dangling surrogate is unreadable |
buildGraph |
0.6 s | real |
| fold + coupling + cycles + export | 0.2–0.6 s each | real |
≈93% of a command is decode-and-revalidate; the analysis itself is under a second. Pushing metrics into SQL would optimise the 6% that is not the problem.
Steps landed:
-
Step 1 — split parse from unify (
12c07f6).loadDecodedModelsruns the unify and diagnose halves only;loadModelskeeps its signature asparseModelthen that, for raw JSON and in-memory models. The CLI reads through core's decoder, so it uses the new entry point. Profile validation, closure over the union, self-edges and cross-model redeclaration all still run —load-equivalence.test.tspins that both entry points agree, and catches divergence (verified by mutation). fineractanalyze: 11.50 s → 9.13 s, 1 438 MB → 1 099 MB, all 32 baseline outputs byte-identical. -
Step 2 — freeze the two order contracts, on pre-DB code, because the store is what will break them. -
fold-order-contract.test.ts: no rendering depends on the order edges ARRIVED in. AFoldedEdgeaggregateskinds/provenancesinto Sets, and a Set iterates in insertion order; today that is the decoder's canonical order, and a query planner's order is not. Asserted over the real pipeline at both levels and under a view, for DOT, PlantUML, CSV, JSON, cycles and coupling — and pinned as non-vacuous, since a single-member Set has no order to get wrong. Unsorting any of the four exporters fails it. -fixtures/unicode/+collation-contract.test.ts: canonical order is by UTF-16 CODE UNIT (compareIds), SQLite'sBINARYcollation is by UTF-8 BYTE. They agree across the BMP and invert above it —𠀀…(U+20000) leads with code unitD840but byteF0, whileA…(U+FF21) leads withFF21butEF. Both are legal Java identifier letters, so it is a corpus someone could write. The committed reference report is the artefact every later step is diffed against.**The rule both contracts state: sorting belongs to the model, never to the storage engine.** A query needing canonical order sorts in TypeScript or `ORDER BY`s a column the importer wrote in canonical order. -
Step 3 — expose core's validated wire-record stream.
ModelDecoderdid two jobs: validating the wire, and materializing aModel. They are nowRecordReaderandModelBuilder, andreadModelRecordsSync(path)yields validated records without building anything — the importer writes rows as they go by and never holds the corpus.readModelFileSyncis a consumer of that same stream, which is what makes the two readers ONE reader rather than two that drift.The trait-key rule moved from materialization into the reader, so it is now reported with its line number and holds for every consumer. `record-stream.test.ts` states the property as equivalence: nine broken files — not JSON, not a record, truncated, miscounted eof, dangling edge surrogate, missing trait key, forward module reference, empty, headerless — must be refused by BOTH routes with the identical message on the identical line. Plus the case that motivates the whole step: a truncated file is still a sequence of perfectly good JSON lines, so a hand-rolled `JSON.parse`-per-line importer accepts it happily and stores a corpus silently missing its tail. Measured on fineract (241 101 entities / 782 046 edges), separate processes: **stream 4.6 s at 140 MB peak; materialize 6.8 s at 555 MB.** Validating costs a quarter of the memory of keeping. -
Step 4 — one load site for Node's SQLite builtin.
loadSqlite()inanalyzer/src/store/sqlite.tspatchesprocess.emitWarning, loads viacreateRequire(import.meta.url), restores in afinally, and memoizes. Three independent reasons it cannot be a static import, in increasing severity:1. ESM evaluates every `import` declaration before any statement of the importing module body, so the ExperimentalWarning is already on stderr by the time a patch could exist — and several CLI end-to-end tests assert `stderr === ""` on the built binary. 2. `await import()` loads late enough but is async, and the read path is synchronous end to end; one `await` would reach `main`. 3. esbuild rewrites a static `from "node:sqlite"` to `from "sqlite"` in the tsup output, stripping the `node:` prefix. Harmless for most builtins — `node:fs` and `fs` both resolve — and fatal for this one, which is prefix-only. **Verified by mutation: the built CLI then does not start at all.** The specifier inside `require()` is opaque to the bundler and survives verbatim. The returned `SqliteApi` is written from the store's needs (`exec`, `prepare`, `run`/`get`/`all`/`iterate`, `close`) rather than re-exporting the builtin's types, so PLAN's `better-sqlite3` fallback stays a matter of satisfying one interface at one site. `open()` passes `options ?? {}`: the builtin validates by ARITY, so an explicit `undefined` is not an omitted argument and throws. Two tests, because neither sees the other's failure. `store-sqlite.test.ts` spawns processes — stderr is only observable in one — and asserts the load is silent, that a static import of the same builtin is NOT silent (the control, so the guard cannot die quietly on a future Node), and that `emitWarning` is restored identically. `source-hygiene.test.ts` asserts that exactly one source file in the workspace names the builtin at all, which is the failure a stderr assertion cannot see: a second, equally quiet loader free to drift from the first. Also stated against the real engine for the first time: `ORDER BY` under BINARY collation really does return the unicode fixture's ids in a different order than `compareIds`, and ordering by the surrogate the importer assigned restores it. Step 2 proved that with `Buffer.compare`, a model of SQLite; this proves it with SQLite. -
Step 5 — freeze the DDL, against M6 rather than the sketch.
analyzer/src/store/schema.tsholds the schema, the open options, themetakey set, and the key→storage mapping. Four things the M6 wire changed about docs/model-encoding.md's earlier sketch:1. `module_id INTEGER **NOT NULL**` — `m` is required on every entity record and a module names itself, so the sketch's "NULL for root modules" case no longer exists. 2. `entity_trait(entity_id, trait_id)` → an interned `trait_set`. Measured: **15 338 entities / 15 distinct trait sets** on commons-lang, **241 101 / 18** on fineract, so the join table would carry ~1.3M rows to say 18 things. Size is the smaller half — MM-4 makes profile validity a function of `(kind, trait set)` and core's validator already memoizes on that key, so the interned id IS the key. `ord` is kept so an entity's `tr` array re-encodes exactly. 3. **No UNIQUE natural key and no `CHECK (to_id <> from_id)`.** Duplicate identities and self-edges are conformance FINDINGS; a store that refused to cache them would make `import` fail on exactly the models `validate` exists to report on. Same rule as step 3: two gates on one format drift. 4. Array-valued keys get ordered tables (`entity_comment`, `entity_defined_in`, `entity_parameter`, `entity_local_variable`, `edge_candidate`) rather than JSON, so they stay joinable; `extra` holds only keys `core` does not type, since records are loose by design. Two things the implementation turned up that reasoning had not. **Foreign keys are a decision, not a default:** SQLite's default is OFF and Node's builtin overrides it to ON, so `STORE_OPEN_OPTIONS` states it — off, because `parent`/`declaredType` legitimately point forward and immediate constraints would reject valid models, while closure is already the reader's guarantee. `PRAGMA foreign_key_check` still audits a suspect cache, which `foreign_keys = ON` would not. And one addition to `core`: **MM-3 says a vocabulary is a SET**, so `RecordReader` now refuses a header dictionary that repeats an entry — a repeat makes two indices name one thing, which any interning store breaks on. It is a property of the file, so it belongs in the reader; stated there, both read routes refuse it identically. `store-schema.test.ts` (17) asserts everything against a REAL database created from the DDL, never against its source text: a `sqlite_master` snapshot of the effective schema; the invariant-4 guard (`edge_to` and `entity_parent` are indexes, no table name reads as an inverse); the "caches what the reader accepts" rule as behaviour, by inserting a duplicate natural key and a self-edge; and the drift guard — every key `WIRE_TRAITS` and the record schemas can carry maps to a real column of a real table, so adding a trait key to `core` fails the store until someone decides where it goes. Five mutations, each caught by the intended test — and S3 (making the natural key UNIQUE) exposed a defect in the test itself: the duplicate it inserted differed only by a NULL disambiguator, and SQLite treats NULLs as distinct in a unique index, so it would have passed either way. Now the duplicate carries a disambiguator, with the NULL case asserted alongside.
Remaining steps (from the synthesis, unchanged):
Role split (encoding doc §1): .jsonl is the CONTRACT — schema-validated,
diffable, language-agnostic; model.db is the WORKBENCH — a derived,
disposable cache, regenerable at any time, never committed, never the
interchange.
-
Step 6 —
importModel(jsonlPath) → model.db, and its exact inverse. Driven byreadModelRecordsSync, one transaction, prepared statements; the corpus is never held.readStoreRecords(db)yields the wire records back out, so losslessness is an EQUALITY rather than a checklist:[...readStoreRecords(db)]must equal[...readModelRecordsSync(path)], record for record. 1 029 842 records identical on fineract.hydrateModelis that stream fed to core's ownModelBuilder— the same onereadModelFileSyncuses, so a model from the cache cannot diverge from one off disk.The acid test earned its name on the first run, finding two real bugs of one kind: **an empty array and an absent key are the same rows.** `definedIn: []` writes nothing to `entity_defined_in`, exactly like an entity with no `TModule` — and the fixture has a stub module that proves it. Presence now comes from the TRAIT SET, which is where the wire keeps it (`TComment` contributes `comments`, so carrying the trait IS carrying the key). `candidates` is the one array-valued key no trait contributes, so `edge.candidate_count` records its presence: NULL absent, 0 empty. Measured on fineract (241 101 entities / 782 046 edges): | | time | peak RSS | |---|---|---| | import (once) | 9.0 s | 289 MB | | read the model from `.jsonl` | 6.4 s | 701 MB | | hydrate the model from `.db` | 3.6 s | 592 MB | | fan-in top 20, in SQL | 0.04 s | — | **The finding that redirects step 7: the win is not hydrating faster, it is not hydrating at all.** Full hydration was first measured at 7.6s — SLOWER than reading the text — and only beats it after switching the bulk queries to `setReturnArrays(true)` (1 023 147 rows: 3.83s as objects, 1.46s as arrays). Even at 3.6s that is a 1.8× constant, where the same question asked in SQL is 0.04s against 6.4s. A DB-backed facade that begins by rebuilding the whole `Model` gives up the actual win. Two smaller decisions, both measured: indexes are created AFTER the inserts (10.75s → 9.0s, 178.4MB → 169.8MB), and atomicity comes from a rename rather than from the journal, which is what makes `journal_mode = OFF` safe. Four mutations. Dropping `space`, inferring list presence from rows, and ordering entities by `symbol` instead of by surrogate were all caught. The fourth — writing straight to the target with no temp file — was NOT, because deleting the target on failure also "leaves no database". The property that separates them is the one users feel: a failed re-import must not cost you the cache you had. Now asserted. -
Step 7 — fold in SQL.
foldFromStore(db, options)answers in SQL whatfoldGraphanswers in JavaScript. Fold is the FUNNEL — coupling, cycles, DOT, PlantUML, CSV and JSON all consume aFoldedGraphand look at nothing else — so this is the one place worth moving, and everything downstream gets it unchanged.Measured on fineract, all eight (level × view) combinations in parity: | | in memory | from the store | |---|---|---| | prepare (build graph / open) | 9.0 s | ~0 s | | fold, module level | 0.4 s | 1.4 s | | fold, type level | 0.5–1.0 s | 1.5–1.6 s | | **total** | **9.4 s** | **1.4 s** | Stated honestly: the SQL fold is SLOWER per call — the win is entirely in not needing the 9s graph build, so a caller that folded many times would amortize the other way. Each CLI invocation folds once. The 782 046 base edges never enter JavaScript; only the ~20 000 aggregated ones do. Containers are resolved by a FIXPOINT over `parent_id`, not a recursive CTE: measured 0.38s against 1.06s, because the CTE re-walks each chain from every descendant. The loop ends when a pass adds nothing, so a malformed parent cycle terminates — its members just never resolve, which is what the in-memory walk also does. **It refuses rather than approximates.** A view is an arbitrary predicate pair; SQL cannot run one. `translateView` translates the ones a view NAMES through `ViewDescriptor.filters` (`internalOnly`, `declaredOnly`, `provenance:…`) and returns `undefined` for anything else, so the caller hydrates. A facade that quietly answered a slightly different question would be worse than none: the numbers would still look like numbers. `assembleFoldedGraph` was extracted from `foldGraph` so both producers sort, index and freeze through one function — the step-3 rule applied to a second producer. `store-fold.test.ts` (22) compares whole graphs — nodes, edges, counts, kind/provenance sets, diagnostics — for both levels × five views, plus self-loop dropping, edge-kind filtering, and the unicode fixture. Then it proves the payoff rather than assuming it: DOT, PlantUML, CSV, JSON, coupling and cycles byte-identical from either fold. Four mutations; two exposed gaps rather than confirming coverage. Silently accepting an untranslatable view, and miscounting `droppedEdges`, were caught. Restricting the container walk to view-included entities was caught only after being rewritten to break chains DURING resolution — the first version deleted already-resolved rows and was a no-op. And deciding "is this its own module?" by comparing SYMBOLS instead of surrogates passed everything: no fixture had a type whose symbol equals its module's path. That case now exists, because getting it wrong renders a class as its own package — two entities, one id. -
Step 8 — diagnose from the store.
diagnoseStore(db)returns theLoadDiagnosticsloadDecodedModelsreturns, computed by query rather than by walking entities. The store is not asked to REMEMBER a diagnosis — a cached verdict inside a cache is a second thing to invalidate — it is asked to answer one.**MM-4 is what makes it cheap.** Profile validity is a function of `(kind, trait set, isStub)`, and the store interns trait sets, so `SELECT DISTINCT kind_id, trait_set_id, is_stub` is 23 rows over fineract's 241 101 entities. Each verdict comes from `entityCompositionVerdict` — now exported from core, the same memoized function `validateEntity` calls — and a verdict with no issues needs no entities at all, so a clean corpus enumerates nothing. `edge-kind-not- allowed` is likewise a function of six values, not 782 046 edges. Measured on fineract: **0.46 s from the store against 2.10 s in memory, and that 2.10 s needs a 6.21 s read first** — 8.3 s → 0.46 s. One check is skipped and says so: the per-entity Zod parse of each declared trait's keys, which is the bulk of the in-memory cost. It cannot fail on a model that reached a store, because the record reader enforced `WIRE_TRAITS` at import. That is an argument, so it is a test — every trait's key set is pinned to match between `TRAITS` and `WIRE_TRAITS`, and a value the model schema rejects is shown to be refused on the wire. Self-edges are the one diagnostic needing whole `Edge` objects, so when one exists the model is hydrated and core's `selfReferences` answers. A self-edge means the corpus is already broken; paying a hydrate there costs nothing on every correct run, and beats a second edge constructor drifting from `ModelBuilder`'s. **Both real corpora are clean, so parity on them proves almost nothing** — returning "no issues" unconditionally would pass. Every category is therefore exercised on a model broken on purpose, each asserting the issue exists before asserting the two paths agree about it. Four mutations; two exposed gaps, and building the case for a third found a real bug. Comparing interned `trait_set_id`s instead of the canonical trait SET was caught (two declarations differing only in trait ORDER are the same declaration). Dropping the emission order was NOT — until a model existed with issues from two categories landing in the opposite order. Constructing it surfaced the bug: `isStubEntity` is "declares TType or TModule AND isStub is true", and reading the `is_stub` column alone calls an entity a stub that the in-memory path does not — which flips the composition verdict, since the trait lower bound is waived for stubs. And the unknown-profile branch was returning no `duplicateIds`, where the in-memory path computes them regardless of profile; the test passed only because no fixture had a duplicate. -
Step 9 —
codegraph import. The explicit command:codegraph import <model.jsonl...> [--out FILE] [--json]. It is unconditional — the user asked for a store, so they get a fresh one, and nothing here guesses whether the old one was still good. That question belongs to the automatic cache, which has to answer it unasked.**One model per store.** Every other command unions its positionals; this one deliberately does not, because surrogates are file-scoped and are not identity (MM-1), so a union would mean renumbering — and a renumbered corpus is a repointed one. Several paths mean several imports, and `--out` with more than one is a usage error rather than a silent choice. Two decisions the CLI's own contracts forced: - **stdout says what is true of the STORE; stderr says what was true of the RUN.** SQLite promises no byte-determinism (page allocation varies) and a duration never could, so the size and the timing must not reach the stream `e2e-determinism` compares. - **A file that is not a model is a FINDING, not a crash.** Left unhandled, the record reader's `JsonlError` escaped to `main` and was reported as "an internal error — a bug in codegraph", blaming the tool for the user's file. It now exits 3 with the reader's own message, and one run reports every bad path while still importing the good ones — the same choice `loadModelFiles` makes for the reading commands. Exit 3 also covers a model that reads fine but breaks its profile: the store IS written and usable (refusing to cache what `validate` exists to describe would be the second gate this project keeps declining to add), and the exit code still says something is wrong. Step 8 makes that check affordable — `diagnoseStore` on fineract costs ~0.5s, not 8s. Also fixed: the 20 000-entity round-trip from step 6 was FLAKY. It takes ~0.4s alone but exceeded vitest's 5s default under the contention of `pnpm -r test`, where several packages' workers share one machine. A test that fails on a busy CI box and passes on a quiet one teaches nobody anything, so its budget is now stated. -
Step 10 — the automatic cache.
analyzeandexportgiven a.jsonlbuild the sibling.dbif they must, reuse it if they can, and answer from it.--no-cachereads the model directly.Measured on fineract, `analyze --report deps`: | | wall | peak RSS | |---|---|---| | first run (builds the store) | 14.9 s | 268 MB | | **second run (reuses it)** | **1.9 s** | **163 MB** | | `--no-cache` | 11.8 s | 1 094 MB | **6.3× faster and 6.7× less memory**, and the 19 090-line report is byte-identical to the one the model produces. `AnalysisSource` (cli/src/source.ts) is the seam: neither command branches on where the answer came from, so there is no second formatting path to keep in step. The cache is used only when it can be EXACT — one model (a store holds one; a union would mean renumbering), no `--no-cache`, a writable store, and a view SQL can translate. Every other case falls back to what was there before, so the worst case is the old speed, never a different number. `importGraphFromStore` was added so `deps --level module` — the most-used report and the one cross-language layer — is not the single command that has to hydrate. Three things this found, two of them real bugs: 1. **`open: false` does not mean "do not create".** In Node's SQLite it means "construct the handle but defer opening", so every freshness query threw, every store read as unreadable, and **the cache rebuilt itself on every run while looking like it worked.** Only the determinism suite noticed — via the elapsed time it printed. 2. **A model the reader refuses was crashing.** `openCache` rethrew `JsonlError`, which reached `main` as "an internal error — a bug in codegraph", blaming the tool for the user's file. A refused model is now simply a reason to have no cache; the reading path reports it as a finding, with the reader's own message and exit 3. 3. **stderr is a diffed stream.** `e2e-determinism` says so and explains why — an elapsed-time figure turns every CI log into a false diff — so the cache note is one line, `cache: <path>`, identical whether the run built the store or reused it. Whether *this* invocation paid for the import is a performance detail, and `codegraph import` is the command that reports it. Staleness is size and mtime, not a content hash: hashing 133 MB costs more than the parse the cache exists to avoid, which would make the check more expensive than the miss. The trade is stated rather than hidden, and both halves are tested — a changed model AND a model rewritten to the same size with a later timestamp. `*.db` is gitignored: the store is derived and disposable, and a binary beside the `.jsonl` in git would be a second, undiffable answer to the same question. -
Step 11 — the SQL cookbook.
docs/sql-cookbook.md: orientation, two setup views that turn surrogates into rendered ids, and recipes for fan-in, fan-out, facts-only, traits, modules, the import graph, reachability, module cycles, evidence and uncertain dispatch — plus the four rules a query must respect and the list of questions to askcodegraphinstead.**The cookbook is executable.** `sql-cookbook.test.ts` extracts every ```sql block and runs it in document order against a store built from the committed fixture, because a recipe that has rotted still LOOKS like SQL and the reader finds out at a prompt rather than in review. Three claims the document makes about agreeing with `codegraph` are checked as equalities, not for syntax: the `node` view reproduces `renderId` for all 167 ids in order, the fan-in recipe counts what the model's edges count, and the cycle recipe agrees about cycles. Two mutations. Breaking a recipe fails three tests. **Rendering the self-module test by SYMBOL instead of surrogate did not** — the same hazard that survived step 7's parity suite, now living in the documentation, where it would have taught the wrong idiom. The namesake corpus is now built here too. Written while checking rather than asserting: `module_id` and the fold's container walk agree for every entity that resolves on both corpora (0 disagreements), but they differ where the walk ends with NO container and `module_id` always points somewhere — the fixture has 2 such entities. The document says so rather than implying they are the same question.
Every clause verified on apache/fineract (241 101 entities / 782 046 edges), not asserted.
| DoD clause | evidence |
|---|---|
a second analyze opens the cache without re-parsing |
cold 10.9 s / 241 MB → warm 1.96 s / 114 MB; --no-cache 9.8 s / 1 053 MB |
| every report byte-identical to its JSONL-only output | 12 of 12, cmp-verified: 8 analyze variants and 4 export formats, 486 000+ lines including deps --level type at 124 266 lines and export --format json at 134 077 |
| ad-hoc SQL cookbook documented | docs/sql-cookbook.md, every query executed by a test |
The cache is 5× faster and 9× lighter on the report that motivated M7, and
the whole M6 → M7 arc takes analyze on fineract from 11.5 s / 1 438 MB to
1.96 s / 114 MB — 5.9× faster on 12.6× less memory, with the answer
unchanged to the byte.
What the eleven steps actually bought, in order of how much it mattered:
- The wire became streamable (steps 1–3, 6). A 559.5 MB v1 model was unreadable at all — past Node's single-string ceiling. It is 133.7 MB of JSONL now, and validating it costs 140 MB of memory where materializing it costs 555 MB.
- The fold moved into SQL (step 7). Fold is the funnel every report and export goes through, so moving that one thing moved all of them, and the 782 046 base edges never enter JavaScript.
- MM-4 made validation nearly free (step 8). Profile validity is a
function of
(kind, trait set, isStub)and the store interns trait sets, so 23 rows decide a 241 101-entity corpus: 2.1 s → 0.46 s. - The store answers, it does not remember (steps 5, 8). No cached verdicts, no second representation to invalidate — every diagnostic is a query.
And what the process bought, which is most of why the numbers are trustworthy:
of roughly two dozen mutations, five exposed gaps rather than confirming
coverage, and three of those found real bugs that no test had asked about —
open: false silently rebuilding the cache on every run, an empty array
indistinguishable from an absent key, and is_stub read without the trait that
gives it meaning. The two that recur are worth naming: a test that passes
because the fixture has no instance of the case (the namesake type, the
duplicate under an unknown profile), and a mutation written so symmetrically
that it cannot be detected by construction.
- C# via Roslyn — planned in full as Phase 10 (§13, M12): the second real extractor, first-class on Linux/macOS, shipped as one binary per OS.
- Clojure via
clj-kondo --analysis→ thin JSON adapter (near-free; first cross-language test on the Import layer; exercises the fn-var case for real). - TypeScript via the TS compiler API — planned in full as Phase 11
(§14, M13): self-hosting (run codegraph on codegraph),
space: type|value, declaration merging, and thenamespace-style legacy corpus. - SCIP adapter (one effort → Rust + TS + Python via existing indexers).
- Code city visualization (
packages/viz, Three.js + Vite): render the analyzed model as a 3D city — districts = modules, buildings = types (height/footprint/color mapped to documented metrics), edges as flows. Consumes analyzer output only; visual-language rules live inCLAUDE.md.
Documented static limits (all languages, per profile notes): reflection,
Class.forName/Spring XML, service loaders, pre-expansion macro code.
Motivation: model.jsonl is a snapshot — it says what the code IS, not what
happened to it. The crime-scene analyses (hotspots, logical coupling,
knowledge maps — Tornhill, Your Code as a Crime Scene) and a Gource-style
replay of the city both need the time axis. Two prior decisions make this
phase cheaper than it looks: identity is the natural key (invariant 7), so
"the same entity across two snapshots" is key equality — no diff heuristics
for named entities; and Spoon runs noClasspath, so historic commits that no
longer compile still extract.
Three principles, locked up front:
- Evolution facts are a third artifact.
history.jsonlsits besidemodel.jsonl: repo-scoped, language-agnostic, and changing on every commit while structure does not. It never merges into the model file and never grows a fifth provenance value — code facts and history inferences stay unmixable. The join happens in the analyzer, onanchor.file/TModule.definedInpaths relative to the analyzed root. - The miner has no code intelligence — the extractor rule, mirrored.
It emits paths, authors, timestamps, and line deltas from one
git logpass; everything smarter is derived downstream. - Lineage v1 is the natural key, exactly. A renamed symbol is a death
plus a birth; anonymous entities (
file:line:columndisambiguators shift under any edit above them) are not tracked over time. Both restrictions are documented, not silent.
Everything Gource shows comes from git log alone — so the replay
experience ships before any extractor touches the time axis, and the one
genuinely hard viz problem (layout stability) is forced on cheap data.
-
codegraph scm <repo> [--since <date>] --out history.jsonl(scm, notgit: the door stays open for hg/fossil): onegit log --numstat --no-merges --find-renames -zsubprocess, parsed by pure@codegraph/scm. M6 discipline reused: header dictionary (author table), interned path table, surrogate ints, sorted deterministic output (encoder canonicalizes),eofcount trailer. - Two record types:
commit(hash, author ref, timestamp,isFix/isRevertsubject-regex flags — labeled heuristic) andchange(commit ref, path ref, added, deleted, rename-from). Rename chains are resolved at mine time so one path surrogate names one file lineage — unresolved renames corrupt every downstream metric. (Documented approximation: a path recreated after a deletion continues the same lineage — lineages are named by paths.) - Reports, file-level only, no model join yet:
codegraph history --report summary|hotspots|authors— churn, bus factor, bug density, momentum, firefighting frequency. - File-level city replay: buildings = files (height = running LOC sum
of numstat deltas), districts = directories, timeline scrubber in
viz, commits as ticks. (
codegraph history --serve/--city FILE: a laid-out city artifact with areplayblock — frozen union layout and keyframe height series, M9c's shape adopted early.) - Fixture: a scripted git repo built by the test suite in a temp dir
(two authors, a rename, a
fix:commit, a deletion) — deterministic, and it exercises the rename chain. - DoD: miner output byte-identical across runs on the scripted repo; summary reports match hand-counted fixture numbers; file-level replay runs end to end on codegraph's own history.
- Sampled snapshots: extract at K chosen revisions (tags/releases, or
every N commits — 50–200 frames suffice for replay) via
git worktree add, never mutating the main checkout. (codegraph snapshots [repo] --jar F (--every N | --tags) [--store S] [--src DIR]: throwaway worktree per frame,java -jarextraction,import --atappend, per-frame diagnose. RESUMABLE — revisions the store holds are skipped, so an interrupted run continues and a moved source root is handled by composing runs with different--src; a frame that fails to extract is reported and isolated, never fatal.) -
codegraph import --at <sha> [--time <t>]extendsmodel.db(M7) withrevision,entity_key(interned natural keys), andentity_version/edge_versiontables (names as TEXT — dictionary ids are file-scoped). The flat tables mirror the latest import; the cache NEVER regenerates a store holding revisions (statetemporal— it stands aside and the model is read directly). Lifespans (appeared,disappeared) are derived per key at query time — inverse indexes over the time axis, never serialized (invariant 4 applied to time). - Cross-graph queries — the ones only a tool holding BOTH graphs can
ask: hidden coupling (co-change pairs with no path — transitive,
either direction — in the declared graph) and dead weight
(declared file dependencies that never co-change), via
codegraph history --report hidden|deadweight --model M; logical coupling with support/confidence thresholds and a changeset-size cap (--report coupling,--min-support,--min-confidence). The join is by path SUFFIX (model roots sit below repo roots); ambiguous suffixes join nothing and are counted. Ownership map and truck factor shipped file-level in M9a (--report authors); code age and revisions×LOC enrichment remain open. -
codegraph timeline <id>: first/last revision containing the key, LOC series between them, still-present flag. (Exact birth commit via targeted bisect — extract one file at ~log₂ N revisions — is still open; a query-time feature, not an ingest-time cost.) - DoD: property suite (closure, profile validity, determinism) green
at every keyframe unconditionally; timeline and coupling queries
verified on a real corpus history (google/gson). ✅ Verified 2026-08-25:
all 55 gson release tags (2008–2025) in one store via two composed
snapshots --tagsruns (--src src/main/javafor the pre-2.4 layout,--src gson/src/main/javaafter the module move — resume skipping makes the composition free), zero findings at every keyframe;timeline Gsonspans 55/55 from 1.0,MappedObjectConstructordies after gson-1.7.2 (the 2.0 rewrite),Excluderis born at gson-2.1; hidden coupling finds the JsonSerializer/JsonDeserializer and Since/Until twins (no declared path), dead weight finds the stable JsonToken/TypeAdapter interfaces.
- Layout computed ONCE on the union of every key that ever existed;
every plot frozen. Buildings animate in place — rise from zero at
birth, sink at death; land is vacant before its time. Early sparse
frames are the honest picture of a city that will grow, not a
defect. (
buildEntityCity: types are buildings, modules are FLAT districts — nesting a package hierarchy from its name would be an inference; presence GAPS become explicit 0 keyframes so a rebirth scrubs honestly; members and anonymous types raise no building.) - One temporal
city.json: per building{plot, birth, death, series: [{t, height, heat, …}]}— viz interpolates between keyframes and still renders a laid-out artifact only. (codegraph replay [--store S] [--out F] [--serve]: analyzerreadEntityHistory→ citybuildEntityCity→ the M9a replay block withclock: "revisions"; metrics carried per building: loc, peak-loc, revisions, born. Verified on gson's 55 release keyframes: rev 1/55 is one sparse district of 2008, dead types leave vacant plots, JsonReader born at gson-1.6.) - Change heat as color + age as desaturation, documented like every
other channel: keyframes carry
heat(1 = changed at that tick, pre-decayed byREPLAY_HEAT_DECAYwhen a sampled revision saw the entity unchanged); the renderer keeps cooling between keyframes, ages toward gray by timeline fraction lived, and legends both — a 'Time colors' toggle (replay artifacts only) restores the plain palette. Verified on gson: 2.14.0's touched core glows ember, one-release-old changes read brick, the untouched old core grays. - Ownership as a toggleable color mode; co-change arcs visually
distinct from declared edges (they are inferences, and the city
never lies). Fed by the history join:
codegraph replay --history F(andhistory --servefor the file city) runs scm'sfileOwners+logicalCouplingand the analyzer's SUFFIX join — buildings gainowner {name, share}, the replay block gainscoChangebuilding pairs. Viz: a 'Colors' selector (Time / Owner / Plain — Owner offered only when a history was joined; unowned buildings go neutral, never a claimed hue), owner in the details panel, and co-change as DASHED magenta arcs shown for the selected building — a different KIND of line from dependency arrows, legend and help stating the inference. - DoD: replay on a real corpus reviewed as screenshots at user-facing camera angles; no per-frame allocation in the scrub path. ✅ Verified 2026-08-25 on gson (55 release keyframes + 2,088-commit history, 170 of 197 store files joined): sparse 2008 city grows to 2026; time colors read ember/brick/gray at 2.14.0 and 2.9.1; owner mode maps the original-author core against the modern maintainers; TypeAdapters shows owner (29% of added lines) and its dashed co-change fan. Scrub path allocates nothing: caller-owned Float32Array buffers, in-place instance matrix/color rewrites, event-driven repaints only.
- Per-commit incremental extraction (keyframes + deltas): Spoon's resolution is corpus-wide — the stub whitelist depends on all corpus-declared ids — so between-keyframe models are approximate. Build only if sampling proves too coarse; keyframes bound the staleness.
- Symbol rename lineage (same-parent + similar-body matching).
- Lambda/anonymous-entity tracking over time.
.mailmapauthor normalization (start with email identity; add it when it bites).
Motivation: the model says what the code is (structure) and what happened
to it (Phase 8). This phase makes it say where it lives (a permalink for
every anchor), how big and how branchy it is (measured, not guessed),
what it says verbatim (the constant values the source writes — annotation
arguments, constant initializers, defaults), and what frameworks do to it
(the DI wiring the static graph cannot see). All four land on hooks the
codebase already carries: anchors are repo-projectable the moment the header
knows the repo; the city's metric registry was written for extractor-emitted
measures (attribute:/sum: sources, metrics.ts); annotation usage is
already an edge and Spring wiring is already a documented blind spot in the
Java profile notes.
Four principles, locked up front:
- Facts in the model, projections downstream. The header gains the
repository facts (
remote,commit, repo-relativeroot); the blob-URL template of a particular host is presentation, derived by consumers — a stored URL would freeze one host's scheme into the interchange (METAMODEL §8a/§9). - Only the extractor measures.
slocandcyclomaticrequire reading source; nothing downstream may re-parse or approximate them. Gross span length stays a derived proxy (the city'sloc), and the two claims are never conflated (METAMODEL §3.8). - A value is what is written, never runtime state. A
Literal(METAMODEL §1.6) is emitted only when the language fixes it at the declaration — a literal, or an expression folding from constants; anything else isunevaluated(source text kept) or absent. Ids inside values obey closure like edge endpoints. - Framework knowledge is data, like profiles. The analyzer derives DI wiring from declared facts plus a declarative annotation table (METAMODEL §9.1); the extractor stays framework-blind, and derived wiring edges are in-memory only — invariant 4 applied to inference.
- Core: optional
repository {remote, commit, root, provider?}on the header record (Model, wire, container contract),pnpm run gen:schemascommitted with it. Additive —schemaVersionunchanged (M6 decision: the format is internal, bumping a version nobody consumes buys nothing).rootis the analyzed root relative to the repo root — the M9b gson gotcha (--src gson/src/main/java) resurfacing as a data requirement; without it no anchor projects back to a repo path. Every field is a PATTERN, not a refinement:z.toJSONSchema()drops refinements, and the published schema has to refuse an ssh remote or a../root exactly as core does, orschemas/is not the contract it claims to be. - Java extractor: passthrough flags (
--repo-remote,--repo-commit,--repo-root,--repo-provider) copied verbatim into the header — no git knowledge, no metamodel intelligence, one writer per file. The flags travel together (a remote without a sha links nowhere) and are checked against the published patterns at the flag, not written into a header the analyzer would then refuse whole. - CLI derives the values: normalize
git remote get-url origin(ssh,ssh://,git://, embedded credentials → https, strip.git);snapshotssupplies the per-frame sha it already checks out and the--srcprefix, which is already repo-relative. A remote no browser can open (local path,file://, plain http) yields NOTHING and says so — a guessed URL is a link that lies. - Analyzer/city carry
repositoryinto artifact metadata (storemetarow;readEntityHistoryexposes it beside the revisions); viz details panel renders a "view source" link from the selected entity's anchor — GitHub{remote}/blob/{commit}/{root}/{file}#L{s}-L{e}, GitLab{remote}/-/blob/{commit}/{root}/{file}#L{s}-{e}; provider guessed from hostname,providerfield overriding (self-hosted GitLab). An unknown host template renders no link. In a replay the commit is the SCRUBBED tick's sha, retargeted in place so playback churns no DOM. - DoD ✅ verified on gson at
b3f4ca2: the JsonReader link openedgson/src/main/java/com/google/gson/stream/JsonReader.java#L211-L2005on github.com at the extraction sha (screenshot reviewed); the fixture city, whose model carries no block, renders no link at all; a 4-revision gson replay store retargeted the link toed2b25d(2011) when scrubbed to revision 2/4, and that URL resolves too.
- Core:
TMetricstrait contributingmetrics: Record<string, number>(finite values validated — Zod v4'sz.number()rejects NaN/Infinity by construction; keys deliberately open — METAMODEL §3.8), optional on the Java profile's measurable kinds (types + invocables); header trait dictionary grows;gen:schemascommitted. The encoder writes the map KEY-SORTED: canonical order (MM-5) reaches inside the record, or two runs that measured the same thing would differ in bytes. - Java extractor emits
slocper type and invocable (span lines minus blank and comment-only lines) andcyclomaticper invocable: 1 + count ofCtIf,CtFor/CtForEach/CtWhile/CtDo, non-defaultCtCase(one per case expression),CtCatch,CtConditional,CtBinaryOperatorAND/OR, switch-pattern guards. A lambda's branches count toward the lambda — it is its own invocable. Purely syntactic: immune to the noClasspath resolution ceiling.slocruns a small LEXER, not a regex: a"/*"inside a string literal would otherwise open a block comment that never closes and silently blank the rest of the file. - Store: measures are ROWS (
entity_metric), not a JSON blob — the cache exists to be queried and a measure is what one aggregates; presence follows the trait set like every other trait key, sometrics: {}survives.DB_VERSION2 → 3. - City:
numericKeyreads themetricsmap (top-level loose keys stay legal but uncontractual, and the map wins), so--height sum:cyclomaticandattribute:slocwork exactly asmetrics.tspromised. - Fixture: hand-counted
cyclomaticvalues asserted on the fixture corpus (Reporting.max4 = for + if +||,join2,first2 ternary,today1); the constructs the corpus does not contain — switch labels, pattern guards, multi-catch, lambda-in-method — are hand-counted over sources written for them in the extractor'sMeasuresTest. Properties: every measure finite and non-negative,sloc ≤ span, a stub carries none. - DoD ✅ verified on apache/commons-lang at
e073d5d(15 381 entities, 5 200 measured, 9 790 total cyclomatic over 4 809 invocables):--height sum:cyclomatic --footprint locreviewed as screenshots at user-facing angles — ArrayUtils (1 099) and StringUtils (904) tower over a mid-rise of StrBuilder/Conversion/TypeUtils; the 246 unmeasured buildings (239 of them stubs) sit at the channel minimum 1, counted in diagnostics and in the legend, never at zero.sloc ≤ spanheld on all 5 200; a hand-count ofJavaVersion.get(1 + 28 case labels + 4 ifs) matched the emitted 33 exactly.
- Core:
Literalprimitive (METAMODEL §1.6) — tagged union: string, number (canonical decimal text — a Javalongdoes not fit a JSON number), boolean, null, enum (type id + simple name — a member stub is never fabricated, §6 verbatim), type, array, nested annotation, andunevaluated(source text of a constant expression the extractor did not fold).TWithValue { value: Literal }optional onattributeandmethod(annotation element defaults) in the Java profile; new edge kindannotationUse— Entity → annotation Type,arguments: {name, value}[], the implicitvalue =normalized explicit. Header dictionaries grew (one trait, one edge kind);gen:schemascommitted — the recursive union publishes as a NAMED$defs/Literal, so the contract states the tree once and both record schemas reference it. - Encoding: ids inside Literals are file-scoped surrogates, so closure
over values is enforced by the wire exactly as for edge endpoints — a
value pointing at an undeclared entity has no surrogate to name it with
and is unwritable. Property suite extended: the generators build values
from the same id pool as every other reference, so closure over values
is exercised by construction; determinism untouched — argument and array
order is written order, a source fact like parameter order. An EMPTY
argument list is omitted on the wire (the edge kind vouches for the key)
and restored on read, so
@Overridecosts no bytes. - Store:
entity.valueandedge.argumentsare JSON columns — a Literal is a tree, nothing queries inside one, and the ids it holds are the wire's own surrogates, so the blob round-trips exactly.DB_VERSION3 → 4. - Java extractor:
emitAnnotationsupgraded fromreferencetoannotationUsewith arguments (Spoon annotation values + partial evaluator; chars ride as one-character strings; a class literal in an argument still emits its ownreferenceedge — a value never replaces a dependency). Constantattributeinitializers (JLS compile-time constant expressions) and annotation elementdefaults carried viaTWithValue. A field that isfinalbut whose initializer is CODE (new StringBuilder()) carries nothing: absence means "not constant". In-place clean break for the usage edge kind — fixtures regenerated, no dual emission (the M6 precedent). - Analyzer and city: values pass through untouched on load; the details panel shows a constant beside its field, ids rendered as the referenced entity's NAME — presentation, no new derivation.
- DoD ✅ the fixture asserts
@Retention(RetentionPolicy.RUNTIME)onAuditedas anannotationUseedge whose argument is the enum form,value() default ""as aTWithValuestring,MAX_LINES = 4 * 25folded to100, and one honestunevaluated(@Audited(LedgerClient.AUDIT_TAG), a constant noClasspath cannot resolve); property suite green including value-closure; regenerated gson (63 valued entities, 599 annotation uses, 122 with arguments) and commons-lang (458 valued, 1 164 uses) diagnose clean. The discipline is visible on real code:SAFE_MAX_ARRAY_LENGTH(static final) states2147483639, whileSOFT_MAX_ARRAY_LENGTH— the sameInteger.MAX_VALUE - 8initializer, but notfinal— states nothing.
- Framework profile as data in the analyzer (METAMODEL §9.1): annotation
identity → role (
stereotype,injection-point,entry-point,qualifier, andprimary— the one role this list did not name, since@Primarynarrows by PRESENCE on the producer rather than qualifying the injection point; adding it was a row, not code, which is the property the table exists to have). Specifiable without being implemented — the profile robustness test, again: a Micronaut table validates with nothing behind it. - Analyzer DI pass: for each injection point (field / constructor param on
a stereotyped class), derive in-memory
dynamic-candidateedges from the consumer to every corpus implementation of the declared interface — candidates straight from the existinginterfaceImplementationinverse index;@Primarynarrows on presence,@Qualifieron itsannotationUseargument value (M10c) against the bean's names (its stereotype's string argument, or Spring's decapitalized default) — exact strings, not guesswork. Narrowing is never silent: the point says what it narrowed FROM. Injection points and roles are selected onannotationUseedges directly; matching reads the referenced annotation entity'sname+ its module, never a parsed id, and tolerates stub targets (the petclinic case: every Spring type is a stub).implicitSoleConstructorInjectionis profile DATA, so Spring 4.3+ constructor injection with no annotation at all is described by a field rather than by a branch. - Stereotype classification surfaced as a report (
analyze --report wiring, text and--jsonunder the same envelope) and as a semantic city color channel (codegraph city --framework spring→Building.roleplus aroleslegend block; viz gains aRolecolor mode, legended like every channel) — derivable fromannotationUseedges (M10c), no further model change. - DoD ✅ hand-verified on spring-petclinic at
a6e81a5(the last revision with@Autowired): all 6@Autowiredsites found — 9 injection points, of which the 5 injectingClinicServicelist exactlyClinicServiceImpl, the corpus's one implementation (grep -rl "implements ClinicService"returns exactly that file), and the 4 injecting Spring Data repository interfaces list NOTHING, with the reason stated: the container implements them at runtime, so the empty set is a fact about the corpus. The facts-only view is byte-identical before and after the pass —analyze --report deps --level type --declared-only --jsonis the same 107 359 bytes either side, 286 edges, provenancedeclaredonly. Also run on petclinic HEAD (818c413), which has no@Autowiredat all: 6 implicit sole-constructor injection points found by the profile's own rule.
Motivation: the second real extractor is the first test of the claim
CLAUDE.md makes on every page — that an extractor in any language can conform
using nothing but schemas/ and the container contract. C# is the right second
language: the csharp profile has existed as data since M1 without an
implementation (invariant 8 says that must be possible; M12 is where it is
checked), Roslyn gives Spoon-grade semantic binding (symbols, overload
resolution, extension-method binding), and the .NET ecosystem's legacy corpora
(Framework 4.x, WebForms, packages.config) are exactly the non-compilable
kind the pipeline exists for. The extractor must run on Linux and macOS as
first-class hosts and ship as ONE self-contained binary per OS, so a user needs
no SDK on the machine that runs it.
Three principles, locked up front:
- Roslyn without MSBuild — the noClasspath of .NET. The extractor never
opens a
.sln/.csprojand never callsMSBuildWorkspace: it walks--srcfor*.cs, parses each file, and binds oneCSharpCompilationagainst the BCL reference assemblies it carries inside itself. A missing NuGet package is a stub, not a build failure — the same degraded-honesty contract as Spoon's noClasspath (§5.2).MSBuildWorkspaceneeds an installed SDK, a restore, and an evaluable project graph: three things a legacy corpus does not offer and a self-contained binary cannot assume. A project-aware mode (--references) is an optional later enrichment, never the baseline. - The extractor holds no metamodel intelligence — the Java rule, mirrored.
It knows the C# id scheme, the kind→traits table, and how to write bytes.
Trait vocabulary, profile validation and closure live in
core; the cross-language gate (packages/core/test/fixtures-csharp.test.ts, the twin of the Java one) is where the two halves meet. - Byte-identity is the cross-OS contract. Two runs on one unchanged corpus
produce the same bytes (contract §6) — on any of the three OSes. That
single property turns "does it work on macOS/Windows?" into
cmpagainst the committed fixture snapshot, which is the only cross-OS test the Linux-only CI can delegate to a human with a laptop.
extractors/csharp/
global.json pins the SDK band (10.0.x, LTS, roll-forward latestPatch)
Directory.Build.props Deterministic, ContinuousIntegrationBuild, InvariantGlobalization,
Nullable=enable, TreatWarningsAsErrors, LangVersion latest
Codegraph.CSharp.sln
src/Codegraph.CSharp/ console project → `codegraph-csharp`
Program.cs CLI: same flags and exit codes as the Java jar (§13.5)
CorpusLoader.cs pass 0: file walk (ordinal order) + parse + one compilation
ReferenceAssemblies.cs the embedded BCL ref pack → MetadataReference.CreateFromImage
CorpusWhitelist.cs pass 1: the declared-type set (INamedTypeSymbol from source)
EntityIds.cs THE C# id scheme (§13.3)
EntityExtractor.cs pass 2
EdgeExtractor.cs pass 3
StubSynthesizer.cs pass 4
Measures.cs sloc (trivia-based) + cyclomatic (syntax-based)
Literals.cs attribute arguments / const initializers → Literal (M10c shape)
Model/ Entity, Edge, NaturalKey, JsonlWriter (Utf8JsonWriter), Progress
tests/Codegraph.CSharp.Tests/ xUnit; the same test names as extractors/java where the property
is the same (SnapshotTest, DeterminismTest, StubDisciplineTest,
ModelSchemaValidationTest, EntityTraitConformanceTest, …)
fixtures/csharp/src/ the reference corpus (§13.6) — `Acme.Order`, the Java corpus's twin
fixtures/csharp/expected/model.jsonl
Toolchain facts that bite first, mirroring extractors/java/README.md:
dotnetis not installed on the dev box (checked 2026-09-07). Install user-locally, sdkman-style:curl -sSL https://dot.net/v1/dotnet-install.sh | bash -s -- --channel 10.0 --install-dir ~/.dotnet, thenDOTNET_ROOT=~/.dotnetand~/.dotnetonPATH.scripts/lib.shgainsensure_dotnetbesideensure_jdk: PATH first, then~/.dotnet/dotnet, then adiethat prints the one-liner. Same script on macOS.DOTNET_CLI_TELEMETRY_OPTOUT=1,DOTNET_NOLOGO=1,DOTNET_SKIP_FIRST_TIME_EXPERIENCE=1exported by the scripts so a first build is not a telemetry conversation.InvariantGlobalization=true: the extractor does no culture-sensitive work, and the flag removes thelibicuruntime dependency that makes a .NET binary fail on a minimal Linux (Couldn't find a valid ICU package) — the single most common "works on my Mac, dies in the container" failure. It also makes ordinal string behaviour identical on all three OSes, which §13.4 needs.- Roslyn packages:
Microsoft.CodeAnalysis.CSharp(syntax + semantics) only. NOTMicrosoft.CodeAnalysis.Workspaces.MSBuild, NOTMicrosoft.Build.Locator(principle 1; both are also incompatible with single-file publishing). - BCL reference assemblies: the
Microsoft.NETCore.App.Refpack'sref/net10.0/*.dllare embedded as resources at build time and loaded withMetadataReference.CreateFromImage. Why nottypeof(object).Assembly.Locationlike every Roslyn tutorial: in a single-file bundleAssembly.Locationis the empty string, so the "obvious" approach works indotnet runand silently binds nothing in the shipped binary — everystringwould become an unresolved stub and the resolution rate would collapse only in production.ReferenceAssembliesTestpins thatSystem.Stringresolves to a stub in moduleSystem, not<unresolved>, and runs against the published binary in CI (§13.7).
The profile (packages/core/src/profiles/csharp.ts) predates M6 and M10; the
first M12 commit is feat(core): csharp profile v2, a data change with the
same justification each Java change had:
| Change | Why |
|---|---|
TWithChildren added to method, constructor, lambda, delegate, property |
M6's executable-containment decision (§9.2): parameters and locals carry parent, so the container must be licensed too, or the model states a containment its own profile forbids. property because accessor bodies hold locals and lambdas. |
TMetrics optional on every type and invocable |
M10b: sloc + cyclomatic, the same two measures Java emits, so the city's --height sum:cyclomatic works on a C# corpus unchanged. |
TWithValue optional on field (const, enum members) and parameter (default values) |
M10c, the value door. An enum member is a field whose value is its constant, exactly the Java shape. |
edges annotationUse and throws added |
Attributes ARE annotation usage (M10c's edge kind, with arguments); throw statements are M10d-era evidence the insights walk consumes. |
kind event added: TNamed, TStructural, TTypedEntity, TChildOf, TSourceAnchor |
An event is a value-shaped member with its own declaration site; folding it into field would lie about the kind and into property about accessors. |
lambda disambiguator note: (file, line, column) |
The M7 lesson (§5.3): two nameless entities on one line — Chain(() => a, () => b), or a lambda inside an anonymous method — collide on (file, line). A column is a source fact. |
the dynamic note reworded: a dynamic call site is dropped and COUNTED, not emitted as dynamic-candidate |
The M10d decision moved candidate generation to the analyzer, which alone has whole-corpus implementor knowledge. An extractor inventing a candidates list from the file's usings would be the prefix-filter sin at the edge level. |
Construct → kind, on top of the profile's table:
| C# construct | kind | notes |
|---|---|---|
namespace (block or file-scoped), global namespace |
namespace |
module; global namespace is csharp:<global>; block-nested namespaces get TChildOf |
class, interface, struct, record, record struct, enum, delegate |
as named; record struct → record |
static class is a class; partial merges by id (§13.3) |
method, operator, conversion, finalizer, local function |
method |
operators use the metadata name (op_Addition, op_Implicit); a local function's parent is the enclosing invocable |
| constructor, static constructor, primary constructor (records, C# 12 classes) | constructor |
primary constructor: anchor = the parameter list; a positional record's synthesized Deconstruct/Equals are NOT emitted (never written) |
property, indexer (this[]), auto-property |
property |
indexer symbol Item(params); accessor bodies contribute edges FROM the property |
field, const, enum member |
field |
TWithValue for const and enum members |
| event | event |
+= / -= are access edges with isWrite |
parameter, local (incl. out var, pattern variables, foreach variables) |
parameter / localVariable |
var → TTypedEntity present, declaredType absent when Roslyn's inferred type is anonymous/error |
| lambda, anonymous method, local function expression | lambda |
#file:line:column |
| extension method | method + TAttachedTo → extended type |
the profile's existing rule |
| attribute usage | annotationUse edge with arguments |
attribute classes are ordinary class entities |
using X;, global using, using static X.Y, using A = X.Y |
import → module X (resp. X, the containing namespace of Y) |
folded to namespace level, never a type |
: Base, interface : IOther |
inheritance |
|
: IFoo on class/struct/record |
interfaceImplementation |
|
calls, new, delegate Invoke, nameof-free |
invocation |
virtual/interface dispatch → the declared member (profile note); dynamic receiver → dropped + counted |
| field/property/event reads and writes | access (isRead/isWrite) |
property access is modelled as access, not invocation — the source writes a member access, and the accessor is not an entity |
type usages: declared types, generic arguments, casts, is/switch patterns, typeof, default(T), base lists' generic args |
reference |
erased per §13.3 |
throw statements |
throws |
static type of the thrown expression; rethrow → the caught variable's static type |
Explicitly NOT extracted in M12, stated in the profile notes: source
generators (their output is not in --src), InternalsVisibleTo, XAML/Razor
code-behind partials (the .cs half is extracted; the generated half is
absent), #if branches the default symbol set excludes (Roslyn parses ONE
configuration — --define is a later flag), and cross-assembly DI wiring
(the analyzer's M10d framework table can gain an ASP.NET Core profile later;
the extractor stays framework-blind).
Same shape as Java's (extractors/java/.../EntityIds.java), with the
differences C# forces:
namespace csharp:<Ns.Path> csharp:Acme.Order
global ns csharp:<global>
unresolved ns csharp:<unresolved> (§13.4)
type csharp:<Ns>/<TypeMetadataName> csharp:Acme.Order/OrderService
generic type csharp:<Ns>/<Name>`<arity> csharp:Acme.Order/Repository`1
nested type csharp:<Ns>/<Outer>.<Inner> csharp:Acme.Order/OrderService.Line
method csharp:<Ns>/<Type>.<name>[`<arity>](<erasedFqnParams>) csharp:Acme.Order/OrderService.Bill(Acme.Order.Order)
constructor csharp:<Ns>/<Type>.<init>(<params>) static ctor: .<cctor>()
indexer csharp:<Ns>/<Type>.Item(<params>)
operator csharp:<Ns>/<Type>.op_Addition(<params>)
lambda / anon csharp:<Ns>/<Type>#<file>:<line>:<column>
field/prop/evt csharp:<Ns>/<Type>.<name>
parameter csharp:<Ns>/<Type>.<methodSig>#param:<name>
local csharp:<Ns>/<Type>.<methodSig>#local:<name>:<startLine>
local function csharp:<Ns>/<Type>.<methodSig>#fn:<name>(<params>)
stub type csharp:<Ns>/<TypeMetadataName> same shape as a declared type, on purpose
- Arity is part of the symbol, because C# lets
Foo,Foo<T>andFoo<T,U>coexist in one namespace. Java erases generics to the raw name; doing so here would merge three legal declarations into one entity — the M2 overload collision one level up. Roslyn'sMetadataName(Foo1`) is a source fact, not an invention, and the backtick is not a reserved id character. - Parameter types in signatures are fully-qualified metadata names WITH their
type arguments (
System.Collections.Generic.List1<Acme.Order.Order>) — **corrected in the M12c audit**: the plan first said "erased", and Humanizer overloadsHumanizeonFunc<T,string>versusFunc<T,object>, which C# allows and Java's erasure never could; erasing merged the two into one key. Arrays asT[],ref/out/indropped (they cannot overload by themselves), nullable annotations dropped (string?andstringare one type),NullableasSystem.Nullable1<T>, tuples asSystem.ValueTuplen<…>, pointers asT*, type parameters as their ordinal!0/!!0(ECMA-335's form — type-level vs method-level — computed fromITypeParameterSymbol.Ordinal; a type parameter's *name* is not part of the signature in C#, and two overloads differing only inT`'s name would otherwise get two ids). - Partial types and partial methods are ONE entity. The anchor is the
declaration that sorts first by
(file path, start line)— ordinal order, so the choice is stable across OSes — and every edge carries thesourceFileof the declaration part that produced it (the profile's own note).PartialTypesTestasserts one id, one entity, both files' edges. - Paths in anchors and
definedInuse/on every OS and are relativized against the deepest common ancestor of the--srcroots, as Java does.
Roslyn does not invent FQNs — but it has its own way of lying, and the discipline is the same: membership is the whitelist of corpus-declared type symbols built in pass 1, never a name-prefix test.
- Whitelist = every
INamedTypeSymbolwhoseDeclaringSyntaxReferencesare in the compilation's own trees (compilation.Assembly.GlobalNamespacewalked, or theTypeDeclarationSyntaxset — the test asserts both agree). - Metadata-resolved external types (the BCL from the embedded ref pack;
later,
--referencesDLLs) are stubs with their real namespace:csharp:System/String{kind: class, traits: [TNamed, TType], isStub: true},parent→ the stub namespacecsharp:System{definedIn: [], isStub: true}. The M3 stub-containment decision applies unchanged. - Error types (
IErrorTypeSymbol, a name Roslyn could not bind — a missing NuGet package, a typo, generated code) are stubs in the reserved modulecsharp:<unresolved>, named as written (csharp:<unresolved>/JsonConvert). This is the honest form: the corpus wrote a type of that name, and nothing says where it lives. Guessing a namespace from the file'susingdirectives would be inventing an FQN, which is the one thing the Java extractor exists to be defended against. Theimportedge tocsharp:Newtonsoft.Jsonfrom theusingdirective still says what was imported. - Roslyn's equivalent of Spoon's receiver promotion: an unbound member
access
foo.Bar()wherefooitself is unbound gives an error type namedfoo. It lands in<unresolved>like any other — and, like the Java note says, a stub count is not a count of external types. - Not entities: primitives are BCL types in C# (
intISSystem.Int32), so unlike Java they DO resolve — to stubs incsharp:System.void,dynamic, anonymous types, tuples' element names, pointer/function-pointer types,null/error values and type parameters are not entities;declaredTypeis omitted for them. Arrays fold to the element type ONLY for the written type reference, never for members (xs.Lengthis an access onSystem.Array'sLength— Roslyn binds it there, so the phantom the Java extractor had to fight does not arise, and the test that pins it is kept anyway). - Resolution summary on stderr, in the Java format: type references,
resolved / unresolved (
IErrorTypeSymbol), rate; entities (stubs); edges (self-edges dropped;dynamiccall sites dropped). Type parameters and error values excluded from the denominator, for the reasons §5.3 records.
codegraph snapshots orchestrates extraction with a fixed argument shape; M12
writes that shape down as the contract every extractor must honour, in
schemas/README.md §8, and the C# extractor implements it verbatim:
<extractor> [--src <dir>]… [--out <file>] [--progress auto|plain|none] [--no-progress]
[--repo-remote <url>] [--repo-commit <sha>] [--repo-root <path>] [--help]
exit codes: 0 ok · 1 failure · 2 usage · 3 unimplemented
stdout: nothing but `--help`; stderr: progress (tty only under auto) + the summary
CLI change: --jar FILE becomes --extractor FILE (--jar kept as an alias).
A .jar runs as java -jar FILE …; anything else runs directly. The seam is
already an injected Extract function in snapshots.ts, so the change is one
dispatcher plus its ENOENT messages naming the right runtime. replay/history
usage strings follow. codegraph itself stays extractor-agnostic — there is no
codegraph extract, by design: the Node side never learns a language.
Red-green, in this order, each step a commit:
-
Walking skeleton (M12a, 2026-09-07) —
header/f/eof, namespaces and every type kind (delegates with their parameters), doc comments,import+inheritance+interfaceImplementationedges, stubs; schema-validated per line (JsonSchema.Net, draft 2020-12) plus the sequence rules in the extractor's own suite (51 tests); snapshot atfixtures/csharp/expected/model.jsonl(20 files, 42 entities of which 11 stubs, 24 edges); the core gatefixtures-csharp.test.ts(13 tests: parse →validateModel(csharpProfile)zero issues → byte-identical toencodeModelToStringon the first run → closure → no self-reference → the stub evidence)../bin/codegraph validatereports OK andanalyze --report depsfolds the import layer to 10 namespaces / 6 dependencies. The JSON writer, ordering and the id scheme were proven against core's encoder before a single member existed — which was the point. -
Fixture corpus
fixtures/csharp/src(Acme.Order, the Java corpus translated, plus what C# adds): a partial class across two files; a file-scoped and a block namespace; nested types;Foo/Foo<T>in one namespace; overloads differing by generic arity; an extension method; a lambda and a local function starting on the same line;record,record struct,struct,enum : byte,delegate,event, indexer,operator +,const+ enum members, a primary constructor,using static,global using, an aliasusing; async/LINQ over the BCL (stubs with real namespaces); an unresolvable external (Newtonsoft.Json);throw, rethrow and a guard clause; attributes with positional, named andtypeofarguments; adynamiccall site (dropped + counted). -
Members and edges (M12b, 2026-09-07) — methods, constructors (the implicit parameterless one included; a record's primary constructor at its parameter list), operators, properties and indexers, fields, events, parameters with defaults, locals (
#local:name:line:column), lambdas (#file:line:column), local functions (#fn:);invocation(calls,new,: this()/: base(), user-defined operators, delegate invocations folded to the delegate type, extension calls resolved to the static method),accesswithisRead/isWrite,referencefrom the narrowest declared owner,throws(rethrow → the catch's type),annotationUsewithLiteralarguments named after the bound constructor's parameters; extensionattachedTo;sloc+cyclomatic. Members Roslyn synthesizes (recordEquals, accessors, backing fields) are not entities — a call to one folds to its type. 25 new .NET tests (MembersTest,EdgesTest,MeasuresTest); the snapshot went from 88 to 570 lines (244 entities, 285 edges) and stayed byte-identical to core's encoder. Review of the first full snapshot caught five defects before they were committed: attribute arguments leaking as accesses, a lambda parameter keyed without its lambda's position,base.Poston an unresolved base collapsing into the module key, an unbound receiver (JsonConvert) producing no edge, and aforeachlocal owning its whole loop body. -
Stub discipline tests: a corpus type declared in namespace
System.Acmestays internal (no prefix test);Newtonsoft.Json.JsonConvertlands in<unresolved>as a reference from its call site;stringlands inSystem; every nameless entity on a shared line gets its own id, and so does each of their parameters; no entity is its own parent. -
Determinism: two runs byte-identical; file walk order ordinal;
\r\nsources give the same lines as\nsources (Windows checkouts withcore.autocrlf);Utf8JsonWriterwithJavaScriptEncoder.UnsafeRelaxedJsonEscapingmatched against core'sJSON.stringifyon thefixtures/unicodecharacters — the byte-identity test in core is what catches an escaping mismatch, so the unicode identifiers go into the C# fixture too. -
CLI e2e on the C# fixture:
validateOK,analyze --report deps(55 type nodes / 114 dependencies),coupling,cycles(three module self-dependencies, no tangle),city(9 districts, 55 buildings, 98 arrows),navigator(181 nodes, 219 rows),domain-facts(26 type dossiers, 69 operations),explain --dry-run(an L8 walk plan). The fixture joined the per-fixture suites: analyzerconformance-csharp(the acceptance gate, clean), city and navigatorcsharp-fixture, and the CLIvalidatesuite. -
Real-corpus audit (M12c, 2026-09-07) — Humanizer (
src/Humanizer, the commons-lang analogue),dotnet/eShop(src, the petclinic analogue) and OrchardCore (src, the fineract analogue), extracted with the publishedlinux-x64binary, every modelvalidate-clean:| corpus | files | entities (stubs) | edges | resolution | wall clock | |---|---|---|---|---|---| | Humanizer | 212 | 12 146 (148) | 27 234 | 97.3 % | 6.2 s | | dotnet/eShop | 498 | 7 396 (789) | 13 223 | 63.5 % | 12.2 s | | OrchardCore | 5 193 | 88 007 (2 175) | 225 184 | 93.4 % | 53.5 s | **The first run aborted on all three**, each with a duplicate natural key — the audit's whole purpose. Humanizer overloads `Humanize<T>` on `Func<T,string>` versus `Func<T,object>`, which C# allows and erasure merged: signatures now carry type arguments. eShop's ten services each declare `static class Extensions` and a top-level `Program.cs`: one compilation makes them one type with ten `<Main>$`, and "first wins" would have erased nine services' DI wiring — every duplicate is now kept, re-keyed by its file. OrchardCore's `JsonDynamicValue` has 37 `explicit operator`s differing only in return type (now part of a conversion's signature) and a `(_, _) =>` lambda whose two parameters share a name (the second carries its ordinal). Humanizer then failed `validate` on an empty `name`: C# 14 `extension(T t) { … }` blocks, which Roslyn models as nameless nested types — their members are now members of the enclosing static class, attached to the receiver. **Resolution, categorised.** Before the audit OrchardCore stood at 62.7 % and its most-referenced "unresolved" names were `Task`, `Task<T>`, `IEnumerable<T>`, `List<T>`: not missing packages but the SDK's *implicit global usings*, which live in the generated `obj/` file the extractor skips as build output. Adding Microsoft.NET.Sdk's seven by default (`--implicit-usings sdk`; `web` opt-in — the Web SDK's additions made OrchardCore's own `StartupBase` ambiguous with `Microsoft.AspNetCore.Hosting.StartupBase` 345 times) and embedding the ASP.NET Core reference pack beside the BCL's took OrchardCore to 93.4 %, its edges from 178 k to 225 k, and Humanizer to 97.3 %. What is left is exactly the dependency surface §5.3 predicts: third-party packages (eShop: EF Core, Npgsql, MAUI, CommunityToolkit.Mvvm; OrchardCore: Fluid, YesSql, GraphQL, OpenIddict), generated code whose output is not under the roots (Humanizer's source generators), and one honest limit of a single compilation — two corpus types with one simple name whose projects each `global using` their own namespace (eShop's two `CatalogItem`) are ambiguous once merged. eShop's 63.5 % is that floor: the corpus mixes ASP.NET Core, MAUI and Aspire projects that never compile together. Screenshots of the eShop city and navigator reviewed at user-facing angles (§13.9). Numbers and causes recorded in the profile `notes`.
No second C# extractor exists, so the cross-validation oracle rule does not apply; the resolution-rate categorisation and the BCL-stub namespace check are the substitutes.
Yes — dedicated binaries for macOS, Linux and Windows are one dotnet publish matrix, and all of it cross-compiles from a single Linux job.
| RID | artifact | notes |
|---|---|---|
linux-x64 |
codegraph-csharp-linux-x64 |
the dev box and CI; InvariantGlobalization means no libicu needed |
linux-arm64 |
codegraph-csharp-linux-arm64 |
Graviton / Pi / Docker on Apple silicon |
osx-arm64 |
codegraph-csharp-osx-arm64 |
Apple silicon; chmod +x after download; unsigned ⇒ Gatekeeper quarantine (xattr -d com.apple.quarantine) — documented, notarization is a later decision |
osx-x64 |
codegraph-csharp-osx-x64 |
Intel Macs; also runs under Rosetta on arm64 |
win-x64 |
codegraph-csharp-win-x64.exe |
unsigned; SmartScreen warns once |
Publish shape, decided:
dotnet publish src/Codegraph.CSharp -c Release -r <rid> --self-contained \
-p:PublishSingleFile=true -p:EnableCompressionInSingleFile=true \
-p:PublishReadyToRun=true -p:IncludeNativeLibrariesForSelfExtract=true \
-p:DebugType=none -o dist/<rid>
- Single-file self-contained, not NativeAOT. Roslyn is not AOT-clean
(
Microsoft.CodeAnalysisemits trim/AOT warnings and relies on reflection for its own services), and NativeAOT needs the target OS's native toolchain, so macOS and Windows binaries would each need their own runner. Single-file bundles the runtime (~30 MB compressed) plus Roslyn and the ref pack; ~70–90 MB per RID, ~1 s cold start with ReadyToRun. Trimming stays OFF (PublishTrimmed=false) for the same Roslyn reason; revisit both when Roslyn ships AOT annotations — that is a build-flag change, not a code change. - Deterministic builds (
Deterministic,ContinuousIntegrationBuild,PathMap) so the binary, like the model, is reproducible from a commit. - Version =
extractor.versionin the header = the assembly's informational version, set from one property inDirectory.Build.props;--versionprints it;HeaderTestpins that the three agree.
Wiring into the repo's scripts and CI:
build.sh --csharp(and--allincludes it whenextractors/csharp/exists —have_csharp_extractor,ensure_dotnet); artifact report listsextractors/csharp/dist/<host-rid>/codegraph-csharp.build.sh --csharp --publish-allruns the five-RID matrix.test.sh --csharp→dotnet test, plus the published-binary smoke test: rundist/<host-rid>/codegraph-csharp --src fixtures/csharp/src --out $tmpandcmpagainst the committed snapshot. That the smoke test runs the published artifact, notdotnet run, is the point — it is the only test that sees the embedded-ref-pack path (§13.1) and the single-fileLocationtrap.- CI moved to GitHub Actions with the repository (M12c, 2026-09-07) —
.github/workflows/ci.ymlreplaces.gitlab-ci.yml, and GitHub's hosted macOS and Windows runners turn the cross-OS check from a laptop ritual into a gate:verify(the TypeScript workspace: build, typecheck, tests,schemas/drift) andjava(./mvnw test) — the old gate, ported;csharp-test—dotnet teston the SDK pinned byglobal.json;csharp-publish— a five-job matrix, one RID each, every job calling./build.sh --csharp --rid <rid>so CI and a developer's machine share one definition of the publish flags; each binary is uploaded as an artifact (14 days);csharp-smoke— a second matrix downloading each artifact onto a runner of ITS OWN OS (ubuntu-latest,ubuntu-24.04-arm,macos-latestfor Apple silicon,macos-15-intel,windows-latest), running it onfixtures/csharp/srcfrom the repo root with the relative--src, andcmp-ing the output with the committed snapshot. Nothing is installed on those runners for the extractor: the binary carries the runtime. A.gitattributes(* text=auto eol=lf,*.jsonl -text) pluscore.autocrlf falsekeep the snapshot's bytes intact on Windows;release— on av*tag, the five binaries renamed after their RID plus aSHA256SUMSare attached to a GitHub Release. Measured locally first: the five-RID matrix cross-publishes from one Linux host in 287 s (57 s per RID, ReadyToRun included), andfileconfirms each artifact is a native executable of its target (ELF x86-64/aarch64, Mach-O x86_64/arm64, PE32+).
README.md+CLAUDE.mdarchitecture block gainextractors/csharp/;extractors/csharp/README.mdmirrors the Java one (build, run, progress, the stderr summary, the toolchain gotchas above).
| Hazard | Where it shows | Guard |
|---|---|---|
Assembly.Location == "" in single-file bundles |
published binary binds no BCL; dotnet run is fine |
embedded ref pack + smoke test on the published artifact |
libicu missing |
Linux containers, Alpine | InvariantGlobalization=true; the CI image test |
| Case-insensitive file systems | macOS, Windows: Foo.cs and foo.cs cannot coexist, and enumeration order differs |
ordinal sort of the walked paths before parsing; DeterminismTest |
\ path separators |
Windows anchors and definedIn |
always emit /; PathsTest on a \-joined input |
\r\n sources |
Windows checkouts | Roslyn line map handles it; sloc scanner test with both endings |
Culture-sensitive string.Compare/ToUpper |
wrong canonical order on a Turkish locale | StringComparer.Ordinal everywhere; an analyzer rule (CA1309) as an error |
Utf8JsonWriter escaping <, >, &, non-ASCII by default |
<global>, <init>, unicode names |
UnsafeRelaxedJsonEscaping + the core byte-identity gate |
| Gatekeeper quarantine | first run on macOS after a download | documented; not a code problem |
| Long paths | Windows MAX_PATH on deep corpora |
\\?\ not needed on .NET 10 with long-path awareness; --src a deep fixture in the Windows manual check |
- M12a — skeleton + contract ✅ (2026-09-07): profile v2 (core), toolchain
scripts (
ensure_dotnet,build.sh --csharp [--publish-all],test.sh --csharpwith the published-binarycmp), walking skeleton end to end (§13.6 first bullet),snapshots --extractor(a.jarruns underjava -jar, anything else directly;--jarkept as an alias), and the extractor command-line contract asschemas/README.md §8, generated from core. - M12b — the model ✅ (2026-09-07): members, all edge kinds, stubs, measures, literals, the full fixture and its snapshot, stub-discipline and determinism tests, CLI e2e, and the fixture in every per-fixture suite.
- M12c — distribution + audit ✅ (2026-09-07): five-RID publish matrix
(cross-published from one Linux host in 287 s, each artifact a native
executable of its target), GitHub Actions gate with the per-OS smoke test
and tagged releases, the three-corpus audit (five defects fixed, resolution
causes measured, implicit usings and the ASP.NET Core pack added), eShop
city (187 districts, 1 176 buildings, 3 738 arrows) and navigator (4 493
nodes, 10 751 rows, 7 cycles) reviewed as screenshots, profile
notesrewritten from measurements,docs/csharp-extractor.mdfor running it with the SDK, the runtime, or nothing.
Definition of done: fixtures/csharp/expected/model.jsonl byte-identical to
core's encoder and profile-valid with zero issues; the same bytes from the
published linux-x64, osx-arm64 and win-x64 binaries on their hosts; the
audit numbers in the profile notes; ./test.sh and CI green with the C#
fixture in every per-fixture suite.
The Spoon extractor ships the same way: one executable per platform, no runtime to install. The mechanism is not the same, and the difference is not cosmetic.
native-image cannot cross-compile. dotnet publish -r osx-arm64 works from
Linux because .NET ships prebuilt per-RID runtimes and the app stays portable IL.
GraalVM instead runs an AOT compiler and the host's native linker, so a macOS
binary is built on macOS. The C# matrix is five RIDs in one Linux job plus five
smoke jobs; the Java matrix is five jobs, each building and proving its own
RID (java-native in .github/workflows/ci.yml).
| RID | runner | artifact |
|---|---|---|
linux-x64 |
ubuntu-latest |
codegraph-java-linux-x64 |
linux-arm64 |
ubuntu-24.04-arm |
codegraph-java-linux-arm64 |
osx-arm64 |
macos-latest |
codegraph-java-osx-arm64 |
osx-x64 |
macos-15-intel |
codegraph-java-osx-x64 |
win-x64 |
windows-latest |
codegraph-java-win-x64.exe |
Build shape, decided:
native-image -jar target/codegraph-java.jar -o dist/<rid>/codegraph-java \
--no-fallback -march=compatibility
- From the shaded jar, deliberately: the binary and
java -jarare then the same artifact compiled two ways, and cannot drift. Reachability metadata and the platform reference travel inside that jar. --no-fallback— a missing reflection entry must fail the BUILD, never produce an image that silently needs a JVM.-march=compatibility— the baseline ISA, not the builder's CPU. A released binary runs on the machine that downloads it.- ~42 MB, ~3 ms startup, and 13× faster than the jar on the fixture corpus (0.13 s vs 1.78 s) — no JIT warm-up on a process that exits in a second.
The hard part was not the compiler: it was the platform library. ECJ, given
no -bootclasspath, resolves java.* against the class library of the JVM it
runs inside. A native image has no JVM, no java.home, and no boot classpath —
and Spoon's noClasspath mode does not degrade gracefully when java.lang is
absent, it INVENTS: System.out.println(…) yields a type named out, a type
variable T becomes an entity. Measured on fixtures/java/src: 84.9%
resolution and 40 stubs, against 94.5% and 27.
So the binary carries its own, exactly as the C# extractor embeds the BCL
reference pack (§13.1). Two build outputs, both generated from the build JDK's
lib/ct.sym by PlatformReferenceJar and both landing inside the shaded jar:
java-api.jar— the Java 17 public API as signature-only class files, 4,721 classes / 2.6 MB (a JDK image is ~130 MB). Handed to Spoon as the source classpath, which ECJ searches LAST — bootclasspath, extdirs, sourcepath, classpath. On a JVM the VM's own library still wins, sojava -jarbehaviour is untouched and the committed snapshot did not move.- A reachability-metadata file naming
java.base's 1,352 types. Spoon resolves an import through the runtime classloader, not through ECJ — proved by running the jar under--limit-modules java.base, which flips exactly thejava.sql.*imports fromdeclaredtoderivedwhile ECJ still sees them.java.baseonly: registering the rest pulls AWT, fontmanager and sound natives into the image and the single file becomes a directory of shared objects.
Two smaller obstacles, both recorded because they are invisible until they bite:
ECJ's handleExtdirs dereferences getJavaHome() unconditionally, so the image
NPEs before parsing a single file unless java.ext.dirs is set (empty is the
truth — extension directories are a JDK 8 relic); and the reflection metadata for
Spoon/ECJ itself is agent-traced over three corpora and committed under
src/main/resources/META-INF/native-image/, because a trace of one corpus misses
what another needs.
The residual, stated exactly. On fixtures/java/src the binary reproduces
the committed snapshot byte for byte. On gson (86 files, 3,635 entities, 9,260
edges) the entity table is byte-identical to the jar's and 3 of 9,260 edges
differ: imports of java.sql.* carry derived instead of declared. The
package they point at is unchanged, no entity is affected, and it is the same
answer a JVM limited to java.base gives. That is provenance doing its job —
the metamodel has a word for "inferred, not read" precisely so this can be
reported rather than hidden.
Wiring, mirroring the C# story so the two cannot disagree:
./build.sh --java --native→extractors/java/dist/<host-rid>/codegraph-java;ensure_native_imagefinds GraalVM (GRAALVM_HOME, JAVA_HOME, PATH, sdkman)../test.sh --javaruns./mvnw testand then, if a binary exists, the native smoke test: the binary re-extracts the fixture corpus and must reproduce the snapshot. It is the only test that sees the image's platform-library path, and--no-fallbackcannot catch a wrong one.- CI builds and smoke-tests all five RIDs and attaches them to a
v*release beside the C# binaries.
Motivation: the third real extractor closes the loop the plan opened in §10 —
run codegraph on codegraph. It is also the first extractor written in the
same language as core, which makes it the sharpest test of the boundary rule:
when importing @codegraph/core is one line away, an extractor must still
conform using nothing but schemas/. TypeScript brings the two metamodel
features no other profile exercises — Entity.space (type vs value) and
declaration merging — and its own flavour of non-compilable legacy: the
pre-ES-module codebase of namespace blocks and /// <reference path>
directives compiled with outFile, which no modern toolchain builds any more.
What is the Spoon / Roslyn of TypeScript? The answer is the language's own
compiler, used as a library — the typescript npm package, the same package
that ships tsc:
| Roslyn | TypeScript compiler API | Role |
|---|---|---|
CSharpCompilation |
ts.Program (ts.createProgram) |
one compilation over a set of files and options |
SyntaxTree / SyntaxNode |
ts.SourceFile / ts.Node |
the concrete syntax tree, positions included |
SemanticModel |
ts.TypeChecker (program.getTypeChecker()) |
binding and typing on demand: getSymbolAtLocation, getTypeAtLocation, getResolvedSignature |
ISymbol |
ts.Symbol — with declarations[], plural |
the binder's view; the plural is declaration merging made explicit |
MetadataReference (the BCL ref pack) |
lib.*.d.ts inside the package |
the standard library, resolved from the package's own directory |
IErrorTypeSymbol |
the any/error type from an unresolved name |
binding failure, tolerated, never fatal |
It is error-tolerant by construction: a Program over files with missing
modules, missing types and type errors still binds everything that binds —
tsc prints diagnostics and emits anyway. So "noClasspath" is not a mode to
switch on, it is the default; the extractor's job is to keep the honest
distinction between "resolved to a declaration" and "resolved to any".
The alternatives, and why they are not the baseline:
| Candidate | What it is | Verdict |
|---|---|---|
| ts-morph | a convenience wrapper over the compiler API (Spoon-like navigation, findReferences) |
rejected — wraps every node in an object, so memory and time scale badly on the corpora this pipeline exists for; lags TypeScript releases; adds nothing the extractor needs beyond the checker it already wraps |
| tree-sitter, swc, oxc, Babel, esbuild | parsers — fast, error-tolerant, no binder | rejected as a baseline — without a checker every call is a dynamic-candidate and every import an unresolved string; the model would be a lexical approximation of the graph. A syntactic pass is not needed for speed either: the compiler API parses a million lines in seconds; the checker is what costs |
| scip-typescript (Sourcegraph) | an indexer built on the same compiler API, emitting SCIP occurrences | not for this extractor — SCIP carries definition/reference/implementation occurrences, not kinds, traits, provenance, access vs invocation, or spaces. It remains the cheap multi-language adapter path §10 lists, whose output is a lower bound of this one |
ts.createLanguageService |
the editor-facing incremental layer | unnecessary — one batch program per run; incremental extraction is the temporal track's problem (§11.4) |
Three principles, locked up front:
- The checker without a build — the noClasspath of TypeScript. The
extractor never runs
tsc, never builds project references, and never needsnode_modules. It walks--srcfor*.ts,*.tsx,*.mts,*.cts(and*.js/*.jsxunder--allow-js), skippingnode_modulesand build output, and creates ONEProgramover all of them. Atsconfig.jsonis read for resolution options only —paths,baseUrl,rootDirs,jsx,lib,target— never forfiles/include(the roots define the corpus) and never forreferences. A module specifier that resolves to nothing is a stub module keyed by the specifier, not a dropped edge; a package that is not installed is a stub, not a failure. One resolution the standard host does not do, and this one does: a bare specifier that names a package declared inside the roots (@codegraph/core→ thepackage.jsonunderpackages/core) resolves to that package's source entry even with nothing installed and nothing built — the monorepo's own packages are corpus, not dependencies. - No metamodel intelligence — and no
@codegraph/*at runtime. The extractor's only runtime dependency istypescript. A boundary test (the twin ofpackages/llm/test/boundary.test.ts) pins that nothing underextractors/typescript/srcimports@codegraph/; core is a devDependency for the extractor's own tests, where validating one's output against the reference implementation is exactly right. Two reasons: the core-side gate — "byte-identical to what core's encoder writes" — is a statement about two independent encoders only if they are independent; and a package that depends ontypescriptalone can be published and run withnpxon a machine that has never seen this repo. - Byte-identity across OS and across install state. Two runs over one
unchanged corpus write the same bytes on Linux, macOS and Windows — the C#
contract — and, because external entities are keyed by package name
(from the nearest
package.json) and never by a resolvednode_modules/…/index.d.tspath, the keys of a model do not depend on what happens to be installed. Edge counts do (an uninstalled package's exports areany, and a call throughanyis dropped and counted), and the fixture snapshot is produced with nonode_modulesat all, so the committed bytes are reproducible from a bare checkout.
extractors/typescript/ pnpm workspace package `codegraph-typescript` (public name; bin of the same name)
package.json dependencies: typescript (catalog) — NOTHING else at runtime
devDependencies: @codegraph/core, vitest, fast-check, tsup
tsconfig.json strict, NodeNext, extends ../../tsconfig.base.json
src/
main.ts CLI: the extractor contract (schemas/README.md §8) + --tsconfig, --allow-js, --ignore-node-modules
corpus.ts pass 0: walk (ordinal order), tsconfig resolution options, the workspace-package resolver, ts.createProgram
whitelist.ts pass 1: the declared-symbol set (every declaration under the roots and outside node_modules)
ids.ts THE TypeScript id scheme (§14.3): keys, escaping, spaces
entities.ts pass 2
edges.ts pass 3
stubs.ts pass 4: lib / package / <unresolved> stubs, stub modules for specifiers
measures.ts sloc (ts.createScanner — the compiler's own lexer, JSX and template literals included) + cyclomatic
literals.ts const initializers, enum values (checker.getConstantValue), decorator arguments, parameter defaults
model/ NaturalKey, Entity, Edge, canonical order, JsonlWriter (JSON.stringify — the reference bytes, natively), Progress
test/ vitest; the C# suite's names where the property is the same (snapshot, determinism, stub-discipline,
ids, schema-per-line, cli, boundary) + spaces, merging, escaping
bin/codegraph-typescript `#!/usr/bin/env node` → dist/main.js
fixtures/typescript/src/ the reference corpus (§14.6): `acme-order`, the Java and C# corpora's twin, plus a legacy script half
fixtures/typescript/expected/model.jsonl
pnpm-workspace.yamlgainsextractors/typescript;pnpm -r build/test/typecheckcover it with no new tooling.build.sh --tsalready builds it;test.sh --tsgains the built-bundle smoke test: runbin/codegraph-typescriptonfixtures/typescript/srcandcmpwith the snapshot — the one test that sees the bundled code path rather than vitest's source aliasing.typescriptstays external to the tsup bundle.ts.createProgramlocateslib.*.d.tsrelative totypescript.js(ts.getDefaultLibFilePath); a bundle that inlines the compiler binds no standard library and everyArray,Promiseandstringmethod becomes<unresolved>— theAssembly.Locationtrap of §13.1 in its Node form. A test pins thatArrayresolves to a stub in module<lib>, and runs against the built bin, not the sources.- The pinned compiler version is the workspace catalog's (
typescript^5.9); the header'sextractor.versionis the package version and the compiler version rides inextractoras an extra key (the header schema allows it), so a model says which checker produced it. codegraph snapshots --extractorlearns one more launcher rule beside.jar → java -jar: a.js/.mjs/.cjsfile runs undernode. Anything else,bin/codegraph-typescriptincluded, still runs directly.- Node ≥ 22 is already a requirement of codegraph itself, so "nothing to install" holds for anyone who can run the analyzer. A Node single-executable build is deferred (§14.7).
The profile (packages/core/src/profiles/typescript.ts) predates M6, M10 and
M12; the first M13 commit is feat(core): typescript profile v2, data changes
with the M12 pattern of justification:
| Change | Why |
|---|---|
kind constructor added: TInvocable, TWithChildren, TWithParameters, TWithLocalVariables, TWithInvocations, TWithAccesses, TChildOf, TSourceAnchor (+ TComment, TMetrics); no TNamed, no TTypedEntity |
the Java/C# shape; a class constructor is not a method named constructor — it has no return type and its parameter properties declare fields |
TMetrics optional on every type kind and every invocable; module too |
M10b: sloc + cyclomatic, so --height sum:cyclomatic works on a TypeScript city unchanged. A module's top level is executable code, hence measured |
TWithValue optional on variable (const initializers), property (enum members, readonly literals), parameter (defaults) |
M10c, the value door; an enum member is a property whose value is its constant |
TWithInvocations, TWithAccesses, TWithLocalVariables optional on module |
a module body IS a scope with statements: app.listen() at top level is an invocation FROM the module, and const x = … at top level is a local of it. Without the licence the model states edges from an entity its profile says cannot have them |
edges annotationUse and throws added |
decorators ARE annotation usage (M10c's edge kind, with arguments); throw sites are the M10d-era evidence the insights walk consumes |
arrowFunction / nameless function disambiguator: line:column |
the M7 lesson (§5.3): two arrows on one line collide on (file, line). The module is the file already, so the disambiguator needs no file component |
| merging note rewritten: a merged symbol is one entity per declaration file, one entity per file for same-file merges | the module is the file and containment is where a thing is written (invariant 5). The old note ("one namespace id may span files") contradicted both. An edge to a merged symbol targets the declaration that owns the referenced member (§14.3) |
new note: key escaping (/, #, % and, in names, .) |
the file path is the module and / is a reserved separator (§14.3) |
space on enum becomes per-entity: ["type"] for const enum |
the profile already says const enum members leave no runtime entity; the entity should say so where analyses read it |
any-receiver note reworded: dropped and COUNTED, never emitted as dynamic-candidate |
the M10d/M12 decision: candidate generation is the analyzer's, which alone has whole-corpus knowledge |
Construct → kind, on top of the profile's table:
| TypeScript construct | kind | notes |
|---|---|---|
| a source file, module or script | module |
definedIn = the one file; space: ["value"]. A script (no import/export) puts its declarations in the global scope semantically, but they are still children of the file (invariant 5); nothing lives in a <global> module |
namespace X / module X {} (internal module), nested X.Y |
namespace |
child of the file or the enclosing namespace; space from what it declares (["type"] if it exports only types) |
declare module "pkg" {} (ambient external module) |
module |
keyed by the quoted name (ts:pkg), definedIn = the .d.ts, isStub: false — it IS declared by the corpus; import "pkg" resolves to it |
declare global {} and module augmentations declare module "express" { interface Request { user: User } } |
no entity for the block | its members are ordinary declared entities parented by the augmented type — a stub type when the package is external (§14.4) |
class, abstract class, class expression bound to a name (const A = class {}, export default class {}) |
class / abstractClass |
name from the binding (default for a nameless default export); a class expression with no binding is not an entity (dropped and counted), its members neither |
interface |
interface |
extends → inheritance (multiple); space: ["type"] |
type T = … |
typeAlias |
constituents → reference edges |
enum, const enum |
enum |
members → property children with TWithValue (checker.getConstantValue); const enum → space: ["type"] |
| function declaration, overload signatures + implementation | function |
ONE entity, anchored at the implementation (or the sole ambient signature); overload signatures are not entities. signature is the implementation's |
| function expression, arrow function | function (no TNamed) / arrowFunction |
#line:column; const f = () => {} yields the variable f AND its child arrow (profile rule); call sites target the arrow |
method, get/set accessor, static member, abstract method, object-literal method |
method |
accessors: #get / #set; static: #static (§14.3) |
constructor(…) |
constructor |
parameter properties (constructor(private x: T)) yield BOTH a parameter and a property |
property declaration, accessor field, object-literal property, enum member |
property |
index signatures are not entities |
#private member |
method / property |
name escaped (%23secret), never the unescaped # |
var/let/const at any level, every binding of a destructuring pattern |
variable |
parent = module / invocable / block-owning invocable; TWithLocalVariables on the owner lists them |
parameter, this parameter excluded |
parameter |
#param:name below the owner, below the owner's own disambiguator when it has one |
| type parameter | not an entity | its constraint and default → reference edges from the declaring entity |
decorator @Dec(args) |
annotationUse edge with arguments (literals) |
legacy (experimentalDecorators) and TC39 decorators alike — both parse; the decorator is an ordinary function/variable entity |
import x from "m", import { a as b }, import * as ns, import type, export … from, export * from, import("m") / require("m") with a literal, /// <reference path="…"> |
import → the module |
module-level, never a member (the Java fold). import type is an ordinary import edge whose erasure shows in the target's space. A computed specifier is dropped and counted |
class A extends B |
inheritance |
extends Mixin(Base): the checker's base type if it names a declaration, else a reference to what is named, counted |
class A implements I |
interfaceImplementation, provenance declared |
structural conformance is NOT computed by the extractor — derived conformance is an analyzer enrichment, never mixed in |
call, new, tagged template, optional call f?.(), JSX element <Comp/> with a component tag |
invocation |
target = the resolved signature's declaration, else every declaration of the callee's symbol (a union receiver's synthetic symbol has the real ones); receiver typed any/unknown/error → dropped and counted. A JSX element is a call by the language definition, so the edge is declared; intrinsic tags (<div>) yield nothing |
property read / write / compound / delete, destructuring reads, obj["lit"] |
access (isRead / isWrite) |
computed non-literal keys dropped and counted; a property access that runs an accessor is access, not invocation (the C# rule) |
type positions: annotations, generic arguments, as, satisfies, typeof X in a type, keyof T, import("m").T, conditional/mapped-type operands, heritage clause type arguments, instanceof X |
reference |
to named types only; primitives, literal types, type parameters and any name nothing |
throw expr |
throws |
the static type of expr when it names a declaration; rethrow → the caught binding's static type; else dropped and counted |
| JSDoc / leading doc comment | comments |
the declaration's own JSDoc, not every leading comment |
Explicitly NOT extracted in M13, stated in the profile notes: structural
(derived) interface conformance; emitDecoratorMetadata synthesis; CommonJS
module.exports reassignment shapes beyond a literal require; monkey
patching (the JS profile's rules apply verbatim); Vue/Svelte/Angular template
files (only .ts/.tsx bodies are read — a .vue SFC's script block is a
later --extract-sfc flag, never a default); declare d code whose
implementation is in a language the pipeline does not model (native
addons).
module (file) ts:src%2Facme%2Forder.ts path relative to root, escaped
ambient module ts:express declare module "express" in the corpus
external package ts:zod/ZodType ts:@types%2Fnode/Buffer nearest package.json `name` under node_modules
external subpath ts:zod%2Fv4 an import edge's target: the specifier as written
lib ts:<lib>/Array lib.*.d.ts inside the typescript package
unresolved name ts:<unresolved>/Ledger a name that binds to nothing, as written
unresolved specifier ts:.%2Fmissing ts:@megacorp%2Fledger a stub module for an import nothing resolves
type / value ts:src%2Facme%2Forder.ts/OrderService nested namespaces: Ns.Inner.Type
member ts:src%2Facme%2Forder.ts/OrderService.bill no parameter list — TypeScript overloads are one declaration
static / accessors …/OrderService.create#static …/OrderService.total#get …/OrderService.total#set
constructor …/OrderService.constructor
nameless invocable …/OrderService.bill#12:7 top level: ts:src%2Fa.ts#3:15
parameter / local …/OrderService.bill#param:order …/OrderService.bill#12:7#param:x
- Escaping, the one rule.
renderIdreserves/and#in a module and#in a symbol so rendering stays injective; a file path is made of/. Every path segment and every non-identifier name entering a key is percent-encoded for exactly/→%2F,#→%23,%→%25, and — for names only, where.is the nesting separator —.→%2E. Identifiers cannot contain any of the four, so an identifier is written as-is and the encoding is injective and reversible; the extractor'sescape/unescapepair is pinned by a fast-check round-trip property. Rejected alternatives: relaxing core's reservation (breaks rendering injectivity for every language, since a module entity renders with no/after it); a look-alike separator such as U+2215 (not typeable on a CLI or in a SQL query); dropping the extension or dotting the path (a.b/c.tsanda/b.c.tscollide). Thenameof the module entity is the unescaped path, which is what the city and navigator display. - Module = file, always. The
<global>module of C# has no counterpart: a script file's globals are children of the file (invariant 5), and an entity's module is the file it is written in. Consequence for declaration merging: aninterface Orderin two files is two entities; two in one file is one entity anchored at the first. A reference to a merged symbol targets the declaration that owns the referenced member (checker.getSymbolAtLocationon the member gives one declaration); a reference to the merged container itself targets its first declaration in canonical file order — a deterministic choice, named in the profile note as one. - Overloads are one entity — TypeScript has no overloading by parameter type at the declaration level; the overload list is one function's signature set. No parameter-list component, unlike Java and C#.
- Same-name members that TypeScript allows to coexist get a
disambiguator:
staticfor static members (a class may declarestatic parse()andparse()),get/setfor accessor pairs. The disambiguator is the only place these flags are identity, so a class with only an instanceparserenders with no suffix — the absent-first canonical order puts the instance member first. - Nameless invocables carry
line:columnof their first token, below the nearest named ancestor's symbol and below that ancestor's own disambiguator when it has one — so two arrows on one line are two entities, and an arrow inside an arrow does not become its own parent (the METAMODEL §1.1 column rule). At module top level the symbol is empty and the disambiguator alone identifies the entity:ts:src%2Fa.ts#3:15. METAMODEL §1.1 says the symbol is "empty only for a module"; M13a loosens that by one clause ("or for a nameless entity at module top level, which then carries a disambiguator"). Verified (2026-09-08): the JSONL reader recognises a module record bym === i, andrenderIdonly omits the/for an empty symbol — nothing incoreor the analyzer treats an empty symbol as the module marker, so the form needs no new encoding. - Stub keys have the same shape as declared keys — membership is the
whitelist, never the key. An external type's module is the npm package
name (walk up from the declaring file to the nearest
package.json), never the resolved path undernode_modules, so keys are the same on every machine; an import edge's stub module is the specifier as written (zodandzod/v4are two entry points, honestly two modules). Node built-ins are normalised to thenode:form (fsandnode:fsare one module in fact, andmodule.builtinModulessays which names those are).
Pass 1 builds the whitelist: every ts.Symbol at least one of whose
declarations lies in a file under the roots and outside node_modules. Then:
| The checker says | The model says |
|---|---|
| declared in the corpus | a declared entity |
declared only under node_modules/<pkg> |
stub type in module <pkg>; a member reference into it retargets to the type stub (the C# fold); a free function or variable exported by the package retargets to the stub module — the smallest degraded container a module-level value has |
declared in lib.*.d.ts |
stub type in <lib>; string/number/boolean/symbol/bigint/void/null/undefined and literal types are not entities (the Java primitive rule) |
an error type / cannot find name |
stub in <unresolved>, named as written — never a guess from the file's imports |
| a specifier that resolves to nothing | a stub module keyed by the specifier; the import edge survives (import fan-out stays honest) |
any / unknown / an index signature on a receiver |
the edge is dropped and counted; the count is the profile's stated resolution ceiling |
| a corpus declaration merged into an external one (module augmentation) | the corpus members are declared entities parented by the stub type — closure holds because the stub exists; isStub is a fact about the type, not about its members |
the typescript package itself, when a corpus imports it |
an external package like any other (ts:typescript/Node) — the extractor's own dependency is not corpus |
Two things the checker resolves that are still external: the standard library
(<lib>) and installed packages. Resolvability is not membership — the C#
rule, verbatim.
The workspace-package resolver (principle 1) is the one place the
extractor adds resolution the compiler host lacks, and it is bounded: a bare
specifier whose package name matches a package.json name under the roots
resolves to that package's source entry — the source/module/main/types
field that points at a file under the roots, else src/index.ts when it
exists — and only when standard resolution failed. It never reads
node_modules to do so, never follows exports conditions into dist/
(build output, not corpus), and names every such resolution on stderr under
its own counter, because it is a fact about the repository layout rather than
about the language. A subpath into a workspace package (@acme/pricing/rules)
resolves the same way below that package's source root, else stubs.
Same flag shape and exit codes as the Java jar and the C# binary
(schemas/README.md §8) — codegraph snapshots must stay language-blind.
Three additions, all optional:
| Flag | Meaning |
|---|---|
--tsconfig <file> |
the tsconfig.json whose resolution options apply (paths, baseUrl, rootDirs, jsx, lib, target, allowJs); default: the nearest tsconfig.json at or above each --src root, the first root's winning on conflict (named on stderr); none for the synthesized defaults (target esnext, moduleResolution bundler, lib esnext + dom, jsx preserve, skipLibCheck, noEmit) |
--allow-js |
also walk *.js/*.jsx/*.mjs/*.cjs; JSDoc types feed declaredType (the JS profile's rule); the model still claims lang: "ts" — a mixed corpus is a TypeScript program with JavaScript files in it, which is what the checker models too |
--ignore-node-modules |
never read node_modules, even when present: every external package is an unresolved stub module. What the fixture uses, and what makes two machines' models key-and-edge identical |
stdout carries nothing but --help/--version; progress and the resolution
summary go to stderr:
✓ walk 310 files 0.1s
✓ program 310 files, tsconfig ./tsconfig.base.json, lib esnext 2.4s
✓ whitelist 4 812 symbols 0.3s
✓ entities 9 340 entities 1.1s
✓ edges 21 907 edges 1.8s
✓ stubs 417 stubs (lib 96, packages 288, <unresolved> 33) 0.1s
✓ write 31 664 records 0.4s
RESOLUTION SUMMARY
type references : 21 907
resolved : 20 588
unresolved : 1 319
resolution rate : 94.0%
any-typed receivers (dropped) : 212 ← the honest ceiling (profile note)
imports : 1 204 (unresolved: 41, workspace-resolved: 388)
entities : 9 757 (stubs: 417)
edges : 21 907 (self-edges dropped: 3, computed accesses dropped: 19, nameless classes dropped: 1)
wrote model.jsonl
-
Walking skeleton first: modules and imports (resolved, workspace-resolved, unresolved → stub modules), every type kind with
space, inheritance / implements, stubs — one file in,codegraph validategreen, snapshot committed, the core gatepackages/core/test/fixtures-typescript.test.ts(parses as aModel; byte-identical to core's encoder; ZERO profile issues; closed; no self-edges; the stub-discipline evidence by id). -
The fixture corpus
fixtures/typescript/src—acme-orderas an ES module package tree plus a legacy script half — around 18 files, 350 lines, each pinning one hazard (its README lists them, the Java README's form):tsconfig.jsonwithpaths(@acme/*) and a second workspace packagepackages/pricingimported by name with nonode_modulesand nodist— the workspace resolver's case;@megacorp/ledgerimported and never present — the stub-module case;Ledgerextended from it — inheritance from a stub; a call through itsanyexport — the dropped-and-counted case;import typeand an inlinetypespecifier — erased imports whose edge is the same kind;interface Ordermerged in one file and again across two files; anamespace Acme.Orderin two script files joined by/// <reference path>— the legacy internal-module style; a module augmentation of the absent package;const enum Channeland a plainenumwith computed and literal members —space: ["type"]on one andTWithValueon both;- two arrow functions on one line, an arrow inside an arrow, a top-level
arrow (
ts:…#line:col), an IIFE; - overloads with one implementation;
static parse()besideparse(); aget/setpair; a#privatefield; a string-literal member"a.b"and a computed[Symbol.iterator]— every escaping rule; - a
.tsxcomponent tree (<OrderTable rows={…}/>) — JSX invocations; - decorators with literal arguments (both syntaxes, two files);
throw new OrderError(...), a rethrow, athrow "string"(dropped);- parameter properties, destructured parameters and locals, a
declare module "legacy-lib"ambient in a corpus.d.ts,export default class {},export * from, a dynamicimport("./lazy")with a literal and one with a template (dropped and counted), a compound assignment (isReadandisWriteboth).
-
The extractor's own suite (vitest): per-line schema validation against
schemas/*.schema.jsonand the sequence rules; the trait-key rule of contract §4; snapshot byte-identity; determinism (two runs, shuffled walk order, CRLF-converted sources,--srcgiven asa bandb a); stub discipline by id; the escaping round-trip property; the spaces of every fixture kind; merging (which declaration an edge lands on); the built bin's CLI (exit codes 0/1/2, silent stdout,--version); the boundary test;Arrayresolving to<lib>from the built bundle. -
The fixture in every per-fixture suite: analyzer (
fixture.tsloaders, conformance, coupling, cycles, store import/fold/diagnose), city, navigator, CLI e2e (validate,analyze,export,import,city,navigator,domain-facts,explain --dry-run). -
Self-hosting — the DoD of the phase.
codegraph-typescript --src packages --src extractors/typescript --out codegraph.jsonlon this repository:validateclean, zero<unresolved>entities that name a corpus symbol (every one must be a genuinely absent dependency, named in the audit),analyze --report cyclesagreeing with the package boundaries CLAUDE.md states (no cycle crosses a package;vizandnavigator-uiimport their model packages for types only — the boundary tests restated as a graph query), and the city and navigator of codegraph reviewed as screenshots at user-facing angles. -
Three-corpus audit, the M12c pattern, with resolution causes measured and the residue named in the profile
notes:- TypeScript 4.9's own
src/compiler— the legacy corpus:namespace ts {}in a hundred files merged by/// <reference>,outFile-style, no ES modules. Merging, script-file globals and the per-file entity rule at scale; - nestjs/nest — decorators everywhere, a pnpm monorepo of packages
importing each other by name: the workspace resolver and
annotationUseat scale; - excalidraw — a large
.tsxReact corpus: JSX invocations,anydensity,import typeerasure, path aliases.
No second TypeScript extractor exists, so the oracle rule does not apply; the resolution-rate categorisation, the
<lib>stub check and the self-hosting graph query are the substitutes. - TypeScript 4.9's own
npx codegraph-typescript --src . --out model.jsonl on any machine with
Node ≥ 22 — the same floor as codegraph itself. The published package
bundles the extractor into one file (tsup, platform: node, typescript
external) and declares typescript as its one dependency, so an install is
the compiler plus a few hundred kilobytes. bin/codegraph-typescript at the
repository root runs the same bundle from a checkout, and is what
test.sh --ts smoke-tests and what codegraph snapshots --extractor is
pointed at.
A Node single-executable application (SEA) per OS — the C# distribution
shape — is deferred, with the trap named now so it is not rediscovered: a SEA
embeds the bundle but not the typescript package's lib/*.d.ts files, and
ts.getDefaultLibFilePath then points into a directory that does not exist,
which binds no standard library and passes every test run from sources. Doing
it means embedding the lib files as assets and a custom CompilerHost that
serves them from memory — a build-and-host change, not a model change — and
it is worth doing only once someone needs the extractor on a machine without
Node, which today is nobody who can run the rest of the pipeline.
Wiring into the repo's scripts and CI:
build.sh --ts(alreadypnpm -r build) listsextractors/typescript/distin its artifact report;test.sh --tsadds the built-bincmpagainst the snapshot..github/workflows/ci.yml: theverifyjob already builds and tests the package as part of the workspace; atypescript-smokematrix runs the built bin onfixtures/typescript/srconubuntu-latest,macos-latestandwindows-latestandcmps the output with the snapshot — Windows is the one that matters (\paths, CRLF checkouts, case-insensitive walks), andactions/setup-nodemakes it a two-step job. On av*tag the package is published to npm with provenance (npm publish --provenance) beside the C# binaries on the release.README.md+CLAUDE.mdarchitecture block gainextractors/typescript/;docs/typescript-extractor.mdmirrorsdocs/csharp-extractor.md(run withnpx, from a checkout, the flags, reading the summary, troubleshooting — the tsconfig-conflict and<lib>-empty symptoms first).
| Hazard | Where it shows | Guard |
|---|---|---|
typescript inlined into the bundle |
built bin binds no lib: every Array is <unresolved>; sources pass |
tsup external: ["typescript"]; the <lib> test runs against the built bin |
\ path separators and drive letters |
Windows anchors, definedIn, module keys |
ts.sys paths normalised with / before they enter a key; paths.test.ts on a \-joined input; the Windows smoke job |
| case-insensitive file systems | macOS/Windows: the checker's useCaseSensitiveFileNames differs, Foo.ts/foo.ts cannot coexist, enumeration order differs |
ordinal sort of walked paths; the host is created with useCaseSensitiveFileNames: true so keys keep the written case; DeterminismTest shuffles the walk |
\r\n sources |
Windows checkouts, .gitattributes-less corpora |
ts.getLineAndCharacterOfPosition is EOL-agnostic; the sloc scanner test with both endings; the CRLF determinism case |
tsconfig conflicts in a monorepo |
two roots with incompatible paths |
first root wins, the conflict named on stderr; --tsconfig overrides; a fixture with two roots |
Intl/locale-sensitive sorting |
canonical order on a machine with a different locale | comparisons by UTF-16 code unit (< on strings), never localeCompare; an eslint rule forbidding localeCompare in the extractor |
JSON.stringify of lone surrogates and non-BMP names |
byte identity with core | it IS core's encoder, natively — the one hazard this extractor lacks |
| memory on very large programs | the checker holds every type; excalidraw and nest are fine, vscode-sized corpora are not |
--max-old-space-size documented; Program created with skipLibCheck and no emit; an explicit "one program per --src root" flag is a later enrichment if a corpus needs it |
- M13a — profile v2 + skeleton + contract ✅ (2026-09-08):
feat(core): typescript profile v2(§14.2) with the METAMODEL §1.1 clause for nameless top-level entities; the workspace package with its boundary test; the walking skeleton (§14.6 first bullet) →fixtures/typescript/expected/model.jsonlbyte-identical to core's encoder and profile-valid with zero issues; the core gate;snapshots --extractorrunning.jsundernode;bin/codegraph-typescript;test.sh --tswith the built-bincmp. - M13b — the model ✅ (2026-09-08): members (methods, accessors as
#get/#set, statics as#staticbeside an instance twin, constructors, properties incl. parameter properties, enum members with computed values, parameters, destructured bindings, locals as#local:name:line:column, nested functions as#fn:), nameless invocables byline:column, bound object literals and class expressions, overloads folded to the implementation, cross-kind merging by rank; every edge kind incl.annotationUsewith written arguments named after the factory's parameters andthrowsat the site; JSX elements as invocations; accessor writes on the setter;sloc+cyclomatic; constant-shaped values (4 * 25unevaluated); the full 23-file fixture and its README; the fixture in the analyzer, city, navigator and CLI suites; every CLI command verified on it. Snapshot: 193 entities / 127 edges. - M13c — self-hosting + audit + distribution ✅ (2026-09-09):
-
Codegraph on codegraph (
--src packages --src extractors/typescript): 24 887 entities / 51 255 edges in 12 s,validateclean, 96.5 % resolution, ZERO<unresolved>, zero duplicates, zero unclosable edges; 168 imports resolved to workspace packages by name. The architecture rules of CLAUDE.md hold as graph queries (test/self-hosting.test.ts): no dependency cycle crosses a package;vizandnavigator-uireach their model packages only through imports and through TYPE-space entities (interface members ARE type-space — the audit's first correction to the profile);@openrouter/sdkis imported bypackages/llm/src/openrouter.tsalone,threebyvizalone,reactbynavigator-uialone. City (355 stub buildings unmeasured, as designed) and navigator reviewed as screenshots. -
Three-corpus audit, every model
validate-clean:corpus files entities / edges rate <unresolved>any-receivers time microsoft/TypeScript 4.9.5 src/compiler(namespaces +/// <reference>,outFilestyle)69 52 341 / 120 127 97.4 % 3 ( Profile,Session,Stats— node types with no@types/node)2 684 21 s nestjs/nest packages(decorators, a monorepo importing itself by name)902 29 992 / 32 537 88.6 % 78 (fastify/express/class-transformer types, not installed) 20 745 12 s excalidraw packages+excalidraw-app(.tsx, path aliases)628 43 706 / 62 557 94.4 % 41 (react/mermaid/vitest types, not installed) 24 954 23 s All three ran with NO
node_modules: the any-receiver counts are the ceiling the profile note describes — every call into an uninstalled package is a call throughany, dropped and counted — and nest's 508 workspace-resolved imports are its own@nestjs/*packages reached bypackage.jsonname. excalidraw's 303 stub modules are mostly font and asset imports (./Excalifont-Regular-….woff2), honest imports of non-code. Six defects found and fixed by the audit: members of an unbound object literal (array elements, arguments) were entities and collided (2 260 re-keyings on codegraph itself); a parameter owned an arrow written in its default value; methods of an unbound literal keyed their parameters under the enclosing function; members of an unbound class expression (return class extends Base {}, nest's dynamic modules) surfaced as members of the enclosing method and aborted the write with a dangling constructor; overloaded signatures with no implementation (an interface's, an ambient function's) each emitted parameters (74 on the compiler); a class expression bound to a PROPERTY (static readonly ConfigProxy = class …) was both a property and a class. Plus the safety net the contract asks for: an edge the model cannot close is dropped and counted (unclosable), never written dangling and never an abort. -
Distribution:
typescript-smokein CI runs the built bundle on Ubuntu, macOS and Windows andcmps the fixture snapshot;npm-publishon av*tag publishescodegraph-typescriptwith provenance when the repository holdsNPM_TOKEN(skipped, not failed, otherwise);docs/typescript-extractor.md; README and CLAUDE.md name the extractor.
-
Definition of done: fixtures/typescript/expected/model.jsonl byte-identical
to core's encoder and profile-valid with zero issues; the same bytes from the
built bin on Ubuntu, macOS and Windows runners; codegraph's own model
validate-clean with its package boundaries recovered as a graph query; the
audit numbers in the profile notes; ./test.sh and CI green with the
TypeScript fixture in every per-fixture suite.
| # | Milestone | Definition of done |
|---|---|---|
| M0 | Bootstrap | workspace builds, CI green |
| M1 | Core metamodel | traits + 9 profiles + validation + JSON Schema, tested |
| M2 | Java extractor | ✅ fixture corpus → valid model.json, schema-validated and profile-validated across the language boundary, closed graph, 94.1% resolution on the fixtures |
| M3 | Analyzer | ✅ import graph, type deps, cycles, coupling metrics, DOT/CSV/JSON exports; 252 tests; verified end to end on google/gson (3 624 entities) and apache/commons-lang (15 338 entities) |
| M4 | CLI + properties | ✅ validate/analyze/export/profiles shipped; conformance gate + property suite green; 1 007 TS tests; verified end to end on apache/commons-lang (15 338 entities / 24 631 edges, every command < 1 s) |
| M5 | Metamodel v2 | ✅ METAMODEL.md §1.1/§4 restated + §5.1/§5.2 added + §8 model/encoding split; core natural-key vocabulary, renderId, memoized validation; 1 089 TS tests, wire format and schemas/ byte-identical |
| M6 | JSONL interchange | ✅ in-place clean break (schemaVersion unchanged): streaming reader/writer, per-record schemas + generated container contract, extractor emits .jsonl, fixtures regenerated, v1 deleted; fineract 559.5MB → 127.4MB and analyzable end to end; 1 149 TS + 153 Java tests |
| M7 | SQLite store | codegraph import → model.db cache; DB-backed analyzer facade; repeat runs skip parsing; reports byte-identical to M6 outputs |
| M8 | 2nd language | clj-kondo adapter; cross-language import-graph query works |
| M9a | SCM miner + Gource replay | codegraph scm → deterministic history.jsonl; history summary/hotspots/authors reports match hand-counted fixture numbers; file-level city replay with timeline scrubber runs on codegraph's own history |
| M9b | Temporal store | ✅ sampled import --at revisions in model.db (orchestrated by codegraph snapshots); lifespans + codegraph timeline; hidden-coupling/ownership queries verified on gson history (55 release keyframes, 2008–2025); property suite green at every keyframe |
| M9c | Entity-level city replay | ✅ frozen union layout; temporal city.json with per-building series (codegraph replay); time colors (heat + age), owner color mode and dashed co-change arcs via the --history join; scrubbed replay reviewed as screenshots on gson at user-facing angles, allocation-free scrub path |
| NV | Navigator frontend | ✅ codegraph navigator --serve (since folded into codegraph serve: one page, the navigator with the city as a tab, a building's panel opening its navigator entry): @codegraph/navigator builds an index-addressed browsable artifact (tree + one classified dependency row per base edge, reference sub-roles recovered from the source entity); @codegraph/navigator-ui renders it — virtualized tree with search, fan-in/fan-out sectioned by role with member, provenance and anchor. Design record: docs/navigator.md. Verified on fineract (102 972 nodes / 668 286 rows, 100 MB artifact loading in 1.3 s, 45 rows mounted after scrolling) |
| M10a | Source links | ✅ header repository facts (remote, commit, repo-relative root, provider?) in core + schemas/ as patterns; extractor passthrough flags; CLI-derived normalization carried per snapshot frame; store/city/viz carry them and the details panel links at the scrubbed sha — verified on gson (JsonReader.java#L211-L2005 at b3f4ca2, replay retargeting to ed2b25d, fixture city linkless) |
| M10b | Measures | ✅ TMetrics (open keys, finite values) in core, schemas/ and the store (entity_metric rows, DB_VERSION 3); Java extractor emits sloc (lexer-based) + cyclomatic with a lambda's branches charged to the lambda; fixture CC values hand-counted, sloc ≤ span a property; --height sum:cyclomatic --footprint loc reviewed as screenshots on commons-lang (JavaVersion.get hand-count 33 = emitted 33) |
| M10c | Literal values | ✅ Literal + TWithValue + annotationUse in core, schemas/ (a named $defs/Literal) and the store (JSON columns, DB_VERSION 4); Java extractor carries annotation arguments, constant initializers and element defaults, unevaluated where it cannot fold; closure reaches inside values by ENCODING; fixture asserts the four Audited/MAX_LINES cases; gson and commons-lang diagnose clean |
| M10d | Framework semantics | ✅ data-driven Spring/Jakarta framework profile (5 roles, matching on name + module, stub-tolerant); derived DI dynamic-candidate wiring with @Primary/@Qualifier narrowing that states what it narrowed from; analyze --report wiring + city --framework role channel with its legend; hand-verified on spring-petclinic (a6e81a5: 5 injections → the one corpus impl, 4 → honest empty; HEAD: 6 implicit constructor injections), declared view byte-identical before and after |
| M11 | Insights walk | ✅ codegraph explain: @codegraph/insights (pure) + @codegraph/llm (the one SDK importer); units = operations → types → modules; one SCC-condensed dependency graph (calls, type deps, imports, downward containment) so mutually dependent packages/types/methods are ONE unit, Kahn-layered; context packs with dependency explanations at --depth; Specy-vocabulary blocks validated by Zod and sent as strict JSON Schema; Merkle fingerprints (inputs + dependency fingerprints + missing deps, never explanation text) make re-runs incremental; --dry-run/--max-calls/--scope/--concurrency/--max-scc; journal + sorted side-car <model>.insights.jsonl; cycle suite pinned to the analyzer's cycle report (opt-in real-corpus run via CODEGRAPH_CORPUS_MODEL; Fineract: 53 207 units, 18 package tangles, largest 509). Verified live on the Java fixture with gpt-5.6-luna: 77 units, 68 calls, $0.06 all-in, blocks in the Specy vocabulary (Order → entity with identity, StockGuard.ensure → precondition + error event, com.acme.order → APIs/SPI Ledger); the specy:domain-extract-from-code skill consumes the side-car (heuristics/codegraph.md). Design record: docs/insights.md |
| M12a | C# extractor — skeleton | ✅ extractors/csharp/ (Roslyn 5.9 on .NET 10, no MSBuild, BCL ref pack embedded — §13); csharp profile v2 in core; walking skeleton (namespaces, every type kind, delegates + parameters, doc comments, import/inheritance/implements, stubs) → fixtures/csharp/expected/model.jsonl byte-identical to core's encoder and profile-valid with zero issues; 51 .NET tests (per-line schema + sequence rules, stub discipline, determinism incl. CRLF and walk order, id scheme, CLI) + 13 core gate tests; codegraph validate OK; codegraph snapshots --extractor (jar or binary); extractor CLI contract as schemas/README.md §8; build.sh --csharp [--publish-all] / test.sh --csharp with the published-binary cmp |
| M12b | C# extractor — model | ✅ members (incl. implicit and primary constructors, operators, indexers, events, locals, lambdas, local functions), every profile edge kind incl. annotationUse with written values and throws, extension attachedTo, sloc + cyclomatic; synthesized record members fold to their type; stub discipline (BCL stubs in real namespaces, error types in <unresolved>, unbound receivers referenced by name); snapshot 244 entities / 285 edges, byte-identical to core's encoder; 76 .NET tests + 20 core gate tests; the fixture in the analyzer, city, navigator and CLI suites; every CLI command verified on it |
| M12c | C# extractor — binaries + audit | ✅ five-RID dotnet publish matrix cross-published from one Linux host (287 s; ELF x64/aarch64, Mach-O x64/arm64, PE32+); GitHub Actions gate (verify, java, csharp-test, csharp-publish ×5, csharp-smoke on Ubuntu x64/arm64, macOS arm64/Intel and Windows — each binary must reproduce the snapshot byte for byte — and tagged releases with SHA256SUMS); three-corpus audit: Humanizer 97.3 % / 12 146 entities / 6 s, dotnet/eShop 63.5 % / 7 396 / 12 s, OrchardCore 93.4 % / 88 007 entities / 225 184 edges / 54 s, every model validate-clean; five defects found and fixed (signatures carry type arguments, conversion operators their return type, duplicate parameter names their ordinal, same-keyed declarations across projects kept and re-keyed by file, C# 14 extension blocks); resolution causes measured — the SDK's implicit usings and the ASP.NET Core reference pack now in — and the residue named in the profile notes; eShop city and navigator screenshots reviewed; docs/csharp-extractor.md |
| M13a | TypeScript extractor — skeleton | ✅ extractors/typescript/ (the compiler API as the front end, no build, typescript the only runtime dependency — §14); typescript profile v2; walking skeleton → fixtures/typescript/expected/model.jsonl byte-identical to core's encoder and profile-valid with zero issues; core gate; boundary test; snapshots --extractor runs .js under node; test.sh --ts built-bin cmp |
| M13b | TypeScript extractor — model | ✅ members, every edge kind incl. annotationUse with written arguments and throws, space per entity, declaration merging per file, key escaping, JSX invocations, workspace-package resolution without node_modules, sloc + cyclomatic, literals; the full fixture with its README; determinism and stub-discipline tests; the fixture in every per-fixture suite |
| M13c | TypeScript extractor — self-hosting + audit | ✅ codegraph's own model (24 887 entities / 51 255 edges, 12 s, 96.5 %, zero <unresolved>) validate-clean with the package boundaries, the frontend types-only rule and the three import boundaries recovered as graph queries, city and navigator screenshots reviewed; TypeScript 4.9 compiler (52 341 / 120 127, 97.4 %) / nestjs (29 992 / 32 537, 88.6 %) / excalidraw (43 706 / 62 557, 94.4 %) audit with six defects fixed and the causes in the profile notes; typescript-smoke on three OSes and npm-publish on a tag; docs/typescript-extractor.md |
| Topic | Decision | Rationale |
|---|---|---|
| Schema lib | Zod v4 (not Malli) | types + runtime validation + JSON Schema export from one source |
| Interchange | JSON (not EDN), schemaVersion-ed |
TS-native; JSON Schema is the polyglot contract |
| Traits/profile equality (open point §2) | required ⊆ traits ⊆ required ∪ optional |
strict equality breaks on TComment; free subset hides extractor bugs |
| Marker traits | TWithInvocations etc. contribute no keys |
edge lists live in edges[], not on entities — keeps entities flat and avoids duplication |
| Java extractor language | Java (Maven) subproject, JSON out | Spoon is a JVM lib; the TS side stays extractor-agnostic |
| Lang ids (M1 review) | frozen as declared, abbreviations kept (clj/js/ts) |
the lang id is the EntityId prefix — renaming one invalidates every id an extractor has emitted |
| Stub containment (M3 review) | a stub type carries TChildOf → its stub module; never a corpus one, and never for primitives or in-corpus phantoms |
the analyzer folds to module level by walking parent and may not parse ids; the alternative — a stub being its own container at every level — put classes and primitives into module graphs (88% of gson's module nodes were not modules) |
Profile space (M1 audit) |
space? declared per kind on the Profile, not only on the Entity |
METAMODEL §1.4's "only meaningful in profiles that declare it" is otherwise unenforceable |
| Trait keys in the published schema (M1 audit) | re-stated as if/then conditionals generated from TRAITS |
Zod refinements do not survive z.toJSONSchema(); without them the contract accepted {traits:["TNamed"]} with no name |
| Identity v2 (fineract audit) | structured key (lang, module, symbol, disambiguator?); rendered id strings are display-only, never stored or compared |
559MB model.json broke Node's 512MB string ceiling; ≈76% of the bytes were repeated id/path strings |
| Interchange v2 | JSONL: surrogate ints for all intra-model refs, header dictionaries for closed vocabularies, file-path table, eof count trailer; in-place clean break, schemaVersion kept at 1.0.0, no old-format reader |
streaming in both directions kills the ceiling; ~6× smaller; extractor bar stays "anything that prints JSON lines"; the format is internal-only — bumping a version nobody consumes buys nothing |
children serialization (v2) |
dropped — derived from parent like every other inverse index |
invariant 4 already forbade serialized inverses; v1 carrying it was an inherited inconsistency (and 12MB on fineract) |
| Closure (M6) | enforced by the ENCODING: a reference is a surrogate, so a dangling one is unwritable and unreadable | the check moves from "report it afterwards" to "it cannot exist"; the extractor drops and counts what it cannot close |
| Executable containment (M6) | method/constructor/lambda declare TWithChildren |
their parameters and locals already carried parent; only one direction was licensed, so the model stated a containment its own profile forbade |
| Sync chunked reader (M6) | readModelFileSync reads 1MB at a time rather than making the CLI async |
the ceiling is one JavaScript string, not synchronous I/O — going async would change every command signature and fix nothing |
| SQLite (v2) | model.db is a derived, disposable analyzer cache built by codegraph import — never the contract, never committed |
queryable/incremental/random access for CLI + future viz without sacrificing diffable fixtures, byte-determinism, or the any-language extractor bar |
| Key separators (M5) | / and # reserved in the key's components; renderId validates and throws |
rendering must be injective, or two distinct keys merge into one entity with no error — the M2 overload collision one level up |
| Module component (M5) | a module names ITSELF, with an empty symbol — not its parent module | the parent form breaks the frozen java:com.acme.order id shape and needs a fabricated java module to place the stub package java:java.util |
| Trait-set interning (v2) | rejected — traits ride inline as int arrays; MM-4's validate-once-per-set is reader-side memoization | set indirection saved ~4MB on a ~90MB file but cost a record type, a dedup pass in every extractor, and lines unreadable in isolation |
| Evolution facts (Phase 8) | separate history.jsonl joined on anchor.file paths — never merged into model.jsonl, no fifth provenance value |
repo-scoped facts with a per-commit lifecycle don't belong in a language-scoped structural contract; the provenance set keeps code facts and history inferences unmixable |
| Lineage over time (Phase 8) | natural-key equality across snapshots; a rename is a death + a birth; anonymous entities untracked | invariant 7 makes temporal identity free for named entities; rename matching is heuristic machinery, deferred until a corpus proves it necessary |
| Replay layout (Phase 8) | one layout over the union of all keys that ever existed, plots frozen; buildings animate in place | shelf packing is chaotic — one insertion reshuffles the city; a readable replay needs positional stability more than land density |
| Snapshot strategy (Phase 8) | sampled keyframes via git worktree; per-commit incremental extraction deferred |
full extraction × thousands of commits is prohibitive, 50–200 frames give the replay effect; noClasspath makes non-compiling historic commits extractable |
| Repository provenance (Phase 9) | facts in the header — remote (normalized https), commit (sha), repo-relative root; host URL templates derived by consumers, provider only when the hostname lies |
a serialized URL freezes one host's scheme into the interchange; a sha is a permalink where a branch moves; anchors are relative to the analyzed root, which sits below the repo root (gson) — without the prefix no anchor projects back |
| Measures (Phase 9) | TMetrics open numeric map; canonical key names documented (METAMODEL §3.8), values validated finite, keys deliberately NOT a closed MM-3 vocabulary |
measures are the extractor-innovation surface — a closed set gates every new measure on a core release; loose top-level keys stay legal (container contract) but uncontractual, and only a trait makes sloc/cyclomatic comparable across extractors |
| Measure storage (M10b) | entity_metric(entity_id, key, value) rows in model.db, not a JSON column; the wire writes the map key-sorted, so reading back ORDER BY key restores it |
the store exists to be QUERIED and a measure is precisely what one aggregates ("cyclomatic by package"); a blob would make the one thing measures are for a full-table JSON scan |
| DI wiring (Phase 9) | derived by the analyzer from declared facts + a framework data table; dynamic-candidate provenance, in-memory only, matched by entity name — never id parsing |
the extractor stays framework-blind; the candidate set needs whole-corpus implementor knowledge only the analyzer holds; Spring dispatch is §1.3's dynamic-candidate definition verbatim |
| Literal values (Phase 9) | tagged-union Literal: numbers as canonical decimal text, enum values as type id + simple name, unfoldable constant expressions kept as unevaluated source text; ids inside values obey closure |
a JSON number loses a Java long; a fabricated enum-member stub is the one thing §6 forbids; dropping an unfoldable expression erases a written fact — degraded honesty over silent loss, the stub discipline applied to values |
| Annotation usage (Phase 9) | dedicated annotationUse edge kind carrying arguments, replacing the plain reference — in-place clean break, fixtures regenerated |
overloading reference would make arguments meaningful on one disguised subset of a kind; consumers cannot select annotation usages today without guessing from the target's kind, which a stub target cannot answer |
| C# corpus loading (M12) | Roslyn syntax + semantic model over a hand-built CSharpCompilation; never MSBuildWorkspace/Build.Locator |
the noClasspath contract: legacy corpora have no restorable project graph; MSBuild needs an installed SDK and breaks single-file publishing; a missing package must be a stub, not a build failure |
| C# unresolved types (M12) | IErrorTypeSymbol → stub in the reserved module csharp:<unresolved>, named as written; BCL/metadata types → stubs in their real namespace |
guessing a namespace from using directives would invent an FQN — the exact fabrication the whitelist rule exists to refuse; the using itself still yields the import edge |
| C# type identity (M12) | symbol carries generic arity via Roslyn MetadataName (Foo1`); signatures use erased FQ metadata names with type parameters as ordinals |
Foo, Foo<T>, Foo<T,U> legally coexist in one namespace — Java-style erasure would merge three declarations into one entity (the M2 overload collision one level up) |
| C# BCL references (M12) | Microsoft.NETCore.App.Ref embedded as resources, loaded via CreateFromImage; never Assembly.Location |
Location is empty inside a single-file bundle, so the tutorial approach binds nothing in the shipped binary while passing under dotnet run |
| C# distribution (M12) | self-contained single-file + ReadyToRun per RID, cross-published from Linux; NativeAOT and trimming deferred | Roslyn is not AOT/trim-clean, and NativeAOT needs each target OS's native toolchain; single-file needs only one runner for all five RIDs |
| Extractor CLI contract (M12) | one flag shape and exit-code set for every extractor, written in schemas/README.md; snapshots --extractor (jar → java -jar, else run directly) |
the Node side must stay language-blind: it orchestrates a process, not a language |
| Cross-OS acceptance (M12) | cmp of the published binary's output against the committed snapshot — run in CI on a runner of each OS since the move to GitHub Actions |
byte-determinism (§6 of the contract) makes "works on macOS/Windows" a one-line check; GitHub's hosted macOS and Windows runners make it a gate rather than a laptop ritual |
| C# signatures carry type arguments (M12c) | System.Func2<!!0,System.String>, not the erased System.Func2; conversion operators append their return type |
C# overloads on type arguments alone (Humanizer) and conversions on the return type alone (OrchardCore, 37 on one type); erasure merged written methods into one key and aborted the extraction |
| Same-keyed declarations across projects (M12c) | every one kept: the first in ordinal file order owns the plain key, each later one is re-keyed #in:<file>, each re-keying named on stderr |
a corpus is not a compilation unit (eShop's ten Program.<Main>$, its per-service Extensions); the JDT "first wins" rule the Java profile documents would erase nine services' DI wiring, and a file is a source fact like a lambda's position |
| Implicit usings (M12c) | Microsoft.NET.Sdk's seven global usings added as a synthetic tree by default; the Web SDK's opt-in (--implicit-usings web); none write an import edge |
they live in the generated obj/ file the extractor skips as build output; without them Task/List<T> were OrchardCore's top unresolved names (62.7 % → 93.4 %); the Web set on a mixed corpus makes names ambiguous (OrchardCore's own StartupBase, 345 times) |
| Reference packs (M12c) | the ASP.NET Core shared framework's pack embedded beside the BCL's when the building SDK has it | ILogger<T>, IServiceCollection, WebApplication topped eShop's and OrchardCore's unresolved lists; a shared framework is not a NuGet package and ships with every SDK |
| C# 14 extension blocks (M12c) | members of extension(T t) { … } are members of the enclosing static class with TAttachedTo → the receiver; the block itself is no entity |
Roslyn models the block as a nameless nested type, which failed validate on an empty name (Humanizer); the block names no type the source can reference |
| TypeScript front end (M13) | the typescript compiler API (ts.createProgram + TypeChecker) used directly; ts-morph, parser-only front ends (tree-sitter, swc, oxc, Babel) and scip-typescript rejected |
it is Roslyn's twin — a compilation plus a semantic model, error-tolerant by default; ts-morph wraps every node and lags releases; a parser has no binder, so every call would be a guess; SCIP carries occurrences, not kinds, provenance or spaces |
| TypeScript corpus loading (M13) | one Program over every source file under the roots; tsconfig read for resolution options only; a bare specifier naming a package declared under the roots resolves by package.json name to its source entry when standard resolution fails; a missing package is a stub module |
the noClasspath contract: no build, no node_modules, no project references; a monorepo's own packages are corpus, not dependencies |
| TypeScript extractor boundary (M13) | runtime dependency typescript only; @codegraph/core a devDependency of its tests; a boundary test scans extractors/typescript/src |
the two-encoder gate ("byte-identical to core's encoder") is a statement only while the encoders are independent; the temptation to import core is greatest in core's own language; a one-dependency package runs under npx anywhere |
| TypeScript module identity (M13) | the module is the file, always; every path segment and non-identifier name entering a key percent-encodes /, #, % (and . in names); declaration merging yields one entity per declaring file, edges land on the declaration owning the referenced member |
/ is a reserved separator and rendering must stay injective for every language; a look-alike separator is not typeable in a CLI or a SQL query; containment is where a thing is written (invariant 5) — a <global> module would state otherwise |
| TypeScript external keys (M13) | stubs keyed by npm package name (nearest package.json), lib types in <lib>, unbound names in <unresolved>, an unresolved specifier a stub module keyed by the specifier; --ignore-node-modules produces the fixture |
keys must not depend on what happens to be installed; resolvability is not membership (the C# rule); a dropped import edge would understate fan-out |
| TypeScript distribution (M13) | an npm package run with npx, typescript external to the bundle; a Node single-executable per OS deferred |
Node ≥ 22 is already the pipeline's floor; a bundle that inlines the compiler loses lib.*.d.ts and binds no standard library — the Assembly.Location trap in its Node form, pinned by a test on the built bin |
| Interface members are type-space (M13c) | a method or property written in an interface body carries space: ["type"]; the profile licenses both spaces on those kinds |
the self-hosting query "frontends depend on model packages for types only" was false with interface members in the value space — viz reads city.roles, a shape, not a value; an access to an interface member is a dependency on the interface's shape and is erased at runtime like the interface |
| What an object literal's parts are (M13c) | members of a literal BOUND to a name (const Ops = {…}, static x = {…}) are entities below the binding; members of an unbound literal (array elements, arguments, return values) are none, and a method written there is keyed positionally like an arrow; a class expression bound to a variable or property IS that binding |
2 260 re-keyed collisions on codegraph's own tests came from [{hash, time}, {hash, time}]; an unbound literal is no entity, so its parts cannot be either, while the functions written inside it are still code |
| Unclosable edges (M13c) | the extractor drops an edge whose endpoint no entity declares, counts it on stderr as unclosable, and never aborts |
schemas/README.md §5: a producer that cannot close a reference drops it and says so; two audited corpora aborted the write on a dangling constructor before this net existed — zero on every corpus is the goal, and a non-zero count names an id-scheme gap |