Skip to content

fix(explore): a symbol written with its package resolves, like refresh.Discovery.refresh - #2750

Merged
colbymchenry merged 1 commit into
mainfrom
claude/explore-go-package-tokens
Oct 21, 2026
Merged

colbymchenry merged 1 commit into
mainfrom
claude/explore-go-package-tokens

Conversation

@colbymchenry

Copy link
Copy Markdown
Owner

Summary

Follow-up found while validating #2748. Explore's token resolution, codegraph_node's lookup and codegraph_callers / codegraph_callees / codegraph_impact dropped every Go symbol written the way Go code names it:

  • refresh.Discovery.refresh, storage.Queryable.Querier and gin.Context.Next resolved to nothing, and so did binding.Default;
  • so a package-qualified question lost its symbols, and explore's Flow had nothing to connect.

That's also why the orchestrator's prometheus check (Discovery.refresh hetzner.refresher.refresh hcloudDiscovery.refresh) could never list robotDiscovery: hetzner.refresher.refresh named no symbol.

On current main:

gin  gin.Engine.ServeHTTP gin.Engine.handleHTTPRequest gin.Context.Next   → no Flow (generic search output)
prom promql.Engine.execEvalStmt storage.Queryable.Querier storage.fanout.Querier storage.NewMergeQuerier → no Flow

With this PR both render the full call path. The hetzner query's links now open with refresher.refresh → robotDiscovery.refresh @discovery/hetzner/robot.go:73.

Three gaps

  1. A package followed by a type. A qualified name was matched against the qualifiedName, which holds the type and not the package, or entirely against the path, which holds the package and not the type. It was never read as a package followed by a type.
    • New matchesSymbol stage 3 reads pkg.Type.method as leading qualifiers that name the package (path segments, as stage 2 does) and trailing ones that end the node's scope.
    • A Python pkg.mod.Class.method and a Rust module::Type::method read the same way.
  2. The package a Go file declares. A module's root package has no directory: gin's context.go is package gin.
    • A Go package qualifier now also matches the file's package clause, read by the new src/graph/go-package.ts.
    • That also covers versioned v2/ dirs and generated packages like prometheus's io_prometheus_client.
    • The clause is read last, and only where it can decide: a top-level function (gin.New), or a node whose scope already ends with the written type. A common name's hundreds of definitions cost no file reads; an earlier version without that guard added 40 ms to targetgroup.Group.String.
  3. A definition below the search cut.
    • Explore and codegraph_node took a qualified token's matches from the full-text search's top 50 only. For gin's binding.Default, those were three binding/default_validator*.go files.
    • Both now go through one helper, qualifiedSymbolMatches, which keeps the search order and falls back to the exact-name index when the search has no match. The CLI's lookupSymbolNodes already started at the exact-name index.

Every surface passes the same Go package reader, so they agree. The surfaces are:

  • explore: tokens, gather and path pins;
  • codegraph_node;
  • the callers/callees/impact tools;
  • the CLI.

Validation

Tests: __tests__/symbol-lookup-go-packages.test.ts (11 tests; unit plus an indexed Go fixture)

The fixture holds:

  • a root package (context.go, package gin);
  • binding.Default beside binding/default_validator.go, so the search answers with other nodes (asserted as a precondition);
  • two packages, each with its own Discovery.refresh.
variant pkg.Type.method root package Python Go-only clause search-cut fallback explore / node / CLI
main's sources fail fail fail pass fail fail ×3
this PR pass pass pass pass pass pass
no stage 3 fail fail fail pass pass fail ×3
no package clause pass fail pass pass pass fail ×3
clause for every language pass pass pass fail pass pass
no exact-name fallback pass pass pass pass fail pass

Token resolution: base = main, fix = this PR, one index per repo (both arms read it)

For every method and function, the spellings an agent writes, sampled (≤300 each):

  • Type.method: the control;
  • <dir>.Type.method;
  • <file stem>.Type.method;
  • <dir>.Func;
  • for Go, <package clause>.… where the clause isn't the directory.

Each runs through explore's resolver, codegraph_node's and the CLI's. The table shows how often the symbol the token was written from is found, explore's resolver, main → this PR:

repo Type.method (control) dir.Type.method stem.Type.method dir.Func Go package clause changed results
gin (Go) 300 identical 0 → 186 / 186 0 → 300 / 300 296 → 300 / 300 3 → 300 / 300 0
cobra (Go, root pkg) 169 identical 0 → 3 / 3 0 → 169 / 169 60 → 60 / 60 5 → 300 / 300 0
prometheus (Go) 300 identical 11 → 299 / 300 6 → 300 / 300 293 → 299 / 300 0 → 300 / 300 0
flask (Python) 300 identical 0 → 300 / 300 0 → 300 / 300 292 → 300 / 300 — 0
zod (TypeScript) 300 identical 14 → 300 / 300 18 → 300 / 300 284 → 300 / 300 — 0
spring-petclinic (Java) 192 identical 175 → 192 / 192 192 → 192 / 192 — — 0
ripgrep (Rust) 300 identical 0 → 299 / 300 0 → 300 / 300 292 → 300 / 300 — 0
  • "Changed results" counts every token, across all three resolvers, whose result was non-empty on main and differs at all, in nodes or order: 0 everywhere. A name that resolved before keeps its result.
  • The CLI already found 300/300 dir.Func (it starts at the exact-name index); explore and codegraph_node now match it.
  • The two misses are the same case.
    • prometheus prompb.Sample.String and ripgrep src.RegexMatcher.new: the qualifier fits several definitions (two prompb trees; every crate's src/).
    • The search found one of them, so the exact-name fallback didn't run. The CLI returns both.
    • I left explore/node search-first on purpose. Starting them at the exact-name index, as the CLI does, would change which six candidates an ambiguous token keeps, and so change existing explore answers. That deserves its own A/B.

Explore, real queries (main → this PR)

  • gin gin.Engine.ServeHTTP gin.Engine.handleHTTPRequest gin.Context.Next: no Flow → ServeHTTP → handleHTTPRequest → Next, plus the middleware-chain links.
  • gin gin.Context.ShouldBindWith binding.Binding.Bind binding.formBinding.Bind binding.mapForm: no Flow → ShouldBindWith → Binding.Bind → formBinding.Bind → mapForm.
  • prometheus promql.Engine.execEvalStmt storage.Queryable.Querier storage.fanout.Querier storage.NewMergeQuerier: no Flow → execEvalStmt → Queryable.Querier → fanout.Querier → NewMergeQuerier.
  • prometheus Discovery.refresh hetzner.refresher.refresh hcloudDiscovery.refresh: the links now open with refresher.refresh → robotDiscovery.refresh.

Controls and cost

  • Controls. The fix(explore): every dynamic-dispatch link gets its own line, the ones the query names first #2748 sweep's 601 gin and 524 prometheus queries, written as Type.method / bare names, produce byte-identical Flow text (0 differ).
  • Cost.
    • Explore resolver time per token, fix/main: median 0.75–1.02 across the seven repos. Base runs first and warms the search, so read that as "no measurable cost", not a speedup.
    • Worst extra +32 ms, on prometheus client.init: init is declared in many Go packages, and each candidate file's clause is read once.
    • The explore sweep's median is 0.96–0.98.

Suites

Full suite on this commit over main fc293f0, on Windows, with the kernel built, CODEGRAPH_KERNEL_EXPECT=1 and --maxWorkers=4 --minWorkers=1:

  • 694 files, 0 failed;
  • 8,303 tests: 8,221 passed, 0 failed, 82 skipped;
  • 19 kernel suites ran, 306 tests, all passed.

Follow-up found (next PR)

In the precise hetzner query (refresh.Discovery.refresh hetzner.refresher.refresh hcloudDiscovery.refresh), every link weighs the same. The generic refresher's 28 func-field targets come first in edge order and fill all six lines before hetzner's robotDiscovery. Interleaving links across the named symbols they hang from would list each named symbol's dispatch. That's the #2748 section's ordering, so it goes in its own PR.

🤖 Generated with Claude Code

…sh.Discovery.refresh`

Explore's token resolution, codegraph_node's lookup and the callers,
callees and impact lookups dropped every Go symbol written the way Go
code names it: `refresh.Discovery.refresh`, `storage.Queryable.Querier`
and `gin.Context.Next` resolved to nothing, and so did `binding.Default`.
Three gaps:

- A qualified name was matched against the qualifiedName, which holds
  the type and not the package, or entirely against the path, which
  holds the package and not the type, never as a package followed by a
  type. A new matcher stage reads `pkg.Type.method` as leading
  qualifiers that name the package by path segment and trailing ones
  that end the node's scope. A Python `pkg.mod.Class.method` and a Rust
  `module::Type::method` read the same way.
- A module's root package has no directory: gin's `context.go` is
  `package gin`. A Go package qualifier now also matches the package the
  file declares, read from its `package` clause (src/graph/go-package.ts).
  That covers versioned `v2/` directories and generated packages like
  prometheus's `io_prometheus_client` as well. The clause is read last
  and only where it can decide, a top-level function or a node whose
  scope already ends with the written type, so a common name's hundreds
  of definitions cost no file reads.
- Explore and codegraph_node took a qualified token's matches from the
  full-text search's top 50 only, and for gin's `binding.Default` those
  were three `binding/default_validator*.go` files. Through one shared
  helper they now fall back to the exact-name index when the search has
  no match; the CLI's lookup already started there.

The new stage runs only after the existing ones fail, and the search
keeps its order, so a name that resolved before keeps its result.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant