diff --git a/CHANGELOG.md b/CHANGELOG.md index f58218b4c7..e63ba6b180 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,15 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - CodeGraph's refusal to use a sensitive folder such as `~/.ssh`, `~/.config`, `/etc` or `C:\Windows` as a project can no longer be sidestepped: a link or junction that leads to one, a differently capitalized name on Windows and macOS, a `\\?\` spelling on Windows, and a path to a missing folder inside one are now refused like the folder itself. Before, any of these let an AI agent's query read the files inside. Thanks @mixxer for finding it. +### New Features + +- **Verilog and SystemVerilog are indexed.** Modules, packages, interfaces and modports, named instances, ports and signals, always/assign blocks and generate scopes are searchable, and `codegraph_explore` surfaces the instantiation path from a top module down to a nested one; based on the extractor contributed in #402 by @FHYQ-Dong. Re-index projects that contain Verilog files. +- **HDL port bindings resolve to their formal ports.** Named, positional and wildcard (`.*`) connections link an instance's local signals to the instantiated module's ports, and bit or part selects keep their base signal. +- **Signal readers and writers on request.** `codegraph_explore` with `hdlAccess` (CLI: `codegraph explore --hdl-access`) lists where a signal is read, written, used as a control condition or as a clock/reset event, including the direction of arguments passed to known functions and tasks. +- **HDL build profiles.** An optional `hdl` section in `codegraph.json` selects source files or filelists, include directories and defines; conditional-compilation branches are indexed under the active profile, `codegraph status` reports the configured versus the indexed profile, and the file watcher follows filelists and included headers. +- **Computed parameters and port widths.** `codegraph hdl-semantic` runs an installed slang (or pyslang through `--python`) on the active profile and returns evaluated parameters, port widths and macro origins as a separate answer, without touching the indexed graph. +- **A derived parameter depends on the parameters it is computed from.** `localparam K = (N + 1) * 2;` links `K` to `N`, so impact on a width or depth parameter follows the whole chain of `localparam`s built on it, inside its own module. Suggested by @FHYQ-Dong. + ### Fixes - In TypeScript, an awaited call with type arguments like `await fetchApi("/me")` or `await api.get(url)` now links to the function it calls. Before, the call was dropped, so typed fetch wrappers looked like they had no callers. (#2586) diff --git a/README.md b/README.md index baac93178c..92b48930e6 100644 --- a/README.md +++ b/README.md @@ -278,7 +278,7 @@ CodeGraph's parsing engine is a **native Rust kernel**: 20 languages — TypeScr | **Full-Text Search** | Find code by name instantly across your entire codebase, powered by FTS5 | | **Impact Analysis** | Trace callers, callees, and the full impact radius of any symbol before making changes | | **Always Fresh** | File watcher uses native OS events (FSEvents/inotify/ReadDirectoryChangesW) with debounced auto-sync — the graph stays current as you code, zero config | -| **20+ Languages** | TypeScript, JavaScript, ArkTS, Python, Go, Rust, Java, C#, VB.NET, PHP, Ruby, C, C++, CUDA, Objective-C, Metal, Swift, Kotlin, Scala, Dart, Lua, Luau, R, Nix, Erlang, CFML, COBOL, Solidity, Terraform/OpenTofu, Svelte, Vue, Astro, Liquid, Pascal/Delphi | +| **20+ Languages** | TypeScript, JavaScript, ArkTS, Python, Go, Rust, Java, C#, VB.NET, PHP, Ruby, C, C++, CUDA, Objective-C, Metal, Swift, Kotlin, Scala, Dart, Lua, Luau, R, Nix, Erlang, CFML, COBOL, Solidity, Terraform/OpenTofu, Svelte, Vue, Astro, Liquid, Pascal/Delphi, Verilog/SystemVerilog | | **Framework-aware Routes** | Recognizes web-framework routing files and links URL patterns to their handlers across 17 frameworks | | **Mixed iOS / React Native / Expo** | Closes cross-language flows that static parsing misses: Swift ↔ ObjC bridging, React Native legacy bridge + TurboModules + Fabric view components, native → JS event emitters, Expo Modules | | **100% Local** | No data leaves your machine. No API keys. No external services. SQLite database only | @@ -835,6 +835,7 @@ is written): | Astro | `.astro` | Full support (frontmatter + script extraction, template component/call references, `src/pages/` routes) | | Liquid | `.liquid` | Full support | | Pascal / Delphi | `.pas`, `.dpr`, `.dpk`, `.lpr` | Full support (classes, records, interfaces, enums, DFM/FMX form files) | +| Verilog / SystemVerilog | `.v`, `.vh`, `.sv`, `.svh` | Modules, packages, interfaces/modports and named instances; ports and signals with named/positional/wildcard port bindings, bit/part selects and always/assign blocks; lexical and generate scopes; read/write/control/event signal-access queries; optional build profiles (filelists/includes/defines, see [docs/hdl-profiles.md](docs/hdl-profiles.md)). Source graph only — no elaboration or timing; `codegraph hdl-semantic` computes parameters and widths through an installed [slang](docs/hdl-semantics.md) | | Lua | `.lua` | Full support (functions, methods with receivers, local variables, `require` imports, call edges) | | R | `.R` `.r` | Full support (functions in every assignment form, S4/R5/R6 classes with methods, `library`/`require` imports, `source()` file references, call edges) | | Luau | `.luau` | Full support (everything in Lua, plus `type`/`export type` aliases, typed signatures, and Roblox instance-path `require`) | diff --git a/__tests__/extraction.test.ts b/__tests__/extraction.test.ts index 2814a4a2c3..591c5a4d2c 100644 --- a/__tests__/extraction.test.ts +++ b/__tests__/extraction.test.ts @@ -14415,3 +14415,188 @@ describe('C++ COM interface declarations (#1519)', () => { expect(Buffer.byteLength(expected)).toBe(Buffer.byteLength(source)); }); }); + +describe('Verilog / SystemVerilog Extraction', () => { + it('should report Verilog as supported', () => { + expect(isLanguageSupported('verilog')).toBe(true); + expect(getSupportedLanguages()).toContain('verilog'); + }); + + it('should extract modules as containers with scoped functions and tasks', () => { + const code = ` +module top (input logic clk, output logic [7:0] z); + function automatic int square(int n); + return n * n; + endfunction + task automatic do_reset(); + z = '0; + endtask +endmodule +`; + const result = extractFromSource('top.sv', code); + + const mod = result.nodes.find((n) => n.kind === 'class' && n.name === 'top'); + expect(mod).toBeDefined(); + expect(mod?.language).toBe('verilog'); + + const fn = result.nodes.find((n) => n.kind === 'function' && n.name === 'square'); + expect(fn).toBeDefined(); + expect(fn?.qualifiedName).toBe('top::square'); // scoped under the module + + const task = result.nodes.find((n) => n.kind === 'function' && n.name === 'do_reset'); + expect(task).toBeDefined(); + }); + + it('should emit an instantiates reference from a module to its submodule type', () => { + const code = ` +module alu #(parameter int WIDTH = 8) (input logic [WIDTH-1:0] a, output logic [WIDTH-1:0] y); +endmodule + +module top (input logic [7:0] x, output logic [7:0] z); + alu #(.WIDTH(8)) u_alu (.a(x), .y(z)); +endmodule +`; + const result = extractFromSource('soc.sv', code); + + const top = result.nodes.find((n) => n.kind === 'class' && n.name === 'top'); + expect(top).toBeDefined(); + + const inst = result.unresolvedReferences.find( + (r) => r.referenceKind === 'instantiates' && r.referenceName === 'alu' + ); + expect(inst).toBeDefined(); + expect(inst?.fromNodeId).toBe(top?.id); // the parent module is the source + + // parameter is captured as a constant + const param = result.nodes.find((n) => n.kind === 'constant' && n.name === 'WIDTH'); + expect(param).toBeDefined(); + }); + + it('should extract package typedefs, functions, imports and call edges', () => { + const code = ` +package math_pkg; + typedef enum logic [1:0] { IDLE, RUN, DONE } state_t; + function automatic int add(int a, int b); + return a + b; + endfunction +endpackage + +module worker (input logic clk); + import math_pkg::*; + function automatic int caller(int y); + return add(y); + endfunction +endmodule +`; + const result = extractFromSource('pkg.sv', code); + + const pkg = result.nodes.find((n) => n.kind === 'class' && n.name === 'math_pkg'); + expect(pkg).toBeDefined(); + + const typedef = result.nodes.find((n) => n.kind === 'type_alias' && n.name === 'state_t'); + expect(typedef).toBeDefined(); + expect(typedef?.qualifiedName).toBe('math_pkg::state_t'); + + const importNode = result.nodes.find((n) => n.kind === 'import' && n.name === 'math_pkg'); + expect(importNode).toBeDefined(); + + const callRef = result.unresolvedReferences.find( + (r) => r.referenceKind === 'calls' && r.referenceName === 'add' + ); + expect(callRef).toBeDefined(); + }); + + it('should extract every package in a multi-package import statement', () => { + const code = ` +package a_pkg; + function automatic int fa(); return 1; endfunction +endpackage + +package b_pkg; + function automatic int fb(); return 2; endfunction +endpackage + +module worker (input logic clk); + import a_pkg::*, b_pkg::*; + function automatic int caller(); + return fa() + fb(); + endfunction +endmodule +`; + const result = extractFromSource('multi.sv', code); + + const imports = result.nodes.filter((n) => n.kind === 'import').map((n) => n.name); + expect(imports).toContain('a_pkg'); + expect(imports).toContain('b_pkg'); // the second import must not be dropped + + // call edges still resolve when multiple imports share one statement + const calls = result.unresolvedReferences + .filter((r) => r.referenceKind === 'calls') + .map((r) => r.referenceName); + expect(calls).toContain('fa'); + expect(calls).toContain('fb'); + }); + + it('should capture function calls inside instantiation port/param expressions', () => { + const code = ` +module sub #(parameter int W = 8) (input logic [W-1:0] a); +endmodule + +module top (input logic [7:0] x); + function automatic int dbl(int v); return v * 2; endfunction + function automatic int wid(); return 8; endfunction + sub #(.W(wid())) u_sub (.a(dbl(x))); +endmodule +`; + const result = extractFromSource('inst_calls.sv', code); + + const calls = result.unresolvedReferences + .filter((r) => r.referenceKind === 'calls') + .map((r) => r.referenceName); + expect(calls).toContain('dbl'); // call inside a port connection + expect(calls).toContain('wid'); // call inside a parameter override + + // the instantiation edge itself is still emitted + const inst = result.unresolvedReferences.find( + (r) => r.referenceKind === 'instantiates' && r.referenceName === 'sub' + ); + expect(inst).toBeDefined(); + }); + + it('should emit an import for each `include directive, named by the included path', () => { + const code = ` +\`include "defs.vh" +\`include "axi/typedef.svh" +\`include +module top (input logic clk); +endmodule +`; + const result = extractFromSource('top.sv', code); + + const includes = result.nodes.filter((n) => n.kind === 'import').map((n) => n.name); + expect(includes).toEqual(['defs.vh', 'axi/typedef.svh', 'uvm_macros.svh']); + + // the path is what the import ref carries, so the file-path matcher can + // land it on the header by suffix + const refs = result.unresolvedReferences + .filter((r) => r.referenceKind === 'imports') + .map((r) => r.referenceName); + expect(refs).toEqual(['defs.vh', 'axi/typedef.svh', 'uvm_macros.svh']); + }); + + it('should extract `define macros and top-level parameters from a header', () => { + const code = ` +\`ifndef DEFS_VH +\`define DEFS_VH +\`define DATA_W 16 +\`define ADDR(x) (x + 1) +parameter CLK_HZ = 27_000_000; +localparam BAUD = 115200; +\`endif +`; + const result = extractFromSource('defs.vh', code); + + const constants = result.nodes.filter((n) => n.kind === 'constant').map((n) => n.name); + expect(constants).toEqual(['DEFS_VH', 'DATA_W', 'ADDR', 'CLK_HZ', 'BAUD']); + }); +}); diff --git a/__tests__/hdl-access-query.test.ts b/__tests__/hdl-access-query.test.ts new file mode 100644 index 0000000000..f2ffb0644f --- /dev/null +++ b/__tests__/hdl-access-query.test.ts @@ -0,0 +1,130 @@ +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; +import { spawnSync } from 'node:child_process'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { CodeGraph } from '../src'; +import { ToolHandler } from '../src/mcp/tools'; +import { formatHdlAccess } from '../src/mcp/hdl-access'; + +let root: string, cg: CodeGraph, handler: ToolHandler; +const rtl = `module top(input clk, input enable, input d, output logic q); + always @(posedge clk) begin + if (enable) q <= q + d; + end + leaf u(.d(d)); +endmodule +module leaf(input d); endmodule +module other(input d, output q); assign q=d; endmodule +module updates(output int q); initial q += 1; endmodule +`; +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'hdl-access-query-')); + fs.writeFileSync(path.join(root, 'top.sv'), rtl); + // Integral seconds survive Date/fs.utimes round trips on all supported filesystems. + const fixtureTime = new Date('2024-01-01T00:00:00.000Z'); + fs.utimesSync(path.join(root, 'top.sv'), fixtureTime, fixtureTime); + fs.writeFileSync(path.join(root, 'lib.sv'), 'package lib; task automatic fill(output int y); y=1; endtask endpackage'); + fs.writeFileSync(path.join(root, 'call.sv'), 'module call_top; int y; initial lib::fill(y); endmodule'); + fs.writeFileSync(path.join(root, 'plain.ts'), 'export function hello() { return 42; }\n'); + cg = CodeGraph.initSync(root); + expect((await cg.indexAll()).success).toBe(true); + handler = new ToolHandler(cg); +}); +afterAll(() => { cg?.close(); if (root) fs.rmSync(root, { recursive: true, force: true }); }); +async function query(signal: string, hdlAccess?: string) { + const result = await handler.execute('codegraph_explore', { query: signal, ...(hdlAccess ? { hdlAccess } : {}) }); + expect(result.isError).not.toBe(true); + return result.content.map(c => c.text).join('\n'); +} + +describe('HDL access MCP query', () => { + it('returns actual source sites and retains reads and writes on the same signal and line', async () => { + const readers = await query('top::q', 'read'); + const writers = await query('top::q', 'write'); + for (const result of [readers, writers]) { + expect(result).toContain('1 matching access site(s).'); + expect(result).toContain('3\t if (enable) q <= q + d;'); + expect(result).toContain('top.sv:3'); + } + expect(readers).toContain('[read]'); + expect(writers).toContain('[write]'); + expect(await query('top.q', 'write')).toContain('**top::q**'); + expect(await query('top::q', 'all')).toContain('2 matching access site(s).'); + expect(await query('updates::q', 'readwrite')).toContain('[readwrite]'); + }); + it('explains writes proven by a known task output formal', async () => { + const result = await query('call_top::y', 'write'); + expect(result).toContain('1 matching access site(s).'); + expect(result).toContain('[write]'); + expect(result).toContain('Formal direction: lib::fill(output y).'); + expect(result).toContain('initial lib::fill(y);'); + expect(result).toContain("known callable formal's argument direction"); + }); + it('includes control and event uses as readers and permits exact category filters', async () => { + expect(await query('top::clk', 'read')).toContain('[event]'); + expect(await query('top::clk', 'event')).toContain('[event]'); + expect(await query('top::enable', 'control')).toContain('[control]'); + expect(await query('top::enable', 'read')).toContain('[control]'); + expect(await query('top::d', 'write')).toContain('0 matching access site(s).'); + }); + it('keeps unclassified connections separate from access evidence and disambiguates exact targets', async () => { + expect(await query('top::d', 'all')).toContain('unclassified reference(s) excluded'); + const all = await query('q', 'write'); + expect(all).toContain('**top::q**'); + expect(all).toContain('**other::q**'); + expect(await query('top::q', 'write')).not.toContain('**other::q**'); + expect(await query('where is q written', 'write')).toContain('No exact HDL signal matches'); + }); + it('does not change normal non-HDL exploration and validates the filter', async () => { + const normal = await query('hello'); + expect(normal).toContain('hello'); + expect(normal).toContain('return 42'); + expect(normal).not.toContain('HDL access:'); + const bad = await handler.execute('codegraph_explore', { query: 'q', hdlAccess: 'driver' }); + expect(bad.isError).toBe(true); + expect(bad.content[0]!.text).toContain('hdlAccess must be one of'); + }); + it('rejects same-size edited source with restored mtime and labels the indexed definition', () => { + const file = path.join(root, 'top.sv'); + const stat = fs.statSync(file); + try { + fs.writeFileSync(file, rtl.replace('q <= q + d', 'q <= q - d')); + fs.utimesSync(file, stat.atime, stat.mtime); + const changed = fs.statSync(file); + expect(changed.size).toBe(stat.size); + expect(Math.floor(changed.mtimeMs)).toBe(Math.floor(stat.mtimeMs)); + const result = formatHdlAccess(cg, 'top::q', 'write'); + expect(result).toContain('1 matching access site(s).'); + expect(result).toContain('Definition location is from the index; current source is unverified'); + expect(result).toContain('Source omitted: changed/unavailable since indexing'); + expect(result).not.toContain('q <= q - d'); + expect(result).not.toContain('q <= q + d'); + } finally { fs.writeFileSync(file, rtl); fs.utimesSync(file, stat.atime, stat.mtime); } + }); + it('does not attach stale indexed access positions to modified source', () => { + const file = path.join(root, 'top.sv'); + const stat = fs.statSync(file); + try { + fs.writeFileSync(file, `// changed source\n${rtl}`); + const result = formatHdlAccess(cg, 'top::q', 'write'); + expect(result).toContain('Source omitted: changed/unavailable since indexing'); + expect(result).not.toContain('3\t if (enable)'); + } finally { fs.writeFileSync(file, rtl); fs.utimesSync(file, stat.atime, stat.mtime); } + }); +}); + +it('HDL access CLI forwards the existing explore filter and rejects invalid values', () => { + const bin = path.resolve(__dirname, '../dist/bin/codegraph.js'); + const run = (filter: string) => spawnSync(process.execPath, [bin, 'explore', '--path', root, '--hdl-access', filter, 'top::q'], { + encoding: 'utf8', timeout: 30000, + env: { ...process.env, CODEGRAPH_NO_DAEMON: '1', CODEGRAPH_WASM_RELAUNCHED: '1', NO_COLOR: '1' }, + }); + const result = run('write'); + expect(result.status, result.stderr).toBe(0); + expect(result.stdout).toContain('HDL access: top::q — write'); + expect(result.stdout).toContain('q <= q + d;'); + const invalid = run('driver'); + expect(invalid.status).toBe(1); + expect(invalid.stdout + invalid.stderr).toContain('hdlAccess must be one of'); +}); diff --git a/__tests__/hdl-expression-origin.test.ts b/__tests__/hdl-expression-origin.test.ts new file mode 100644 index 0000000000..7709f93821 --- /dev/null +++ b/__tests__/hdl-expression-origin.test.ts @@ -0,0 +1,74 @@ +import { afterEach, expect, it, vi } from 'vitest'; +import { createHash } from 'node:crypto'; +import { importMappedSemantics } from '../src/hdl/source-map-import'; +import { analyzeHdlSemantics, type HdlSemanticSourceGraph } from '../src/hdl/semantics'; +import type { HdlProfileStatus } from '../src/hdl/status'; +import type { FileRecord, Node } from '../src/types'; +const {run}=vi.hoisted(()=>({run:vi.fn()})); +vi.mock('../src/hdl/semantic-runner',()=>({runSlangSemantics:run})); +afterEach(()=>run.mockReset()); +const sources={'defs.svh':'`define WIDTH 8\r\n','top.sv':'// Ω\r\nmodule top #(parameter W = `WIDTH)(); endmodule\r\n'}; +function point(file:keyof typeof sources,byteOffset:number){const prefix=Buffer.from(sources[file]).subarray(0,byteOffset);return {file,byteOffset,line:[...prefix].filter(n=>n===10).length+1,column:byteOffset-prefix.lastIndexOf(10)};} +const start=Buffer.from(sources['top.sv']).indexOf('`WIDTH'); +function origin(role='initializer') {return {role,sourceOrigin:'macro',source:{file:'top.sv',line:2,column:28},macroExpansionComplete:true, + macroExpansion:[{name:'WIDTH',argument:false,spelling:point('defs.svh',Buffer.from(sources['defs.svh']).indexOf('8')),invocation:{start:point('top.sv',start),end:point('top.sv',start+6)}}]};} +function fact(){return {kind:'parameter',name:'W',instancePath:'top',value:'8',type:'int',width:32,sourceOrigin:'direct',source:{file:'top.sv',line:2,column:24}, + expressionOrigins:[origin()],expressionOriginCoverage:{initializer:'checked',declaredInitializer:'not-applicable',type:'not-applicable',truncated:false}};} +const payload=(f:unknown=fact())=>({codegraphSemanticVersion:1,facts:[f]}); + +it('keeps a direct declaration link when its initializer comes from a macro, preserving distinct scopes of provenance',async()=>{ + const graph:HdlSemanticSourceGraph={profile:()=>({state:'matches',indexed:{configurationFingerprint:'cfg'}} as HdlProfileStatus),stale:()=>false, + file:()=>({contentHash:createHash('sha256').update(sources['top.sv']).digest('hex')} as FileRecord),nodes:()=>[{id:'decl-W',kind:'constant',name:'W',startLine:2} as Node]}; + run.mockResolvedValue({frontend:'pyslang',ast:payload(),sources,sourceRoot:'/snapshot',profileName:'p',configurationFingerprint:'cfg',fingerprint:'snapshot', + version:'fixture',executableSha256:'python',exporterSha256:'exporter',librarySha256:'native',top:'top',parameters:{},allowUseBeforeDeclare:false, + languageStandard:'1800-2023',compilationUnitMode:'separate',runnerVersion:'test',compilerLimits:[],diagnostics:[]}); + const result=await analyzeHdlSemantics('/project',{pythonExecutable:'python',query:'top.W'},graph); + expect(result.facts[0]!.sourceNodeId).toBe('decl-W'); + expect(result.facts[0]!.sourceOrigin).toBe('direct'); + expect(result.facts[0]!.expressionOrigins![0]!.sourceOrigin).toBe('macro'); + expect(result.facts[0]!.expressionOrigins![0]!.macroExpansion[0]!.spelling!.file).toBe('defs.svh'); +}); + +it('preserves separate declared-initializer and type coverage when the effective initializer is supplied on the command line',()=>{ + const f=fact();f.expressionOriginCoverage={initializer:'command-line',declaredInitializer:'checked',type:'checked',truncated:true}; + f.expressionOrigins=[origin('declared-initializer'),origin('type')]; + const normalized=importMappedSemantics(payload(f),sources)[0]!; + expect(normalized.expressionOrigins!.map(e=>e.role)).toEqual(['declared-initializer','type']); + expect(normalized.expressionOriginCoverage).toEqual(f.expressionOriginCoverage); +}); + +it.each(['initializer','declaredInitializer','type'])('rejects non-string coverage even when coercion would resemble a known enum: %s',key=>{ + const f:any=fact();f.expressionOrigins=[];f.expressionOriginCoverage[key]=['checked']; + expect(()=>importMappedSemantics(payload(f),sources)).toThrow('coverage'); +}); + +it('rejects missing coverage, invalid roles/origins, contradictory coverage and malformed truncation',()=>{ + let f:any=fact();delete f.expressionOriginCoverage;expect(()=>importMappedSemantics(payload(f),sources)).toThrow('origins'); + f=fact();f.expressionOrigins[0].sourceOrigin='direct';expect(()=>importMappedSemantics(payload(f),sources)).toThrow('role'); + f=fact();f.expressionOrigins[0].role='expression';expect(()=>importMappedSemantics(payload(f),sources)).toThrow('role'); + f=fact();f.expressionOriginCoverage.initializer='unavailable';expect(()=>importMappedSemantics(payload(f),sources)).toThrow('contradicts'); + f=fact();f.expressionOriginCoverage.declaredInitializer='command-line';expect(()=>importMappedSemantics(payload(f),sources)).toThrow('coverage'); + f=fact();f.expressionOriginCoverage.truncated='false';expect(()=>importMappedSemantics(payload(f),sources)).toThrow('coverage'); +}); + +it('uses immutable source and byte-coordinate checks for every expression macro frame and source',()=>{ + let f:any=fact();f.expressionOrigins[0].macroExpansion[0].spelling.file='../defs.svh';expect(()=>importMappedSemantics(payload(f),sources)).toThrow('snapshot'); + f=fact();f.expressionOrigins[0].macroExpansion[0].spelling.byteOffset++;expect(()=>importMappedSemantics(payload(f),sources)).toThrow('disagree'); + f=fact();f.expressionOrigins[0].source={file:'top.sv',line:1,column:5};expect(()=>importMappedSemantics(payload(f),sources)).toThrow('UTF-8'); + f=fact();f.expressionOrigins[0].macroExpansion[0].invocation.end={...point('top.sv',start),file:'defs.svh'};expect(()=>importMappedSemantics(payload(f),sources)).toThrow(); +}); + +it('rejects duplicate normalized entries even if their raw objects differ by unrecognized fields',()=>{ + const f:any=fact();f.expressionOrigins.push({...origin(),ignored:'not-an-identity'}); + expect(()=>importMappedSemantics(payload(f),sources)).toThrow('duplicate expression'); +}); + +it('enforces the per-fact expression cap and one shared frame budget for declaration and expression origins',()=>{ + const f:any=fact();f.expressionOrigins=Array.from({length:33},()=>origin());expect(()=>importMappedSemantics(payload(f),sources)).toThrow('origins'); + const chain=(name:string)=>Array.from({length:128},(_,i)=>({name:i===0?name:'nested',argument:false})); + const facts=Array.from({length:25},(_,i)=>({...fact(),name:'W'+i,sourceOrigin:'macro',macroExpansion:chain('declaration'),macroExpansionComplete:false, + expressionOrigins:Array.from({length:31},(_,j)=>({role:'initializer',sourceOrigin:'macro',macroExpansion:chain('expression'+j),macroExpansionComplete:false}))})); + // Expressions alone have 99,200 frames; only a shared declaration+expression + // budget rejects this 102,400-frame payload. + expect(()=>importMappedSemantics({codegraphSemanticVersion:1,facts},sources)).toThrow('frame count'); +}); diff --git a/__tests__/hdl-instances.test.ts b/__tests__/hdl-instances.test.ts new file mode 100644 index 0000000000..70e74656f1 --- /dev/null +++ b/__tests__/hdl-instances.test.ts @@ -0,0 +1,46 @@ +import { it, expect } from 'vitest'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { CodeGraph } from '../src'; + +it('keeps named HDL instances, source connections and generate scopes without elaborating arrays', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-hdl-instances-')); + let cg: CodeGraph | undefined; + try { + fs.writeFileSync(path.join(root, 'top.sv'), `module leaf #(parameter W=8) (input a); endmodule +module top(input a); + leaf #(.W(4)) first(.a(a)), bank[3:0](a); + if (1) begin : yes leaf u(.a(a)); end else begin : no leaf u(.*); end + for(genvar i=0;i<4;i++) begin : lanes leaf repeated(.a); end + if (1) begin leaf anonymous(.a(a)); end +endmodule +`); + cg = CodeGraph.initSync(root); + expect((await cg.indexAll()).success).toBe(true); + const instances = ['first', 'bank', 'u', 'repeated', 'anonymous'].flatMap(name => + cg!.getNodesByName(name).filter(n => n.kind === 'variable')); + expect(instances).toHaveLength(6); + expect(new Set(instances.map(n => n.id)).size).toBe(6); + expect(instances.find(n => n.name === 'first')?.signature).toBe('leaf #(.W(4)) first(.a(a))'); + expect(instances.find(n => n.name === 'bank')?.signature).toBe('leaf #(.W(4)) bank[3:0](a)'); + expect(instances.filter(n => n.name === 'u').map(n => n.qualifiedName).sort()).toEqual(['top::no::u', 'top::yes::u']); + expect(instances.find(n => n.name === 'repeated')?.qualifiedName).toBe('top::lanes::repeated'); + expect(instances.find(n => n.name === 'anonymous')?.qualifiedName).toMatch(/^top::generate@6:\d+::anonymous$/); + const targets = (id: string) => cg!.getCallees(id).filter(e => e.edge.kind === 'instantiates').map(e => e.node.name); + for (const instance of instances) expect(targets(instance.id)).toEqual(['leaf']); + expect(targets(cg.getNodesByName('top')[0].id)).toEqual(['leaf']); + for (const instance of instances.filter(n => n.signature?.includes('.a(') || n.name === 'bank' || n.name === 'repeated')) { + expect(cg.getCallees(instance.id).filter(e => e.edge.kind === 'references').map(e => e.node.qualifiedName)).toEqual(['top::a']); + } + const wildcard = instances.find(n => n.signature === 'leaf u(.*)')!; + expect(cg.getCallees(wildcard.id).filter(e => e.edge.kind === 'references')).toEqual([]); + const before = instances.map(n => n.id).sort(); + await cg.indexAll(); + expect(['first', 'bank', 'u', 'repeated', 'anonymous'].flatMap(name => + cg!.getNodesByName(name).filter(n => n.kind === 'variable').map(n => n.id)).sort()).toEqual(before); + } finally { + cg?.close(); + fs.rmSync(root, { recursive: true, force: true }); + } +}); diff --git a/__tests__/hdl-port-bindings.test.ts b/__tests__/hdl-port-bindings.test.ts new file mode 100644 index 0000000000..e906b39d85 --- /dev/null +++ b/__tests__/hdl-port-bindings.test.ts @@ -0,0 +1,112 @@ +import { it, expect } from 'vitest'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { CodeGraph } from '../src'; + +async function graph(files: Record, check: (cg: CodeGraph, root: string) => Promise | void): Promise { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'hdl-ports-')); + let cg: CodeGraph | undefined; + try { + for (const [file, source] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, file)), { recursive: true }); + fs.writeFileSync(path.join(root, file), source); + } + cg = CodeGraph.initSync(root); + expect((await cg.indexAll()).success).toBe(true); + await check(cg, root); + } finally { cg?.close(); fs.rmSync(root, { recursive: true, force: true }); } +} +function bindings(cg: CodeGraph) { + return (cg as any).db.db.prepare("select id, name, qualified_name qualifiedName, signature from nodes where kind='property' and language='verilog' and decorators LIKE '%hdl:named%'").all() as Array<{ id: string; name: string; qualifiedName: string; signature: string }>; +} +const leaf = 'module leaf(input rx, output tx); endmodule'; + +it('maps named and shorthand ports to formal declarations and local expressions, including explicit open ports', async () => { + await graph({ 'leaf.sv': leaf, 'top.sv': `module top(input wire rx, output wire tx); +leaf u(.rx(rx), .tx()); leaf bank[3:0](.rx, .tx(tx)); +leaf positional(rx,tx); leaf wildcard(.*); +endmodule` }, cg => { + const ports = bindings(cg); + expect(ports).toHaveLength(4); + const endpoints = (q: string) => cg.getCallees(ports.find(n => n.qualifiedName === q)!.id) + .filter(e => e.edge.kind === 'references').map(e => e.node.qualifiedName).sort(); + expect(endpoints('top::u::rx')).toEqual(['leaf::rx', 'top::rx']); + expect(endpoints('top::u::tx')).toEqual(['leaf::tx']); + expect(endpoints('top::bank::rx')).toEqual(['leaf::rx', 'top::rx']); + expect(endpoints('top::bank::tx')).toEqual(['leaf::tx', 'top::tx']); + expect(ports.find(n => n.qualifiedName === 'top::u::tx')!.signature).toBe('.tx()'); + expect(cg.getNodesByName('bank').filter(n => n.kind === 'variable')).toHaveLength(1); + }); +}); + +it('does not borrow missing formal ports from another module and drops removed targets after reindex', async () => { + await graph({ 'leaf.sv': leaf, 'foreign.sv': 'module foreign(input missing); endmodule', + 'top.sv': 'module top(input sig); leaf u(.rx(sig), .missing(sig)); endmodule' }, async (cg, root) => { + const formalTargets = (name: string) => cg.getCallees(bindings(cg).find(n => n.name === name)!.id) + .filter(e => e.edge.metadata?.binding === 'hdl-named-port').map(e => e.node.qualifiedName); + expect(formalTargets('rx')).toEqual(['leaf::rx']); + expect(formalTargets('missing')).toEqual([]); + const before = bindings(cg).map(n => n.id).sort(); + fs.writeFileSync(path.join(root, 'leaf.sv'), 'module leaf(output tx); endmodule'); + expect((await cg.indexAll()).success).toBe(true); + expect(formalTargets('rx')).toEqual([]); + expect(bindings(cg).map(n => n.id).sort()).toEqual(before); + }); +}); + +it('uses the same synthesis versus simulation target as the instance', async () => { + await graph({ 'rtl/leaf.sv': leaf, 'sim/leaf_stub.sv': leaf, + 'top.sv': 'module top(input sig); leaf u(.rx(sig)); endmodule', + 'sim/tb_top.sv': 'module tb_top(input sig); leaf dut(.rx(sig)); endmodule' }, cg => { + for (const binding of bindings(cg)) { + const instanceName = binding.qualifiedName.startsWith('top::') ? 'u' : 'dut'; + const instance = cg.getNodesByName(instanceName).find(n => n.kind === 'variable')!; + const target = cg.getCallees(instance.id).find(e => e.edge.kind === 'instantiates')!.node; + const formal = cg.getCallees(binding.id).find(e => e.edge.metadata?.binding === 'hdl-named-port')!.node; + expect(formal.filePath).toBe(target.filePath); + } + }); +}); + +it.each([ + { 'a.sv': leaf, 'b.sv': leaf }, + { 'a.sv': `${leaf} ${leaf}` }, +])('leaves formal endpoints unresolved for duplicate module definitions: %j', async definitions => { + await graph({ ...definitions, 'top.sv': 'module top(input sig); leaf u(.rx(sig)); endmodule' } as Record, cg => { + expect(cg.getCallees(bindings(cg)[0]!.id).filter(e => e.edge.metadata?.binding === 'hdl-named-port')).toEqual([]); + }); +}); + +it('does not use fuzzy module-name guesses for a formal endpoint', async () => { + await graph({ 'leaf.sv': leaf, 'top.sv': 'module top(input sig); leaff u(.rx(sig)); endmodule' }, cg => { + expect(cg.getCallees(bindings(cg)[0]!.id).filter(e => e.edge.metadata?.binding === 'hdl-named-port')).toEqual([]); + }); +}); + +it('rehydrates named-port tokens when only the target changes during incremental sync', async () => { + const top = 'module top(input sig); leaf u(.rx(sig)); endmodule'; + await graph({ 'leaf.sv': leaf, 'top.sv': top }, async (cg, root) => { + const binding = bindings(cg)[0]!; + const formal = () => cg.getCallees(binding.id).filter(e => e.edge.metadata?.binding === 'hdl-named-port'); + const originalTarget = formal()[0]!.node; + expect(originalTarget.qualifiedName).toBe('leaf::rx'); + fs.writeFileSync(path.join(root, 'leaf.sv'), 'module leaf(output tx); endmodule'); + const removed = await cg.sync(); + expect(removed.filesModified).toBe(1); + expect(formal()).toEqual([]); + expect(bindings(cg)[0]!.id).toBe(binding.id); + // Restore at a different source position: a stale retained edge cannot pass. + fs.writeFileSync(path.join(root, 'leaf.sv'), `// moved port declaration\n${leaf}`); + const restored = await cg.sync(); + expect(restored.filesModified).toBe(1); + const restoredFormal = formal(); + expect(restoredFormal).toHaveLength(1); + expect(restoredFormal[0]!.node.qualifiedName).toBe('leaf::rx'); + expect(restoredFormal[0]!.node.filePath).toBe('leaf.sv'); + expect(restoredFormal[0]!.node.id).not.toBe(originalTarget.id); + expect(bindings(cg)[0]!.id).toBe(binding.id); + expect(fs.readFileSync(path.join(root, 'top.sv'), 'utf8')).toBe(top); + expect(cg.getCallees(binding.id).some(e => e.node.qualifiedName === 'top::sig')).toBe(true); + }); +}); diff --git a/__tests__/hdl-positional-ports.test.ts b/__tests__/hdl-positional-ports.test.ts new file mode 100644 index 0000000000..6df3bad257 --- /dev/null +++ b/__tests__/hdl-positional-ports.test.ts @@ -0,0 +1,106 @@ +import { it, expect } from 'vitest'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { CodeGraph } from '../src'; + +async function graph(leaf: string, top: string, check: (cg: CodeGraph, root: string) => void | Promise) { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'hdl-positional-')); + let cg: CodeGraph | undefined; + try { + fs.writeFileSync(path.join(root, 'leaf.sv'), leaf); + fs.writeFileSync(path.join(root, 'top.sv'), top); + cg = CodeGraph.initSync(root); expect((await cg.indexAll()).success).toBe(true); + await check(cg, root); + } finally { cg?.close(); fs.rmSync(root, { recursive: true, force: true }); } +} +function slots(cg: CodeGraph) { + return (cg as any).db.db.prepare("select id,name,signature from nodes where kind='property' and decorators like '%hdl:position:%' order by name").all() as Array<{id:string;name:string;signature:string}>; +} +function formal(cg: CodeGraph, id: string) { + return cg.getCallees(id).filter(e => e.edge.metadata?.binding === 'hdl-positional-port').map(e => e.node.name); +} +const ansi = 'module leaf(input rx, output tx, input enable); endmodule'; + +it('maps ANSI positions, preserving empty slots and local expressions without expanding instance arrays', async () => { + await graph(ansi, 'module top(input sig); leaf bank[3:0](sig,,sig); endmodule', cg => { + const connections = slots(cg); + expect(connections).toHaveLength(3); + expect(connections.map(n => formal(cg,n.id))).toEqual([['rx'], ['tx'], ['enable']]); + expect(connections[1]!.signature).toBe(''); + expect(cg.getCallees(connections[1]!.id).filter(e => e.node.qualifiedName === 'top::sig')).toEqual([]); + expect(cg.getCallees(connections[2]!.id).some(e => e.node.qualifiedName === 'top::sig')).toBe(true); + expect(cg.getNodesByName('bank').filter(n => n.kind === 'variable')).toHaveLength(1); + }); +}); + +it('uses the non-ANSI header order rather than declaration body order', async () => { + await graph('module leaf(enable, rx, tx); output tx; input rx; input enable; endmodule', + 'module top(input sig); leaf u(sig,sig,sig); endmodule', cg => { + expect(slots(cg).map(n => formal(cg,n.id))).toEqual([['enable'], ['rx'], ['tx']]); + }); +}); + +it.each([ + ['extra arguments', ansi, 'leaf u(sig,sig,sig,sig);'], + ['mixed named and positional', ansi, 'leaf u(sig,.tx(sig));'], + ['complex non-ANSI header', 'module leaf(.rx({rx0,rx1})); input rx0,rx1; endmodule', 'leaf u(sig);'], +])('does not guess formal endpoints for %s', async (_label, declaration, instance) => { + await graph(declaration, `module top(input sig); ${instance} endmodule`, cg => { + expect(slots(cg).flatMap(n => formal(cg,n.id))).toEqual([]); + }); +}); + +it('rebinds unchanged positional callers after target order changes, removal and restoration through sync', async () => { + const top = 'module top(input sig); leaf u(sig,sig); endmodule'; + await graph('module leaf(rx,tx); input rx; output tx; endmodule', top, async (cg, root) => { + const ids = slots(cg).map(n => n.id); + const targets = () => slots(cg).map(n => formal(cg,n.id)); + expect(targets()).toEqual([['rx'], ['tx']]); + // Keep names and body declaration positions identical; only the header order changes. + fs.writeFileSync(path.join(root,'leaf.sv'), 'module leaf(tx,rx); input rx; output tx; endmodule'); + expect((await cg.sync()).filesModified).toBe(1); + expect(targets()).toEqual([['tx'], ['rx']]); + fs.writeFileSync(path.join(root,'leaf.sv'), 'module leaf(rx); input rx; endmodule'); + expect((await cg.sync()).filesModified).toBe(1); + expect(targets()).toEqual([[], []]); + fs.writeFileSync(path.join(root,'leaf.sv'), 'module leaf(rx,tx); input rx; output tx; endmodule'); + expect((await cg.sync()).filesModified).toBe(1); + expect(targets()).toEqual([['rx'], ['tx']]); + expect(slots(cg).map(n => n.id)).toEqual(ids); + expect(fs.readFileSync(path.join(root,'top.sv'),'utf8')).toBe(top); + }); +}); + +it('keeps leading and trailing empty slots while allowing an omitted trailing port list', async () => { + await graph(ansi, 'module top(input sig); leaf u(,sig,); leaf short_list(sig); endmodule', cg => { + const nodes = (cg as any).db.db.prepare("select id,qualified_name q from nodes where kind='property' and decorators like '%hdl:position:%' order by qualified_name").all(); + expect(nodes.map((n: any) => [n.q, formal(cg,n.id)])).toEqual([ + ['top::short_list::port[0]', ['rx']], ['top::u::port[0]', ['rx']], + ['top::u::port[1]', ['tx']], ['top::u::port[2]', ['enable']], + ]); + }); +}); + +it('invalidates formal mappings when an equally near empty duplicate module is added and recovers when removed', async () => { + await graph(ansi, 'module top(input sig); leaf u(sig); endmodule', async (cg, root) => { + const slot = slots(cg)[0]!; + expect(formal(cg,slot.id)).toEqual(['rx']); + fs.writeFileSync(path.join(root,'duplicate.sv'),'module leaf; endmodule'); + await cg.sync(); + expect(formal(cg,slot.id)).toEqual([]); + fs.rmSync(path.join(root,'duplicate.sv')); + await cg.sync(); + expect(formal(cg,slot.id)).toEqual(['rx']); + }); +}); + +it.each(['/* second slot */', '// second slot\n'])('keeps positional slot order across comments: %s', async comment => { + await graph(ansi, `module top(input sig); leaf u(sig, ${comment} sig,); endmodule`, cg => { + const connections = slots(cg); + expect(connections.map(n => formal(cg,n.id))).toEqual([['rx'], ['tx'], ['enable']]); + expect(connections[1]!.signature).toContain(comment); + expect(cg.getCallees(connections[1]!.id).some(e => e.node.qualifiedName === 'top::sig')).toBe(true); + expect(cg.getCallees(connections[2]!.id).some(e => e.node.qualifiedName === 'top::sig')).toBe(false); + }); +}); diff --git a/__tests__/hdl-preprocess.test.ts b/__tests__/hdl-preprocess.test.ts new file mode 100644 index 0000000000..631c325554 --- /dev/null +++ b/__tests__/hdl-preprocess.test.ts @@ -0,0 +1,80 @@ +import { describe, expect, it } from 'vitest'; +import { preprocessVerilog } from '../src/hdl/preprocess'; +const run=(source:string,defines:Record={})=>preprocessVerilog(source,{filePath:'top.sv',defines}); +const unchangedOffsets=(before:string,after:string)=>{ + expect(after.length).toBe(before.length); + expect([...after.matchAll(/\r|\n/g)].map(m=>m.index)).toEqual([...before.matchAll(/\r|\n/g)].map(m=>m.index)); +}; +describe('bounded HDL profile preprocessing',()=>{ + it('selects nested branches with per-unit defines and exact UTF16/CRLF positions',()=>{ + const source='// 😀\r\nmodule m(\r\n`ifdef VCS\r\ninput a\r\n`ifndef OMIT\r\n, input b\r\n`endif\r\n`else\r\ninput c\r\n`endif\r\n); endmodule'; + const selected=run(source,{VCS:null});unchangedOffsets(source,selected.source); + expect(selected.source).toContain('input a');expect(selected.source).toContain('input b');expect(selected.source).not.toContain('input c'); + expect(selected.source.indexOf('input a')).toBe(source.indexOf('input a'));expect(selected.complete).toBe(true); + const other=run(source);expect(other.source).toContain('input c');expect(other.source).not.toContain('input a'); + }); + it('supports inline conditionals, elsif and active define/undef without deleting following code',()=>{ + const source='`define A 1\nmodule m; `ifdef A wire a; `elsif B wire b; `else wire c; `endif\n`undef A\n`ifdef A wire bad; `else wire good; `endif endmodule'; + const result=run(source,{B:null});unchangedOffsets(source,result.source); + expect(result.source).toContain('`define A 1');expect(result.source).toContain('`undef A'); + expect(result.source).toContain('wire a;');expect(result.source).toContain('wire good;');expect(result.source).not.toMatch(/wire (bad|b|c);/); + }); + it('ignores directive-looking comments, strings, escaped identifiers and macro bodies',()=>{ + const source='/* `ifdef BAD */\n// `else\nmodule m; string s="`endif"; wire \\`ifdef ; endmodule\n`define MULTI(x) x \\\n `ifdef not_a_directive\n'; + const result=run(source);expect(result.source).toBe(source);expect(result.diagnostics).toEqual([]); + }); + it('does not leak macro definitions across translation units or inactive branches',()=>{ + const source='`ifdef MISSING\n`define LEAK 1\n`endif\n`ifdef LEAK\nwire bad;\n`else\nwire good;\n`endif'; + expect(run(source).source).toContain('wire good;');expect(run('`define LEAK 1').complete).toBe(true); + expect(run('`ifdef LEAK\nwire bad;\n`else\nwire good;\n`endif').source).toContain('wire good;'); + }); + it('uses nested includes for macro state while retaining original include source',()=>{ + const files:Record={ + 'a.svh':'`ifndef A_GUARD\n`define A_GUARD\n`include "b.svh"\n`endif', + 'b.svh':'`define FEATURE 1\n`include "a.svh"', + }; + const source='`include "a.svh"\n`ifdef FEATURE\nwire yes;\n`else\nwire no;\n`endif'; + const calls:string[]=[]; + const result=preprocessVerilog(source,{filePath:'top.sv',includeDirs:['inc'],readInclude:(request,_from,dirs)=>{ + expect(dirs).toEqual(['inc']);calls.push(request);return files[request]===undefined?null:{filePath:request,source:files[request]!}; + }}); + expect(result.source).toContain('`include "a.svh"');expect(result.source).toContain('wire yes;');expect(result.source).not.toContain('wire no;'); + expect(result.dependencies.sort()).toEqual(['a.svh','b.svh']);expect(result.complete).toBe(true);expect(calls.length).toBeLessThan(5); + }); + it('fails closed on unknown include macro state but keeps unrelated source',()=>{ + const source='`include "missing.svh"\nwire outside;\n`ifdef X\nwire first;\n`else\nwire second;\n`endif\n`define CERTAIN\n`ifdef CERTAIN\nwire restored;\n`endif'; + const result=run(source);expect(result.source).toContain('wire outside;');expect(result.source).toContain('wire restored;'); + expect(result.source).not.toMatch(/wire (first|second);/);expect(result.complete).toBe(false); + expect(result.diagnostics.map(d=>d.code)).toContain('unknown-conditional');unchangedOffsets(source,result.source); + }); + it('bounds unguarded include cycles with explicit diagnostics',()=>{ + const result=preprocessVerilog('`include "cycle.svh"',{filePath:'top.sv',readInclude:()=>({filePath:'cycle.svh',source:'`include "cycle.svh"'})}); + expect(result.diagnostics.map(d=>d.code)).toContain('include-cycle-or-depth');expect(result.complete).toBe(false); + }); + it.each(['`else\nwire unsafe;','`ifdef X\nwire unsafe;','`ifdef X\n`else\n`else\nwire unsafe;\n`endif'])('blanks structurally malformed conditional units: %s',source=>{ + const result=run(source);expect(result.source.trim()).toBe('');unchangedOffsets(source,result.source); + expect(result.diagnostics.map(d=>d.code)).toContain('malformed-conditional'); + }); + it('does not choose a stale branch after an opaque macro can define another macro',()=>{ + const source='`define ENABLE `define FLAG\n`ENABLE\n`ifdef FLAG\nmodule chosen; endmodule\n`else\nmodule wrong; endmodule\n`endif'; + const result=run(source);unchangedOffsets(source,result.source); + expect(result.source).toContain('`ENABLE');expect(result.source).not.toMatch(/module (chosen|wrong)/); + expect(result.diagnostics.map(d=>d.code)).toContain('unknown-conditional'); + }); + it('invalidates later nested and sibling conditions after an active opaque invocation',()=>{ + const source='`ifdef ACTIVE\n`SIDE_EFFECT\n`ifdef FLAG\nwire first;\n`else\nwire second;\n`endif\n`endif\n`ifdef ACTIVE\nwire stale;\n`endif'; + const result=run(source,{ACTIVE:null});unchangedOffsets(source,result.source); + expect(result.source).not.toMatch(/wire (first|second|stale)/);expect(result.complete).toBe(false); + }); + it('caps diagnostics while continuing macro state changes and source selection',()=>{ + const source='`OPAQUE\n'.repeat(540)+'`define RESTORED\n`ifdef RESTORED\nmodule good; endmodule\n`else\nmodule bad; endmodule\n`endif'; + const result=run(source);unchangedOffsets(source,result.source); + expect(result.diagnostics).toHaveLength(513);expect(result.diagnostics.filter(d=>d.code==='diagnostic-limit')).toHaveLength(1); + expect(result.complete).toBe(false);expect(result.source).toContain('module good;');expect(result.source).not.toContain('module bad;'); + }); + it('preserves unexpanded macros and unknown directives with diagnostics',()=>{ + const source='`define ASSUME(name, expr) name: assume property(expr);\nmodule m; `ASSUME(ok, x) `timescale 1ns/1ps\nendmodule'; + const result=run(source);expect(result.source).toBe(source); + expect(result.diagnostics.map(d=>d.code)).toEqual(['unsupported-macro-expansion','unsupported-directive']); + }); +}); diff --git a/__tests__/hdl-profile-cli.test.ts b/__tests__/hdl-profile-cli.test.ts new file mode 100644 index 0000000000..6fd1ee65d4 --- /dev/null +++ b/__tests__/hdl-profile-cli.test.ts @@ -0,0 +1,35 @@ +import { it, expect } from 'vitest'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { createHash } from 'node:crypto'; +import { spawnSync } from 'node:child_process'; +import { CodeGraph } from '../src'; + +it('passes selected source through compiled workers and preserves raw hashes across CLI index/sync/status', () => { + const root=fs.mkdtempSync(path.join(os.tmpdir(),'hdl-profile-cli-')); + const bin=path.resolve(__dirname,'../dist/bin/codegraph.js'); + const run=(args:string[])=>{const r=spawnSync(process.execPath,[bin,...args],{encoding:'utf8',timeout:30000, + env:{...process.env,CODEGRAPH_NO_DAEMON:'1',NO_COLOR:'1'}});expect(r.status,r.stderr).toBe(0);return r.stdout;}; + const source='`include "flags.svh"\nmodule top;\n`ifdef ENABLED\nfunction int selected_fn(); return 1; endfunction\n`else\nfunction int fallback_fn(); return 0; endfunction\n`endif\nendmodule'; + try{ + fs.mkdirSync(path.join(root,'inc'));fs.writeFileSync(path.join(root,'inc/flags.svh'),'`define ENABLED\n'); + fs.writeFileSync(path.join(root,'top.sv'),source);fs.writeFileSync(path.join(root,'rtl.f'),'top.sv\n'); + fs.writeFileSync(path.join(root,'codegraph.json'),JSON.stringify({hdl:{activeProfile:'synth',profiles:{synth:{filelists:['rtl.f'],includeDirs:['inc'],topModules:['top']}}}})); + CodeGraph.initSync(root).close(); + run(['index',root,'--quiet']); + let cg=CodeGraph.openSync(root);try{ + expect(cg.getNodesByName('selected_fn')).toHaveLength(1);expect(cg.getNodesByName('fallback_fn')).toHaveLength(0); + expect(cg.getFile('top.sv')?.contentHash).toBe(createHash('sha256').update(source).digest('hex')); + }finally{cg.close();} + let status=JSON.parse(run(['status',root,'--json']));expect(status.index.hdlProfile.indexed.name).toBe('synth'); + fs.writeFileSync(path.join(root,'inc/flags.svh'),'`undef ENABLED\n'); + status=JSON.parse(run(['status',root,'--json']));expect(status.index.hdlProfile.reindexRecommended).toBe(true); + run(['sync',root,'--quiet']); + cg=CodeGraph.openSync(root);try{ + expect(cg.getNodesByName('selected_fn')).toHaveLength(0);expect(cg.getNodesByName('fallback_fn')).toHaveLength(1); + }finally{cg.close();} + status=JSON.parse(run(['status',root,'--json']));expect(status.index.hdlProfile.state).toBe('matches'); + expect(run(['explore','fallback_fn','--path',root])).toContain('HDL context: indexed "synth"'); + }finally{fs.rmSync(root,{recursive:true,force:true});} +}, 120_000); diff --git a/__tests__/hdl-profile-grammar-failure.test.ts b/__tests__/hdl-profile-grammar-failure.test.ts new file mode 100644 index 0000000000..fce841a93e --- /dev/null +++ b/__tests__/hdl-profile-grammar-failure.test.ts @@ -0,0 +1,56 @@ +/** + * An HDL profile change re-indexes every profile source, since the same bytes + * parse differently under other defines. If the Verilog grammar cannot be + * loaded during that sync, nothing is stored for those files (#2335) — so the + * index must not record the new profile as indexed either, or no later sync + * would revisit files whose size, mtime and hash all still match. + */ +import { it, expect, vi } from 'vitest'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { CodeGraph } from '../src'; +import type { Language } from '../src/types'; + +const { failing } = vi.hoisted(() => ({ failing: new Set() })); + +vi.mock('../src/extraction/grammars', async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + getParser: (language: Language) => (failing.has(language) ? null : actual.getParser(language)), + }; +}); + +it('a profile switch whose sources could not be parsed stays pending until they are', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'hdl-profile-grammar-')); + let cg: CodeGraph | undefined; + const config = (activeProfile: string) => JSON.stringify({ hdl: { activeProfile, profiles: { + synth: { files: ['top.sv'], defines: { SYNTHESIS: '1' } }, + sim: { files: ['top.sv'] }, + } } }); + const functions = () => cg!.getNodesInFile('top.sv').filter((n) => n.kind === 'function').map((n) => n.name); + try { + fs.writeFileSync(path.join(root, 'top.sv'), + 'module top;\n`ifdef SYNTHESIS\nfunction int hardware(); return 1; endfunction\n`else\nfunction int software(); return 2; endfunction\n`endif\nendmodule\n'); + fs.writeFileSync(path.join(root, 'codegraph.json'), config('synth')); + cg = CodeGraph.initSync(root); + expect((await cg.indexAll()).success).toBe(true); + expect(functions()).toEqual(['hardware']); + + fs.writeFileSync(path.join(root, 'codegraph.json'), config('sim')); + failing.add('verilog'); + await cg.sync(); + expect(functions()).toEqual(['hardware']); + expect(cg.getHdlProfileStatus()?.state).toBe('mismatch'); + + failing.clear(); + await cg.sync(); + expect(functions()).toEqual(['software']); + expect(cg.getHdlProfileStatus()?.state).toBe('matches'); + } finally { + failing.clear(); + cg?.close(); + fs.rmSync(root, { recursive: true, force: true }); + } +}); diff --git a/__tests__/hdl-profile-index.test.ts b/__tests__/hdl-profile-index.test.ts new file mode 100644 index 0000000000..7c10edd4ac --- /dev/null +++ b/__tests__/hdl-profile-index.test.ts @@ -0,0 +1,87 @@ +import { it, expect } from 'vitest'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { CodeGraph } from '../src'; + +it('switches profiles and include/filelist dependencies without mixing HDL graphs or changing source offsets', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'hdl-profile-index-')); + let cg: CodeGraph | undefined; + const write = (file: string, text: string) => { fs.mkdirSync(path.dirname(path.join(root, file)), {recursive:true}); fs.writeFileSync(path.join(root,file),text); }; + const config = (activeProfile: string) => JSON.stringify({hdl:{activeProfile,profiles:{synth:{filelists:['rtl.f'],includeDirs:['inc'],defines:{SYNTHESIS:'1'},topModules:['top']},sim:{files:['top.sv','sim.sv'],includeDirs:['inc']}}}}); + const source = '`include "defs.svh"\nmodule top;\n`ifdef SYNTHESIS\nfunction int hardware(); return 1; endfunction\n`else\nfunction int software(); return 2; endfunction\n`endif\n`ifdef EXTRA\nfunction int extra(); return 3; endfunction\n`endif\nendmodule\n'; + try { + write('top.sv',source); write('rtl.f','top.sv\n'); write('sim.sv','module sim; endmodule'); write('inc/defs.svh','// empty\n'); + write('plain.ts','export function ordinary(){return 1;}'); write('codegraph.json',config('synth')); + cg = CodeGraph.initSync(root); expect((await cg.indexAll()).success).toBe(true); + const names = () => (cg as any).db.db.prepare("SELECT name FROM nodes WHERE kind IN ('function','class') ORDER BY name").all().map((n:any)=>n.name); + expect(names()).toEqual(['hardware','ordinary','top']); + expect(cg.getNodesByName('hardware')[0].startLine).toBe(4); + expect(cg.getHdlProfileStatus()?.state).toBe('matches'); + expect(cg.getChangedFiles()).toEqual({added:[],modified:[],removed:[]}); + write('codegraph.json',config('sim')); + expect(cg.getHdlProfileStatus()?.state).toBe('mismatch'); + await cg.sync({paths:['codegraph.json']}); + expect(names()).toEqual(['ordinary','sim','software','top']); + expect(cg.getHdlProfileStatus()?.state).toBe('matches'); + write('inc/defs.svh','`define EXTRA\n'); + expect(cg.getHdlProfileStatus()?.reindexRecommended).toBe(true); + await cg.sync({paths:['inc/defs.svh']}); + expect(names()).toContain('extra'); + write('codegraph.json',config('synth')); write('rtl.f','top.sv\nsim.sv\n'); + await cg.sync({paths:['rtl.f']}); + expect(names()).toEqual(['extra','hardware','ordinary','sim','top']); + const snapshot=names(); await cg.indexAll(); expect(names()).toEqual(snapshot); + expect(fs.readFileSync(path.join(root,'top.sv'),'utf8')).toBe(source); + write('codegraph.json','{}'); await cg.sync({paths:['codegraph.json']}); + expect(names()).toContain('software'); expect(names()).toContain('hardware'); + expect(cg.getHdlProfileStatus()?.indexed.mode).toBe('raw'); + } finally { cg?.close(); fs.rmSync(root,{recursive:true,force:true}); } +}); + +it('fails closed on invalid active profile and preserves the previously indexed graph', async () => { + const root=fs.mkdtempSync(path.join(os.tmpdir(),'hdl-profile-invalid-'));let cg:CodeGraph|undefined; + try { + fs.writeFileSync(path.join(root,'top.sv'),'module top; endmodule'); + fs.writeFileSync(path.join(root,'codegraph.json'),JSON.stringify({hdl:{activeProfile:'ok',profiles:{ok:{files:['top.sv']}}}})); + cg=CodeGraph.initSync(root);await cg.indexAll(); + fs.writeFileSync(path.join(root,'codegraph.json'),JSON.stringify({hdl:{activeProfile:'missing',profiles:{ok:{files:['top.sv']}}}})); + await expect(cg.sync()).rejects.toThrow('Invalid HDL profile'); + expect(cg.getNodesByName('top')).toHaveLength(1); + expect(cg.getHdlProfileStatus()?.state).toBe('configuration-error'); + fs.writeFileSync(path.join(root,'codegraph.json'),'{'); + await expect(cg.sync()).rejects.toThrow('Invalid HDL profile'); + expect(cg.getHdlProfileStatus()?.indexed.name).toBe('ok'); + }finally{cg?.close();fs.rmSync(root,{recursive:true,force:true});} +}); + +it('stores one coherent source snapshot when a profile unit changes during a bulk index', async () => { + const root=fs.mkdtempSync(path.join(os.tmpdir(),'hdl-profile-race-'));let cg:CodeGraph|undefined; + try { + const file=path.join(root,'top.sv'); + fs.writeFileSync(file,'module old_unit; endmodule'); + fs.writeFileSync(path.join(root,'codegraph.json'),JSON.stringify({hdl:{activeProfile:'p',profiles:{p:{files:['top.sv']}}}})); + cg=CodeGraph.initSync(root);await cg.indexAll();let changed=false; + const result=await cg.indexAll({onProgress:p=>{if(!changed&&p.phase==='scanning'){changed=true;fs.writeFileSync(file,'module new_unit; endmodule');}}}); + expect(result.success).toBe(true);expect(cg.getNodesByName('old_unit')).toHaveLength(1); + expect(cg.getChangedFiles().modified).toContain('top.sv'); + await cg.sync();expect(cg.getNodesByName('new_unit')).toHaveLength(1);expect(cg.getNodesByName('old_unit')).toHaveLength(0); + }finally{cg?.close();fs.rmSync(root,{recursive:true,force:true});} +}); + +it('publishes only diagnostics for units actually updated by partial operations', async () => { + const root=fs.mkdtempSync(path.join(os.tmpdir(),'hdl-profile-partial-'));let cg:CodeGraph|undefined; + const opaque='`OPAQUE\nmodule other; endmodule'; + try{ + fs.writeFileSync(path.join(root,'top.sv'),'module top; endmodule'); + fs.writeFileSync(path.join(root,'other.sv'),opaque); + fs.writeFileSync(path.join(root,'codegraph.json'),JSON.stringify({hdl:{activeProfile:'p',profiles:{p:{files:['top.sv','other.sv']}}}})); + cg=CodeGraph.initSync(root);await cg.indexAll();expect(cg.getHdlProfileStatus()?.incomplete).toBe(true); + fs.writeFileSync(path.join(root,'other.sv'),'module other; endmodule'); + fs.writeFileSync(path.join(root,'top.sv'),'module top; wire value; endmodule'); + await cg.indexFiles(['top.sv']);expect(cg.getHdlProfileStatus()?.incomplete).toBe(true); + await cg.sync({paths:['other.sv']});expect(cg.getHdlProfileStatus()?.incomplete).toBe(false); + fs.writeFileSync(path.join(root,'top.sv'),'`OPAQUE\nmodule top; endmodule'); + await cg.indexFiles(['top.sv']);expect(cg.getHdlProfileStatus()?.incomplete).toBe(true); + }finally{cg?.close();fs.rmSync(root,{recursive:true,force:true});} +}); diff --git a/__tests__/hdl-profile-status.test.ts b/__tests__/hdl-profile-status.test.ts new file mode 100644 index 0000000000..e14e559db8 --- /dev/null +++ b/__tests__/hdl-profile-status.test.ts @@ -0,0 +1,126 @@ +import { describe, expect, it } from 'vitest'; +import { buildHdlProfileStatus, formatHdlProfileStatus, type HdlProfileConfiguration, type HdlIndexedProfileMetadata } from '../src/hdl/status'; +const active = (name='simulation',fingerprint='configured-sim'): HdlProfileConfiguration => ({status:'active',profile:{name,fingerprint},diagnostics:[]}); +const indexed = (name='simulation',configurationFingerprint='configured-sim',effective='effective-with-headers'):HdlIndexedProfileMetadata => ({ + name,fingerprint:effective,context:JSON.stringify({mode:'profile',configurationFingerprint,dependencies:[{path:'defs.svh',sha256:'header-sha'}]}), +}); +const none:HdlProfileConfiguration={status:'none',profile:null,diagnostics:[]}; +const raw:HdlIndexedProfileMetadata={name:null,fingerprint:'raw',context:JSON.stringify({mode:'raw'})}; +const legacy:HdlIndexedProfileMetadata={name:null,fingerprint:null,context:null}; + +describe('HDL configured versus indexed profile visibility',()=>{ + it('compares configured fingerprint against stored configuration context, not the effective header hash',()=>{ + const status=buildHdlProfileStatus(active(),indexed(),true)!; + expect(status.state).toBe('matches'); + expect(status.mismatch).toBe(false); + expect(status.reindexRecommended).toBe(false); + expect(status.indexed.fingerprint).toBe('effective-with-headers'); + expect(status.indexed.configurationFingerprint).toBe('configured-sim'); + }); + it('keeps indexed identity when configuration changes to another real build variant',()=>{ + const status=buildHdlProfileStatus(active('synthesis','configured-synth'),indexed(),true)!; + expect(status.state).toBe('mismatch'); + expect(status.reindexRecommended).toBe(true); + expect(status.indexed.name).toBe('simulation'); + expect(status.configured.name).toBe('synthesis'); + const text=formatHdlProfileStatus(status); + expect(text).toContain('indexed "simulation"'); + expect(text).toContain('configured "synthesis"'); + expect(text).toContain('rebuild the index'); + expect(text).toContain('not fully expanded'); + }); + it('detects defines or filelists changing under the same profile name',()=>{ + expect(buildHdlProfileStatus(active('simulation','changed-config'),indexed(),true)?.mismatch).toBe(true); + }); + it('distinguishes explicit raw-source indexing from missing legacy provenance',()=>{ + expect(buildHdlProfileStatus(none,raw,true)?.state).toBe('matches'); + expect(buildHdlProfileStatus(none,legacy,true)?.state).toBe('unknown'); + expect(buildHdlProfileStatus(none,legacy,true)?.reindexRecommended).toBe(true); + expect(formatHdlProfileStatus(buildHdlProfileStatus(none,raw,true))).toContain('multiple conditional alternatives'); + expect(buildHdlProfileStatus(active(),raw,true)?.state).toBe('mismatch'); + expect(buildHdlProfileStatus(none,indexed(),true)?.state).toBe('mismatch'); + }); + it('reports invalid configuration without claiming a rebuild alone can fix it',()=>{ + const status=buildHdlProfileStatus({status:'invalid-profile',profile:null,diagnostics:['Missing filelist build/sim.f']},indexed(),true)!; + expect(status.state).toBe('configuration-error'); + expect(status.reindexRecommended).toBe(false); + expect(status.indexed.name).toBe('simulation'); + expect(formatHdlProfileStatus(status)).toContain('Fix the HDL profile configuration'); + expect(formatHdlProfileStatus(status)).toContain('Missing filelist build/sim.f'); + }); + it('marks corrupt or partial stored context as unknown instead of trusting current configuration',()=>{ + for(const context of ['{broken','[]','{}']) { + const status=buildHdlProfileStatus(active(),{...indexed(),context},true)!; + expect(status.indexed.mode).toBe('unknown'); + expect(status.reindexRecommended).toBe(true); + } + }); + it('retains indexed preprocessing diagnostics and bounds their text rendering',()=>{ + const metadata=indexed(); + metadata.context=JSON.stringify({mode:'profile',configurationFingerprint:'configured-sim',incomplete:true, + diagnostics:[{filePath:'rtl/top.sv',line:4,message:'Unknown include macro'},'Macro expansion unsupported','Third diagnostic','Fourth diagnostic']}); + const status=buildHdlProfileStatus(active(),metadata,true)!; + expect(status.incomplete).toBe(true); + expect(status.indexedDiagnostics).toHaveLength(4); + const text=formatHdlProfileStatus(status); + expect(text).toContain('rtl/top.sv:4: Unknown include macro'); + expect(text).toContain('1 additional diagnostics'); + expect(text).not.toContain('Fourth diagnostic'); + expect(status.reindexRecommended).toBe(false); + }); + it('keeps unrelated projects quiet',()=>{ + expect(buildHdlProfileStatus(none,legacy,false)).toBeNull(); + expect(buildHdlProfileStatus(none,raw,false)).toBeNull(); + expect(formatHdlProfileStatus(null)).toBe(''); + }); +}); + +it('prefixes normal, empty and access-filtered MCP exploration with the indexed context', async()=>{ + const fs=await import('node:fs'), path=await import('node:path'), os=await import('node:os'); + const {CodeGraph}=await import('../src'); + const {ToolHandler}=await import('../src/mcp/tools'); + const root=fs.mkdtempSync(path.join(os.tmpdir(),'hdl-profile-notice-')); + let cg: InstanceType | undefined; + try { + fs.writeFileSync(path.join(root,'top.sv'),'module top(input a, output y); assign y=a; endmodule'); + cg=CodeGraph.initSync(root); expect((await cg.indexAll()).success).toBe(true); + const state=buildHdlProfileStatus(active('synthesis','configured-synth'),indexed(),true)!; + Object.assign(cg,{getHdlProfileStatus:()=>state}); + const handler=new ToolHandler(cg); + for (const args of [{query:'top'}, {query:'missing_name_that_does_not_exist'}, {query:'top::a',hdlAccess:'all'}]) { + const result=await handler.executeReadTool('codegraph_explore',args); + expect(result.isError).not.toBe(true); + const text=result.content.map(c=>c.text ?? '').join('\n'); + expect(text).toContain('HDL context: indexed "simulation"'); + expect(text).toContain('configured "synthesis"'); + expect(text).toContain('Profile mismatch: rebuild the index'); + } + } finally {cg?.close();fs.rmSync(root,{recursive:true,force:true});} +}, 30_000); + +it('summarizes configured and indexed settings separately without exposing defines values or source filelists',()=>{ + const configuration=active('synthesis','configured-synth'); + configuration.profile={...configuration.profile!,files:['rtl/new_top.v'],includeDirs:['rtl/include'], + defines:{SYNTHESIS:'1',PRIVATE_VALUE:'do-not-print-this-value'},topModules:['new_top'],languageMode:'verilog'}; + const metadata=indexed(); + metadata.context=JSON.stringify({mode:'profile',configurationFingerprint:'configured-sim', + files:['sim/testbench.sv','rtl/old_top.sv'],includeDirs:['sim/include','common/include'], + defines:{SIMULATION:'1',OTHER_SECRET:'never-display-this'},topModules:['tb_top'],languageMode:'systemverilog'}); + const before=JSON.stringify({configuration,metadata}); + const status=buildHdlProfileStatus(configuration,metadata,true)!; + expect(status.configured).toMatchObject({fileCount:1,includeDirs:['rtl/include'],defineNames:['PRIVATE_VALUE','SYNTHESIS'],topModules:['new_top'],languageMode:'verilog'}); + expect(status.indexed).toMatchObject({fileCount:2,includeDirs:['sim/include','common/include'],defineNames:['OTHER_SECRET','SIMULATION'],topModules:['tb_top'],languageMode:'systemverilog'}); + const json=JSON.stringify(status), text=formatHdlProfileStatus(status); + for(const hidden of ['do-not-print-this-value','never-display-this','sim/testbench.sv','rtl/old_top.sv']) { + expect(json).not.toContain(hidden);expect(text).not.toContain(hidden); + } + expect(text).toContain('2 file(s)'); + expect(text).toContain('intended dialect systemverilog'); + expect(text).toContain('does not enforce Verilog-only conformance'); + expect(text).toContain('topModules are recorded roots'); + expect(text).not.toContain('sim/include'); + expect(text).not.toContain('OTHER_SECRET'); + expect(JSON.stringify({configuration,metadata})).toBe(before); + status.configured.includeDirs!.push('local-only-change'); + expect(configuration.profile.includeDirs).toEqual(['rtl/include']); +}); diff --git a/__tests__/hdl-profile-watch.test.ts b/__tests__/hdl-profile-watch.test.ts new file mode 100644 index 0000000000..849ef2c8c2 --- /dev/null +++ b/__tests__/hdl-profile-watch.test.ts @@ -0,0 +1,90 @@ +import { it, expect } from 'vitest'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { CodeGraph } from '../src'; +import { FileWatcher } from '../src/sync/watcher'; +import { HdlWatchScope } from '../src/sync/hdl-watch-scope'; + +async function waitFor(check: () => boolean, timeout = 8000) { + const end = Date.now() + timeout; + while (!check()) { if (Date.now() > end) throw new Error('HDL watcher did not converge'); await new Promise(r => setTimeout(r, 25)); } +} +function project() { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'hdl-profile-watch-')); + const write = (file: string, text: string) => { fs.mkdirSync(path.dirname(path.join(root, file)), {recursive:true}); fs.writeFileSync(path.join(root,file),text); }; + const config = (activeProfile: string) => JSON.stringify({hdl:{activeProfile,profiles:{ + first:{filelists:['build/units.f']},second:{filelists:['alt/units.f'],includeDirs:['alt/inc']}, + }}}); + write('.gitignore', 'build/\nalt/\nprivate/\nunrelated/\n'); + write('build/top.sv', '`include "../private/flags.inc"\nmodule top;\n`ifdef EXTRA\nfunction int extra();return 1;endfunction\n`endif\nendmodule'); + write('build/spare.sv', 'module spare; endmodule'); write('build/units.f','build/top.sv\n'); + write('private/flags.inc','// empty\n'); + write('alt/top.sv','`include "flags.txt"\nmodule other;\n`ifdef SECOND\nfunction int second();return 2;endfunction\n`endif\nendmodule'); + write('alt/units.f','alt/top.sv\n'); write('alt/inc/flags.txt','// empty\n'); + write('codegraph.json',config('first')); + return {root,write,config}; +} + +it('watches profile filelists, explicit ignored sources and external-to-source header dependencies through the real event filter', async () => { + const {root,write} = project(); const calls: Array = []; + const watcher = new FileWatcher(root, async paths => { calls.push(paths); return {filesChanged:1,durationMs:1}; }, + {inertForTests:true,debounceMs:5}, undefined, () => ['private/flags.inc']); + try { + expect(watcher.start()).toBe(true); + watcher.ingestEventForTests('private/flags.inc'); + await waitFor(() => calls.length === 1); + expect(calls[0]).toContain('private/flags.inc'); + write('build/units.f','build/top.sv\nbuild/spare.sv\n'); watcher.ingestEventForTests('build/units.f'); + await waitFor(() => calls.length === 2); + expect(calls[1]).toBeUndefined(); // profile membership needs full HDL reconciliation + watcher.ingestEventForTests('build/spare.sv'); await waitFor(() => calls.length === 3); + expect(calls[2]).toContain('build/spare.sv'); + watcher.ingestEventForTests('unrelated/noise.txt'); watcher.ingestEventForTests('.codegraph/graph.db'); + await new Promise(r => setTimeout(r, 30)); expect(calls).toHaveLength(3); + } finally { watcher.stop(); fs.rmSync(root,{recursive:true,force:true}); } +}); + +it('retains the last valid HDL watch scope through missing filelists and malformed configuration', () => { + const {root,write,config} = project(); const scope = new HdlWatchScope(root, () => ['private/flags.inc']); + try { + scope.refresh(); expect(scope.matchesDirectory('build')).toBe(true); expect(scope.matchesDirectory('private')).toBe(true); + fs.unlinkSync(path.join(root,'build/units.f')); scope.refresh(); expect(scope.isFilelist('build/units.f')).toBe(true); + write('codegraph.json','{'); scope.refresh(); expect(scope.matchesFile('private/flags.inc')).toBe(true); + write('build/units.f','build/top.sv\n'); write('codegraph.json',config('second')); scope.refresh(); + expect(scope.matchesDirectory('alt/inc/nested')).toBe(true); expect(scope.matchesFile('alt/inc/new.flags')).toBe(true); + write('codegraph.json','{}'); scope.refresh(); expect(scope.matchesFile('private/flags.inc')).toBe(false); + } finally { fs.rmSync(root,{recursive:true,force:true}); } +}); + +it('auto-syncs native fs.watch events for ignored filelists, arbitrary-extension includes and newly selected directories', async () => { + const {root,write,config} = project(); let cg: CodeGraph|undefined; const errors: string[]=[]; + try { + cg=CodeGraph.initSync(root); expect((await cg.indexAll()).success).toBe(true); + expect(cg.getNodesByName('top')).toHaveLength(1); expect(cg.getNodesByName('spare')).toHaveLength(0); + expect(cg.watch({debounceMs:30,onSyncError:e=>errors.push(e.message)})).toBe(true); + write('build/units.f','build/top.sv\nbuild/spare.sv\n'); + await waitFor(()=>cg!.getNodesByName('spare').length===1); + write('private/flags.inc','`define EXTRA\n'); + await waitFor(()=>cg!.getNodesByName('extra').length===1); + write('codegraph.json',config('second')); + await waitFor(()=>cg!.getNodesByName('other').length===1&&cg!.getNodesByName('top').length===0); + write('alt/inc/flags.txt','`define SECOND\n'); + await waitFor(()=>cg!.getNodesByName('second').length===1); + expect(errors).toEqual([]); + await waitFor(()=>cg!.getHdlProfileStatus()?.state==='matches'); + } finally { cg?.close(); fs.rmSync(root,{recursive:true,force:true}); } +}); + +it('does not admit reserved or outside paths from untrusted indexed dependency metadata', () => { + const {root} = project(); + const scope = new HdlWatchScope(root, () => ['../outside/flags.inc','.codegraph-other/flags.inc','vendor/.git/config','.jj/state','private/flags.inc',123] as string[]); + try { + scope.refresh(); + expect(scope.matchesFile('private/flags.inc')).toBe(true); + expect(scope.matchesDirectory('outside')).toBe(false); + expect(scope.matchesDirectory('.codegraph-other')).toBe(false); + expect(scope.matchesFile('vendor/.git/config')).toBe(false); + expect(scope.matchesDirectory('.jj')).toBe(false); + } finally { fs.rmSync(root,{recursive:true,force:true}); } +}); diff --git a/__tests__/hdl-profile.test.ts b/__tests__/hdl-profile.test.ts new file mode 100644 index 0000000000..2df1d9d137 --- /dev/null +++ b/__tests__/hdl-profile.test.ts @@ -0,0 +1,119 @@ +import { afterEach, beforeEach, expect, it } from 'vitest'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { loadHdlProfile } from '../src/hdl/profile'; + +let root: string; +beforeEach(() => { root = fs.mkdtempSync(path.join(os.tmpdir(), 'hdl-profile-')); }); +afterEach(() => { fs.rmSync(root, { recursive: true, force: true }); }); +function write(file: string, content: string) { + fs.mkdirSync(path.dirname(path.join(root, file)), { recursive: true }); + fs.writeFileSync(path.join(root, file), content); +} +function config(profile: Record, extra: Record = {}) { + write('codegraph.json', JSON.stringify({ ...extra, hdl: { activeProfile: 'synth', profiles: { synth: profile } } })); +} +function active() { + const result = loadHdlProfile(root); + expect(result.status, result.diagnostics.join('\n')).toBe('active'); + return result.profile!; +} + +it('preserves no-HDL legacy behavior and reports malformed overall JSON separately', () => { + expect(loadHdlProfile(root)).toMatchObject({ status: 'none', profile: null, configPresent: false }); + write('codegraph.json', JSON.stringify({ exclude: ['cache'] })); + expect(loadHdlProfile(root)).toMatchObject({ status: 'none', configPresent: true }); + write('codegraph.json', '{'); + expect(loadHdlProfile(root)).toMatchObject({ status: 'invalid-config', configInvalid: true, profile: null }); +}); + +it('loads explicit files in order with root-relative canonical identity and no implicit includes as source files', () => { + write('rtl/a.sv', 'module a; endmodule'); write('rtl/b.v', 'module b; endmodule'); write('inc/defs.svh', '`define W 2'); + config({ files: ['rtl/b.v', './rtl/a.sv', 'rtl/b.v'], includeDirs: ['inc'], defines: { SYNTHESIS: '1' }, topModules: ['a'], languageMode: 'systemverilog' }); + expect(active()).toMatchObject({ name: 'synth', files: ['rtl/b.v', 'rtl/a.sv'], includeDirs: ['inc'], defines: { SYNTHESIS: '1' }, topModules: ['a'], languageMode: 'systemverilog', dependencies: [] }); +}); + +it('treats an explicitly empty profile as an empty design', () => { + write('unrelated.sv', 'module unrelated; endmodule'); config({ files: [] }); + expect(active().files).toEqual([]); +}); + +it('handles nested -f/-F bases, quotes, comments and multiple include/define entries without executing anything', () => { + write('rtl/a.sv', 'module a; endmodule'); write('lists/local file.sv', 'module local_file; endmodule'); + write('inc one/defs.svh', ''); write('inc+two/defs.svh', ''); + write('root.f', '// comment\n+incdir+"inc one"+"inc+two"\n+define+ONE+BITS=1\'b1+TEXT="a+b c"\n-f lists/rootpaths.f\n-F lists/local.F\n'); + write('lists/rootpaths.f', '# comment\nrtl/a.sv /* end comment */\n'); + write('lists/local.F', '"local file.sv"\n'); + config({ files: [], filelists: ['root.f'] }); + const p = active(); + expect(p.files).toEqual(['rtl/a.sv', 'lists/local file.sv']); + expect(p.includeDirs).toEqual(['inc one', 'inc+two']); + expect(p.defines).toEqual({ ONE: '1', BITS: "1'b1", TEXT: 'a+b c' }); + expect(p.dependencies.map(d => d.path)).toEqual(['lists/local.F', 'lists/rootpaths.f', 'root.f']); + expect(p.dependencies.every(d => /^[a-f0-9]{64}$/.test(d.sha256))).toBe(true); +}); + +it('uses top-level .F paths relative to that list and rejects source paths missing under -f root semantics', () => { + write('lists/only.sv', 'module only; endmodule'); write('lists/relative.F', 'only.sv'); + config({ filelists: ['lists/relative.F'] }); expect(active().files).toEqual(['lists/only.sv']); + write('lists/root.f', 'only.sv'); config({ filelists: ['lists/root.f'] }); + expect(loadHdlProfile(root).status).toBe('invalid-profile'); +}); + +it.each([ + { files: 'rtl/a.sv' }, { files: [], unexpected: true }, { files: [], defines: { FLAG: true } }, + { files: [], languageMode: 'vhdl' }, { files: ['missing.sv'] }, +])('rejects invalid recognized HDL profiles instead of scanning everything: %j', profile => { + config(profile); expect(loadHdlProfile(root)).toMatchObject({ status: 'invalid-profile', profile: null, configInvalid: false }); +}); + +it.each(['-sv rtl/a.sv', '+libext+.v', '-f', '+incdir+', '+define+', '"unfinished', '/* unfinished'])('rejects unsupported/malformed filelist syntax: %s', text => { + write('list.f', text); config({ filelists: ['list.f'] }); + expect(loadHdlProfile(root)).toMatchObject({ status: 'invalid-profile', profile: null }); +}); + +it('rejects recursive filelists, conflicting defines and external/symlink escapes', () => { + write('a.f', '-f b.f'); write('b.f', '-f a.f'); config({ filelists: ['a.f'] }); + expect(loadHdlProfile(root).diagnostics.join()).toContain('cycle'); + write('a.f', '+define+FLAG=2'); config({ filelists: ['a.f'], defines: { FLAG: '1' } }); + expect(loadHdlProfile(root).diagnostics.join()).toContain('Conflicting'); + const external = fs.mkdtempSync(path.join(os.tmpdir(), 'hdl-profile-external-')); + try { + fs.writeFileSync(path.join(external, 'foreign.sv'), 'module foreign; endmodule'); + config({ files: [path.join(external, 'foreign.sv')] }); + expect(loadHdlProfile(root).diagnostics.join()).toContain('outside'); + fs.symlinkSync(external, path.join(root, 'escape'), 'dir'); config({ includeDirs: ['escape'], files: [] }); + expect(loadHdlProfile(root).diagnostics.join()).toContain('outside'); + } finally { fs.rmSync(external, { recursive: true, force: true }); } +}); + +it('fingerprints profile switching, filelist content and include configuration, preserving inactive/whitespace changes', () => { + write('a.sv', 'module a; endmodule'); write('src.f', 'a.sv\n'); write('inc/defs.svh', ''); write('alt/defs.svh', ''); + const cfg = { hdl: { activeProfile: 'synth', profiles: { synth: { filelists: ['src.f'], includeDirs: ['inc'] }, sim: { files: ['a.sv'] } } } }; + write('codegraph.json', JSON.stringify(cfg)); const a = active().fingerprint; + write('codegraph.json', JSON.stringify(cfg, null, 4)); expect(active().fingerprint).toBe(a); + cfg.hdl.profiles.sim.files = []; write('codegraph.json', JSON.stringify(cfg)); expect(active().fingerprint).toBe(a); + write('src.f', '// dependency edit\na.sv\n'); const b = active().fingerprint; expect(b).not.toBe(a); + cfg.hdl.profiles.synth.includeDirs = ['alt']; write('codegraph.json', JSON.stringify(cfg)); const c = active().fingerprint; expect(c).not.toBe(b); + cfg.hdl.activeProfile = 'sim'; write('codegraph.json', JSON.stringify(cfg)); expect(active().fingerprint).not.toBe(c); expect(active().files).toEqual([]); +}); + +it('enforces filelist resource limits', () => { + write('huge.f', ' '.repeat(1024 * 1024 + 1)); config({ filelists: ['huge.f'] }); + expect(loadHdlProfile(root).diagnostics.join()).toContain('1 MiB'); +}); + +it('rejects reserved VCS and CodeGraph files and include directories before indexing', () => { + write('.git/units.f',''); write('.codegraph-alt/unit.sv','module unit; endmodule'); write('.jj/headers/a.svh',''); + for (const profile of [{filelists:['.git/units.f']},{files:['.codegraph-alt/unit.sv']},{files:[],includeDirs:['.jj/headers']}]) { + config(profile); expect(loadHdlProfile(root).status).toBe('invalid-profile'); + expect(loadHdlProfile(root).diagnostics.join()).toContain('Reserved'); + } +}); + +it('rejects ordinary aliases resolving into reserved CodeGraph paths', () => { + write('.codegraph-cache/unit.sv','module unit; endmodule'); + fs.symlinkSync(path.join(root,'.codegraph-cache'),path.join(root,'alias'),'dir'); + config({files:['alias/unit.sv']}); expect(loadHdlProfile(root).diagnostics.join()).toContain('Reserved'); +}); diff --git a/__tests__/hdl-semantic-cli.test.ts b/__tests__/hdl-semantic-cli.test.ts new file mode 100644 index 0000000000..9058c51ac7 --- /dev/null +++ b/__tests__/hdl-semantic-cli.test.ts @@ -0,0 +1,53 @@ +import { expect, it } from 'vitest'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { spawnSync } from 'node:child_process'; +import CodeGraph from '../src/index'; + +it.skipIf(process.platform === 'win32')('compiled hdl-semantic CLI passes explicit overrides and rejects duplicate/invalid options', () => { + const temp = fs.mkdtempSync(path.join(os.tmpdir(), 'hdl-semantic-cli-')); + const root = path.join(temp, 'project'); fs.mkdirSync(root); + const tool = path.join(temp, 'fake-slang'); + fs.writeFileSync(tool, `#!${process.execPath} +const fs=require('node:fs'),a=process.argv.slice(2),python=a[0]==='-I'; +if(a.includes('--version')) console.log(python?'codegraph pyslang 11.0.0 exporter 1 native '+'a'.repeat(64):'slang version 11.0.448'); +else { + const override=a[a.indexOf('-G')+1],value=override?.startsWith('W=')?override.slice(2):'8'; + const output=python?{codegraphSemanticVersion:1,facts:[{kind:'parameter',name:'W',instancePath:'top',value,sourceOrigin:'direct',source:{file:'top.sv',line:1,column:24}}]}: + {design:{kind:'Root',members:[{kind:'Instance',name:'top',body:{kind:'InstanceBody',members:[{kind:'Parameter',name:'W',value,type:{kind:'PredefinedIntegerType',name:'int'},source_file:'top.sv',source_line:1,source_column:24}]}}]}}; + fs.writeFileSync(a[a.indexOf('--ast-json')+1],JSON.stringify(output)); +} +`, { mode: 0o700 }); + fs.writeFileSync(path.join(root, 'top.sv'), 'module top #(parameter W=8)(); endmodule\n'); + fs.writeFileSync(path.join(root, 'codegraph.json'), JSON.stringify({ hdl: { activeProfile: 'synth', profiles: { + synth: { files: ['top.sv'], topModules: ['top'] }, + } } })); + const bin = path.resolve(__dirname, '../dist/bin/codegraph.js'); + const run = (args: string[]) => spawnSync(process.execPath, [bin, ...args], { + encoding: 'utf8', timeout: 30000, env: { ...process.env, CODEGRAPH_NO_DAEMON: '1', + CODEGRAPH_SEMANTIC_TMPDIR: path.join(temp, 'snapshots'), NO_COLOR: '1' }, + }); + try { + CodeGraph.initSync(root).close(); + const indexed = run(['index', root, '--quiet']); expect(indexed.status, indexed.stderr).toBe(0); + const result = run(['hdl-semantic', 'top.W', '--path', root, '--slang', tool, '--parameter', 'W=16']); + expect(result.status, result.stderr).toBe(0); + const body = JSON.parse(result.stdout); + expect(body.provenance.parameters).toEqual({ W: '16' }); + expect(body.provenance.compilationUnitMode).toBe('separate'); + expect(body.facts[0]).toMatchObject({ name: 'W', value: '16' }); + expect(body.facts[0].sourceNodeId).toBeTruthy(); + const mapped = run(['hdl-semantic', 'W', '--path', root, '--python', tool, '--parameter', 'W=12']); + expect(mapped.status, mapped.stderr).toBe(0); + expect(JSON.parse(mapped.stdout).provenance.frontend).toBe('pyslang'); + expect(JSON.parse(mapped.stdout).facts[0]).toMatchObject({ value: '12', sourceOrigin: 'direct' }); + const dual = run(['hdl-semantic', '--path', root, '--slang', tool, '--python', tool]); + expect(dual.status).toBe(1); expect(dual.stderr).toContain('exactly one'); + const duplicate = run(['hdl-semantic', '--path', root, '--slang', tool, '--parameter', 'W=1', 'W=2']); + expect(duplicate.status).toBe(1); expect(duplicate.stderr).toContain('Duplicate HDL parameter'); + const invalid = run(['hdl-semantic', '--path', root, '--slang', tool, '--limit', '1junk']); + expect(invalid.status).toBe(1); expect(invalid.stderr).toContain('1..1000'); + expect(fs.readdirSync(path.join(temp, 'snapshots'))).toHaveLength(0); + } finally { fs.rmSync(temp, { recursive: true, force: true }); } +}); diff --git a/__tests__/hdl-semantic-import.test.ts b/__tests__/hdl-semantic-import.test.ts new file mode 100644 index 0000000000..a41abbaff6 --- /dev/null +++ b/__tests__/hdl-semantic-import.test.ts @@ -0,0 +1,80 @@ +import { describe, expect, it } from 'vitest'; +import { importSlangSemantics } from '../src/hdl/semantic-import'; +const scalar={kind:'ScalarType',name:'logic',addr:1,isSigned:false}; +const parameter=(name:string,value:string)=>({kind:'Parameter',name,value,type:{kind:'PredefinedIntegerType',name:'int'}}); +const instance=(name:string,body:unknown)=>({kind:'Instance',name,body}); +const body=(members:unknown[],addr?:number)=>({kind:'InstanceBody',name:'leaf',members,...(addr?{addr}:{})}); +const design=(...members:unknown[])=>({design:{kind:'Root',name:'$root',members}}); + +describe('bounded slang semantic importer',()=>{ + it('imports primitive facts with source points and detailed packed widths',()=>{ + const facts=importSlangSemantics(design(instance('top',body([ + {...parameter('N',"32'd2"),source_file:'rtl/top.sv',source_line:4,source_column:9}, + {kind:'Port',name:'q',direction:'Out',type:{kind:'PackedArrayType',name:'',elementType:scalar,range:'[15:0]'},source_file:'rtl/top.sv',source_line:8,source_column:0}, + ]))),'/project'); + expect(facts).toEqual([ + {kind:'parameter',name:'N',instancePath:'top',value:"32'd2",type:'int',width:32,source:{file:'rtl/top.sv',line:4,column:9}}, + {kind:'port',name:'q',instancePath:'top',type:'logic[15:0]',width:16,direction:'Out',source:{file:'rtl/top.sv',line:8,column:null}}, + ]); + }); + it('revisits shared bodies per instance and keeps specialized parameter values separate',()=>{ + const shared=body([parameter('P','2')],10); + const ast=design(instance('top',body([instance('u1',shared),instance('u2','10 leaf'),instance('u3',body([parameter('P','4')],11))]))); + expect(importSlangSemantics(ast,'/project').map(f=>[f.instancePath,f.value])).toEqual([['top.u1','2'],['top.u2','2'],['top.u3','4']]); + }); + it('uses implicit generate index values, not iteration ordinals, and skips uninstantiated branches',()=>{ + const loop={kind:'GenerateBlockArray',name:'lanes',loopVariable:'999 i',members:[{kind:'Genvar',name:'i'}, + {kind:'GenerateBlock',name:'',constructIndex:0,isUninstantiated:false,members:[parameter('i','2'),instance('u',body([parameter('P','8')]))]}, + {kind:'GenerateBlock',name:'',constructIndex:1,isUninstantiated:false,members:[parameter('i','4'),instance('u',body([parameter('P','8')]))]}, + ]}; + const ast=design(instance('top',body([loop,{kind:'GenerateBlock',name:'inactive',isUninstantiated:true,members:[instance('bad','404 missing')]}]))); + expect(importSlangSemantics(ast,'/project').filter(f=>f.name==='P').map(f=>f.instancePath)).toEqual(['top.lanes[2].u','top.lanes[4].u']); + }); + it('maps instance arrays in Slang numeric ascending element order',()=>{ + const array={kind:'InstanceArray',name:'a',range:'[3:2]',members:[instance('',body([parameter('P','10')])),instance('',body([parameter('P','20')]))]}; + expect(importSlangSemantics(design(instance('top',body([array]))),'/project').map(f=>[f.instancePath,f.value])).toEqual([['top.a[2]','10'],['top.a[3]','20']]); + }); + it('does not guess complex/unpacked widths or retain process addresses in type text',()=>{ + const ast=design(instance('top',body([ + {kind:'Port',name:'s',type:{kind:'PackedStructType',name:'Packet',members:[]}}, + {kind:'Port',name:'a',type:{kind:'FixedSizeUnpackedArrayType',name:'',elementType:scalar,range:'[7:0]'}}, + {kind:'Port',name:'pretty',type:'logic[7:0]'}, + {kind:'Port',name:'alias',type:{kind:'TypeAlias',name:'word_t',addr:5,target:{kind:'PackedArrayType',name:'',elementType:scalar,range:'[3:0]'}}}, + {kind:'Port',name:'ref',type:'5 top.word_t'}, + ]))); + const facts=importSlangSemantics(ast,'/project');expect(facts.slice(0,3).map(f=>f.width)).toEqual([undefined,undefined,undefined]); + expect(facts[4]).toMatchObject({type:'top.word_t',width:4});expect(JSON.stringify(facts)).not.toContain('5 top'); + }); + it('confines sources and preserves frontend Unicode columns without conversion',()=>{ + const facts=importSlangSemantics(design(instance('top',body([ + {...parameter('unicode','1'),source_file:'rtl/😀.sv',source_line:2,source_column:17}, + {...parameter('outside','1'),source_file:'../external.sv',source_line:2,source_column:17}, + {...parameter('absolute','1'),source_file:'/elsewhere/file.sv',source_line:2,source_column:17}, + {...parameter('windows','1'),source_file:'C:\\elsewhere\\file.sv',source_line:2,source_column:17}, + {...parameter('relativeWindows','1'),source_file:'..\\elsewhere\\file.sv',source_line:2,source_column:17}, + ]))),'/project'); + expect(facts[0]?.source).toEqual({file:'rtl/😀.sv',line:2,column:17});expect(facts.slice(1).map(f=>f.source)).toEqual([undefined,undefined,undefined,undefined]); + }); + it('does not treat Object prototype names as known integer widths',()=>{ + const facts=importSlangSemantics(design(instance('top',body([{kind:'Port',name:'x',type:'constructor'}]))),'/project'); + expect(facts[0]?.width).toBeUndefined(); + }); + it('escapes hierarchy segments that contain literal dots',()=>{ + const facts=importSlangSemantics(design(instance('top',body([instance('a.b',body([parameter('P','1')]))]))),'/project'); + expect(facts[0]?.instancePath).toBe('top.\\a.b '); + }); + it('rejects malformed schema, missing references, cycles and excessive hierarchy names',()=>{ + expect(()=>importSlangSemantics({},'/project')).toThrow(); + expect(()=>importSlangSemantics({design:{kind:'Root',members:new Array(1_000_001)}},'/project')).toThrow('node limit'); + expect(()=>importSlangSemantics(design(instance('top','99 missing')),'/project')).toThrow('unresolved'); + expect(()=>importSlangSemantics(design(instance('x'.repeat(4097),body([]))),'/project')).toThrow('oversized'); + const cyclic=body([],10);cyclic.members.push(instance('again','10 leaf')); + expect(()=>importSlangSemantics(design(instance('top',cyclic)),'/project')).toThrow('cyclic instance'); + const input:any=design();input.extra=input; + expect(()=>importSlangSemantics(input,'/project')).toThrow('cyclic input'); + }); + it('rejects ambiguous generated indices and array shape mismatches',()=>{ + expect(()=>importSlangSemantics(design(instance('top',body([{kind:'GenerateBlockArray',name:'g',loopVariable:'1 i',members:[{kind:'GenerateBlock',name:'',members:[]}]}]))),'/project')).toThrow('index parameter'); + expect(()=>importSlangSemantics(design(instance('top',body([{kind:'InstanceArray',name:'a',range:'[3:0]',members:[]}]))),'/project')).toThrow('length disagrees'); + }); +}); diff --git a/__tests__/hdl-semantic-runner.test.ts b/__tests__/hdl-semantic-runner.test.ts new file mode 100644 index 0000000000..06b43e8f6c --- /dev/null +++ b/__tests__/hdl-semantic-runner.test.ts @@ -0,0 +1,144 @@ +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { runSlangSemantics } from '../src/hdl/semantic-runner'; + +const suite = process.platform==='win32' ? describe.skip : describe; +suite('isolated optional slang runner',()=>{ + let base:string,root:string,temp:string,exe:string,oldTemp:string|undefined; + const write=(file:string,text:string)=>{fs.mkdirSync(path.dirname(path.join(root,file)),{recursive:true});fs.writeFileSync(path.join(root,file),text);}; + const fake=(body='')=>{ + const script=`#!${process.execPath}\nconst fs=require('node:fs'),cp=require('node:child_process');const args=process.argv.slice(2);if(args.includes('--version')){console.log('slang version 11.0.448+e222e7dc0');process.exit(0);}const output=args[args.indexOf('--ast-json')+1];${body}\nfs.writeFileSync(output,JSON.stringify({kind:'Root',argv:args,source:fs.readFileSync('rtl/top.sv','utf8'),include:fs.readFileSync('inc/common.flags','utf8'),cwd:process.cwd()}));`; + fs.writeFileSync(exe,script,{mode:0o700}); + }; + const fakePython=(body='',version=`codegraph pyslang 11.0.0 exporter 1 native ${'a'.repeat(64)}`)=>{ + const script=`#!${process.execPath}\nconst fs=require('node:fs');const args=process.argv.slice(2);if(args.includes('--version')){console.log(${JSON.stringify(version)});process.exit(0);}if(args[0]!=='-I'||!args[1].endsWith('source-map-export.py'))process.exit(3);const output=args[args.indexOf('--ast-json')+1];${body}\nfs.writeFileSync(output,JSON.stringify({codegraphSemanticVersion:1,facts:[],argv:args,launcher:process.argv[1]}));`; + fs.writeFileSync(exe,script,{mode:0o700}); + }; + beforeEach(()=>{ + const storage=path.join(os.homedir(),'storage');fs.mkdirSync(storage,{recursive:true}); + base=fs.mkdtempSync(path.join(storage,'cg-semantic-test-'));root=path.join(base,'project');temp=path.join(base,'snapshots');exe=path.join(base,'fake-slang'); + fs.mkdirSync(root);oldTemp=process.env.CODEGRAPH_SEMANTIC_TMPDIR;process.env.CODEGRAPH_SEMANTIC_TMPDIR=temp; + write('rtl/top.sv','`include "../inc/common.flags"\nmodule top; endmodule\n');write('inc/common.flags','`define VALUE 1\n'); + write('codegraph.json',JSON.stringify({hdl:{activeProfile:'synth',profiles:{synth:{files:['rtl/top.sv'],includeDirs:['inc'],topModules:['top'],defines:{SAFE:'1'}}}}}));fake(); + }); + afterEach(()=>{if(oldTemp===undefined)delete process.env.CODEGRAPH_SEMANTIC_TMPDIR;else process.env.CODEGRAPH_SEMANTIC_TMPDIR=oldTemp;fs.rmSync(base,{recursive:true,force:true});}); + it('requires exactly one explicit backend selector',async()=>{ + await expect(runSlangSemantics(root,{})).rejects.toThrow('exactly one'); + await expect(runSlangSemantics(root,{executable:exe,pythonExecutable:exe})).rejects.toThrow('exactly one'); + }); + it('runs only isolated Python and records exporter/native hashes with normalized facts',async()=>{ + fakePython(); + const launcher=path.join(base,'venv/bin/python');fs.mkdirSync(path.dirname(launcher),{recursive:true});fs.symlinkSync(exe,launcher); + const result=await runSlangSemantics(root,{pythonExecutable:launcher,parameters:{WIDTH:'4'}}); + expect(result.frontend).toBe('pyslang');expect(result.exporterSha256).toMatch(/^[a-f0-9]{64}$/);expect(result.librarySha256).toBe('a'.repeat(64)); + expect((result.ast as any).codegraphSemanticVersion).toBe(1);expect((result.ast as any).argv[0]).toBe('-I'); + expect((result.ast as any).launcher).toBe(launcher);expect((result.ast as any).argv).toContain('WIDTH=4');expect(fs.readdirSync(temp)).toEqual([]); + }); + it('rejects unsupported Python exporter/library versions and malformed envelopes',async()=>{ + fakePython('','codegraph pyslang 12.0.0 exporter 1 native '+ 'a'.repeat(64)); + await expect(runSlangSemantics(root,{pythonExecutable:exe})).rejects.toThrow('schema family'); + fakePython("fs.writeFileSync(output,JSON.stringify({codegraphSemanticVersion:2,facts:[]}));process.exit(0);"); + await expect(runSlangSemantics(root,{pythonExecutable:exe})).rejects.toThrow('exporter envelope'); + }); + it('rejects a Python native library fingerprint that changes across the run',async()=>{ + const marker=path.join(base,'library-changed'); + const script=`#!${process.execPath}\nconst fs=require('node:fs'),args=process.argv.slice(2);if(args.includes('--version')){console.log('codegraph pyslang 11.0.0 exporter 1 native '+(fs.existsSync(${JSON.stringify(marker)})?'b':'a').repeat(64));process.exit(0);}fs.writeFileSync(${JSON.stringify(marker)},'changed');fs.writeFileSync(args[args.indexOf('--ast-json')+1],JSON.stringify({codegraphSemanticVersion:1,facts:[]}));`; + fs.writeFileSync(exe,script,{mode:0o700});await expect(runSlangSemantics(root,{pythonExecutable:exe})).rejects.toThrow('library changed'); + expect(fs.readdirSync(temp)).toEqual([]); + }); + it('applies the same output and cancellation bounds to Python',async()=>{ + fakePython("process.stdout.write('x'.repeat(1024*1024+1));setInterval(()=>{},1000);return;"); + await expect(runSlangSemantics(root,{pythonExecutable:exe})).rejects.toThrow('output exceeds'); + fakePython("setInterval(()=>{},1000);return;"); + const controller=new AbortController(),pending=runSlangSemantics(root,{pythonExecutable:exe,signal:controller.signal}); + setTimeout(()=>controller.abort(),100);await expect(pending).rejects.toThrow('cancelled');expect(fs.readdirSync(temp)).toEqual([]); + }); + it('copies literal headers and argv safely, returns provenance/source map and removes snapshots',async()=>{ + const result=await runSlangSemantics(root,{executable:exe,parameters:{WIDTH:'8'},allowUseBeforeDeclare:true}); + const ast=result.ast as any; + expect(ast.source).toBe(result.sources['rtl/top.sv']);expect(ast.include).toBe(result.sources['inc/common.flags']); + expect(ast.argv).toContain('-DSAFE=1');expect(ast.argv).toContain('WIDTH=8');expect(ast.argv).toContain('--allow-use-before-declare'); + expect(result.configurationFingerprint).toMatch(/^[a-f0-9]{64}$/);expect(result.executableSha256).toMatch(/^[a-f0-9]{64}$/); + expect(result.sourceRoot).toBe(ast.cwd);expect(fs.existsSync(result.sourceRoot)).toBe(false);expect(fs.readdirSync(temp)).toEqual([]); + expect((await runSlangSemantics(root,{executable:exe,parameters:{WIDTH:'8'},allowUseBeforeDeclare:true})).fingerprint).toBe(result.fingerprint); + }); + it('captures caller options before awaits and records the explicit compilation mode',async()=>{ + const replacement=new AbortController();replacement.abort(); + const options={executable:exe,allowUseBeforeDeclare:false,parameters:{WIDTH:'4'},signal:new AbortController().signal}; + const pending=runSlangSemantics(root,options); + options.allowUseBeforeDeclare=true;options.parameters.WIDTH='9';options.signal=replacement.signal; + const result=await pending;expect(result.allowUseBeforeDeclare).toBe(false);expect(result.parameters.WIDTH).toBe('4'); + expect(result.languageStandard).toBe('1800-2023');expect(result.compilationUnitMode).toBe('separate'); + expect((result.ast as any).argv).not.toContain('--allow-use-before-declare'); + expect((result.ast as any).argv).toContain('--max-generate-steps=10000'); + }); + it('rejects include directives injected through configured macro values',async()=>{ + const file=path.join(root,'codegraph.json'),cfg=JSON.parse(fs.readFileSync(file,'utf8')); + cfg.hdl.profiles.synth.defines.LOAD='`include "/etc/passwd"';fs.writeFileSync(file,JSON.stringify(cfg)); + await expect(runSlangSemantics(root,{executable:exe})).rejects.toThrow('Absolute include'); + }); + it('preserves UTF-8 BOM and source-relative headers outside include directories',async()=>{ + write('rtl/top.sv','\ufeff`include "../private/local.inc"\nmodule top; endmodule\n');write('private/local.inc','// header\n'); + const result=await runSlangSemantics(root,{executable:exe});expect(result.sources['rtl/top.sv'].charCodeAt(0)).toBe(0xfeff); + expect(result.sources['private/local.inc']).toBe('// header\n'); + }); + it('requires an available approved version and an explicit profile/top',async()=>{ + await expect(runSlangSemantics(root,{executable:path.join(base,'missing')})).rejects.toThrow('unavailable'); + fs.writeFileSync(exe,`#!${process.execPath}\nconsole.log('slang version 12.0');`,{mode:0o700}); + await expect(runSlangSemantics(root,{executable:exe})).rejects.toThrow('schema family'); + write('codegraph.json','{}');await expect(runSlangSemantics(root,{executable:exe,top:'top'})).rejects.toThrow('active HDL profile'); + }); + it.each(['`include "/etc/passwd"','`include "../../../outside"','`include `FILE'])('rejects unsafe/unresolved snapshot include forms %s',async line=>{ + write('rtl/top.sv',line+'\nmodule top; endmodule');await expect(runSlangSemantics(root,{executable:exe})).rejects.toThrow(/include|escapes/); + }); + it('rejects symlink headers and oversized individual inputs',async()=>{ + fs.symlinkSync(path.join(base,'external'),path.join(root,'inc/link.flags')); + await expect(runSlangSemantics(root,{executable:exe})).rejects.toThrow('Symbolic links'); + fs.unlinkSync(path.join(root,'inc/link.flags'));write('inc/large.flags','x'.repeat(1024*1024+1)); + await expect(runSlangSemantics(root,{executable:exe})).rejects.toThrow('input size'); + }); + it('rejects original-content changes even with restored size/mtime',async()=>{ + const file=path.join(root,'inc/common.flags'); + fake(`const p=${JSON.stringify(file)},s=fs.statSync(p);fs.writeFileSync(p,'\x60define VALUE 2\\n');fs.utimesSync(p,s.atime,s.mtime);`); + await expect(runSlangSemantics(root,{executable:exe})).rejects.toThrow('sources or configuration changed'); + expect(fs.readdirSync(temp)).toEqual([]); + }); + it('rejects creation of a previously missing source-relative shadow candidate',async()=>{ + write('rtl/top.sv','`include "common.flags"\nmodule top; endmodule'); + fake(`fs.writeFileSync(${JSON.stringify(path.join(root,'rtl/common.flags'))},'// new shadow');`); + await expect(runSlangSemantics(root,{executable:exe})).rejects.toThrow('sources or configuration changed'); + }); + it('rejects a frontend that edits the snapshot, malformed AST and oversized output',async()=>{ + fake("fs.writeFileSync('rtl/top.sv','module changed; endmodule');"); + await expect(runSlangSemantics(root,{executable:exe})).rejects.toThrow('modified its source snapshot'); + fake("fs.writeFileSync(output,'not JSON');process.exit(0);"); + await expect(runSlangSemantics(root,{executable:exe})).rejects.toThrow(); + fake("fs.writeFileSync(output,'');fs.truncateSync(output,64*1024*1024+1);process.exit(0);"); + await expect(runSlangSemantics(root,{executable:exe})).rejects.toThrow(/oversized|exceeds/); + expect(fs.readdirSync(temp)).toEqual([]); + }); + it('times out and cancels its own process group without leaving snapshots',async()=>{ + fake("setInterval(()=>{},1000);return;"); + await expect(runSlangSemantics(root,{executable:exe,timeoutMs:100})).rejects.toThrow('timed out'); + const controller=new AbortController();const promise=runSlangSemantics(root,{executable:exe,signal:controller.signal}); + setTimeout(()=>controller.abort(),100);await expect(promise).rejects.toThrow('cancelled');expect(fs.readdirSync(temp)).toEqual([]); + }); + it('cancels descendants in the owned process group',async()=>{ + const heartbeat=path.join(base,'heartbeat'); + const childCode=`setInterval(()=>require('node:fs').appendFileSync(${JSON.stringify(heartbeat)},'x'),20)`; + fake(`cp.spawn(process.execPath,['-e',${JSON.stringify(childCode)}],{stdio:'ignore'});setInterval(()=>{},1000);return;`); + const controller=new AbortController(),pending=runSlangSemantics(root,{executable:exe,signal:controller.signal}); + const end=Date.now()+4000;while(!fs.existsSync(heartbeat)){if(Date.now()>end)throw new Error('descendant did not start');await new Promise(r=>setTimeout(r,20));} + controller.abort();await expect(pending).rejects.toThrow('cancelled'); + const before=fs.statSync(heartbeat).size;await new Promise(r=>setTimeout(r,100));expect(fs.statSync(heartbeat).size).toBe(before); + expect(fs.readdirSync(temp)).toEqual([]); + }); + it('bounds stdout/stderr and rejects frontend errors',async()=>{ + fake("process.stdout.write('x'.repeat(1024*1024+1));setInterval(()=>{},1000);return;"); + await expect(runSlangSemantics(root,{executable:exe})).rejects.toThrow('output exceeds'); + fake("process.stderr.write('invalid design');process.exit(2);"); + await expect(runSlangSemantics(root,{executable:exe})).rejects.toThrow('invalid design'); + }); +}); diff --git a/__tests__/hdl-semantics-review.test.ts b/__tests__/hdl-semantics-review.test.ts new file mode 100644 index 0000000000..9cb4191640 --- /dev/null +++ b/__tests__/hdl-semantics-review.test.ts @@ -0,0 +1,57 @@ +import { afterEach, expect, it, vi } from 'vitest'; +import { createHash } from 'node:crypto'; +import type { Node, FileRecord } from '../src/types'; +import type { HdlProfileStatus } from '../src/hdl/status'; +import { analyzeHdlSemantics, type HdlSemanticOptions, type HdlSemanticSourceGraph } from '../src/hdl/semantics'; +const { run }=vi.hoisted(()=>({run:vi.fn()})); +vi.mock('../src/hdl/semantic-runner',()=>({runSlangSemantics:run})); +afterEach(()=>run.mockReset()); +const text='module top #(parameter W = 8)(input logic data); endmodule'; +function result(source=text,column=24,kind:'Parameter'|'Port'='Parameter',name='W') { + return {ast:{design:{kind:'Root',members:[{kind:'Instance',name:'top',body:{kind:'InstanceBody',members:[{ + kind,name,...(kind==='Parameter'?{value:'8',type:'int'}:{type:'logic',direction:'In'}), + source_file:'/snapshot/top.sv',source_line:1,source_column:column, + }]}}]}},sourceRoot:'/snapshot',profileName:'p',configurationFingerprint:'cfg',fingerprint:'captured', + version:'slang fixture',executableSha256:'sha',sources:{'top.sv':source},top:'top',parameters:{W:'8'},allowUseBeforeDeclare:false,diagnostics:[]}; +} +function sourceGraph(nodes:Partial[],source=text):HdlSemanticSourceGraph { + return {profile:()=>({state:'matches',indexed:{configurationFingerprint:'cfg'}} as HdlProfileStatus),stale:()=>false, + file:()=>({contentHash:createHash('sha256').update(source).digest('hex')} as FileRecord), + nodes:()=>nodes.map(n=>({id:'node',name:'W',startLine:1,filePath:'top.sv',language:'verilog',...n} as Node))}; +} + +it('reports the captured compiler options and initial query even if the caller mutates options during the run',async()=>{ + const options:HdlSemanticOptions={executable:'slang',query:'top.W',parameters:{W:'8'},allowUseBeforeDeclare:false}; + run.mockImplementation(async()=>{ + options.parameters!.W='12';options.allowUseBeforeDeclare=true;options.query='missing'; + return result(); + }); + const value=await analyzeHdlSemantics('/project',options,sourceGraph([{kind:'constant'}])); + expect(value.facts).toHaveLength(1); + expect(value.facts[0]!.value).toBe('8'); + expect(value.provenance.parameters).toEqual({W:'8'}); + expect(value.provenance.allowUseBeforeDeclare).toBe(false); +}); + +it.each([ + {kind:'Parameter' as const,name:'W',column:24,nodes:[{kind:'function' as const}],linked:false}, + {kind:'Port' as const,name:'data',column:42,nodes:[{kind:'field' as const,name:'data'}],linked:false}, + {kind:'Port' as const,name:'data',column:42,nodes:[{kind:'field' as const,name:'data',decorators:['hdl:port']}],linked:true}, + {kind:'Parameter' as const,name:'W',column:100000,nodes:[{kind:'constant' as const}],linked:false}, + {kind:'Parameter' as const,name:'W',column:0,nodes:[{kind:'constant' as const}],linked:false}, + {kind:'Parameter' as const,name:'W',column:24,nodes:[{id:'one',kind:'constant' as const},{id:'two',kind:'constant' as const}],linked:false}, +])('links only compatible unambiguous source declarations: %j',async({kind,name,column,nodes,linked})=>{ + run.mockResolvedValue(result(text,column,kind,name)); + const value=await analyzeHdlSemantics('/project',{executable:'slang'},sourceGraph(nodes)); + expect(!!value.facts[0]!.sourceNodeId).toBe(linked); +}); + +it('accepts frontend byte coordinates beyond the UTF-16 line length without reinterpreting their units',async()=>{ + const unicode='/* '+'😀'.repeat(40)+' */ '+text; + const column=Buffer.byteLength(unicode.slice(0,unicode.indexOf('W')))+1; + expect(column).toBeGreaterThan(unicode.length); + run.mockResolvedValue(result(unicode,column)); + const value=await analyzeHdlSemantics('/project',{executable:'slang'},sourceGraph([{kind:'constant'}],unicode)); + expect(value.facts[0]!.sourceNodeId).toBe('node'); + expect(value.facts[0]!.source?.column).toBe(column); +}); diff --git a/__tests__/hdl-semantics.test.ts b/__tests__/hdl-semantics.test.ts new file mode 100644 index 0000000000..bce988a0cc --- /dev/null +++ b/__tests__/hdl-semantics.test.ts @@ -0,0 +1,72 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import CodeGraph from '../src/index'; +import { loadHdlProfile } from '../src/hdl/profile'; +const { run } = vi.hoisted(() => ({ run: vi.fn() })); +vi.mock('../src/hdl/semantic-runner', () => ({ runSlangSemantics: run })); + +let root: string; +let cg: CodeGraph; +const source = 'module top #(parameter W = 8)(input logic [W-1:0] data);\nendmodule\n'; +function ast(width = 8, column: number = 24) { + return { design: { kind: 'Root', name: '$root', members: [{ kind: 'Instance', name: 'top', body: { + kind: 'InstanceBody', name: 'top', members: [ + { kind: 'Parameter', name: 'W', value: String(width), type: { kind: 'PredefinedIntegerType', name: 'int' }, + source_file: path.join(root, 'top.sv'), source_line: 1, source_column: column }, + { kind: 'Port', name: 'data', type: { kind: 'PackedArrayType', range: `[${width - 1}:0]`, elementType: { kind: 'ScalarType', name: 'logic' } }, + direction: 'In', source_file: path.join(root, 'top.sv'), source_line: 1, source_column: 49 }, + ], + } }] } }; +} +function result(value = ast()) { + return { ast: value, sourceRoot: root, profileName: 'synth', configurationFingerprint: loadHdlProfile(root).profile!.fingerprint, + fingerprint: 'semantic-snapshot', version: 'slang version 11.0.448', executableSha256: 'executable-hash', + sources: { 'top.sv': source }, top: 'top', diagnostics: [], parameters: {}, allowUseBeforeDeclare: false }; +} +beforeEach(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'hdl-semantics-')); + fs.writeFileSync(path.join(root, 'top.sv'), source); + fs.writeFileSync(path.join(root, 'codegraph.json'), JSON.stringify({ hdl: { activeProfile: 'synth', profiles: { + synth: { files: ['top.sv'], topModules: ['top'] }, + } } })); + cg = await CodeGraph.init(root, { index: true }); + run.mockResolvedValue(result()); +}); +afterEach(() => { cg?.close(); fs.rmSync(root, { recursive: true, force: true }); run.mockReset(); }); + +describe('on-demand HDL semantics on the source graph', () => { + it('returns computed facts and links the matching source declaration', async () => { + const value = await cg.getHdlSemantics({ executable: 'slang', query: 'top.W' }); + expect(value.sourceGraphProfileMatches).toBe(true); + expect(value.facts).toHaveLength(1); + expect(value.facts[0]).toMatchObject({ name: 'W', value: '8', sourceNodeId: cg.getNodesByName('W')[0].id }); + expect(value.provenance.top).toBe('top'); + }); + it('does not link an index from another profile or different source bytes', async () => { + const changed = result(); changed.configurationFingerprint = 'different-profile'; + run.mockResolvedValue(changed); + expect((await cg.getHdlSemantics({ executable: 'slang' })).facts.every(f => !f.sourceNodeId)).toBe(true); + run.mockResolvedValue({ ...result(), sources: { 'top.sv': source.replace('8', '9') } }); + expect((await cg.getHdlSemantics({ executable: 'slang' })).facts.every(f => !f.sourceNodeId)).toBe(true); + }); + it('preserves unknown macro columns, exact queries and truncation', async () => { + run.mockResolvedValue(result(ast(8, 0))); + const value = await cg.getHdlSemantics({ executable: 'slang', query: 'top', limit: 1 }); + expect(value.totalMatches).toBe(2); expect(value.truncated).toBe(true); + expect(value.facts[0].source?.column).toBeNull(); expect(value.facts[0].sourceNodeId).toBeUndefined(); + expect((await cg.getHdlSemantics({ executable: 'slang', query: 'missing' })).facts).toHaveLength(0); + await expect(cg.getHdlSemantics({ executable: 'slang', limit: 0 })).rejects.toThrow('1..1000'); + }); + it('recomputes overrides and keeps source graph after compiler failure', async () => { + run.mockResolvedValue(result(ast(12))); + expect((await cg.getHdlSemantics({ executable: 'slang', parameters: { W: '12' }, query: 'W' })).facts[0].value).toBe('12'); + expect(run.mock.calls[0][1].parameters).toEqual({ W: '12' }); + const before = cg.getNodesByName('W'); + run.mockRejectedValue(new Error('slang unavailable')); + await expect(cg.getHdlSemantics({ executable: 'missing' })).rejects.toThrow('unavailable'); + expect(cg.getNodesByName('W')).toEqual(before); + expect(cg.getHdlProfileStatus()?.state).toBe('matches'); + }); +}); diff --git a/__tests__/hdl-source-map-asset.test.ts b/__tests__/hdl-source-map-asset.test.ts new file mode 100644 index 0000000000..6888c3847d --- /dev/null +++ b/__tests__/hdl-source-map-asset.test.ts @@ -0,0 +1,8 @@ +import { expect, it } from 'vitest'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +it('ships the Python exporter byte-for-byte with the compiled engine', () => { + const root = path.resolve(__dirname, '..'); + expect(fs.readFileSync(path.join(root, 'dist/hdl/source-map-export.py'))) + .toEqual(fs.readFileSync(path.join(root, 'src/hdl/source-map-export.py'))); +}); diff --git a/__tests__/hdl-source-map-import.test.ts b/__tests__/hdl-source-map-import.test.ts new file mode 100644 index 0000000000..7d9da77f6a --- /dev/null +++ b/__tests__/hdl-source-map-import.test.ts @@ -0,0 +1,41 @@ +import { expect, it } from 'vitest'; +import { importMappedSemantics } from '../src/hdl/source-map-import'; +const sources = { 'inc/ports.svh': '`define P(N) input logic N\r\n', 'top.sv': '// Ω\r\nmodule top(`P(data)); endmodule\r\n' }; +const point = (file: keyof typeof sources, offset: number) => { + const prefix = Buffer.from(sources[file]).subarray(0, offset); + const line = [...prefix].filter(v => v === 10).length + 1; + return { file, line, column: offset - prefix.lastIndexOf(10), byteOffset: offset }; +}; +const start = Buffer.from(sources['top.sv']).indexOf('`P'); +const payload = () => ({ codegraphSemanticVersion: 1, facts: [{ kind: 'port', name: 'data', instancePath: 'top', type: 'logic', width: 1, + sourceOrigin: 'macro', source: { file: 'top.sv', line: 2, column: 15 }, macroExpansionComplete: true, + macroExpansion: [{ name: 'P', argument: false, spelling: point('inc/ports.svh', 24), invocation: { start: point('top.sv', start), end: point('top.sv', start + 8) } }], +}] }); +it('preserves compiler byte coordinates and macro call/spelling across headers, UTF8 and CRLF', () => { + const result = importMappedSemantics(payload(), sources); + expect(result[0].macroExpansion?.[0]).toEqual(payload().facts[0]!.macroExpansion[0]); + expect(result[0].sourceOrigin).toBe('macro'); +}); +it('rejects tampered source paths, byte offsets, reversed ranges and UTF8 continuation positions', () => { + let p = payload(); p.facts[0]!.macroExpansion[0]!.spelling.file = '../secret' as any; + expect(() => importMappedSemantics(p, sources)).toThrow('outside snapshot'); + p = payload(); p.facts[0]!.macroExpansion[0]!.spelling.byteOffset++; + expect(() => importMappedSemantics(p, sources)).toThrow('disagree'); + p = payload(); const range = p.facts[0]!.macroExpansion[0]!.invocation; [range.start, range.end] = [range.end, range.start]; + expect(() => importMappedSemantics(p, sources)).toThrow('order'); + p = payload(); p.facts[0]!.macroExpansion[0]!.spelling = point('top.sv', 4); + expect(() => importMappedSemantics(p, sources)).toThrow('UTF-8'); +}); +it('does not accept invented complete chains or duplicate identities', () => { + const p = payload(); p.facts[0]!.macroExpansion = []; + expect(() => importMappedSemantics(p, sources)).toThrow('complete'); + p.facts[0]!.macroExpansionComplete = false; + expect(importMappedSemantics(p, sources)[0].macroExpansionComplete).toBe(false); + p.facts.push(p.facts[0]!); + expect(() => importMappedSemantics(p, sources)).toThrow('duplicate'); +}); +it('rejects line/offset disagreement at a newline and unsupported schema', () => { + const p = payload(); p.facts[0]!.macroExpansion[0]!.spelling = { file: 'top.sv', line: 1, column: 8, byteOffset: 7 }; + expect(() => importMappedSemantics(p, sources)).toThrow('column'); + expect(() => importMappedSemantics({ codegraphSemanticVersion: 2, facts: [] }, sources)).toThrow('schema'); +}); diff --git a/__tests__/hdl-source-map-review.test.ts b/__tests__/hdl-source-map-review.test.ts new file mode 100644 index 0000000000..1c448ba200 --- /dev/null +++ b/__tests__/hdl-source-map-review.test.ts @@ -0,0 +1,67 @@ +import { afterEach, expect, it, vi } from 'vitest'; +import { createHash } from 'node:crypto'; +import { importMappedSemantics } from '../src/hdl/source-map-import'; +import { analyzeHdlSemantics, type HdlSemanticSourceGraph } from '../src/hdl/semantics'; +import type { FileRecord, Node } from '../src/types'; +import type { HdlProfileStatus } from '../src/hdl/status'; +const {run}=vi.hoisted(()=>({run:vi.fn()})); +vi.mock('../src/hdl/semantic-runner',()=>({runSlangSemantics:run})); +afterEach(()=>run.mockReset()); +const source='// Ω\r\nmodule top(input logic data);\r\nendmodule\r\n'; +const sources={'top.sv':source}; +const fact=()=>({kind:'port',name:'data',instancePath:'top',width:1,direction:'In',sourceOrigin:'direct', + source:{file:'top.sv',line:2,column:24}}); +const payload=(item:unknown=fact())=>({codegraphSemanticVersion:1,facts:[item]}); +const graph:HdlSemanticSourceGraph={profile:()=>({state:'matches',indexed:{configurationFingerprint:'cfg'}} as HdlProfileStatus),stale:()=>false, + file:()=>({contentHash:createHash('sha256').update(source).digest('hex')} as FileRecord), + nodes:()=>[{id:'decl-data',kind:'field',name:'data',startLine:2,decorators:['hdl:port']} as Node]}; +function result(ast:unknown){return {ast,frontend:'pyslang',sourceRoot:'/deleted-snapshot',profileName:'p',configurationFingerprint:'cfg',fingerprint:'snapshot', + version:'pyslang fixture',executableSha256:'python-sha',exporterSha256:'exporter-sha',librarySha256:'native-sha',sources,top:'top',parameters:{},allowUseBeforeDeclare:false, + diagnostics:[],languageStandard:'1800-2023',compilationUnitMode:'separate',runnerVersion:'review',compilerLimits:[]};} + +it('never links macro-origin facts despite precise same-point coordinates, while direct declarations can link',async()=>{ + run.mockResolvedValue(result(payload())); + const direct=await analyzeHdlSemantics('/project',{pythonExecutable:'python'},graph); + expect(direct.facts[0]!.sourceNodeId).toBe('decl-data'); + run.mockResolvedValue(result(payload({...fact(),sourceOrigin:'macro',macroExpansion:[],macroExpansionComplete:false}))); + const macro=await analyzeHdlSemantics('/project',{pythonExecutable:'python'},graph); + expect(macro.facts[0]!.source).toEqual(direct.facts[0]!.source); + expect(macro.facts[0]!.sourceOrigin).toBe('macro'); + expect(macro.facts[0]!.sourceNodeId).toBeUndefined(); + expect(macro.provenance).toMatchObject({frontend:'pyslang',exporterSha256:'exporter-sha',librarySha256:'native-sha'}); +}); + +it('normalizes away unsolicited source links and does not trust unknown snapshot filenames',()=>{ + const normalized=importMappedSemantics(payload({...fact(),sourceNodeId:'forged',extra:'untrusted'}),sources)[0]!; + expect(normalized).not.toHaveProperty('sourceNodeId');expect(normalized).not.toHaveProperty('extra'); + for(const file of ['../top.sv','./top.sv','top.sv/../top.sv','__proto__','C:/top.sv','top\\file.sv']) { + expect(()=>importMappedSemantics(payload({...fact(),source:{...fact().source,file}}),sources)).toThrow('snapshot'); + } +}); + +it('rejects UTF8 continuation and previous-line aliases of the next CRLF line, including macro byte-offset tampering',()=>{ + expect(()=>importMappedSemantics(payload({...fact(),source:{file:'top.sv',line:1,column:5}}),sources)).toThrow('UTF-8'); + expect(()=>importMappedSemantics(payload({...fact(),source:{file:'top.sv',line:1,column:8}}),sources)).toThrow('column'); + const point={file:'top.sv',line:2,column:1,byteOffset:7}; + const mapped={...fact(),sourceOrigin:'macro',macroExpansionComplete:false, + macroExpansion:[{name:'P',argument:false,spelling:point,invocation:{start:point,end:point}}]}; + expect(importMappedSemantics(payload(mapped),sources)[0]!.macroExpansion![0]!.spelling!.byteOffset).toBe(7); + expect(()=>importMappedSemantics(payload({...mapped,macroExpansionComplete:true}),sources)).toThrow('complete'); + const tampered={...mapped,macroExpansion:[{...mapped.macroExpansion[0]!,spelling:{...point,byteOffset:8}}]}; + expect(()=>importMappedSemantics(payload(tampered),sources)).toThrow('disagree'); +}); + +it('bounds normalized fact counts, fields and chains before accepting exporter output',()=>{ + expect(()=>importMappedSemantics({codegraphSemanticVersion:1,facts:Array(100001).fill(fact())},sources)).toThrow('count'); + expect(()=>importMappedSemantics(payload({...fact(),value:'x'.repeat(65537)}),sources)).toThrow('value'); + expect(()=>importMappedSemantics(payload({...fact(),sourceOrigin:'macro',macroExpansionComplete:false,macroExpansion:Array(129).fill({name:null,argument:false})}),sources)).toThrow('chain'); + expect(()=>importMappedSemantics(payload({...fact(),sourceOrigin:'macro',macroExpansionComplete:true,macroExpansion:[]}),sources)).toThrow('complete'); +}); + +it('links direct typedefs by type_alias kind, never through a same-name field',async()=>{ + const typeFact={...fact(),kind:'type',type:'logic[31:0]',width:32}; + run.mockResolvedValue(result(payload(typeFact))); + expect((await analyzeHdlSemantics('/project',{pythonExecutable:'python'},graph)).facts[0]!.sourceNodeId).toBeUndefined(); + const types={...graph,nodes:()=>[{id:'type-data',kind:'type_alias',name:'data',startLine:2} as Node]}; + expect((await analyzeHdlSemantics('/project',{pythonExecutable:'python'},types)).facts[0]!.sourceNodeId).toBe('type-data'); +}); diff --git a/__tests__/hdl-wildcard-ports.test.ts b/__tests__/hdl-wildcard-ports.test.ts new file mode 100644 index 0000000000..5a783e5c15 --- /dev/null +++ b/__tests__/hdl-wildcard-ports.test.ts @@ -0,0 +1,131 @@ +import { it, expect } from 'vitest'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { CodeGraph } from '../src'; + +async function graph(files: Record, check: (cg: CodeGraph, root: string) => Promise | void): Promise { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'hdl-wildcard-')); + let cg: CodeGraph | undefined; + try { + for (const [file, source] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, file)), { recursive: true }); + fs.writeFileSync(path.join(root, file), source); + } + cg = CodeGraph.initSync(root); + expect((await cg.indexAll()).success).toBe(true); + await check(cg, root); + } finally { cg?.close(); fs.rmSync(root, { recursive: true, force: true }); } +} +function wildcard(cg: CodeGraph, qname = 'top::u::*') { + const node = cg.getNodesByName('*').find(n => n.qualifiedName === qname)!; + expect(node).toBeDefined(); + return cg.getCallees(node.id).filter(e => e.edge.metadata?.binding === 'hdl-wildcard-port') + .map(e => ({ target: e.node.qualifiedName, endpoint: e.edge.metadata!.endpoint, + port: e.edge.metadata!.portName })).sort((a, b) => a.target.localeCompare(b.target)); +} +const pair = (name: string) => [ + { target: `leaf::${name}`, endpoint: 'formal', port: name }, + { target: `top::${name}`, endpoint: 'actual', port: name }, +]; + +it('pairs wildcard ports only with the exact local signal and keeps endpoint identity', async () => { + await graph({ 'leaf.sv': 'module leaf(input rx,output tx,input missing); endmodule', + 'top.sv': 'module top(input rx,output tx,input extra); leaf u(.*); endmodule', + 'other.sv': 'module other(input missing); endmodule' }, cg => { + expect(wildcard(cg)).toEqual([...pair('rx'), ...pair('tx')].sort((a, b) => a.target.localeCompare(b.target))); + }); +}); + +it('explicit named and open connections override .* independently of source order', async () => { + await graph({ 'leaf.sv': 'module leaf(input a,b,c); endmodule', + 'top.sv': 'module top(input a,b,c,x); leaf u(.a(x),.*, .b()); endmodule' }, cg => { + expect(wildcard(cg)).toEqual(pair('c')); + }); +}); + +it('does not bind an outer signal through an enclosing genvar or parameter', async () => { + await graph({ 'leaf.sv': 'module leaf(input i,input a); endmodule', + 'top.sv': 'module top(input i,input a); for(genvar i=0;i<2;i++) begin: lanes localparam a=1; leaf u(.*); end endmodule' }, cg => { + expect(wildcard(cg, 'top::lanes::u::*')).toEqual([]); + }); +}); + +it.each([ + { 'a.sv': 'module leaf(input a); endmodule', 'b.sv': 'module leaf(input a); endmodule' }, + { 'a.sv': 'module leaf(input a); endmodule module leaf(input a); endmodule' }, +])('rejects ambiguous module definitions %j', async definitions => { + await graph({ ...definitions, 'top.sv': 'module top(input a); leaf u(.*); endmodule' } as Record, cg => { + expect(wildcard(cg)).toEqual([]); + }); +}); + +it('refreshes added and removed formal endpoints when the wildcard source is unchanged', async () => { + await graph({ 'leaf.sv': 'module leaf(input a); endmodule', + 'top.sv': 'module top(input a,b); leaf u(.*); endmodule' }, async (cg, root) => { + expect(wildcard(cg)).toEqual(pair('a')); + fs.writeFileSync(path.join(root, 'leaf.sv'), 'module leaf(input a,b); endmodule'); + await cg.sync(); + expect(wildcard(cg)).toEqual([...pair('a'), ...pair('b')].sort((a,b)=>a.target.localeCompare(b.target))); + fs.writeFileSync(path.join(root, 'leaf.sv'), 'module leaf(input b); endmodule'); + await cg.sync(); + expect(wildcard(cg)).toEqual(pair('b')); + }); +}); + +it('retries a known module with no matches and updates added or removed local signals', async () => { + await graph({ 'leaf.sv': 'module leaf(input absent); endmodule', + 'top.sv': 'module top(input b); leaf u(.*); endmodule' }, async (cg, root) => { + expect(wildcard(cg)).toEqual([]); + fs.writeFileSync(path.join(root, 'leaf.sv'), 'module leaf(input b); endmodule'); + await cg.sync(); + expect(wildcard(cg)).toEqual(pair('b')); + fs.writeFileSync(path.join(root, 'top.sv'), 'module top; leaf u(.*); endmodule'); + await cg.sync(); + expect(wildcard(cg)).toEqual([]); + fs.writeFileSync(path.join(root, 'top.sv'), 'module top(input b); leaf u(.*); endmodule'); + await cg.sync(); + expect(wildcard(cg)).toEqual(pair('b')); + }); +}); + +it('discovers a previously unknown module and removes bindings when a duplicate appears', async () => { + await graph({ 'top.sv': 'module top(input a); leaf u(.*); endmodule' }, async (cg, root) => { + expect(wildcard(cg)).toEqual([]); + fs.writeFileSync(path.join(root, 'leaf.sv'), 'module leaf(input a); endmodule'); + await cg.sync(); + expect(wildcard(cg)).toEqual(pair('a')); + fs.writeFileSync(path.join(root, 'duplicate.sv'), 'module leaf(input a); endmodule'); + await cg.sync(); + expect(wildcard(cg)).toEqual([]); + fs.unlinkSync(path.join(root, 'duplicate.sv')); + await cg.sync(); + expect(wildcard(cg)).toEqual(pair('a')); + fs.unlinkSync(path.join(root, 'leaf.sv')); + await cg.sync(); + expect(wildcard(cg)).toEqual([]); + }); +}); + +it('normalizes escaped source identifiers for matching and explicit exclusions', async () => { + await graph({ 'leaf.sv': 'module leaf(input \\a , input b); endmodule', + 'top.sv': 'module top(input a, input \\b ); leaf u(.*, .\\a ()); endmodule' }, cg => { + expect(wildcard(cg)).toEqual([ + { target: 'leaf::b', endpoint: 'formal', port: 'b' }, + { target: 'top::\\b', endpoint: 'actual', port: 'b' }, + ]); + }); +}); + +it.each([ + 'module leaf(.p(a)); input a; endmodule', + 'module leaf(.p({a,b})); input a,b; endmodule', + '`define PORTS input a\nmodule leaf(`PORTS); endmodule', +])('keeps only the module dependency for unknown or complex non-ANSI headers: %s', async leaf => { + await graph({ 'leaf.sv': leaf, 'top.sv': 'module top(input a,b,p); leaf u(.*); endmodule' }, cg => { + expect(wildcard(cg)).toEqual([]); + const node = cg.getNodesByName('*').find(n => n.qualifiedName === 'top::u::*')!; + expect(cg.getCallees(node.id).filter(e => e.edge.metadata?.binding === 'hdl-wildcard-dependency') + .map(e => e.node.qualifiedName)).toEqual(['leaf']); + }); +}); diff --git a/__tests__/verilog-access-sync.test.ts b/__tests__/verilog-access-sync.test.ts new file mode 100644 index 0000000000..3913c8c9c6 --- /dev/null +++ b/__tests__/verilog-access-sync.test.ts @@ -0,0 +1,47 @@ +import { it, expect } from 'vitest'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { CodeGraph } from '../src'; + +it('reclassifies untouched caller arguments after signature changes, deletion and reappearance', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'hdl-access-sync-')); + let cg: CodeGraph | undefined; + const pkg = (dir: string) => `package lib; task move(${dir} int x); x=1; endtask endpackage`; + const top = 'module top; int value; initial lib::move(value); endmodule'; + try { + fs.writeFileSync(path.join(root, 'pkg.sv'), pkg('output')); + fs.writeFileSync(path.join(root, 'top.sv'), top); + cg = CodeGraph.initSync(root); + await cg.indexAll(); + const rows = () => (cg as any).db.db.prepare("SELECT e.metadata FROM edges e JOIN nodes t ON t.id=e.target JOIN nodes s ON s.id=e.source WHERE e.kind='references' AND t.qualified_name='top::value' AND s.file_path='top.sv'").all(); + const roles = () => rows().flatMap((r: any) => JSON.parse(r.metadata)?.hdlAccess ?? []).sort(); + expect(roles()).toEqual(['write']); + fs.writeFileSync(path.join(root, 'pkg.sv'), pkg('input')); + await cg.sync(); + expect(roles()).toEqual(['read']); + fs.unlinkSync(path.join(root, 'pkg.sv')); + await cg.sync(); + expect(rows()).toHaveLength(1); // Keep the ordinary reference without inferred direction. + expect(roles()).toEqual([]); + fs.writeFileSync(path.join(root, 'pkg.sv'), pkg('inout')); + await cg.sync(); + expect(roles()).toEqual(['readwrite']); + expect(fs.readFileSync(path.join(root, 'top.sv'), 'utf8')).toBe(top); + await cg.indexAll(); + expect(roles()).toEqual(['readwrite']); + } finally { cg?.close(); fs.rmSync(root, { recursive: true, force: true }); } +}); + +it('preserves distinct same-line read/write occurrences and their columns', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'hdl-access-sites-')); + let cg: CodeGraph | undefined; + try { + fs.writeFileSync(path.join(root, 'top.sv'), 'module top; int x; initial x=x+1; endmodule'); + cg = CodeGraph.initSync(root); await cg.indexAll(); + const rows = (cg as any).db.db.prepare("SELECT e.col, e.metadata FROM edges e JOIN nodes t ON t.id=e.target WHERE e.kind='references' AND t.qualified_name='top::x' ORDER BY e.col").all(); + expect(rows).toHaveLength(2); + expect(rows[0].col).not.toBe(rows[1].col); + expect(rows.map((r: any) => JSON.parse(r.metadata).hdlAccess)).toEqual([['write'], ['read']]); + } finally { cg?.close(); fs.rmSync(root, { recursive: true, force: true }); } +}); diff --git a/__tests__/verilog-access.test.ts b/__tests__/verilog-access.test.ts new file mode 100644 index 0000000000..7a9c278ab1 --- /dev/null +++ b/__tests__/verilog-access.test.ts @@ -0,0 +1,111 @@ +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { CodeGraph } from '../src'; +import { beforeAll, describe, expect, it } from 'vitest'; +import { extractFromSource } from '../src/extraction'; +import { initGrammars, loadGrammarsForLanguages } from '../src/extraction/grammars'; +beforeAll(async()=>{await initGrammars();await loadGrammarsForLanguages(['verilog']);}); +const access=(source:string)=>extractFromSource('access.sv',source).unresolvedReferences + .filter(r=>r.referenceName.startsWith('hdl:signal:')) + .map(r=>({name:r.referenceName.slice('hdl:signal:'.length),line:r.line,access:r.candidates?.find(c=>c.startsWith('hdl:access:'))?.slice('hdl:access:'.length),event:r.candidates?.find(c=>c.startsWith('hdl:event:'))?.slice('hdl:event:'.length)})); +const compact=(source:string)=>access(source).map(({name,access})=>[name,access]); +describe('syntactic HDL signal access',()=>{ + it('distinguishes assignment destinations from indexes and RHS expressions',()=>{ + expect(compact('module m; assign x[i+:W]=y; endmodule')).toEqual([['x','write'],['i','read'],['W','read'],['y','read']]); + expect(compact('module m; initial x[i] <= y; endmodule')).toEqual([['x','write'],['i','read'],['y','read']]); + expect(compact('module m; assign {a,b[j]}=v; endmodule')).toEqual([['a','write'],['b','write'],['j','read'],['v','read']]); + }); + it('labels compound and increment accesses without classifying their indexes as writes',()=>{ + expect(compact('module m; initial begin x[i]+=y; x[j]++; --x[k]; end endmodule')).toEqual([['x','readwrite'],['i','read'],['y','read'],['x','readwrite'],['j','read'],['x','readwrite'],['k','read']]); + }); + it('separates conditions and edge event expressions from data reads',()=>{ + const refs=access(`module m; +always @(posedge clk or negedge rst) begin + if(en) q=d; + case(sel) choice: q=d; endcase + while(run) q=d; +end endmodule`); + expect(refs.filter(r=>r.line===2).map(r=>[r.name,r.access,r.event])).toEqual([['clk','event','posedge'],['rst','event','negedge']]); + expect(refs.filter(r=>r.access==='control').map(r=>r.name)).toEqual(['en','sel','choice','run']); + expect(refs.filter(r=>r.name==='q').every(r=>r.access==='write')).toBe(true); + expect(refs.filter(r=>r.name==='d').every(r=>r.access==='read')).toBe(true); + }); + it('separates loop initialization, condition, increment, and indexed body assignment',()=>{ + expect(compact('module m; initial for(i=0;i{ + expect(access('module m; always @(posedge clk iff en) q=d; endmodule').map(r=>[r.name,r.access,r.event])).toEqual([ + ['clk','event','posedge'],['en','control',undefined],['q','write',undefined],['d','read',undefined], + ]); + }); + it('records formal order and directions for grouped ANSI and legacy task declarations',()=>{ + const result=extractFromSource('formals.sv','module m; task f(output int x, int y, const ref int z, input q=0); endtask task g; output a; input b,c; endtask endmodule'); + const formals=result.nodes.filter(n=>n.decorators?.includes('hdl:formal')); + expect(formals.map(n=>[n.qualifiedName,n.decorators?.find(d=>d.startsWith('hdl:formal-index:')),n.decorators?.find(d=>d.startsWith('hdl:direction:'))])).toEqual([ + ['m::f::x','hdl:formal-index:0','hdl:direction:output'],['m::f::y','hdl:formal-index:1','hdl:direction:output'], + ['m::f::z','hdl:formal-index:2','hdl:direction:const-ref'],['m::f::q','hdl:formal-index:3','hdl:direction:input'], + ['m::g::a','hdl:formal-index:0','hdl:direction:output'],['m::g::b','hdl:formal-index:1','hdl:direction:input'],['m::g::c','hdl:formal-index:2','hdl:direction:input'], + ]); + expect(formals.find(n=>n.name==='q')?.decorators).toContain('hdl:default'); + }); + it('preserves explicit increment side effects inside call arguments',()=>{ + const result=extractFromSource('effect.sv','module m; initial consume(i++); endmodule'); + const ref=result.unresolvedReferences.find(r=>r.referenceName==='hdl:signal:i')!; + expect(ref.candidates).toContain('hdl:access:readwrite'); + expect(ref.candidates?.some(c=>c.startsWith('hdl:call-arg:'))).toBe(false); + }); + it('does not certify a recovered callable signature for access inference',()=>{ + const result=extractFromSource('recovered.sv','module m; function int f(input int x); for(int i=0,j=0;i<3;i++) x=i; return x; endfunction endmodule'); + const formal=result.nodes.find(n=>n.qualifiedName==='m::f::x')!; + expect(formal.decorators).toContain('hdl:formal'); + expect(formal.decorators?.some(d=>d.startsWith('hdl:formal-index:')||d.startsWith('hdl:direction:'))).toBe(false); + }); + it('classifies returned values and delay operands as reads',()=>{ + const refs=access(`module m; +function int f(); return d; endfunction +initial #(d) q=d; +initial q = #(delay_value) d; +initial #wait_time q=d; +endmodule`); + expect(refs.filter(r=>r.line===2).map(r=>[r.name,r.access])).toEqual([['d','read']]); + expect(refs.filter(r=>r.line===3).map(r=>[r.name,r.access])).toEqual([['d','read'],['q','write'],['d','read']]); + expect(refs.filter(r=>r.line===4).map(r=>[r.name,r.access])).toEqual([['q','write'],['delay_value','read'],['d','read']]); + expect(refs.filter(r=>r.line===5).map(r=>[r.name,r.access])).toEqual([['wait_time','read'],['q','write'],['d','read']]); + }); + it('retains readwrite and control roles for increments inside conditions',()=>{ + const refs=extractFromSource('effect-control.sv','module m; initial while(i++) q=d; initial if(j++) q=d; endmodule').unresolvedReferences; + for(const name of ['i','j']) { + const ref=refs.find(r=>r.referenceName===`hdl:signal:${name}`)!; + expect(ref.candidates).toContain('hdl:access:readwrite'); + expect(ref.candidates).toContain('hdl:access:control'); + } + }); + it('records an explicit initializer write only on initialized net/data declarators',()=>{ + const source='module m; wire feedback = crc[15] ^ bit_in; logic q=0; reg idle, ready=1; localparam WIDTH=8; logic array[WIDTH]; endmodule'; + const result=extractFromSource('initializers.sv',source); + const writes=result.unresolvedReferences.filter(r=>r.candidates?.includes('hdl:access:write')); + expect(writes.map(r=>r.referenceName)).toEqual(['hdl:signal:feedback','hdl:signal:q','hdl:signal:ready']); + for(const ref of writes) expect(result.nodes.find(n=>n.id===ref.fromNodeId)?.name).toBe(ref.referenceName.slice('hdl:signal:'.length)); + expect(result.unresolvedReferences.filter(r=>r.candidates?.includes('hdl:access:read')).map(r=>r.referenceName)).toContain('hdl:signal:crc'); + }); + it('persists declarator self-writes alongside RHS reads in the graph',async()=>{ + const root=fs.mkdtempSync(path.join(os.tmpdir(),'hdl-initializer-write-')); + let cg:CodeGraph|undefined; + try { + fs.writeFileSync(path.join(root,'crc.sv'),'module crc(input [15:0] bits, input bit_in); wire feedback=bits[15]^bit_in; logic q=0; logic idle; endmodule'); + cg=CodeGraph.initSync(root);await cg.indexAll(); + const feedback=cg.getNodesByName('feedback')[0]!; + const refs=cg.getCallees(feedback.id).filter(e=>e.edge.kind==='references'); + const self=(cg as any).db.db.prepare('SELECT metadata FROM edges WHERE source=? AND target=? AND kind=?').get(feedback.id,feedback.id,'references'); + expect(JSON.parse(self.metadata).hdlAccess).toContain('write'); + expect(refs.filter(e=>e.node.id!==feedback.id).map(e=>e.node.name).sort()).toEqual(['bit_in','bits']); + const idle=cg.getNodesByName('idle')[0]!; + expect((cg as any).db.db.prepare('SELECT id FROM edges WHERE source=? AND target=?').get(idle.id,idle.id)).toBeUndefined(); + } finally {cg?.close();fs.rmSync(root,{recursive:true,force:true});} + }); + it('keeps qualified targets suppressed while classifying their local indexes and RHS',()=>{ + expect(compact('module m; initial p::x[i]=y; endmodule')).toEqual([['i','read'],['y','read']]); + expect(compact('module m; assign obj.x[i]=y; endmodule')).toEqual([['i','read'],['y','read']]); + }); +}); diff --git a/__tests__/verilog-call-access.test.ts b/__tests__/verilog-call-access.test.ts new file mode 100644 index 0000000000..95b028868a --- /dev/null +++ b/__tests__/verilog-call-access.test.ts @@ -0,0 +1,101 @@ +import { beforeAll, it, expect } from 'vitest'; +import { extractFromSource } from '../src/extraction'; +import { initGrammars, loadGrammarsForLanguages } from '../src/extraction/grammars'; +import { resolveVerilogCallArgumentAccess } from '../src/resolution/verilog-call-access'; +import { matchVerilogMember } from '../src/resolution/verilog-members'; +import type { ResolutionContext, UnresolvedRef } from '../src/resolution/types'; +import { VERILOG_CALL_ARGUMENT_PREFIX } from '../src/extraction/languages/verilog-call-access'; +beforeAll(async()=>{await initGrammars();await loadGrammarsForLanguages(['verilog']);}); + +function accesses(files: Record) { + const results = Object.entries(files).map(([file,source]) => ({file,result:extractFromSource(file,source)})); + const nodes = results.flatMap(r=>r.result.nodes); + const context = { + getNodeById: (id:string)=>nodes.find(n=>n.id===id) ?? null, + getNodesInFile: (file:string)=>nodes.filter(n=>n.filePath===file), + getNodesByName: (name:string)=>nodes.filter(n=>n.name===name), + getNodesByQualifiedName: (q:string)=>nodes.filter(n=>n.qualifiedName===q), + } as ResolutionContext; + return results.flatMap(({file,result})=>result.unresolvedReferences + .filter(r=>r.candidates?.some(c=>c.startsWith(VERILOG_CALL_ARGUMENT_PREFIX))) + .map(r=>{ + const ref={...r,filePath:file,language:'verilog'} as UnresolvedRef; + const access=resolveVerilogCallArgumentAccess(ref,context,matchVerilogMember); + return {name:r.referenceName.replace('hdl:signal:',''),access:access?.access, + callable:nodes.find(n=>n.id===access?.callableId)?.qualifiedName}; + })); +} + +it('classifies input, output, inout, ref and const ref actuals using known formal directions',()=>{ + expect(accesses({'top.sv':`module top; +task t(input int a, output int b, inout int c, ref int d, const ref int e); endtask +int ai,ao,aio,ar,acr; +initial t(ai,ao,aio,ar,acr); +endmodule`}).map(({name,access})=>[name,access])).toEqual([ + ['ai','read'],['ao','write'],['aio','readwrite'],['ar','readwrite'],['acr','read'], + ]); +}); + +it('supports omitted defaults and named actuals with default input direction',()=>{ + expect(accesses({'top.sv':`module top; +task t(int a=0, output int b); endtask +int out; +initial begin t(,out); t(.b(out)); end +endmodule`}).map(({name,access})=>[name,access])).toEqual([['out','write'],['out','write']]); +}); + +it('keeps output indexes as reads and never marks a non-lvalue output expression as a write',()=>{ + expect(accesses({'top.sv':`module top; +task t(output int x); endtask +int data[8],idx,a,b; +initial begin t(data[idx]); t(a+b); end +endmodule`}).map(({name,access})=>[name,access])).toEqual([ + ['data','write'],['idx','read'],['a',undefined],['b',undefined], + ]); +}); + +it('uses the nearest nested call and retains unknown-call occurrences without access claims',()=>{ + expect(accesses({'top.sv':`module top; +function int inner(input int x); return x; endfunction +task outer(input int x); endtask +int a,b; +initial begin outer(inner(a)); external_call(b); end +endmodule`}).map(({name,access,callable})=>[name,access,callable])).toEqual([ + ['a','read','top::inner'],['b',undefined,undefined], + ]); +}); + +it('respects lexical package imports and does not borrow a direction from a same-name task elsewhere',()=>{ + expect(accesses({'pkg.sv':'package p; task t(output int x); endtask endpackage', + 'top.sv':`module top; +task t(input int x); endtask +int a,b; +initial begin + begin : imported import p::*; t(a); end + begin : local_scope t(b); end +end +endmodule`}).map(({name,access,callable})=>[name,access,callable])).toEqual([ + ['a','write','p::t'],['b','read','top::t'], + ]); +}); + +it('rejects too many actuals and missing required formals while retaining their references',()=>{ + expect(accesses({'top.sv':`module top; +task t(input int a, output int b); endtask +int x,y,z; +initial begin t(x); t(x,y,z); end +endmodule`}).map(({name,access})=>[name,access])).toEqual([ + ['x',undefined],['x',undefined],['y',undefined],['z',undefined], + ]); +}); + +it('permits output concatenations but does not invent a reference-variable binding for a concat actual',()=>{ + expect(accesses({'top.sv':`module top; +task output_task(output int x); endtask +task reference_task(ref int x); endtask +int a,b,c,d; +initial begin output_task({a,b}); reference_task({c,d}); end +endmodule`}).map(({name,access})=>[name,access])).toEqual([ + ['a','write'],['b','write'],['c',undefined],['d',undefined], + ]); +}); diff --git a/__tests__/verilog-call-scope.test.ts b/__tests__/verilog-call-scope.test.ts new file mode 100644 index 0000000000..3e4f1247e8 --- /dev/null +++ b/__tests__/verilog-call-scope.test.ts @@ -0,0 +1,53 @@ +import { it, expect } from 'vitest'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { CodeGraph } from '../src'; + +it('retains callable owners and block imports when package definitions change through sync', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'hdl-call-scope-')); + let cg: CodeGraph | undefined; + const pkg = 'package p; function int run(); return 1; endfunction endpackage\npackage q; function int run(); return 2; endfunction endpackage'; + try { + fs.writeFileSync(path.join(root, 'pkg.sv'), pkg); + fs.writeFileSync(path.join(root, 'top.sv'), `module top; +function int f(); begin : chosen import p::*; f = run(); end endfunction +function int g(); begin : other import q::*; g = run(); end endfunction +initial begin : direct p::run(); end +endmodule`); + cg = CodeGraph.initSync(root); + expect((await cg.indexAll()).success).toBe(true); + const calls = () => (cg as any).db.db.prepare("SELECT s.qualified_name source,t.qualified_name target FROM edges e JOIN nodes s ON s.id=e.source JOIN nodes t ON t.id=e.target WHERE e.kind='calls' ORDER BY source,target").all(); + const before = calls(); + expect(before).toEqual([ + { source: 'top::f', target: 'p::run' }, + { source: 'top::g', target: 'q::run' }, + { source: 'top::initial@4:0', target: 'p::run' }, + ]); + fs.writeFileSync(path.join(root, 'pkg.sv'), '// move declarations\n' + pkg); + await cg.sync(); + expect(calls()).toEqual(before); + fs.writeFileSync(path.join(root, 'pkg.sv'), pkg.replace('int run()', 'int renamed()')); + await cg.sync(); + expect(calls()).toEqual([{ source: 'top::g', target: 'q::run' }]); + fs.writeFileSync(path.join(root, 'pkg.sv'), pkg); + await cg.sync(); + expect(calls()).toEqual(before); + await cg.indexAll(); + expect(calls()).toEqual(before); + } finally { cg?.close(); fs.rmSync(root, { recursive: true, force: true }); } +}); + +it.each(['import p::*;', 'function int run(); return 1; endfunction'])( + 'prefers an inner package import over outer declarations: %s', async outer => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'hdl-import-shadow-')); + let cg: CodeGraph | undefined; + try { + fs.writeFileSync(path.join(root, 'pkg.sv'), 'package p; function int run(); return 1; endfunction endpackage\npackage q; function int run(); return 2; endfunction endpackage'); + fs.writeFileSync(path.join(root, 'top.sv'), `module top; ${outer} initial begin : local_scope import q::*; $display(run()); end endmodule`); + cg = CodeGraph.initSync(root); + await cg.indexAll(); + const calls = (cg as any).db.db.prepare("SELECT t.qualified_name target FROM edges e JOIN nodes t ON t.id=e.target WHERE e.kind='calls'").all(); + expect(calls).toEqual([{ target: 'q::run' }]); + } finally { cg?.close(); fs.rmSync(root, { recursive: true, force: true }); } + }); diff --git a/__tests__/verilog-flow.test.ts b/__tests__/verilog-flow.test.ts new file mode 100644 index 0000000000..57563c14a2 --- /dev/null +++ b/__tests__/verilog-flow.test.ts @@ -0,0 +1,56 @@ +/** + * In an HDL the design hierarchy IS the flow: how `top` reaches `uart_rx` is + * `top` instantiating `uart_bridge` instantiating `uart_rx`. The flow finder + * rides `instantiates` edges between Verilog modules (indexed as `class`), + * and only there — a TS `new X()` stays a dependency, not a step. + */ + +import { describe, it, expect, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import CodeGraph from '../src/index'; +import { resolveNamedSymbolFlow } from '../src/graph/named-symbol-flow'; + +let tempDir: string; +let cg: CodeGraph | null = null; + +afterEach(() => { + cg?.close(); + cg = null; + fs.rmSync(tempDir, { recursive: true, force: true }); +}); + +async function index(files: Record): Promise { + tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-verilog-flow-')); + for (const [rel, content] of Object.entries(files)) fs.writeFileSync(path.join(tempDir, rel), content); + cg = await CodeGraph.init(tempDir, { index: true }); + cg.resolveReferences(); + return cg; +} + +describe('Verilog: the module-instantiation hierarchy is the flow', () => { + it('connects top to a leaf module through an unnamed bridge module', async () => { + const g = await index({ + 'top.v': 'module top(input clk, input rx, output tx);\n uart_bridge bridge (.clk(clk), .rx(rx), .tx(tx));\nendmodule\n', + 'uart_bridge.v': 'module uart_bridge(input clk, input rx, output tx);\n uart_rx rx_inst (.clk(clk), .rx(rx));\n uart_tx tx_inst (.clk(clk), .tx(tx));\nendmodule\n', + 'uart_rx.v': 'module uart_rx(input clk, input rx);\nendmodule\n', + 'uart_tx.v': 'module uart_tx(input clk, output tx);\nendmodule\n', + }); + const flow = resolveNamedSymbolFlow(g, 'top uart_rx'); + expect(flow.chains.length).toBeGreaterThan(0); + const steps = flow.chains[0]!.steps; + expect(steps.map((s) => s.node.name)).toEqual(['top', 'uart_bridge', 'uart_rx']); + expect(steps.slice(1).map((s) => s.edge?.kind)).toEqual(['instantiates', 'instantiates']); + }); + + it('does not ride a TypeScript `new X()` the same way', async () => { + const g = await index({ + 'top.ts': "import { Bridge } from './bridge';\nexport function top() { return new Bridge(); }\n", + 'bridge.ts': "import { Leaf } from './leaf';\nexport class Bridge { leaf = new Leaf(); }\n", + 'leaf.ts': 'export class Leaf {}\n', + }); + const flow = resolveNamedSymbolFlow(g, 'top Leaf'); + expect(flow.chains.map((c) => c.steps.map((s) => s.edge?.kind))).not.toContainEqual([undefined, 'instantiates', 'instantiates']); + }); +}); diff --git a/__tests__/verilog-generate-scopes.test.ts b/__tests__/verilog-generate-scopes.test.ts new file mode 100644 index 0000000000..0a934e949a --- /dev/null +++ b/__tests__/verilog-generate-scopes.test.ts @@ -0,0 +1,88 @@ +import { it, expect } from 'vitest'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { CodeGraph } from '../src'; + +async function graph(source: string, check: (cg: CodeGraph, root: string) => void | Promise) { + const root = fs.mkdtempSync(path.join(os.tmpdir(),'hdl-generate-')); + let cg: CodeGraph | undefined; + try { + fs.writeFileSync(path.join(root,'top.sv'),source); + cg = CodeGraph.initSync(root); expect((await cg.indexAll()).success).toBe(true); + await check(cg,root); + } finally { cg?.close(); fs.rmSync(root,{recursive:true,force:true}); } +} +function targets(cg: CodeGraph, name: string) { + const source = cg.getNodesByName(name).find(n => n.kind === 'variable')!; + return cg.getCallees(source.id).filter(e => e.edge.kind === 'references').map(e => e.node.qualifiedName).sort(); +} + +it('binds inline iteration templates in nested and sibling generate scopes instead of outer ports', async () => { + await graph(`module leaf(input i); endmodule +module top(input i); +for(genvar i=0;i<2;i++) begin: outer + leaf a(.i(i)); + for(genvar i=0;i<3;i++) begin: inner leaf b(.i); end +end +for(genvar i=0;i<4;i++) begin: sibling leaf c(i); end +leaf outside(i); +endmodule`, cg => { + expect(targets(cg,'a')).toEqual(['top::outer::i']); + expect(targets(cg,'b')).toEqual(['top::outer::inner::i']); + expect(targets(cg,'c')).toEqual(['top::sibling::i']); + expect(targets(cg,'outside')).toEqual(['top::i']); + const iterations = cg.getNodesByName('i').filter(n => n.decorators?.includes('hdl:generate-parameter')); + expect(iterations).toHaveLength(3); + expect(new Set(iterations.map(n => n.id)).size).toBe(3); + expect(iterations.every(n => n.kind === 'constant' && n.decorators?.includes('hdl:template'))).toBe(true); + }); +}); + +it('separates predeclared genvars from their per-loop templates and retains anonymous scope identities', async () => { + await graph(`module leaf(input p); endmodule +module top; +genvar i,j; +for(i=0;i<2;i++) leaf a(i); +for(i=0;i<3;i++) leaf b(i); +endmodule`, async cg => { + const declared = cg.getNodesByName('i').filter(n => n.decorators?.includes('hdl:genvar')); + expect(declared.map(n => n.qualifiedName)).toEqual(['top::i']); + const a = targets(cg,'a'), b = targets(cg,'b'); + expect(a[0]).toMatch(/^top::generate@4:\d+::i$/); + expect(b[0]).toMatch(/^top::generate@5:\d+::i$/); + expect(a).not.toEqual(b); + const before = cg.getNodesByName('i').map(n => n.id).sort(); + await cg.indexAll(); + expect(cg.getNodesByName('i').map(n => n.id).sort()).toEqual(before); + }); +}); + +it('keeps references on declarations and assignment expressions in the nearest generated template scope', async () => { + await graph(`module top(input i, output wire[3:0] out); +for(genvar i=0;i<4;i++) begin: lanes +wire local_sig = i; +assign out[i] = local_sig; +end +endmodule`, cg => { + const local = cg.getNodesByName('local_sig')[0]!; + expect(cg.getCallees(local.id).filter(e => e.edge.kind === 'references').map(e => e.node.qualifiedName)).toEqual(['top::lanes::i']); + const db=(cg as any).db.db; + const wrong = db.prepare("select count(*) n from edges e join nodes s on s.id=e.source join nodes t on t.id=e.target where e.kind='references' and s.qualified_name like 'top::lanes::%' and t.qualified_name='top::i'").get(); + expect(wrong.n).toBe(0); + }); +}); + +it('converges sync and clean indexing after a generate variable and scope rename', async () => { + const initial = 'module leaf(input p); endmodule module top; for(genvar i=0;i<2;i++) begin: lanes leaf u(i); end endmodule'; + await graph(initial, async (cg,root) => { + const changed = initial.replace(/genvar i=0;i<2;i\+\+/, 'genvar j=0;j<3;j++').replace('lanes leaf u(i)', 'channels leaf u(j)'); + fs.writeFileSync(path.join(root,'top.sv'),changed); + expect((await cg.sync()).filesModified).toBe(1); + expect(targets(cg,'u')).toEqual(['top::channels::j']); + const snapshot = () => (cg as any).db.db.prepare("select s.qualified_name source,t.qualified_name target,e.kind kind from edges e join nodes s on s.id=e.source join nodes t on t.id=e.target order by source,target,kind").all(); + const synced = snapshot(); + await cg.indexAll(); + expect(snapshot()).toEqual(synced); + }); +}); diff --git a/__tests__/verilog-grammar-version.test.ts b/__tests__/verilog-grammar-version.test.ts new file mode 100644 index 0000000000..861c8397f9 --- /dev/null +++ b/__tests__/verilog-grammar-version.test.ts @@ -0,0 +1,43 @@ +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; +import { Parser, Language } from 'web-tree-sitter'; +import path from 'node:path'; + +let parser: Parser; +let grammar: Language; +beforeAll(async () => { + await Parser.init(); + grammar = await Language.load(path.join(__dirname, '../src/extraction/wasm/tree-sitter-systemverilog.wasm')); + parser = new Parser(); + parser.setLanguage(grammar); +}); +afterAll(() => parser?.delete()); + +describe('SystemVerilog grammar capabilities', () => { + it('loads the shipped grammar with the supported ABI', () => { + expect(grammar.abiVersion).toBe(15); + }); + it.each([ + ['class timeunits', 'class C; timeunit 1ns; timeprecision 1ps; endclass', 'timeunits_declaration'], + ['generate inside expression', 'module m; parameter N=1; if (N inside {1,2}) begin: g wire x; end endmodule', 'inside_expression'], + ['keywords in macro arguments', 'module m; `M(input, output, always_ff) endmodule', 'text_macro_usage'], + ])('parses %s without recovery', (_name, source, expectedType) => { + const tree = parser.parse(source)!; + try { + expect(tree.rootNode.hasError).toBe(false); + expect(tree.rootNode.descendantsOfType(expectedType).length).toBeGreaterThan(0); + } finally { tree.delete(); } + }); + it('rejects whitespace between the backtick and a macro name', () => { + const tree = parser.parse('module m; ` M(x) endmodule')!; + try { expect(tree.rootNode.hasError).toBe(true); } + finally { tree.delete(); } + }); + it('represents replication multipliers as constant expressions', () => { + const tree = parser.parse('module m(input a, output [3:0] q); parameter N=4; assign q={N{a}}; endmodule')!; + try { + expect(tree.rootNode.hasError).toBe(false); + const replication = tree.rootNode.descendantsOfType('multiple_concatenation')[0]!; + expect(replication.namedChildren[0]?.type).toBe('constant_expression'); + } finally { tree.delete(); } + }); +}); diff --git a/__tests__/verilog-packages.test.ts b/__tests__/verilog-packages.test.ts new file mode 100644 index 0000000000..c62c6a6935 --- /dev/null +++ b/__tests__/verilog-packages.test.ts @@ -0,0 +1,61 @@ +import { describe, it, expect } from 'vitest'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { CodeGraph } from '../src'; + +async function graph(sources: Record, check: (db: any) => void) { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'hdl-packages-')); + let cg: CodeGraph | undefined; + try { + for (const [name, source] of Object.entries(sources)) fs.writeFileSync(path.join(dir, name), source); + cg = await CodeGraph.init(dir, { silent: true }); + await cg.indexAll(); + check((cg as any).db.db); + } finally { cg?.destroy(); fs.rmSync(dir, { recursive: true, force: true }); } +} +const functions = 'package a; function int convert(); return 1; endfunction endpackage\npackage b; function int convert(); return 2; endfunction endpackage'; +function calls(db: any) { return db.prepare("SELECT t.qualified_name target FROM edges e JOIN nodes t ON t.id=e.target WHERE e.kind='calls'").all().map((r: any) => r.target); } + +describe('SystemVerilog package and interface identity', () => { + it('keeps the package qualifier and does not match a same-named foreign function', async () => { + await graph({ 'pkg.sv': functions, 'top.sv': 'module top; initial begin b::convert(); missing::convert(); end endmodule' }, db => { + expect(calls(db)).toEqual(['b::convert']); + }); + }); + it('resolves a visible package import but never an unrelated global package member', async () => { + await graph({ 'pkg.sv': functions, 'top.sv': 'module top; import b::*; initial convert(); endmodule\nmodule other; initial convert(); endmodule' }, db => { + expect(calls(db)).toEqual(['b::convert']); + }); + }); + it('does not choose between conflicting package imports', async () => { + await graph({ 'pkg.sv': functions, 'top.sv': 'module top; import a::*, b::*; initial convert(); endmodule' }, db => { + expect(calls(db)).toEqual([]); + }); + }); + it('does not flatten hierarchical calls to an unrelated package function', async () => { + await graph({ 'pkg.sv': functions, 'top.sv': 'module top; initial obj.convert(); endmodule' }, db => { + expect(calls(db)).toEqual([]); + }); + }); + it('does not import compilation-unit functions from an unrelated file', async () => { + await graph({ 'foreign.sv': 'function int convert(); return 1; endfunction', + 'top.sv': 'module top; initial convert(); endmodule' }, db => expect(calls(db)).toEqual([])); + }); + it('keeps nested package calls inside argument expressions', async () => { + await graph({ 'pkg.sv': functions, 'top.sv': 'module top; initial b::convert(a::convert()); endmodule' }, db => { + expect(calls(db).sort()).toEqual(['a::convert', 'b::convert']); + }); + }); + it('exposes modport directions and resolves an interface port to its exact view', async () => { + await graph({ 'bus.sv': 'interface bus; logic valid, ready; modport Master(output valid, input ready), Slave(input valid, output ready); endinterface', + 'top.sv': 'module top(bus.Master axi); endmodule' }, db => { + const views = db.prepare("SELECT qualified_name name, signature FROM nodes WHERE signature LIKE 'modport %' ORDER BY name").all(); + expect(views).toEqual([{ name: 'bus::Master', signature: 'modport Master(output valid, input ready)' }, { name: 'bus::Slave', signature: 'modport Slave(input valid, output ready)' }]); + const types = db.prepare("SELECT s.name source, t.qualified_name target FROM edges e JOIN nodes s ON s.id=e.source JOIN nodes t ON t.id=e.target WHERE e.kind='type_of'").all(); + const signals = db.prepare("SELECT t.qualified_name target FROM edges e JOIN nodes s ON s.id=e.source JOIN nodes t ON t.id=e.target WHERE e.kind='references' AND s.qualified_name='bus::Master' ORDER BY target").all(); + expect(signals).toEqual([{ target: 'bus::ready' }, { target: 'bus::valid' }]); + expect(types).toContainEqual({ source: 'axi', target: 'bus::Master' }); + }); + }); +}); diff --git a/__tests__/verilog-param-deps.test.ts b/__tests__/verilog-param-deps.test.ts new file mode 100644 index 0000000000..310dacb0ef --- /dev/null +++ b/__tests__/verilog-param-deps.test.ts @@ -0,0 +1,64 @@ +import { afterEach, beforeAll, describe, expect, it } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import { extractFromSource } from '../src/extraction'; +import { initGrammars, loadAllGrammars } from '../src/extraction/grammars'; + +beforeAll(async () => { await initGrammars(); await loadAllGrammars(); }); + +describe('HDL parameter dependencies', () => { + let dir: string | undefined; + afterEach(() => { if (dir) fs.rmSync(dir, { recursive: true, force: true }); dir = undefined; }); + + it('a parameter references the identifiers of its value, never its own name or a callee', () => { + const code = `module m; + parameter N = 8; + localparam K = (N + 1) * 2; + localparam M = K << 1, W = $clog2(N) + width_of(K); + localparam P = pkg::BASE + N; +endmodule`; + const result = extractFromSource('m.sv', code); + const byName = Object.fromEntries(result.nodes.filter(n => n.kind === 'constant').map(n => [n.name, n])); + const refsFrom = (name: string) => result.unresolvedReferences + .filter(r => r.fromNodeId === byName[name]!.id && r.referenceName.startsWith('hdl:signal:')) + .map(r => r.referenceName.slice('hdl:signal:'.length)); + expect(refsFrom('N')).toEqual([]); + expect(refsFrom('K')).toEqual(['N']); + expect(refsFrom('M')).toEqual(['K']); + // $clog2 is a system task, width_of a callee: only their arguments count. + expect(refsFrom('W').sort()).toEqual(['K', 'N']); + // pkg::BASE is a package item, not evidence for a local named BASE. + expect(refsFrom('P')).toEqual(['N']); + }); + + it('impact on a parameter follows the derived-parameter chain, and stays inside its module', async () => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-hdl-params-')); + fs.writeFileSync(path.join(dir, 'a.sv'), `module a; + parameter N = 8; + localparam K = (N + 1) * 2; + localparam M = K << 1; +endmodule +module b; + parameter N = 4; + localparam Q = N * 3; +endmodule +`); + const cg = await CodeGraph.init(dir, { index: true }); + try { + const constant = (qualified: string) => cg.getNodesByKind('constant').find(n => n.qualifiedName === qualified)!; + const dependents = (qualified: string) => cg.getCallers(constant(qualified).id) + .filter(c => c.edge.kind === 'references').map(c => c.node.qualifiedName).sort(); + expect(dependents('a::N')).toEqual(['a::K']); + expect(dependents('a::K')).toEqual(['a::M']); + // Same-named parameter of another module: bound lexically, not by name. + expect(dependents('b::N')).toEqual(['b::Q']); + const impacted = [...cg.getImpactRadius(constant('a::N').id, 3).nodes.values()].map(n => n.qualifiedName); + expect(impacted).toEqual(expect.arrayContaining(['a::K', 'a::M'])); + expect(impacted).not.toContain('b::Q'); + } finally { + cg.close(); + } + }); +}); diff --git a/__tests__/verilog-port-order.test.ts b/__tests__/verilog-port-order.test.ts new file mode 100644 index 0000000000..da2ca372af --- /dev/null +++ b/__tests__/verilog-port-order.test.ts @@ -0,0 +1,47 @@ +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; +import { Parser, Language } from 'web-tree-sitter'; +import path from 'node:path'; +import { getVerilogPortOrder } from '../src/extraction/languages/verilog-signals'; + +let parser: Parser; +beforeAll(async () => { + await Parser.init(); + parser = new Parser(); + parser.setLanguage(await Language.load(path.join(__dirname, '../src/extraction/wasm/tree-sitter-systemverilog.wasm'))); +}); +afterAll(() => parser?.delete()); +function order(source: string): string[] | null { + const tree = parser.parse(source)!; + try { return getVerilogPortOrder(tree.rootNode.namedChildren[0]!, source); } + finally { tree.delete(); } +} + +describe('authoritative HDL positional port order', () => { + it('takes ANSI comma continuation ports in header source order, ignoring comments', () => { + expect(order('module m(input a, /* comment `ifdef FALSE */ b, output [3:0] c); endmodule')).toEqual(['a', 'b', 'c']); + }); + it('uses non-ANSI header order even when body declarations are reversed', () => { + expect(order('module m(a,b,c); input c; output b; input a; endmodule')).toEqual(['a', 'b', 'c']); + }); + it('preserves escaped identifiers and supports empty headers', () => { + expect(order('module m(input \\x.y , output z); endmodule')).toEqual(['\\x.y', 'z']); + expect(order('module m(\\x.y ,z); input z; output \\x.y ; endmodule')).toEqual(['\\x.y', 'z']); + expect(order('module m(); endmodule')).toEqual([]); + expect(order('module m; endmodule')).toEqual([]); + }); + it.each([ + 'module m(a,,b); input a,b; endmodule', + 'module m({a,b},c); input a,b,c; endmodule', + 'module m(.a(x),b); input x,b; endmodule', + 'module m(input a, a); endmodule', + 'module m(input a, \\a ); endmodule', + 'module m(input a, `ifdef TEST input b, `endif output c); endmodule', + 'module m(`PORTS); endmodule', + 'module m(input a); assign = ; endmodule', + ])('does not infer order from ambiguous or incomplete syntax: %s', source => { + expect(order(source)).toBeNull(); + }); + it('accepts interfaces and ignores parameter declarations in port numbering', () => { + expect(order('interface bus #(parameter W=8)(input clk, input [W-1:0] data); endinterface')).toEqual(['clk', 'data']); + }); +}); diff --git a/__tests__/verilog-resolution.test.ts b/__tests__/verilog-resolution.test.ts new file mode 100644 index 0000000000..d9a5d9f0ce --- /dev/null +++ b/__tests__/verilog-resolution.test.ts @@ -0,0 +1,87 @@ +/** + * Verilog module resolution across files. + * + * A module instantiation names its target by bare name — no import pins + * `pll u_pll (...)` to a file, the tool's file list does. A design commonly + * keeps a simulation stand-in beside the real module (`src/pll.v` and + * `sim/pll_stub.v`, both `module pll`); a synthesis file must land on the real + * one whatever the directory layout, while a testbench may mean either. And a + * `` `include "defs.vh" `` must become an edge to the header file. + */ +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { CodeGraph } from '../src'; + +type Edge = { kind: string; sn: string; sf: string; tn: string; tf: string }; + +async function edges(dir: string): Promise { + const cg = await CodeGraph.init(dir, { silent: true }); + await cg.indexAll(); + const db = (cg as any).db.db; + const rows = db + .prepare( + `SELECT e.kind kind, s.name sn, s.file_path sf, t.name tn, t.file_path tf + FROM edges e JOIN nodes s ON s.id = e.source JOIN nodes t ON t.id = e.target + WHERE e.kind IN ('instantiates', 'imports') AND s.language = 'verilog'` + ) + .all(); + cg.destroy(); + return rows; +} + +function write(dir: string, rel: string, body: string): void { + fs.mkdirSync(path.dirname(path.join(dir, rel)), { recursive: true }); + fs.writeFileSync(path.join(dir, rel), body); +} + +const PLL = 'module pll (input clk, output lock);\nendmodule\n'; +const TOP = 'module top (input clk, output lock);\n pll u_pll (.clk(clk), .lock(lock));\nendmodule\n'; + +describe('verilog module resolution', () => { + let dir: string; + beforeEach(() => { dir = fs.mkdtempSync(path.join(os.tmpdir(), 'verilog-res-')); }); + afterEach(() => { fs.rmSync(dir, { recursive: true, force: true }); }); + + it('a synthesis file skips the simulation stub even when no directory is closer', async () => { + // `top.v` at the root shares no directory with either candidate, so + // proximity cannot break the tie — the simulation path must. + write(dir, 'top.v', TOP); + write(dir, 'rtl/pll.v', PLL); + write(dir, 'sim/pll_stub.v', PLL); + const inst = (await edges(dir)).filter((e) => e.kind === 'instantiates'); + expect(inst.map(e => [e.sn, e.tn, e.tf]).sort()).toEqual([['top', 'pll', 'rtl/pll.v'], ['u_pll', 'pll', 'rtl/pll.v']]); + }); + + it('a stub named by file suffix is skipped too', async () => { + write(dir, 'top.v', TOP); + write(dir, 'rtl/pll.v', PLL); + write(dir, 'models/pll_stub.v', PLL); + const inst = (await edges(dir)).filter((e) => e.kind === 'instantiates'); + expect(inst.map((e) => e.tf)).toEqual(['rtl/pll.v', 'rtl/pll.v']); + }); + + it('a testbench keeps every candidate and lands on the stub beside it', async () => { + write(dir, 'src/pll.v', PLL); + write(dir, 'sim/pll_stub.v', PLL); + write(dir, 'sim/tb_top.v', 'module tb_top;\n reg clk; wire lock;\n pll dut (.clk(clk), .lock(lock));\nendmodule\n'); + const inst = (await edges(dir)).filter((e) => e.kind === 'instantiates'); + expect(inst.map((e) => e.tf)).toEqual(['sim/pll_stub.v', 'sim/pll_stub.v']); + }); + + it('`include resolves to the header file, by suffix and closest to the includer', async () => { + write(dir, 'src/defs.vh', '`define DATA_W 16\n'); + write(dir, 'include/axi/typedef.svh', '`define AXI_TYPEDEF(x) x\n'); + write( + dir, + 'src/top.v', + '`include "defs.vh"\n`include "axi/typedef.svh"\nmodule top (input clk);\nendmodule\n' + ); + const imports = (await edges(dir)).filter((e) => e.kind === 'imports'); + expect(imports.map((e) => [e.sf, e.tf]).sort()).toEqual([ + ['src/top.v', 'include/axi/typedef.svh'], + ['src/top.v', 'src/defs.vh'], + ]); + }); +}); diff --git a/__tests__/verilog-scopes.test.ts b/__tests__/verilog-scopes.test.ts new file mode 100644 index 0000000000..67e882ffb5 --- /dev/null +++ b/__tests__/verilog-scopes.test.ts @@ -0,0 +1,84 @@ +import { afterEach, beforeEach, expect, it } from 'vitest'; +import fs from 'node:fs'; import os from 'node:os'; import path from 'node:path'; +import { CodeGraph } from '../src'; +let root:string; let cg:CodeGraph|undefined; +beforeEach(()=>{root=fs.mkdtempSync(path.join(os.tmpdir(),'hdl-scopes-'));}); +afterEach(()=>{cg?.close();cg=undefined;fs.rmSync(root,{recursive:true,force:true});}); +const rows=()=> (cg as any).db.db.prepare(`SELECT s.qualified_name source,t.qualified_name target,e.line line FROM edges e JOIN nodes s ON s.id=e.source JOIN nodes t ON t.id=e.target WHERE e.kind='references' AND json_extract(e.metadata,'$.refName') LIKE 'hdl:signal:%' ORDER BY source,target,line`).all() as {source:string,target:string,line:number}[]; +const code=`module crc(input clk, input feedback, output logic [15:0] crc); +always @(posedge clk) begin : update + crc[0] <= feedback; + begin : inner + logic feedback; + feedback = crc[15]; + crc[1] <= feedback; + end + begin : sibling + crc[2] <= feedback; + end +end +endmodule`; +it('binds nested shadows in their own scope and keeps outer/sibling uses',async()=>{ + fs.writeFileSync(path.join(root,'crc.sv'),code);cg=CodeGraph.initSync(root);await cg.indexAll(); + const refs=rows().filter(r=>r.target.endsWith('::feedback')); + expect(refs.filter(r=>r.line===3).map(r=>r.target)).toEqual(['crc::feedback']); + expect(refs.filter(r=>r.line===7).map(r=>r.target)).toEqual(['crc::always@2:0::update::inner::feedback']); + expect(refs.filter(r=>r.line===10).map(r=>r.target)).toEqual(['crc::feedback']); +}); +it('keeps anonymous siblings and for/foreach bindings distinct',async()=>{ + fs.writeFileSync(path.join(root,'uart.sv'),`module uart(input i, input j, input data); logic [7:0] bits; +initial begin + begin integer count; count=1; end + begin integer count; count=2; end + for(int i=0;i<8;i++) bits[i]=data; + foreach(bits[j]) bits[j]=data; + bits[i]=data; +end endmodule`);cg=CodeGraph.initSync(root);await cg.indexAll(); + const counts=cg.getNodesByName('count');expect(counts).toHaveLength(2);expect(new Set(counts.map(n=>n.qualifiedName)).size).toBe(2); + const refs=rows();expect(refs.filter(r=>r.line===5&&r.target.endsWith('::i')).every(r=>r.target.includes('::for@'))).toBe(true); + expect(refs.filter(r=>r.line===5&&r.target.endsWith('::i')).length).toBeGreaterThan(0); + expect(refs.filter(r=>r.line===6&&r.target.endsWith('::j')).every(r=>r.target.includes('::foreach@'))).toBe(true); + expect(refs.filter(r=>r.line===7&&r.target.endsWith('::i')).map(r=>r.target)).toEqual(['uart::i']); +}); +it('rebinds a removed local shadow identically on sync and full rebuild',async()=>{ + const file=path.join(root,'crc.sv');fs.writeFileSync(file,code);cg=CodeGraph.initSync(root);await cg.indexAll(); + fs.writeFileSync(file,code.replace(' logic feedback;',' // feedback now belongs to module')); + await cg.sync();const synced=rows();await cg.indexAll();expect(rows()).toEqual(synced); + expect(synced.filter(r=>r.line===7&&r.target.endsWith('::feedback')).map(r=>r.target)).toEqual(['crc::feedback']); +}); +it('models function/task formal ports and preserves function call owners inside blocks',async()=>{ + fs.writeFileSync(path.join(root,'functions.sv'),`module m(input x); +function int g(input int x); return x; endfunction +function int f(input int x); + begin : work + f = g(x); + end +endfunction +task t; input x; begin x=x; end endtask +endmodule`);cg=CodeGraph.initSync(root);await cg.indexAll(); + const refs=rows();expect(refs.filter(r=>r.line===5&&r.target.endsWith('::x')).map(r=>r.target)).toEqual(['m::f::x']); + expect(refs.filter(r=>r.line===8&&r.target.endsWith('::x')).every(r=>r.target==='m::t::x')).toBe(true); + expect(refs.filter(r=>r.line===8&&r.target.endsWith('::x')).length).toBeGreaterThan(0); + const f=cg.getNodesByName('f').find(n=>n.kind==='function')!; + expect(cg.getCallees(f.id).filter(e=>e.edge.kind==='calls').map(e=>e.node.qualifiedName)).toContain('m::g'); +}); +it('keeps calls and signal dependencies in formal defaults and loop initializers',async()=>{ + fs.writeFileSync(path.join(root,'defaults.sv'),`module m(input seed); +function int g(); return 1; endfunction +function int f(input int x=g(), input int y=seed); return x+y; endfunction +initial for(int i=g();i<3;i++) $display(i); +endmodule`);cg=CodeGraph.initSync(root);await cg.indexAll(); + const f=cg.getNodesByName('f').find(n=>n.kind==='function')!; + const initial=cg.getNodesByName('initial@4:0')[0]!; + const callees=(id:string)=>cg!.getCallees(id).filter(e=>e.edge.kind==='calls').map(e=>e.node.qualifiedName); + expect(callees(f.id)).toEqual(['m::g']);expect(callees(initial.id)).toEqual(['m::g']); + expect(rows().filter(r=>r.source==='m::f::y').map(r=>r.target)).toEqual(['m::seed']); +}); +it('does not invent outer bindings from a loop with recovered declaration syntax',async()=>{ + fs.writeFileSync(path.join(root,'recovery.sv'),`module m(input j); +initial for(int i=0,j=0;i<3;i++) $display(j); +initial $display(j); +endmodule`);cg=CodeGraph.initSync(root);await cg.indexAll(); + expect(rows().filter(r=>r.line===2&&r.target==='m::j')).toEqual([]); + expect(rows().filter(r=>r.line===3&&r.target==='m::j').length).toBeGreaterThan(0); +}); diff --git a/__tests__/verilog-signals.test.ts b/__tests__/verilog-signals.test.ts new file mode 100644 index 0000000000..58076e3fe1 --- /dev/null +++ b/__tests__/verilog-signals.test.ts @@ -0,0 +1,148 @@ +import { beforeAll, describe, expect, it } from 'vitest'; +import { extractFromSource } from '../src/extraction'; +import { initGrammars, loadAllGrammars } from '../src/extraction/grammars'; + +beforeAll(async () => { await initGrammars(); await loadAllGrammars(); }); + +describe('HDL signals and procedural source navigation', () => { + it('keeps ANSI direction, non-ANSI declarations and exact source signatures', () => { + const code = `module top(input logic clk, rst, output reg [3:0] q); +wire a,b=1; logic [7:0] x,y; +endmodule +module old(a,b); input [3:0] a; output reg b; endmodule`; + const result = extractFromSource('ports.sv', code); + const fields = result.nodes.filter(n => n.kind === 'field'); + expect(fields.map(n => n.qualifiedName)).toEqual(['top::clk', 'top::rst', 'top::q', 'top::a', 'top::b', 'top::x', 'top::y', 'old::a', 'old::b']); + expect(fields.find(n => n.name === 'rst')?.decorators).toContain('hdl:input'); + expect(fields.find(n => n.name === 'q')?.signature).toBe('output reg [3:0] q'); + expect(fields.find(n => n.name === 'x')?.signature).toBe('logic [7:0] x,y;'); + expect(fields.find(n => n.qualifiedName === 'old::a')?.signature).toBe('input [3:0] a'); + }); + + it('indexes separate same-line processes and records only local signal candidates', () => { + const code = `module top(input logic clk, output logic q); +logic x; +always_ff @(posedge clk) q <= helper(x); always_comb x = q; +initial x=0; assign q = x; +endmodule`; + const result = extractFromSource('logic.sv', code); + const processes = result.nodes.filter(n => n.decorators?.includes('hdl:process')); + expect(processes.map(n => n.name.split('@')[0])).toEqual(['always_ff', 'always_comb', 'initial', 'assign']); + expect(new Set(processes.map(n => n.id)).size).toBe(4); + expect(processes[0].signature).toBe('always_ff @(posedge clk) q <= helper(x);'); + const signals = result.unresolvedReferences.filter(r => r.referenceName.startsWith('hdl:signal:')); + expect(signals.some(r => r.referenceName === 'hdl:signal:helper')).toBe(false); + expect(signals.filter(r => r.fromNodeId === processes[0].id).map(r => r.referenceName)).toEqual(['hdl:signal:clk', 'hdl:signal:q', 'hdl:signal:x']); + }); + + it('does not inherit a scalar output direction onto an interface port', () => { + const result = extractFromSource('interface.sv', 'module top(output q, bus.master link); endmodule'); + const link = result.nodes.find(n => n.name === 'link')!; + expect(link.decorators).toContain('hdl:interface:bus'); + expect(link.decorators).not.toContain('hdl:output'); + expect(result.unresolvedReferences.find(r => r.fromNodeId === link.id)?.referenceName).toBe('bus.master'); + }); + + it('does not bind a block-local shadow to a module signal', () => { + const result = extractFromSource('shadow.sv', 'module top; logic x,y; initial begin logic x; x = y; end endmodule'); + const local = result.nodes.find(n => n.name === 'x' && n.qualifiedName.includes('::block@'))!; + const refs = result.unresolvedReferences.filter(r => r.referenceName.startsWith('hdl:signal:')); + expect(refs.map(r => r.referenceName)).toEqual(['hdl:signal:x', 'hdl:signal:y']); + expect(result.nodes.find(n => n.id === refs[0].fromNodeId)?.qualifiedName).toBe(local.qualifiedName.split('::').slice(0, -1).join('::')); + }); + + it('merges non-ANSI reg redeclarations and indexes initializer dependencies', () => { + const result = extractFromSource('legacy.sv', 'module top(a); output a; reg a; wire x = a; endmodule'); + expect(result.nodes.filter(n => n.qualifiedName === 'top::a')).toHaveLength(1); + const x = result.nodes.find(n => n.name === 'x')!; + expect(result.unresolvedReferences.filter(r => r.fromNodeId === x.id).map(r => r.referenceName)).toContain('hdl:signal:a'); + }); + + it('does not confuse generate iteration bindings with outer ports', () => { + const result = extractFromSource('generate.sv', `module top(input i, input x); +for(genvar i=0;i<2;i++) begin : lanes leaf u(.a(i), .b(x)); leaf short(.i); end +leaf outside(.a(i)); endmodule`); + const inside = result.nodes.find(n => n.name === 'u')!; + const outside = result.nodes.find(n => n.name === 'outside')!; + const shorthand = result.nodes.find(n => n.name === 'short')!; + expect(result.nodes.find(n => n.qualifiedName === 'top::lanes::i')?.decorators).toContain('hdl:generate-parameter'); + expect(result.unresolvedReferences.filter(r => r.fromNodeId === shorthand.id).map(r => r.referenceName)).toContain('hdl:signal:i'); + expect(result.unresolvedReferences.filter(r => r.fromNodeId === inside.id && r.referenceName.startsWith('hdl:signal:')).map(r => r.referenceName)).toEqual(['hdl:signal:i', 'hdl:signal:x']); + expect(result.unresolvedReferences.filter(r => r.fromNodeId === outside.id).map(r => r.referenceName)).toContain('hdl:signal:i'); + }); + + it('skips package qualifiers while preserving method-call argument signals', () => { + const result = extractFromSource('package-call.sv', `module top(input p, input x, output integer q); +initial q=p::value(x); endmodule`); + expect(result.unresolvedReferences.filter(r => r.referenceName.startsWith('hdl:signal:')).map(r => r.referenceName)).toEqual(['hdl:signal:q', 'hdl:signal:x']); + }); + + it('records bit/part-select bases, bounds and concatenated assignment targets', () => { + const result = extractFromSource('selects.sv', `module top; +assign x[3] = y; +assign x[i+:W] = y; +assign x[msb:lsb] = y; +assign {a, b[j]} = v; +endmodule`); + const refs = (line: number) => result.unresolvedReferences.filter(r => r.line === line && r.referenceName.startsWith('hdl:signal:')).map(r => r.referenceName); + expect(refs(2)).toEqual(['hdl:signal:x', 'hdl:signal:y']); + expect(refs(3)).toEqual(['hdl:signal:x', 'hdl:signal:i', 'hdl:signal:W', 'hdl:signal:y']); + expect(refs(4)).toEqual(['hdl:signal:x', 'hdl:signal:msb', 'hdl:signal:lsb', 'hdl:signal:y']); + expect(refs(5)).toEqual(['hdl:signal:a', 'hdl:signal:b', 'hdl:signal:j', 'hdl:signal:v']); + }); + + it('keeps index expressions but never flattens hierarchical LHS members', () => { + const result = extractFromSource('member-select.sv', `module top; +assign remote.x[i] = y; +assign {a, remote.x[j]} = v; +assign x[bound(i)+:pkg::W] = y; +assign pkg::x[i] = y; +assign x[i].field[j] = y; +assign $root.top.x[i] = y; +endmodule`); + const refs = (line: number) => result.unresolvedReferences.filter(r => r.line === line && r.referenceName.startsWith('hdl:signal:')).map(r => r.referenceName); + expect(refs(2)).toEqual(['hdl:signal:i', 'hdl:signal:y']); + expect(refs(3)).toEqual(['hdl:signal:a', 'hdl:signal:j', 'hdl:signal:v']); + expect(refs(4)).toEqual(['hdl:signal:x', 'hdl:signal:i', 'hdl:signal:y']); + expect(refs(5)).toEqual(['hdl:signal:i', 'hdl:signal:y']); + expect(refs(6)).toEqual(['hdl:signal:i', 'hdl:signal:j', 'hdl:signal:y']); + expect(refs(7)).toEqual(['hdl:signal:i', 'hdl:signal:y']); + }); + + it('does not bind procedural for/foreach variables to matching module ports', () => { + const result = extractFromSource('procedural-loop.sv', `module top(input i, input j, input y); +logic [7:0] x; +initial for (int i=0;i<4;i++) x[i]=y; +initial foreach (x[j]) x[j]=y; +endmodule`); + const refs = result.unresolvedReferences.filter(r => r.referenceName.startsWith('hdl:signal:')).map(r => r.referenceName); + expect(result.nodes.some(n => n.name === 'i' && n.qualifiedName.includes('::for@'))).toBe(true); + expect(result.nodes.some(n => n.name === 'j' && n.qualifiedName.includes('::foreach@'))).toBe(true); + expect(refs).toContain('hdl:signal:i'); + expect(refs).toContain('hdl:signal:j'); + expect(refs).toContain('hdl:signal:x'); + expect(refs).toContain('hdl:signal:y'); + }); + + it('does not bind qualified procedural targets to same-named local ports', () => { + const result = extractFromSource('procedural-package.sv', `package p; logic x; endpackage +module top(input a, input x, input i, input j, output [7:0] y); +always_comb p::x = a; +always_comb p::x[i] = helper(a); +always_comb {p::x[i], y[j]} = a; +always_comb remote.x[i] = a; +assign y = p::x[i]; +endmodule`); + const refs = (line: number) => result.unresolvedReferences.filter(r => r.line === line && r.referenceName.startsWith('hdl:signal:')).map(r => r.referenceName); + expect(refs(3)).toEqual(['hdl:signal:a']); + expect(refs(4)).toEqual(['hdl:signal:i', 'hdl:signal:a']); + expect(refs(5)).toEqual(['hdl:signal:i', 'hdl:signal:y', 'hdl:signal:j', 'hdl:signal:a']); + expect(refs(6)).toEqual(['hdl:signal:i', 'hdl:signal:a']); + expect(refs(7)).toEqual(['hdl:signal:y', 'hdl:signal:i']); + }); + + it('does not flatten hierarchical members into local signal references', () => { + const result = extractFromSource('scope.sv', 'module top; wire x,y; assign x = remote.y; endmodule'); + expect(result.unresolvedReferences.filter(r => r.referenceName.startsWith('hdl:signal:')).map(r => r.referenceName)).toEqual(['hdl:signal:x']); + }); +}); diff --git a/docs/hdl-macro-source-map.md b/docs/hdl-macro-source-map.md new file mode 100644 index 0000000000..3cd426bfc8 --- /dev/null +++ b/docs/hdl-macro-source-map.md @@ -0,0 +1,101 @@ +# HDL macro provenance through pyslang + +An optional backend reports where a macro-generated declaration was spelled and +the ranges of the invocations that produced it. It uses the `SourceManager` of +`pyslang==11.0.0`; it does not search macro definitions with regular +expressions and does not guess unknown columns. + +## Running it + +You need a separate Python >= 3.11 with pyslang 11.0.0 installed (validated with +Python 3.14). CodeGraph does not install it and does not touch the project's +Python dependencies. From the root of an initialized project with an active HDL +profile: + +```sh +codegraph hdl-semantic data --python /absolute/path/to/venv/bin/python +``` + +Through the API: + +```ts +const result = await cg.getHdlSemantics({ + pythonExecutable: '/absolute/path/to/venv/bin/python', + query: 'top.data', +}); +``` + +Choose exactly one frontend: `--python` / `pythonExecutable`, or `--slang` / +`executable`. The Python backend performs the compilation and the fact export +itself; results of different compiler runs are never merged. The active +profile, the overrides, separate compilation units, the limits and the snapshot +checks all apply as with slang. + +## What is returned + +Names of `parameter` / `port` / `typedef` declarations keep their `sourceOrigin` +and `macroExpansion`. A macro inside an initializer or a type does not turn the +name itself into a macro origin: a link to the direct source declaration stays +valid. + +In addition, `expressionOrigins` lists the unique compiler macro-token origins +with the roles `initializer`, `declared-initializer` and `type`. For `type` the +available declared type syntax is checked, including supported dimensions. The +effective initializer is not mixed with the declaration's default after an +override. These are the direct macro tokens of the corresponding syntax, not a +transitive analysis of every constant the result depends on. + +`expressionOriginCoverage` describes `initializer`, `declaredInitializer` and +`type` separately as `checked`, `not-applicable` or `unavailable`; an +initializer may also be `command-line`. An empty list under `unavailable` does +not prove the absence of macros. `truncated=true` means the output limit was +reached. At most 32 expression origins are returned per fact; whether the +individual macro frames are complete is stated by `macroExpansionComplete`. + +- `provenance.frontend = pyslang`, the protocol/exporter version and the hashes + of Python, the exporter and the native pyslang module. Python runs with `-I`; + the venv path is recorded so that another interpreter's site-packages are + never picked up. +- `sourceOrigin`: `direct` or `macro`. `source` is the physical point in the + source or the available invocation point, not automatically the declaration's + full range. +- `macroExpansion`: a bounded set of provenance frames. Each carries the macro + name, whether it is an argument, the spelling point and the compiler's + invocation range, when the frontend could map them. +- Every point has a project-relative file, physical line/column and + `byteOffset`. Columns count from 1 in UTF-8 bytes, the offset from 0; this is + not CodeGraph's UTF-16 column. Coordinates are verified against the snapshot + bytes. +- `macroExpansionComplete` states explicitly whether every frame could be + mapped. Collapsed or unknown ranges make it `false`. A missing part is not + reconstructed heuristically. + +Frames are not a linear stack. `SourceManager` keeps separate original and +expansion links; nested macros need both. A spelling point inside an argument +or a macro body does not mean the whole resulting name is written there +contiguously — in particular with token concatenation: the point of `bus` is +not the range of the name `bus_data`. + +Macro-origin facts carry no `sourceNodeId`, even with an exact spelling column: +it may point at a call argument or a definition body rather than the port's +own declaration. Navigation goes through `macroExpansion`. Direct facts keep +their verified source node links. + +## Verifiability and limits + +The exporter is a small shipped Python helper (`dist/hdl/source-map-export.py`) +that uses the compiler API rather than its own HDL preprocessor. The normalized +JSON carries its own `codegraphSemanticVersion: 1` and is validated before any +fact is emitted. A compiler error, an unknown library version, a corrupt +payload, a source path outside the snapshot, inconsistent coordinates or inputs +that changed during the run are all rejected. + +Computed include paths, files outside the root and the other snapshot-runner +limits still apply. This is not an OS sandbox for running an untrusted +compiler. There is no persistent cache, no transitive dependency map of all +expressions, no complete source map for every constructed token and no +automatic MCP compilation. The Windows backend is disabled for now; the macOS +runtime has not been validated. + +`SourceManager` behaviour follows the +[official API](https://www.sv-lang.com/classslang_1_1_source_manager.html). diff --git a/docs/hdl-profiles.md b/docs/hdl-profiles.md new file mode 100644 index 0000000000..b636689c12 --- /dev/null +++ b/docs/hdl-profiles.md @@ -0,0 +1,96 @@ +# HDL build profiles + +A profile selects the HDL source units and the conditional-compilation context +they are indexed under. One index holds one active profile; the project's other +languages keep their ordinary scope. + +## Configuration + +```json +{ + "hdl": { + "activeProfile": "synth", + "profiles": { + "synth": { + "filelists": ["profiles/synth.f"], + "includeDirs": ["rtl/include"], + "defines": {"SYNTHESIS": "1"}, + "topModules": ["top"], + "languageMode": "systemverilog" + }, + "sim": { + "files": ["rtl/top.sv", "tb/top_tb.sv"], + "includeDirs": ["rtl/include"], + "defines": {}, + "topModules": ["top_tb"] + } + } + } +} +``` + +The file is `codegraph.json` at the project root. `files` and `filelists` are +combined; when both are empty or missing the HDL selection is empty, not "every +file". Explicitly listed HDL sources may live in gitignored or generated +directories. Paths must stay inside the project; VCS directories and CodeGraph's +own data directory are excluded. + +A filelist supports plain paths, quotes and comments, `+incdir+` and `+define+`, +and nested `-f` / `-F`. For `.f` / `-f` relative paths are resolved from the +project root; for `.F` / `-F` from the directory of the containing list. Unknown +flags, cycles and conflicting define values are rejected rather than guessed. +Define values are strings. The current preprocessor only uses whether a macro is +defined; a value of `"0"` still means defined. + +## Working with the index + +For an already initialized project, after adding a profile: + +```sh +codegraph index +codegraph status --json +``` + +To switch `hdl.activeProfile`, change it and run `codegraph sync`. A running +watcher observes the configuration, the filelists and the include dependencies, +including headers with unusual extensions and explicitly selected ignored paths. +A context change re-evaluates the whole HDL selection even during a scoped sync; +a partial `indexFiles` cannot switch context without a full sync or index. + +The CLI and MCP report the configured and the indexed profile separately, with +fingerprint, diagnostics and limitations. An include change is detected by +content, with the recorded size and mtime as a fast check. The fingerprint covers +the selected configuration, the filelists, include snapshots and the extraction +version. With an invalid profile, indexing stops before the graph is modified. + +Units are parsed from a prepared snapshot; stored hashes refer to the original +bytes, not to the masked text. Edits made during a pass stay pending for the +next sync. Partial operations only update the diagnostics of the units they +actually processed. + +## Supported preprocessing and limits + +- `ifdef` / `ifndef` / `elsif` / `else` / `endif` and `define` / `undef` / + `undefineall`. +- An include contributes macro state and a dependency. It does not splice the + header's contents into the source unit; a header can be listed explicitly as + its own source. +- Inactive text is replaced by spaces, preserving UTF-16 offsets and CR/LF. The + snippets shown remain the original source and may contain inactive branches; + the graph reflects the selected profile. +- There is no full macro expansion. An opaque macro invocation may change the + defines: a later conditional branch that cannot be decided is not selected but + diagnosed. Unsupported constructs and an incomplete state are visible in + `status`. +- Every listed source unit starts from the profile's defines; file order is not + used as an implicit shared macro environment across compilation units. +- `languageMode` records the intended dialect; the parser stays SystemVerilog + and does not prove Verilog-only conformance. `topModules` are recorded roots; + no reachability pruning or elaboration is performed. +- Snapshot limits: 1 MiB per source or header, 64 MiB in total; the loader also + bounds filelist size and nesting depth. Diagnostics are capped in volume + without stopping macro-state processing. + +Full include/macro expansion with compiler source mapping is what the separate +[`hdl-semantic`](hdl-semantics.md) command provides. A profile does not prove +synthesis, simulation, width compatibility, timing or CDC. diff --git a/docs/hdl-semantics.md b/docs/hdl-semantics.md new file mode 100644 index 0000000000..7fd403e214 --- /dev/null +++ b/docs/hdl-semantics.md @@ -0,0 +1,89 @@ +# Computed HDL parameters and widths through slang + +The `hdl-semantic` command and the `CodeGraph.getHdlSemantics()` API run an +installed [slang](https://github.com/MikePopoloski/slang) on explicit request. +They need an initialized project and an active HDL build profile in +`codegraph.json`. Facts are returned separately from the source graph; graph +data is never overwritten. slang is not installed automatically. For compiler +macro provenance a [separate pyslang backend](hdl-macro-source-map.md) is +available through `--python`; choose exactly one frontend executable. + +## CLI + +```sh +codegraph hdl-semantic top.W --path /path/to/project --slang /path/to/slang +codegraph hdl-semantic crc_out --path /path/to/project --slang /path/to/slang +codegraph hdl-semantic NoSlvMst --path /path/to/project --slang /path/to/slang \ + --top synth_axi_lite_xbar --parameter NoSlvMst=4 +``` + +The answer is JSON with facts and provenance. The query selects an exact name, +an instance path or `instancePath.name`; without a query the first bounded +selection is returned. `--limit` sets 1..1000 facts (default 100); +`totalMatches` / `truncated` report the cut. For several overrides list +`NAME=VALUE` pairs after `--parameter`. Names and values are passed as separate +arguments, never through a shell. + +If slang rejects a design because of use-before-declare, compatibility can be +enabled explicitly with `--allow-use-before-declare`. It is part of the +provenance and fingerprint and is never switched on automatically after a +compiler error. + +## API + +```ts +const result = await cg.getHdlSemantics({ + executable: '/path/to/slang', + top: 'top', + parameters: { W: '16' }, + query: 'top.data', + limit: 20, + signal: abortController.signal, +}); +``` + +Parameters are returned as strings of evaluated values so that large integers +and unknown bits are not lost. A port width is returned only for types the +importer can compute reliably; a missing width does not mean 0. +Instance-specific facts are not merged across instances of one module. + +Every answer records the slang version, the executable hash, the active profile, +the configuration and semantic fingerprints, the top, the overrides, the language +standard, the separate-compilation-unit mode, the runner version, the compiler +limits and the compatibility mode. `sourceNodeId` is added only when the profile +matches, the file's raw hash matches and exactly one source declaration matches +by name and line. With a stale or different source graph the computed facts stay +available without guessed node links. + +A source location keeps the frontend's coordinates; it is not a promise of +CodeGraph's UTF-16 offsets. A `null` column means the column is unknown, for +example for a macro-generated declaration. A macro invocation line is not +presented as an exact expanded token range. + +## Limits of the first implementation + +- Computation is on demand: no persistent semantic cache and no automatic run + during index, sync or explore. The next query uses a fresh snapshot. +- The supported slang contract is checked by the runner and importer; an + unknown version or AST shape is rejected. An AST written alongside a compiler + error is not accepted. +- Snapshot: 1 MiB per input, 64 MiB in total; the AST at most 64 MiB, stdout + and stderr 1 MiB together. Compiler limits for hierarchy, generate and + constant evaluation are fixed; this is not a hard OS memory limit. Temporary + data lives in the OS temp directory (override with + `CODEGRAPH_SEMANTIC_TMPDIR`, outside the project root) and is removed after + the query. +- The snapshot contains the selected inputs and their literal include + dependencies. Paths that are invalid or leave the project root, and + unsupported include forms, are rejected. This is not a sandbox for running an + untrusted compiler executable. +- Full macro expansion is done by the compiler inside the allowed snapshot + context; the exact expansion chain and spelling locations come from the + pyslang backend only. +- A failure, a missing slang or a cancellation never changes the source graph. + The Windows runner is not enabled yet; process cancellation on that OS needs + its own validation. +- This is about parameters and ports and their compiler context — not + simulation, synthesis, timing, CDC or a proof of hardware correctness. + Semantics are not added to the ordinary MCP explore; use the CLI or API above + for an explicit query. diff --git a/package.json b/package.json index 3faaa192d3..5f89e0dd99 100644 --- a/package.json +++ b/package.json @@ -24,7 +24,7 @@ "build:ui": "cd ui && npm run build && cd .. && node scripts/check-ui-build.mjs", "build:lib": "cd ui && npm run build:lib", "preuninstall": "node dist/bin/uninstall.js", - "copy-assets": "node -e \"const fs=require('fs');fs.mkdirSync('dist/db',{recursive:true});fs.copyFileSync('src/db/schema.sql','dist/db/schema.sql');fs.mkdirSync('dist/extraction/wasm',{recursive:true});fs.readdirSync('src/extraction/wasm').filter(f=>f.endsWith('.wasm')).forEach(f=>fs.copyFileSync('src/extraction/wasm/'+f,'dist/extraction/wasm/'+f))\"", + "copy-assets": "node -e \"const fs=require('fs');fs.mkdirSync('dist/db',{recursive:true});fs.copyFileSync('src/db/schema.sql','dist/db/schema.sql');fs.mkdirSync('dist/extraction/wasm',{recursive:true});fs.readdirSync('src/extraction/wasm').filter(f=>f.endsWith('.wasm')).forEach(f=>fs.copyFileSync('src/extraction/wasm/'+f,'dist/extraction/wasm/'+f));fs.mkdirSync('dist/hdl',{recursive:true});fs.copyFileSync('src/hdl/source-map-export.py','dist/hdl/source-map-export.py')\"", "dev": "tsc --watch", "build:kernel": "bash scripts/build-kernel.sh", "cli": "npm run build && node dist/bin/codegraph.js", diff --git a/scripts/verify-hdl-source-map-export.py b/scripts/verify-hdl-source-map-export.py new file mode 100644 index 0000000000..b343d5c832 --- /dev/null +++ b/scripts/verify-hdl-source-map-export.py @@ -0,0 +1,246 @@ +#!/usr/bin/env python3 +"""Bounded real-world tests for the exporter; run with a Python environment that has pyslang 11.0.0. + +Installs no dependencies. Creates only isolated temporary fixtures. +""" +import argparse +import json +from pathlib import Path +import subprocess +import sys +import tempfile +import unittest + +HELPER = Path(__file__).resolve().parents[1] / 'src/hdl/source-map-export.py' +WORK_ROOT = None + + +class SourceMapExportTests(unittest.TestCase): + def setUp(self): + self.directory = tempfile.TemporaryDirectory(prefix='hdl-source-map-test-', dir=WORK_ROOT) + self.root = Path(self.directory.name) + + def tearDown(self): + self.directory.cleanup() + + def write(self, name, text, crlf=False, bom=False): + target = self.root / name + target.parent.mkdir(parents=True, exist_ok=True) + target.write_bytes((b'\xef\xbb\xbf' if bom else b'') + text.replace('\n', '\r\n' if crlf else '\n').encode('utf8')) + + def compile(self, source, extra=(), success=True): + out = self.root / 'facts.json' + result = subprocess.run([sys.executable, '-I', str(HELPER), '--ast-json', str(out), + '--ast-json-source-info', '--ast-json-detailed-types', '--std=1800-2023', + '--threads=1', '--error-limit=20', '--top', 'top', *extra, '--', source], + cwd=self.root, text=True, capture_output=True, timeout=60) + self.assertEqual(result.returncode == 0, success, result.stderr) + if not success: + self.assertFalse(out.exists(), 'failed frontend must not emit semantic facts') + return result + payload = json.loads(out.read_text()) + self.assertEqual(payload['codegraphSemanticVersion'], 1) + return payload['facts'] + + def assertPoint(self, point): + file = (self.root / point['file']).resolve() + self.assertTrue(file.is_relative_to(self.root)) + data = file.read_bytes() + offset = point['byteOffset'] + self.assertGreaterEqual(offset, 0) + self.assertLessEqual(offset, len(data)) + self.assertEqual(point['line'], data.count(b'\n', 0, offset) + 1) + self.assertEqual(point['column'], offset - data.rfind(b'\n', 0, offset)) + + def test_nested_arguments_include_utf8_crlf_and_four_state_value(self): + self.write('defs/inner.svh', '`define LEAF(name,value) localparam int name=value;\n', crlf=True) + self.write('defs/outer.svh', '`include "inner.svh"\n`define WRAP(name,value) `LEAF(name,value)\n', crlf=True) + source = "top space'quote.sv" + self.write(source, '`include "defs/outer.svh"\nmodule top #(parameter int N=1);\n /* 😀 */ `WRAP(P, 7)\n localparam logic [3:0] XZ=4\'b10xz;\nendmodule\n', crlf=True, bom=True) + facts = self.compile(source, ['-G', "N=32'd4"]) + byname = {fact['name']: fact for fact in facts} + self.assertEqual(byname['N']['value'], '4') + self.assertEqual(byname['P']['value'], '7') + self.assertEqual(byname['XZ']['value'].lower(), "4'b10xz") + p = byname['P'] + self.assertEqual(p['sourceOrigin'], 'macro') + self.assertFalse(p['macroExpansionComplete']) + frames = p['macroExpansion'] + self.assertEqual({frame['name'] for frame in frames}, {'LEAF', 'WRAP'}) + self.assertTrue(any(frame['name'] == 'LEAF' and frame['argument'] for frame in frames)) + self.assertTrue(any(frame['name'] == 'WRAP' and frame['argument'] for frame in frames)) + data = (self.root / source).read_bytes() + offset = data.index(b'P, 7') + self.assertEqual(p['source'], {'file': source, 'line': 3, 'column': offset - data.rfind(b'\n', 0, offset)}) + for frame in frames: + if 'spelling' in frame: + self.assertPoint(frame['spelling']) + if 'invocation' in frame: + self.assertPoint(frame['invocation']['start']) + self.assertPoint(frame['invocation']['end']) + self.assertTrue(any(frame.get('spelling', {}).get('file') == 'defs/inner.svh' for frame in frames)) + + def test_token_concat_is_prefix_point_not_a_generated_name_range(self): + self.write('defs.svh', '`define CAT(a,b) a``b\n`define DECL(p) localparam int `CAT(p,_param)=3;\n') + self.write('top.sv', '`include "defs.svh"\nmodule top;\n `DECL(bus)\nendmodule\n') + fact = next(f for f in self.compile('top.sv') if f['name'] == 'bus_param') + self.assertEqual(fact['value'], '3') + self.assertEqual(fact['sourceOrigin'], 'macro') + self.assertNotIn('nameRange', fact) + self.assertFalse(fact['macroExpansionComplete']) + data = (self.root / 'top.sv').read_bytes() + prefix = [frame['spelling'] for frame in fact['macroExpansion'] + if frame.get('spelling', {}).get('file') == 'top.sv' + and data[frame['spelling']['byteOffset']:].startswith(b'bus')] + self.assertTrue(prefix) + self.assertFalse(data[prefix[0]['byteOffset']:].startswith(b'bus_param')) + + def test_zero_empty_string_and_physical_line_directive_coordinates(self): + self.write('top.sv', '`line 700 "virtual.sv" 0\nmodule top; parameter int ZERO=0; parameter string EMPTY=""; endmodule\n') + facts = {f['name']: f for f in self.compile('top.sv')} + self.assertEqual(facts['ZERO']['value'], '0') + self.assertEqual(facts['EMPTY']['value'], '""') + for fact in facts.values(): + self.assertEqual(fact['sourceOrigin'], 'direct') + self.assertEqual(fact['source']['file'], 'top.sv') + self.assertEqual(fact['source']['line'], 2) + + def test_type_alias_facts_have_integral_width_and_declaration_origin(self): + self.write('types.svh', '`define DECLTYPE(name) typedef logic [7:0] name;\n') + self.write('top.sv', '`include "types.svh"\nmodule top;\n `DECLTYPE(byte_t)\n typedef logic [3:0] nibble_t;\nendmodule\n') + facts = {fact['name']: fact for fact in self.compile('top.sv')} + self.assertEqual(facts['byte_t']['kind'], 'type') + self.assertEqual(facts['byte_t']['width'], 8) + self.assertEqual(facts['byte_t']['sourceOrigin'], 'macro') + self.assertTrue(facts['byte_t']['macroExpansion']) + self.assertEqual(facts['nibble_t']['kind'], 'type') + self.assertEqual(facts['nibble_t']['width'], 4) + self.assertEqual(facts['nibble_t']['sourceOrigin'], 'direct') + self.assertNotIn('direction', facts['byte_t']) + + @staticmethod + def origin_names(fact, role): + return {frame['name'] for origin in fact['expressionOrigins'] if origin['role'] == role + for frame in origin.get('macroExpansion', [])} + + def test_literal_parameter_tracks_its_macro_initializer_not_declaration_name(self): + self.write('defs.svh', '`define VALUE 7\n') + self.write('top.sv', '`include "defs.svh"\nmodule top; localparam int P=`VALUE; endmodule\n') + fact = next(f for f in self.compile('top.sv') if f['name'] == 'P') + self.assertEqual(fact['sourceOrigin'], 'direct') + self.assertEqual(self.origin_names(fact, 'initializer'), {'VALUE'}) + self.assertEqual(fact['expressionOriginCoverage']['initializer'], 'checked') + self.assertEqual(fact['expressionOriginCoverage']['declaredInitializer'], 'not-applicable') + + def test_port_and_type_dimensions_have_compiler_macro_origins_with_physical_coordinates(self): + self.write('defs.svh', '`define WIDTH 8\n', crlf=True) + self.write('top.sv', '`include "defs.svh"\n/* 😀 */ module top(output logic [`WIDTH-1:0] data);\n typedef logic [`WIDTH-1:0] word_t;\nendmodule\n', crlf=True, bom=True) + facts = {f['name']: f for f in self.compile('top.sv')} + for name in ['data', 'word_t']: + fact = facts[name] + self.assertEqual(fact['sourceOrigin'], 'direct') + self.assertEqual(fact['width'], 8) + self.assertEqual(self.origin_names(fact, 'type'), {'WIDTH'}) + self.assertEqual(fact['expressionOriginCoverage']['type'], 'checked') + for origin in fact['expressionOrigins']: + for frame in origin.get('macroExpansion', []): + if 'spelling' in frame: + self.assertPoint(frame['spelling']) + if 'invocation' in frame: + self.assertPoint(frame['invocation']['start']) + self.assertPoint(frame['invocation']['end']) + + def test_nonansi_and_ansi_unpacked_port_dimensions_keep_macro_type_origins(self): + self.write('defs.svh', '`define W 4\n') + for declaration in ['module top(a); input logic a[`W]; endmodule', + 'module top(input logic a[`W]); endmodule']: + self.write('top.sv', '`include "defs.svh"\n' + declaration + '\n') + fact = next(f for f in self.compile('top.sv') if f['kind'] == 'port' and f['name'] == 'a') + self.assertEqual(self.origin_names(fact, 'type'), {'W'}) + self.assertEqual(fact['expressionOriginCoverage']['type'], 'checked') + self.assertFalse(fact['expressionOriginCoverage']['truncated']) + self.assertEqual(len(fact['expressionOrigins']), 1) + for frame in fact['expressionOrigins'][0]['macroExpansion']: + if 'spelling' in frame: + self.assertPoint(frame['spelling']) + if 'invocation' in frame: + self.assertPoint(frame['invocation']['start']) + self.assertPoint(frame['invocation']['end']) + + def test_instance_override_separates_actual_and_declared_default_macro_origins(self): + self.write('defs.svh', '`define DEFAULT 7\n`define ACTUAL 11\n') + self.write('top.sv', '`include "defs.svh"\nmodule leaf #(parameter int P=`DEFAULT); endmodule\nmodule top; leaf #(.P(`ACTUAL)) u(); endmodule\n') + fact = next(f for f in self.compile('top.sv') if f['instancePath'] == 'top.u' and f['name'] == 'P') + self.assertEqual(fact['value'], '11') + self.assertEqual(self.origin_names(fact, 'initializer'), {'ACTUAL'}) + self.assertEqual(self.origin_names(fact, 'declared-initializer'), {'DEFAULT'}) + self.assertEqual(fact['expressionOriginCoverage']['declaredInitializer'], 'checked') + + def test_command_line_override_never_reports_macro_default_as_effective(self): + self.write('top.sv', '`define DEFAULT 7\nmodule top #(parameter int P=`DEFAULT); endmodule\n') + fact = next(f for f in self.compile('top.sv', ['-G', 'P=11']) if f['name'] == 'P') + self.assertEqual(fact['value'], '11') + self.assertEqual(fact['expressionOriginCoverage']['initializer'], 'command-line') + self.assertFalse(self.origin_names(fact, 'initializer')) + self.assertEqual(self.origin_names(fact, 'declared-initializer'), {'DEFAULT'}) + + def test_checked_type_syntax_does_not_claim_transitive_typedef_dependencies(self): + self.write('top.sv', '`define WIDTH 8\ntypedef logic [`WIDTH-1:0] word_t;\nmodule top(output word_t data); endmodule\n') + fact = next(f for f in self.compile('top.sv') if f['name'] == 'data') + self.assertEqual(fact['width'], 8) + self.assertEqual(fact['expressionOriginCoverage']['type'], 'checked') + self.assertEqual(fact['expressionOrigins'], []) + + def test_expression_origin_budget_is_explicit_and_keeps_compiled_value(self): + self.write('top.sv', '`define V 1\nmodule top; localparam int P=' + '+'.join(['`V'] * 40) + '; endmodule\n') + fact = next(f for f in self.compile('top.sv') if f['name'] == 'P') + self.assertEqual(fact['value'], '40') + self.assertEqual(len(fact['expressionOrigins']), 32) + self.assertTrue(fact['expressionOriginCoverage']['truncated']) + + def test_normalized_origins_are_deduplicated_when_frame_budget_is_exhausted(self): + self.write('top.sv', '`define V 1\nmodule top; localparam int P=`V+`V; endmodule\n') + out = self.root / 'facts.json' + probe = """import importlib.util,sys +sys.dont_write_bytecode=True +spec=importlib.util.spec_from_file_location('test_exporter',sys.argv[1]) +module=importlib.util.module_from_spec(spec);spec.loader.exec_module(module) +module.MAX_PROVENANCE_FRAMES=0 +raise SystemExit(module.main(['--ast-json',sys.argv[2],'--std=1800-2023','--top','top','--','top.sv'])) +""" + result = subprocess.run([sys.executable, '-I', '-c', probe, str(HELPER), str(out)], + cwd=self.root, capture_output=True, text=True, timeout=60) + self.assertEqual(result.returncode, 0, result.stderr) + fact = next(f for f in json.loads(out.read_text())['facts'] if f['name'] == 'P') + self.assertEqual(fact['value'], '2') + self.assertEqual(len(fact['expressionOrigins']), 1) + self.assertTrue(fact['expressionOriginCoverage']['truncated']) + self.assertEqual(fact['expressionOrigins'][0]['macroExpansion'], []) + self.assertFalse(fact['expressionOrigins'][0]['macroExpansionComplete']) + + def test_line_directive_cannot_spoof_command_line_origin(self): + self.write('top.sv', '`define VALUE 7\n`line 700 "" 0\nmodule top; parameter int P=`VALUE; endmodule\n') + fact = next(f for f in self.compile('top.sv') if f['name'] == 'P') + self.assertEqual(fact['expressionOriginCoverage']['initializer'], 'checked') + self.assertEqual(self.origin_names(fact, 'initializer'), {'VALUE'}) + + def test_rejects_semantic_errors_instead_of_exporting_partial_ast(self): + self.write('top.sv', 'module top; missing_dependency u(); endmodule\n') + result = self.compile('top.sv', success=False) + self.assertIn('unknown module', result.stderr) + + def test_shared_bodies_and_generate_indices_have_instance_specific_paths(self): + self.write('top.sv', 'module leaf #(parameter int P=2); endmodule\nmodule top; leaf u1(); leaf u2(); for(genvar i=2;i<4;i++) begin : lanes leaf u(); end endmodule\n') + facts = self.compile('top.sv') + paths = {f['instancePath'] for f in facts if f['name'] == 'P'} + self.assertEqual(paths, {'top.u1', 'top.u2', 'top.lanes[2].u', 'top.lanes[3].u'}) + + +if __name__ == '__main__': + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--helper', type=Path, default=HELPER) + parser.add_argument('--work-root', type=Path) + args = parser.parse_args() + HELPER = args.helper.resolve() + WORK_ROOT = args.work_root + unittest.main(argv=[sys.argv[0], '-v']) diff --git a/src/bin/codegraph.ts b/src/bin/codegraph.ts index b041843ff9..fe2fe36d4d 100644 --- a/src/bin/codegraph.ts +++ b/src/bin/codegraph.ts @@ -1,4 +1,5 @@ #!/usr/bin/env node +import { formatHdlProfileStatus } from '../hdl/status'; /** * CodeGraph CLI * @@ -1166,7 +1167,9 @@ program const journalMode = cg.getJournalMode(); const buildInfo = cg.getIndexBuildInfo(); - const reindexRecommended = cg.isIndexStale(); + const hdlProfile = cg.getHdlProfileStatus(); + const engineReindexRecommended = cg.isIndexStale(); + const reindexRecommended = engineReindexRecommended || !!hdlProfile?.reindexRecommended; // Frameworks a sync detected since the last full index (#2516): files // indexed before then were extracted without them. const frameworkChanges = cg.getFrameworkChangesSinceIndex(); @@ -1214,6 +1217,7 @@ program reindexRecommended: reindexRecommended || frameworkChanges !== null, // { added, removed } frameworks since the last full index, or null. frameworksChangedSinceIndex: frameworkChanges, + hdlProfile, // 'complete' | 'partial' (files silently dropped) | 'indexing' // (a run was killed mid-index — the index is truncated) | // 'failed' | null (predates the marker). @@ -1335,14 +1339,23 @@ program } const missingSymbols = describeFilesMissingSymbols(health); for (const line of missingSymbols) warn(line); - if (totalChanges === 0 && missingSymbols.length === 0) { + if (totalChanges === 0 && hdlProfile?.state === 'configuration-error') { + warn('Source files are unchanged; HDL profile configuration requires attention'); + } else if (totalChanges === 0 && hdlProfile?.reindexRecommended) { + warn('Source files are unchanged; indexed HDL context needs a rebuild'); + } else if (totalChanges === 0 && missingSymbols.length === 0) { success('Index is up to date'); } console.log(); // Re-index hint: the index was built by an older engine than the one now // running, so a rebuild would add data a migration can't backfill. - if (reindexRecommended) { + const profileNote = formatHdlProfileStatus(hdlProfile); + if (profileNote) { + console.log(profileNote); + console.log(); + } + if (engineReindexRecommended) { const builtWith = buildInfo.version ? `v${buildInfo.version.replace(/^v/, '')}` : 'an earlier version'; warn(`Index was built by ${builtWith}; re-index to pick up this engine's improvements.`); info('Run "codegraph index" (full rebuild) or "codegraph sync"'); @@ -1455,12 +1468,58 @@ program * subagents (which don't inherit MCP tools, #704) and non-MCP harnesses — * can reach the graph through a plain shell command. */ +program + .command('hdl-semantic [query]') + .description('Compute HDL parameters and port widths with optional slang (exact symbol or instance query)') + .option('--slang ', 'Path to the installed slang executable') + .option('--python ', 'Python with pinned pyslang for compiler macro source mapping (instead of --slang)') + .option('-p, --path ', 'Project path') + .option('--top ', 'Single top module (defaults to the active profile top)') + .option('--parameter ', 'Top-level parameter overrides') + .option('--allow-use-before-declare', 'Explicit slang compatibility mode') + .option('--limit ', 'Maximum facts (1..1000)', '100') + .action(async (query: string | undefined, options: { slang?: string; python?: string; path?: string; top?: string; + parameter?: string[]; allowUseBeforeDeclare?: boolean; limit: string }) => { + const projectPath = resolveProjectPath(options.path); + try { + if (!isInitialized(projectPath)) throw new Error('HDL semantic queries require an initialized CodeGraph project'); + const parameters: Record = Object.create(null); + for (const argument of options.parameter ?? []) { + const separator = argument.indexOf('='); + if (separator < 1 || !argument.slice(separator + 1)) throw new Error('Parameter must be NAME=VALUE'); + const name = argument.slice(0, separator); + if (Object.prototype.hasOwnProperty.call(parameters, name)) throw new Error('Duplicate HDL parameter override: ' + name); + parameters[name] = argument.slice(separator + 1); + } + const { default: CodeGraph } = await loadCodeGraph(); + const cg = await CodeGraph.open(projectPath); + const controller = new AbortController(); + const interrupt = () => controller.abort(); + process.once('SIGINT', interrupt); + process.once('SIGTERM', interrupt); + try { + const result = await cg.getHdlSemantics({ executable: options.slang, pythonExecutable: options.python, top: options.top, + parameters, allowUseBeforeDeclare: options.allowUseBeforeDeclare, query, limit: Number(options.limit), + signal: controller.signal }); + console.log(JSON.stringify(result)); + } finally { + process.removeListener('SIGINT', interrupt); + process.removeListener('SIGTERM', interrupt); + cg.close(); + } + } catch (err) { + error(`HDL semantics failed: ${err instanceof Error ? err.message : String(err)}`); + process.exitCode = 1; + } + }); + program .command('explore ') .description('Explore an area: relevant symbols\' source + call paths in one shot (same output as the codegraph_explore MCP tool)') .option('-p, --path ', 'Project path') .option('--max-files ', 'Maximum number of files to include source from') - .action(async (queryParts: string[], options: { path?: string; maxFiles?: string }) => { + .option('--hdl-access ', 'HDL signal access: read, write, readwrite, control, event, all (query is one exact signal name)') + .action(async (queryParts: string[], options: { path?: string; maxFiles?: string; hdlAccess?: string }) => { const projectPath = resolveProjectPath(options.path); try { @@ -1475,6 +1534,7 @@ program const args: Record = { query: queryParts.join(' ') }; if (options.maxFiles) args.maxFiles = parseInt(options.maxFiles, 10); + if (options.hdlAccess) args.hdlAccess = options.hdlAccess; const result = await handler.execute('codegraph_explore', args); console.log(result.content[0]?.text ?? ''); diff --git a/src/db/queries.ts b/src/db/queries.ts index 4675e3b65e..450c3b8ff8 100644 --- a/src/db/queries.ts +++ b/src/db/queries.ts @@ -5335,12 +5335,46 @@ export class QueryBuilder { refs: UnresolvedReference[] ): number { return this.db.transaction(() => { - const changed = this.deleteEdgesByIds(edgeIds); - this.insertUnresolvedRefsBatch(refs); + let changed = this.deleteEdgesByIds(edgeIds); + const seenHdl = new Set(); + const pending = refs.filter(ref => { + if (!ref.referenceName.startsWith('hdl:wildcard:') && !ref.referenceName.startsWith('hdl:port-position:')) return true; + const key = JSON.stringify([ref.fromNodeId, ref.referenceName, ref.line, ref.column]); + if (seenHdl.has(key)) return false; + seenHdl.add(key); + return true; + }); + // A wildcard/positional binding depends on the entire module header. + // Invalidate its complete fan-out, including actual signals in untouched + // files, before rebuilding it from one original reference. + for (const ref of pending) { + if (ref.referenceName.startsWith('hdl:wildcard:') || ref.referenceName.startsWith('hdl:port-position:')) { + changed += this.db.prepare("DELETE FROM edges WHERE source = ? AND json_extract(metadata, '$.refName') = ?") + .run(ref.fromNodeId, ref.referenceName).changes; + } + } + this.insertUnresolvedRefsBatch(pending); return changed; })(); } + hasLanguage(language: Language): boolean { + return !!this.db.prepare('SELECT 1 FROM nodes WHERE language = ? LIMIT 1').get(language); + } + + /** HDL call arguments may change access when a remote formal changes direction. + * Seek HDL sources first, then their indexed outgoing edges; ordinary projects + * have no rows here. Include unclassified arguments so newly indexed callees + * can provide direction evidence on the next sync. */ + getHdlCallArgumentEdges(): Array { + const rows = this.db.prepare(`SELECT e.*, src.file_path AS source_file_path, src.language AS source_language + FROM nodes src JOIN edges e ON e.source = src.id + WHERE src.language = 'verilog' AND e.kind = 'references' AND e.metadata LIKE '%hdl:call-arg:%'`) + .all() as Array; + return rows.map(row => ({ ...rowToEdge(row), edgeId: row.id, + sourceFilePath: row.source_file_path, sourceLanguage: row.source_language })); + } + /** * Distinct node names present in the given files — the symbol names a sync * pass uses to look up retryable failed refs after those files changed. diff --git a/src/db/reference-tail.ts b/src/db/reference-tail.ts index 29ecb85d01..ffaec199e7 100644 --- a/src/db/reference-tail.ts +++ b/src/db/reference-tail.ts @@ -38,6 +38,22 @@ * division, so only a `references` ref is read as a path. */ export function referenceNameTail(referenceName: string, referenceKind?: string): string { + // Named HDL connections carry structured module/port/instance identity. + // Retry on the formal port name when a previously removed port reappears. + for (const prefix of ['hdl:port-position:', 'hdl:wildcard:']) { + if (referenceName.startsWith(prefix)) { + try { + const parts: unknown = JSON.parse(referenceName.slice(prefix.length)); + if (Array.isArray(parts) && typeof parts[0] === 'string') return parts[0]; + } catch { /* Fall back for malformed internal references. */ } + } + } + if (referenceName.startsWith('hdl:port:')) { + try { + const parts: unknown = JSON.parse(referenceName.slice('hdl:port:'.length)); + if (Array.isArray(parts) && parts.length === 3 && typeof parts[1] === 'string') return parts[1]; + } catch { /* Malformed refs retain the generic tail behavior below. */ } + } if (referenceKind === 'imports' && referenceName.replace(/\/+$/, '').includes('/')) { const stem = pathStem(referenceName); if (stem) return stem; diff --git a/src/extraction/grammars.ts b/src/extraction/grammars.ts index 66c69687b7..3217be7e4f 100644 --- a/src/extraction/grammars.ts +++ b/src/extraction/grammars.ts @@ -24,6 +24,7 @@ const WASM_GRAMMAR_FILES: Record = { javascript: 'tree-sitter-javascript.wasm', jsx: 'tree-sitter-javascript.wasm', python: 'tree-sitter-python.wasm', + verilog: 'tree-sitter-systemverilog.wasm', go: 'tree-sitter-go.wasm', rust: 'tree-sitter-rust.wasm', java: 'tree-sitter-java.wasm', @@ -121,6 +122,11 @@ export const EXTENSION_MAP: Record = { '.scala': 'scala', '.sc': 'scala', '.lua': 'lua', + // Verilog / SystemVerilog (one grammar covers both) + '.v': 'verilog', + '.vh': 'verilog', + '.sv': 'verilog', + '.svh': 'verilog', '.luau': 'luau', '.m': 'objc', '.mm': 'objc', @@ -395,7 +401,7 @@ export async function initGrammars(): Promise { * the vendored wasm together. */ const VENDORED_WASM_LANGS: ReadonlySet = new Set([ - 'pascal', 'scala', 'lua', 'luau', 'csharp', 'r', 'cfml', 'cfscript', 'cfquery', + 'pascal', 'scala', 'lua', 'luau', 'verilog', 'csharp', 'r', 'cfml', 'cfscript', 'cfquery', 'cobol', 'vbnet', 'erlang', 'terraform', 'arkts', 'nix', 'typescript', 'tsx', 'javascript', 'jsx', 'java', 'python', 'go', // R7a (C/C++ kernel port prep): tree-sitter-c v0.24.2 (b780e47) + @@ -929,6 +935,7 @@ export function getLanguageDisplayName(language: Language): string { scala: 'Scala', lua: 'Lua', luau: 'Luau', + verilog: 'Verilog / SystemVerilog', objc: 'Objective-C', solidity: 'Solidity', nix: 'Nix', diff --git a/src/extraction/index.ts b/src/extraction/index.ts index 78b8bf2142..0c9eff3f24 100644 --- a/src/extraction/index.ts +++ b/src/extraction/index.ts @@ -5,6 +5,8 @@ */ import * as fs from 'fs'; +import { HdlIndexContext } from '../hdl/index-context'; +import { loadHdlProfile } from '../hdl/profile'; import * as fsp from 'fs/promises'; import * as path from 'path'; import * as os from 'os'; @@ -2061,6 +2063,54 @@ function pairReindexedNodes( */ export class ExtractionOrchestrator { private rootDir: string; + private hdlContext: HdlIndexContext | null = null; + private hdlForce = false; + /** A source the changed profile re-reads was not stored, so that profile is not the indexed one yet. */ + private hdlForceUnstored = false; + private hdlTouched = new Set(); + + private prepareHdlContext(): void { + this.hdlTouched.clear(); + this.hdlForceUnstored = false; + const previous = this.queries.getMetadata('hdl_profile_fingerprint'); + this.hdlContext = new HdlIndexContext(this.rootDir, !!this.queries.getMetadata('hdl_profile_name')); + this.hdlForce = previous !== this.hdlContext.fingerprint && (!!this.hdlContext.profile || !!this.queries.getMetadata('hdl_profile_name')); + } + private forceHdlFile(file: string): boolean { return this.hdlForce && !!this.hdlContext?.isHdl(file); } + private hdlSource(file: string, content: string): string { + if (!this.hdlContext) this.prepareHdlContext(); + return this.hdlContext!.source(file, content); + } + commitHdlProfile(): void { + if (!this.hdlContext || this.hdlForceUnstored) return; + this.queries.setMetadata('hdl_profile_name', this.hdlContext.profile?.name ?? ''); + this.queries.setMetadata('hdl_profile_fingerprint', this.hdlContext.fingerprint); + const context = JSON.parse(this.hdlContext.context()); + if (this.hdlContext.profile) { + if (!this.hdlForce && this.hdlTouched.size < this.hdlContext.profile.files.length) { + const previous = JSON.parse(this.queries.getMetadata('hdl_profile_context') ?? '{}'); + const units = previous.unitDiagnostics ?? {}; + for (const file of this.hdlTouched) units[file] = context.unitDiagnostics[file] ?? []; + context.unitDiagnostics = units; + const combined = Object.values(units).flat(); + context.diagnosticCount = combined.length; + context.diagnostics = combined.slice(0, 200); + context.incomplete = context.diagnostics.length > 0 || (!previous.unitDiagnostics && previous.incomplete === true); + } + const parseIssues = this.hdlContext.profile.files.flatMap(file => { + const record = this.queries.getFileByPath(file); + return record ? (record.errors ?? []).map(error => ({ filePath: file, ...error })) + : [{ filePath: file, message: 'Profile source has not been indexed.' }]; + }); + if (parseIssues.length) { + context.incomplete = true; + context.diagnostics = [...context.diagnostics, ...parseIssues].slice(0, 200); + } + } + this.queries.setMetadata('hdl_profile_context', JSON.stringify(context)); + this.hdlForce = false; + } + private queries: QueryBuilder; /** * Names of frameworks detected for this project, populated by indexAll(). @@ -2384,6 +2434,7 @@ export class ExtractionOrchestrator { // Threaded into language detection so custom-extension files load the right // grammar and store under the mapped language. const overrides = loadExtensionOverrides(this.rootDir); + this.prepareHdlContext(); const log = verbose ? (msg: string) => { console.log(`[worker] ${msg}`); } @@ -2401,7 +2452,7 @@ export class ExtractionOrchestrator { // attributed — these labels settle scan vs framework-detect vs grammars. const tScan = Date.now(); const skipStats: ScanSkipStats = { unsupportedByExtension: new Map() }; - const files = await scanDirectoryAsync(this.rootDir, (current, file) => { + const scannedFiles = await scanDirectoryAsync(this.rootDir, (current, file) => { onProgress?.({ phase: 'scanning', current, @@ -2409,6 +2460,15 @@ export class ExtractionOrchestrator { currentFile: file, }); }, skipStats); + const files = this.hdlContext!.files(scannedFiles); + const allowed = new Set(files); + for (const tracked of this.queries.getAllFiles()) { + if (tracked.language !== 'verilog' || allowed.has(tracked.path)) continue; + const refs = this.queries.getCrossFileIncomingEdgesWithTarget(tracked.path) + .map(resurrectRefFromDroppedEdge).filter((ref): ref is UnresolvedReference => !!ref); + this.queries.deleteFile(tracked.path); + if (refs.length) this.queries.replaceResolutionEdgesWithUnresolvedRefs([], refs); + } if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[phase-timing] scan: ${Date.now() - tScan}ms (${files.length} files)`); /** Only meaningful when nothing was indexable — see IndexResult (#1502). */ const skipSummary = (): Pick => { @@ -2541,8 +2601,9 @@ export class ExtractionOrchestrator { const parseFile = (filePath: string, content: string): Promise => { const language = detectLanguage(filePath, content, overrides, this.rootDir); const names = this.frameworksForFile(filePath, frameworkNames); - if (!pool) return Promise.resolve(extractFromSource(filePath, content, language, names)); - return pool.requestParse({ filePath, content, language, frameworkNames: names }); + const selected = this.hdlSource(filePath, content); + if (!pool) return Promise.resolve(extractFromSource(filePath, selected, language, names)); + return pool.requestParse({ filePath, content: selected, language, frameworkNames: names }); }; // --- Bounded rolling-window dispatch, ordered commit --- @@ -2572,6 +2633,7 @@ export class ExtractionOrchestrator { const storeResult = async (filePath: string, content: string, stats: fs.Stats, result: ExtractionResult): Promise => { processed++; + if (this.hdlContext?.profile && this.hdlContext.isHdl(filePath)) this.hdlTouched.add(filePath); // WAL hard-cap backstop: between files (never mid-transaction), pause // the store until the off-thread checkpoint catches up. Resolves to @@ -2731,6 +2793,8 @@ export class ExtractionOrchestrator { // Read files in parallel (with path validation before any I/O) const fileContents = await Promise.all( batch.map(async (fp) => { + const snapshot = this.hdlContext?.sources.get(fp); + if (snapshot) return { filePath: fp, content: snapshot.original, stats: snapshot.stats, error: null as Error | null }; try { // Indexing read: follow in-root symlinks the directory walk already // descended into (the `../` guard still applies) so files reached @@ -2989,6 +3053,7 @@ export class ExtractionOrchestrator { .map(line => /^\s*\/\//.test(line) ? '' : line) .join('\n'); + if (this.hdlContext?.profile && this.hdlContext.isHdl(filePath)) continue; let result: ExtractionResult; try { result = await parseFile(filePath, stripped); @@ -3043,6 +3108,11 @@ export class ExtractionOrchestrator { * Index specific files */ async indexFiles(filePaths: string[]): Promise { + this.prepareHdlContext(); + if (this.hdlForce && (this.hdlContext!.profile || this.queries.getMetadata('hdl_profile_name'))) { + throw new Error('HDL profile context changed; use sync or index to update the complete HDL selection'); + } + filePaths = filePaths.filter(file => this.hdlContext!.accepts(file)); const startTime = Date.now(); const errors: ExtractionError[] = []; let filesIndexed = 0; @@ -3105,6 +3175,9 @@ export class ExtractionOrchestrator { }; } + const snapshot = this.hdlContext?.sources.get(relativePath); + if (snapshot) return this.indexFileWithContent(relativePath, snapshot.original, snapshot.stats); + // Read file content and stats let content: string; let stats: fs.Stats; @@ -3200,7 +3273,7 @@ export class ExtractionOrchestrator { // otherwise detect on the spot so single-file re-index paths still emit // route nodes / middleware / etc. const frameworkNames = this.frameworksForFile(relativePath, this.ensureDetectedFrameworks()); - const result = extractFromSource(relativePath, content, language, frameworkNames); + const result = extractFromSource(relativePath, this.hdlSource(relativePath, content), language, frameworkNames); // Store in database await this.storeExtractionResult(relativePath, content, language, stats, result, createYielder()); @@ -3251,13 +3324,17 @@ export class ExtractionOrchestrator { // this would replace the file's symbols with an empty row under the new // content hash, which no hash-based sync revisits. Keep whatever the index // has; the stale (or missing) row makes the next sync retry the file. - if (hasGrammarLoadFailure(result.errors)) return; + if (hasGrammarLoadFailure(result.errors)) { + if (this.forceHdlFile(filePath)) this.hdlForceUnstored = true; + return; + } // A kernel result can arrive as an undecoded buffer transport (empty // node/edge arrays, tables riding in kernelBuffers). Decode it before // storing — persisting the transport as-is records the file as having no // symbols at all (#1541). No-op for already-decoded results. result = materializeKernelResult(result, filePath, language); + if (this.hdlContext?.profile && language === 'verilog') this.hdlTouched.add(filePath); // Bulk inserts run in bounded sub-transactions with a yield between, so a // giant generated file (tens of thousands of symbols) can't block the @@ -3275,7 +3352,7 @@ export class ExtractionOrchestrator { // successful retry's symbols — a permanent empty file presented as // recovered (the #1541 wipe, reintroduced through the marker path). const existingFile = this.queries.getFileByPath(filePath); - if (existingFile && existingFile.contentHash === contentHash) { + if (existingFile && existingFile.contentHash === contentHash && !this.forceHdlFile(filePath)) { const existingIsMarker = existingFile.nodeCount === 0 && (existingFile.errors?.length ?? 0) > 0; const incomingHasContent = result.nodes.length > 0; @@ -3500,6 +3577,14 @@ export class ExtractionOrchestrator { const reinserted: Edge[] = []; const resurrected: UnresolvedReference[] = []; for (const e of crossFileIncomingEdges) { + const name = e.metadata?.refName; + // HDL members and ports often share short names within one file. + // Replay their qualified reference instead of reattaching by kind/name. + if (e.sourceLanguage === 'verilog' && typeof name === 'string') { + const ref = resurrectRefFromDroppedEdge(e); + if (ref) resurrected.push(ref); + continue; + } const newTargetId = replacementOf.get(e.target); if (newTargetId) { reinserted.push({ source: e.source, target: newTargetId, kind: e.kind, metadata: e.metadata, line: e.line, column: e.column, provenance: e.provenance }); @@ -3512,7 +3597,7 @@ export class ExtractionOrchestrator { this.queries.insertEdges(reinserted); } if (resurrected.length > 0) { - this.queries.insertUnresolvedRefsBatch(resurrected); + this.queries.replaceResolutionEdgesWithUnresolvedRefs([], resurrected); } } @@ -3574,6 +3659,27 @@ export class ExtractionOrchestrator { return refs.length; } + /** Reclassify argument accesses against current function/task formals. + * Existing calls are dependencies; missing signatures also need retries when + * a callee appears, so retain and revisit their plain argument references. */ + resurrectHdlCallArgumentEdges(changedFilePaths: string[], removedFiles = false): number { + if (!changedFilePaths.length && !removedFiles) return 0; + const overrides = loadExtensionOverrides(this.rootDir); + // Removed files are absent from changedFilePaths and their language records + // have already cascaded; existing HDL argument sites still need declassification. + if (!removedFiles && !changedFilePaths.some(file => detectLanguage(file, undefined, overrides) === 'verilog')) return 0; + const fresh = new Set(changedFilePaths); + const candidates = this.queries.getHdlCallArgumentEdges().filter(e => !fresh.has(e.sourceFilePath)); + const ids: number[] = []; + const refs: UnresolvedReference[] = []; + for (const edge of candidates) { + const ref = resurrectRefFromDroppedEdge(edge); + if (ref) { ids.push(edge.edgeId); refs.push(ref); } + } + if (refs.length) this.queries.replaceResolutionEdgesWithUnresolvedRefs(ids, refs); + return refs.length; + } + /** * Re-open resolution edges whose answer was READ from a file this sync * touched, although neither end of the edge changed: a python attribute typed @@ -3727,6 +3833,8 @@ export class ExtractionOrchestrator { /** Inspect changed inputs before deletion/re-extraction cascades their edges. */ onFileChange?: (filePath: string, content?: string) => void ): Promise { + this.prepareHdlContext(); + if (this.hdlForce) scopedPaths = undefined; await initGrammars(); // Initialize WASM runtime (grammars loaded lazily below) const startTime = Date.now(); let filesChecked = 0; @@ -3785,7 +3893,7 @@ export class ExtractionOrchestrator { currentFiles = unique.filter( (p) => isSourceFile(p, overrides, this.rootDir) && - !scope.ignores(p) && + (!scope.ignores(p) || this.hdlContext!.sources.has(p)) && fs.existsSync(path.join(this.rootDir, p)) ); trackedFiles = []; @@ -3812,6 +3920,9 @@ export class ExtractionOrchestrator { trackedFiles = this.queries.getAllFiles(); if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[phase-timing] sync-tracked-load: ${Date.now() - tTracked}ms (${trackedFiles.length} tracked)`); } + currentFiles = scopedPaths?.length + ? currentFiles.filter(file => this.hdlContext!.accepts(file)) + : this.hdlContext!.files(currentFiles); const currentSet = new Set(currentFiles); const trackedMap = new Map(); for (const f of trackedFiles) { @@ -3841,7 +3952,7 @@ export class ExtractionOrchestrator { .map((e) => resurrectRefFromDroppedEdge(e)) .filter((r): r is UnresolvedReference => r !== null); if (resurrected.length > 0) { - this.queries.insertUnresolvedRefsBatch(resurrected); + this.queries.replaceResolutionEdgesWithUnresolvedRefs([], resurrected); } } onFileChange?.(tracked.path); @@ -3896,7 +4007,7 @@ export class ExtractionOrchestrator { // change that preserves both exactly is the blind spot every mtime-based // incremental tool accepts; `index --force` is the escape hatch. Git bumps // mtime on every file it writes during checkout/merge, so pulls are caught.) - if (tracked && !neverParsed) { + if (tracked && !neverParsed && !this.forceHdlFile(filePath)) { try { const stat = fs.statSync(fullPath); if (stat.size === tracked.size && Math.floor(stat.mtimeMs) === Math.floor(tracked.modifiedAt)) { @@ -3936,7 +4047,7 @@ export class ExtractionOrchestrator { changedFilePaths.push(filePath); addedFilePaths.push(filePath); filesAdded++; - } else if (tracked.contentHash !== contentHash || neverParsed) { + } else if (tracked.contentHash !== contentHash || neverParsed || this.forceHdlFile(filePath)) { onFileChange?.(filePath, content); filesToIndex.push(filePath); changedFilePaths.push(filePath); @@ -4067,6 +4178,10 @@ export class ExtractionOrchestrator { * Uses git status as a fast path when available, falling back to full scan. */ getChangedFiles(): { added: string[]; modified: string[]; removed: string[] } { + const profile = loadHdlProfile(this.rootDir).profile; + const profileFiles = profile ? new Set(profile.files) : null; + const profileOverrides = loadExtensionOverrides(this.rootDir); + const allowed = (file: string) => !profileFiles || detectLanguage(file, undefined, profileOverrides) !== 'verilog' || profileFiles.has(file); // The commit this index was last brought up to date at. Absent on an index // built before stamping existed — getGitChangedFiles then declines the fast // path and the full scan below answers correctly, once, until a sync or a @@ -4087,12 +4202,26 @@ export class ExtractionOrchestrator { // Git supplies candidates, never the verdict. A committed deletion may // have been recreated locally; a previously indexed dirty path may have // vanished from git status after restore. Classify current disk vs DB once. - const candidates = new Set([...gitChanges.deleted, ...gitChanges.modified, ...gitChanges.added, ...dirtyPaths!]); + const candidates = new Set([...gitChanges.deleted, ...gitChanges.modified, ...gitChanges.added, ...dirtyPaths!, ...(profile?.files ?? [])]); const scope = this.scopedSyncMatcher(); const overrides = loadExtensionOverrides(this.rootDir); for (const filePath of candidates) { const tracked = this.queries.getFileByPath(filePath); + // A profile source outside the git candidate set is still one file to + // classify; a profile source is indexed whatever its size. + if (!allowed(filePath)) continue; const fullPath = path.join(this.rootDir, filePath); + if (profileFiles?.has(filePath)) { + try { + if (fs.statSync(fullPath).size > MAX_SOURCE_FILE_SIZE_BYTES) { + (tracked ? modified : added).push(filePath); + continue; + } + } catch { + if (tracked) removed.push(filePath); + continue; + } + } if (!isSourceFile(filePath, overrides, this.rootDir) || scope.ignores(filePath) || !fs.existsSync(fullPath)) { if (tracked) removed.push(filePath); continue; @@ -4117,7 +4246,8 @@ export class ExtractionOrchestrator { } // === Fallback: full scan (non-git project or git failure) === - const currentFiles = new Set(scanDirectory(this.rootDir)); + const scanned = scanDirectory(this.rootDir).filter(allowed); + const currentFiles = new Set(profile ? [...scanned, ...profile.files] : scanned); const trackedFiles = this.queries.getAllFiles(); // Build Map for O(1) lookups @@ -4140,6 +4270,14 @@ export class ExtractionOrchestrator { // Find added and modified files for (const filePath of currentFiles) { const fullPath = path.join(this.rootDir, filePath); + if (profileFiles?.has(filePath)) { + try { + if (fs.statSync(fullPath).size > MAX_SOURCE_FILE_SIZE_BYTES) { + (trackedMap.has(filePath) ? modified : added).push(filePath); + continue; + } + } catch { continue; } + } let content: string | null; try { content = readSourceOrStamp(fullPath); diff --git a/src/extraction/languages/index.ts b/src/extraction/languages/index.ts index 6b760b01d6..ed11e19084 100644 --- a/src/extraction/languages/index.ts +++ b/src/extraction/languages/index.ts @@ -24,6 +24,7 @@ import { dartExtractor } from './dart'; import { pascalExtractor } from './pascal'; import { scalaExtractor } from './scala'; import { luaExtractor } from './lua'; +import { verilogExtractor } from './verilog'; import { rExtractor } from './r'; import { luauExtractor } from './luau'; import { objcExtractor } from './objc'; @@ -57,6 +58,7 @@ export const EXTRACTORS: Partial> = { pascal: pascalExtractor, scala: scalaExtractor, lua: luaExtractor, + verilog: verilogExtractor, r: rExtractor, luau: luauExtractor, objc: objcExtractor, diff --git a/src/extraction/languages/verilog-call-access.ts b/src/extraction/languages/verilog-call-access.ts new file mode 100644 index 0000000000..7c5b374d5b --- /dev/null +++ b/src/extraction/languages/verilog-call-access.ts @@ -0,0 +1,80 @@ +import type { Node as SyntaxNode } from 'web-tree-sitter'; +import type { ExtractorContext } from '../tree-sitter-types'; + +export const VERILOG_CALL_ARGUMENT_PREFIX = 'hdl:call-arg:'; +export interface VerilogCallArgument { + callee: string; + callableId: string; + scope: string; + argument: number | string; + slots: Array<{ key: number | string; omitted: boolean }>; + role: 'value' | 'index'; + writeTarget: boolean; + referenceTarget: boolean; +} + +function assignable(node: SyntaxNode, allowConcat = true): boolean { + if (node.type === 'expression') return node.namedChildCount === 1 && !!node.firstNamedChild && assignable(node.firstNamedChild, allowConcat); + if (node.type === 'primary') { + const first = node.firstNamedChild; + return !!first && assignable(first, allowConcat) && node.namedChildren.slice(1).every(n => n.type === 'select' + && !n.namedChildren.some(c => ['simple_identifier', 'escaped_identifier'].includes(c.type))); + } + if (node.type === 'hierarchical_identifier') return node.namedChildCount === 1 + && ['simple_identifier', 'escaped_identifier'].includes(node.firstNamedChild?.type ?? ''); + if (node.type === 'concatenation') return allowConcat && node.namedChildren.length > 0 && node.namedChildren.every(n => assignable(n, allowConcat)); + return false; +} + +/** Annotate an already discovered local signal occurrence; never emit a second edge. */ +export function getVerilogCallArgumentCandidates(id: SyntaxNode, ctx: ExtractorContext): string[] { + let argument: SyntaxNode = id; + let role: VerilogCallArgument['role'] = 'value'; + while (argument.parent && argument.parent.type !== 'list_of_arguments') { + argument = argument.parent; + if (['select', 'constant_select', 'bit_select', 'constant_bit_select', 'part_select_range'].includes(argument.type)) role = 'index'; + if (['function_declaration', 'task_declaration', 'module_declaration'].includes(argument.type)) return []; + } + const list = argument.parent; + if (!list || list.hasError || argument.type !== 'expression') return []; + const call = list.parent?.type === 'method_call_body' ? list.parent.parent : list.parent; + if (!call || call.hasError) return []; + let callee: string | undefined; + if (call.type === 'tf_call') callee = call.namedChildren.find(n => n.type === 'hierarchical_identifier')?.text.trim(); + if (call.type === 'method_call') { + const receiver = call.namedChildren.find(n => n.type === 'primary'); + const name = list.parent?.childForFieldName('name'); + if (receiver && name && /^\s*::\s*$/.test(ctx.source.slice(receiver.endIndex, name.startIndex))) callee = `${receiver.text.trim()}::${name.text.trim()}`; + } + if (!callee) return []; + const segments: SyntaxNode[][] = [[]]; + for (const child of list.children) { + if (child.type === ',') segments.push([]); + else if (!['comment', 'block_comment', 'one_line_comment'].includes(child.type)) segments[segments.length - 1]!.push(child); + } + const slots: VerilogCallArgument['slots'] = []; + let selected: number | string | undefined; + let namedSeen = false; + for (let index = 0; index < segments.length; index++) { + const segment = segments[index]!; + const named = segment[0]?.type === '.'; + const expression = segment.find(n => n.type === 'expression'); + let key: number | string = index; + if (named) { + namedSeen = true; + const name = segment.find(n => ['simple_identifier', 'escaped_identifier'].includes(n.type)); + if (!name) return []; + key = name.text.trim(); + } else if (namedSeen) return []; // A positional actual cannot follow named ones. + slots.push({ key, omitted: !expression }); + if (expression?.id === argument.id) selected = key; + } + if (selected === undefined) return []; + const lexical = ctx.nodes.find(n => n.id === ctx.nodeStack[ctx.nodeStack.length - 1]); + const owner = [...ctx.nodeStack].reverse().map(nodeId => ctx.nodes.find(n => n.id === nodeId)) + .find(n => n?.kind === 'function' || n?.kind === 'method') ?? lexical; + if (!owner || !lexical) return []; + const metadata: VerilogCallArgument = { callee, callableId: owner.id, scope: lexical.qualifiedName, + argument: selected, slots, role, writeTarget: assignable(argument), referenceTarget: assignable(argument, false) }; + return [`${VERILOG_CALL_ARGUMENT_PREFIX}${JSON.stringify(metadata)}`]; +} diff --git a/src/extraction/languages/verilog-generate.ts b/src/extraction/languages/verilog-generate.ts new file mode 100644 index 0000000000..6a5c1f55c4 --- /dev/null +++ b/src/extraction/languages/verilog-generate.ts @@ -0,0 +1,33 @@ +import type { Node as SyntaxNode } from 'web-tree-sitter'; +import type { ExtractorContext } from '../tree-sitter-types'; +import { getNodeText } from '../tree-sitter-helpers'; +import { addVerilogSignalReferences } from './verilog-signals'; + +function identifier(node: SyntaxNode): SyntaxNode | undefined { + return node.namedChildren.find(n => ['simple_identifier', 'escaped_identifier'].includes(n.type)); +} + +/** A generate iteration is a declaration template, never a guessed elaborated value. */ +export function initializeVerilogGenerateScope(block: SyntaxNode, ctx: ExtractorContext): void { + const loop = block.parent; + if (loop?.type !== 'loop_generate_construct') return; + const init = loop.namedChildren.find(n => n.type === 'genvar_initialization'); + const name = init && identifier(init); + if (!init || !name) return; + const binding = ctx.createNode('constant', getNodeText(name, ctx.source), init, { + signature: getNodeText(init, ctx.source), decorators: ['hdl:generate-parameter', 'hdl:template'], + }); + if (binding) addVerilogSignalReferences(init, ctx, binding.id); +} + +export function visitVerilogGenvarDeclaration(node: SyntaxNode, ctx: ExtractorContext): boolean { + if (node.type !== 'genvar_declaration') return false; + const list = node.namedChildren.find(n => n.type === 'list_of_genvar_identifiers'); + for (const name of list?.namedChildren ?? []) { + if (!['simple_identifier', 'escaped_identifier'].includes(name.type)) continue; + ctx.createNode('constant', getNodeText(name, ctx.source), name, { + signature: getNodeText(node, ctx.source), decorators: ['hdl:genvar', 'hdl:template'], + }); + } + return true; +} diff --git a/src/extraction/languages/verilog-packages.ts b/src/extraction/languages/verilog-packages.ts new file mode 100644 index 0000000000..5440c479e7 --- /dev/null +++ b/src/extraction/languages/verilog-packages.ts @@ -0,0 +1,46 @@ +import type { Node as SyntaxNode } from 'web-tree-sitter'; +import type { ExtractorContext } from '../tree-sitter-types'; + +/** Keep qualification: a package or hierarchical call must never become a bare global call. */ +export function getVerilogCallName(node: SyntaxNode, _source: string): string | undefined { + const callee = node.namedChildren.find(c => c.type === 'hierarchical_identifier' || c.type === 'simple_identifier'); + return callee?.text.trim(); +} + +/** Package calls use method_call in this grammar; modports are named interface views. */ +export function handleVerilogPackageNode(node: SyntaxNode, ctx: ExtractorContext): boolean { + if (node.type === 'modport_item') { + const name = node.namedChildren.find(c => c.type === 'simple_identifier'); + const view = name ? ctx.createNode('interface', name.text, node, { signature: `modport ${node.text.trim()}` }) : null; + if (view) { + const visit = (part: SyntaxNode): void => { + if (part.type === 'modport_simple_port' && part.namedChildCount === 1 && part.firstNamedChild?.type === 'simple_identifier') { + ctx.addUnresolvedReference({ fromNodeId: view.id, + referenceName: `hdl:signal:${part.firstNamedChild.text}`, referenceKind: 'references', + line: part.startPosition.row + 1, column: part.startPosition.column }); + } else for (const child of part.namedChildren) visit(child); + }; + visit(node); + } + return true; + } + if (node.type !== 'method_call') return false; + const primary = node.namedChildren.find(c => c.type === 'primary'); + const body = node.namedChildren.find(c => c.type === 'method_call_body'); + const name = body?.childForFieldName('name'); + // Require the literal package separator between AST nodes, excluding object methods. + if (primary && name && /^\s*::\s*$/.test(ctx.source.slice(primary.endIndex, name.startIndex))) { + const lexical = ctx.nodes.find(n => n.id === ctx.nodeStack[ctx.nodeStack.length - 1]); + const callable = [...ctx.nodeStack].reverse().map(id => ctx.nodes.find(n => n.id === id)) + .find(n => n && (n.kind === 'function' || n.kind === 'method')); + const fromNodeId = callable?.id ?? lexical?.id; + if (fromNodeId) ctx.addUnresolvedReference({ fromNodeId, + ...(lexical?.kind === 'namespace' && lexical.id !== fromNodeId + ? { candidates: [`hdl:scope:${lexical.qualifiedName}`] } : {}), + referenceName: `${primary.text.trim()}::${name.text}`, referenceKind: 'calls', + line: node.startPosition.row + 1, column: node.startPosition.column }); + for (const child of body!.namedChildren) if (child.id !== name.id) ctx.visitNode(child); + return true; + } + return false; +} diff --git a/src/extraction/languages/verilog-signals.ts b/src/extraction/languages/verilog-signals.ts new file mode 100644 index 0000000000..fdfd40ff92 --- /dev/null +++ b/src/extraction/languages/verilog-signals.ts @@ -0,0 +1,387 @@ +import { getVerilogCallArgumentCandidates } from './verilog-call-access'; +import type { Node as SyntaxNode } from 'web-tree-sitter'; +import type { ExtractorContext } from '../tree-sitter-types'; +import { getNodeText } from '../tree-sitter-helpers'; + +function children(node: SyntaxNode): SyntaxNode[] { + return node.namedChildren; +} +function identifier(node: SyntaxNode): SyntaxNode | undefined { + return children(node).find(n => n.type === 'simple_identifier' || n.type === 'escaped_identifier'); +} +function walkChildren(node: SyntaxNode, ctx: ExtractorContext): void { + for (const child of children(node)) ctx.visitNode(child); +} +function direction(node: SyntaxNode, ctx: ExtractorContext): string | undefined { + for (const child of children(node)) { + if (child.type === 'port_direction') return getNodeText(child, ctx.source); + const nested = direction(child, ctx); + if (nested) return nested; + } + return undefined; +} + +/** Authoritative source header order for positional instance connections. + * null means the syntax is incomplete or needs preprocessing/elaboration. + * [] explicitly describes a module with no ports. Body declaration order is + * never used: legacy non-ANSI declarations can legally be reordered. + */ +export function getVerilogPortOrder(node: SyntaxNode, source: string): string[] | null { + const headers = new Set(['module_ansi_header', 'module_nonansi_header', 'module_header', + 'interface_ansi_header', 'interface_nonansi_header', 'program_ansi_header', 'program_nonansi_header']); + const header = children(node).find(n => headers.has(n.type)); + if (!header || node.hasError) return null; + const uncertain = (current: SyntaxNode): boolean => { + if (current.type.endsWith('_comment') || current.type === 'comment') return false; + return current.isMissing || current.type.includes('directive') || current.type.includes('macro') + || children(current).some(uncertain); + }; + if (uncertain(header)) return null; + const list = children(header).find(n => ['list_of_ports', 'list_of_port_declarations'].includes(n.type)); + if (!list) return []; + const names: string[] = []; + const seen = new Set(); + for (const port of children(list)) { + if (port.type.endsWith('_comment') || port.type === 'comment') continue; + let id: SyntaxNode | null | undefined; + if (port.type === 'ansi_port_declaration') { + if (getNodeText(port, source).trimStart().startsWith('.')) return null; + id = port.childForFieldName('port_name'); + } else if (port.type === 'port') { + id = identifier(port); + if (!id || children(port).length !== 1 || getNodeText(port, source).trim() !== getNodeText(id, source).trim()) return null; + } else return null; + if (!id || !['simple_identifier', 'escaped_identifier'].includes(id.type)) return null; + const name = getNodeText(id, source).trim(); + // Escaping an otherwise simple name does not create a distinct HDL port. + const canonical = name.replace(/^\\/, ''); + if (!canonical || seen.has(canonical)) return null; + seen.add(canonical); + names.push(name); + } + return names; +} + +/** Generate iteration variables are implicit local parameters in their body. + * Do not let an identically named outer signal stand in for that binding. */ +export function isVerilogGenerateBinding(node: SyntaxNode, ctx: ExtractorContext, name: string): boolean { + for (let ancestor = node.parent; ancestor; ancestor = ancestor.parent) { + if (ancestor.type === 'loop_generate_construct') { + const init = children(ancestor).find(n => n.type === 'genvar_initialization'); + const id = init && identifier(init); + if (id && getNodeText(id, ctx.source) === name) { + for (let i = ctx.nodeStack.length - 1; i >= 0; i--) { + const scope = ctx.nodes.find(n => n.id === ctx.nodeStack[i]); + if (scope && ctx.nodes.some(n => n.qualifiedName === `${scope.qualifiedName}::${name}` && n.decorators?.includes('hdl:generate-parameter'))) return false; + } + return true; + } + } + if (['module_declaration', 'interface_declaration', 'program_declaration'].includes(ancestor.type)) break; + } + return false; +} + +/** Describe the syntactic role of an occurrence, not simulated drive/race behavior. + * Calls and other unknown constructs remain ordinary references unless a + * separate argument resolver can prove their formal direction. + */ +function signalAccess(id: SyntaxNode, source: string): string[] { + let lhs = false; + const role = (kind: string): string[] => { + const roles = [`hdl:access:${kind}`]; + if (kind === 'write' || kind === 'readwrite') { + for (let context: SyntaxNode | null = id; context; context = context.parent) { + if (['statement_item', 'list_of_arguments'].includes(context.type)) break; + if (['cond_predicate', 'case_expression', 'case_item_expression'].includes(context.type) + || (context.type === 'expression' && ['loop_statement', 'wait_statement'].includes(context.parent?.type ?? ''))) { + roles.push('hdl:access:control'); + break; + } + } + } + return roles; + }; + for (let current: SyntaxNode | null = id; current; current = current.parent) { + if (['select', 'constant_select', 'bit_select', 'constant_bit_select', 'constant_indexed_range', 'constant_range'].includes(current.type)) return role('read'); + if (['variable_lvalue', 'net_lvalue'].includes(current.type)) lhs = true; + if (current.type === 'list_of_arguments') return []; + if (current.type === 'inc_or_dec_expression') return lhs ? role('readwrite') : []; + if (['operator_assignment', 'blocking_assignment', 'nonblocking_assignment', 'net_assignment', 'variable_assignment'].includes(current.type)) { + const op = children(current).find(n => n.type === 'assignment_operator'); + return role(lhs ? (op && getNodeText(op, source).trim() !== '=' ? 'readwrite' : 'write') : 'read'); + } + if (current.type === 'delay_control') return role('read'); + if (current.type === 'jump_statement' && current.children.some(n => n.type === 'return')) return role('read'); + if (current.type === 'event_expression') { + const iff = current.children.find(n => n.type === 'iff'); + if (iff && id.startIndex > iff.endIndex) return role('control'); + const edge = children(current).find(n => n.type === 'edge_identifier'); + const text = edge && getNodeText(edge, source).trim(); + return [...role('event'), ...(['posedge', 'negedge'].includes(text ?? '') ? [`hdl:event:${text}`] : [])]; + } + if (['cond_predicate', 'case_expression', 'case_item_expression'].includes(current.type) + || (current.type === 'expression' && ['loop_statement', 'wait_statement'].includes(current.parent?.type ?? ''))) return role('control'); + if (['variable_decl_assignment', 'net_decl_assignment', 'for_variable_declaration', 'tf_port_item'].includes(current.type)) return role('read'); + if (['statement_item', 'function_body_declaration', 'task_body_declaration', 'module_declaration'].includes(current.type)) return []; + } + return []; +} + +function hasIncompleteProceduralScope(node: SyntaxNode): boolean { + for (let ancestor: SyntaxNode | null = node; ancestor; ancestor = ancestor.parent) { + if (['always_construct', 'initial_construct', 'final_construct', 'function_declaration', 'task_declaration', 'seq_block', 'par_block', 'loop_statement'].includes(ancestor.type) && ancestor.hasError) return true; + if (['module_declaration', 'interface_declaration', 'program_declaration'].includes(ancestor.type)) break; + } + return false; +} + +/** Emit exact syntactic occurrences and bounded source access roles. + * A dedicated resolver must bind these only in the lexical scope of the owner. + * Dotted hierarchical accesses are deliberately excluded: their root is not + * sufficient evidence that the terminal signal belongs to this module. + */ +export function addVerilogSignalReferences(node: SyntaxNode, ctx: ExtractorContext, ownerId: string, lexicalScope = false): void { + // Parser recovery can lose a declaration while retaining its later uses. + // Keep the source nodes, but do not bind names from an incomplete procedural + // region to outer declarations. An unrelated valid process remains usable. + if (hasIncompleteProceduralScope(node)) return; + + const skip = new Set(['data_declaration', 'net_declaration', 'for_variable_declaration', 'tf_port_list', 'tf_port_declaration', 'tf_call', 'ps_or_hierarchical_function_identifier']); + const emit = (id: SyntaxNode): void => { + const name = getNodeText(id, ctx.source).trim(); + if (isVerilogGenerateBinding(id, ctx, name)) return; + const syntax = signalAccess(id, ctx.source); + // i++ remains a read/write occurrence even when its resulting value is + // passed to an input formal. Formal direction cannot erase that side effect. + const explicitEffect = syntax.some(c => c === 'hdl:access:write' || c === 'hdl:access:readwrite'); + ctx.addUnresolvedReference({ fromNodeId: ownerId, referenceName: `hdl:signal:${name}`, + referenceKind: 'references', line: id.startPosition.row + 1, column: id.startPosition.column, + candidates: [...syntax, ...(explicitEffect ? [] : getVerilogCallArgumentCandidates(id, ctx))] }); + }; + const visit = (current: SyntaxNode): void => { + if (lexicalScope && current.id !== node.id && ['seq_block', 'par_block', 'loop_statement'].includes(current.type)) return; + // A package-qualified call is also parsed as method_call. Its receiver + // may name a package, never evidence for an identically named local port. + if (current.type === 'method_call') { + const body = children(current).find(n => n.type === 'method_call_body'); + const args = body?.namedChildren.find(n => n.type === 'list_of_arguments'); + if (args) visit(args); + return; + } + if (skip.has(current.type)) { + // Function arguments still contain ordinary signal expressions. + if (current.type === 'tf_call') { + for (const child of children(current)) if (child.type === 'list_of_arguments') visit(child); + } + return; + } + if (['variable_lvalue', 'primary', 'constant_primary'].includes(current.type) && children(current).some(n => n.type.endsWith('_scope') || n.type === 'implicit_class_handle')) { + // p::x (on either side) is a package variable, even though the grammar wraps x in a + // single hierarchical_identifier. Keep index expressions, not its name. + for (const child of children(current)) { + if (['select', 'constant_select', 'variable_lvalue'].includes(child.type)) visit(child); + } + return; + } + if (current.type === 'net_lvalue') { + const id = identifier(current); + const select = children(current).find(n => n.type === 'constant_select'); + // The grammar puts `.member` identifiers directly inside constant_select, + // whereas bit/part-select expressions have their own nested AST nodes. + const hasMember = select && children(select).some(n => ['simple_identifier', 'escaped_identifier'].includes(n.type)); + const hasQualifier = children(current).some(n => n.id !== id?.id && !['constant_select', 'net_lvalue'].includes(n.type)); + if (id && !hasMember && !hasQualifier) emit(id); + // Concatenations recurse through nested net_lvalue; bounds recurse through + // constant_select. Bare member identifiers are never emitted by this walk. + for (const child of children(current)) if (child.id !== id?.id) visit(child); + return; + } + if (current.type === 'hierarchical_identifier' || current.type === 'constant_primary' || current.type === 'delay_value') { + const id = identifier(current); + // A single AST identifier is local evidence; qualified/member expressions + // must not collapse to a terminal name merely because that signal exists. + if (id && children(current).length === 1 && getNodeText(id, ctx.source).trim() === getNodeText(current, ctx.source).trim()) { + emit(id); + return; + } + if (current.type === 'hierarchical_identifier') { + for (const child of children(current)) { + if (['constant_bit_select', 'bit_select', 'select'].includes(child.type)) visit(child); + } + return; + } + } + for (const child of children(current)) visit(child); + }; + visit(node); +} + +/** Source-order/direction evidence for function/task actual arguments. */ +function formalMetadata(node: SyntaxNode, id: SyntaxNode, source: string): string[] { + for (let ancestor: SyntaxNode | null = node; ancestor; ancestor = ancestor.parent) { + if (['function_declaration', 'task_declaration'].includes(ancestor.type)) { + if (ancestor.hasError) return []; + break; + } + } + const uncertain = (part: SyntaxNode): boolean => !part.type.endsWith('_comment') + && (part.type.includes('directive') || part.type.includes('macro') || children(part).some(uncertain)); + if (node.type === 'tf_port_item' && node.parent && uncertain(node.parent)) return []; + const declarations: SyntaxNode[] = []; + if (node.type === 'tf_port_item') { + declarations.push(...(node.parent?.namedChildren.filter(n => n.type === 'tf_port_item') ?? [])); + } else if (node.type === 'tf_port_declaration') { + const body = node.parent?.parent; + if (!body || !['task_body_declaration', 'function_body_declaration'].includes(body.type) || uncertain(body)) return []; + for (const item of children(body)) { + declarations.push(...children(item).filter(n => n.type === 'tf_port_declaration')); + } + } else return []; + let index = 0; + let inherited = 'input'; + for (const declaration of declarations) { + const directionNode = children(declaration).find(n => n.type === 'tf_port_direction'); + const direct = directionNode && getNodeText(directionNode, source).trim().replace(/\s+/g, '-'); + const direction = direct ?? (node.type === 'tf_port_item' ? inherited : 'input'); + inherited = direction; + const list = declaration.type === 'tf_port_declaration' + ? children(declaration).find(n => n.type === 'list_of_tf_variable_identifiers') : declaration; + for (const name of list ? children(list).filter(n => ['simple_identifier', 'escaped_identifier'].includes(n.type)) : []) { + if (name.id === id.id) { + const hasDefault = name.nextNamedSibling?.type === 'expression'; + return [`hdl:formal-index:${index}`, ...(['input', 'output', 'inout', 'ref', 'const-ref'].includes(direction) ? [`hdl:direction:${direction}`] : []), ...(hasDefault ? ['hdl:default'] : [])]; + } + index++; + } + } + return []; +} + +/** HDL declarations and executable blocks; called before generic recursion. */ +export function visitVerilogSignals(node: SyntaxNode, ctx: ExtractorContext): boolean { + if (['seq_block', 'par_block', 'loop_statement'].includes(node.type)) { + // Generate blocks have their own handler; these nodes are procedural scopes. + const keyword = node.children[0]?.text ?? 'block'; + const label = node.type === 'loop_statement' ? undefined : identifier(node); + const kind = node.type === 'loop_statement' ? keyword : node.type === 'par_block' ? 'fork' : 'block'; + const name = label ? getNodeText(label, ctx.source) : `${kind}@${node.startPosition.row + 1}:${node.startPosition.column}`; + const created = ctx.createNode('namespace', name, node, { decorators: ['hdl:procedural-scope'] }); + if (created) { + ctx.pushScope(created.id); + addVerilogSignalReferences(node, ctx, created.id, true); + } + walkChildren(node, ctx); + if (created) ctx.popScope(); + return true; + } + if (['tf_port_item', 'tf_port_declaration', 'for_variable_declaration', 'loop_variables'].includes(node.type)) { + const list = node.type === 'tf_port_declaration' + ? children(node).find(n => n.type === 'list_of_tf_variable_identifiers') : node; + const ids = list ? children(list).filter(n => ['simple_identifier', 'escaped_identifier'].includes(n.type)) : []; + for (const id of ids) { + const created = ctx.createNode('field', getNodeText(id, ctx.source), id, + { signature: getNodeText(node, ctx.source), decorators: [node.type.startsWith('tf_') ? 'hdl:formal' : 'hdl:signal', ...formalMetadata(node, id, ctx.source)] }); + // Declarator initializers belong to this declaration's lexical scope. + if (created) { + for (let next = id.nextNamedSibling; next && !ids.some(n => n.id === next!.id); next = next.nextNamedSibling) { + if (next.type === 'expression' || next.type === 'constant_expression') addVerilogSignalReferences(next, ctx, created.id, true); + } + } + } + // Visit each initializer once with the enclosing callable as call owner. + for (const child of list ? children(list) : []) { + if (child.type === 'expression' || child.type === 'constant_expression') ctx.visitNode(child); + } + return true; + } + + if (node.type === 'ansi_port_declaration') { + const name = node.childForFieldName('port_name'); + if (name) { + let dir = direction(node, ctx); + // ANSI comma continuations inherit direction from the preceding port. + let previous = node.previousNamedSibling; + const hasHeader = children(node).some(n => n.type.endsWith('_port_header')); + while (!dir && !hasHeader && previous?.type === 'ansi_port_declaration') { + dir = direction(previous, ctx); + if (children(previous).some(n => n.type.endsWith('_port_header'))) break; + previous = previous.previousNamedSibling; + } + const decorators = ['hdl:port', ...(dir ? [`hdl:${dir}`] : [])]; + const iface = children(node).find(n => n.type === 'interface_port_header'); + for (const [field, tag] of [['interface_name', 'interface'], ['modport_name', 'modport']] as const) { + const value = iface?.childForFieldName(field); + if (value) decorators.push(`hdl:${tag}:${getNodeText(value, ctx.source)}`); + } + const created = ctx.createNode('field', getNodeText(name, ctx.source), node, + { signature: getNodeText(node, ctx.source), decorators }); + const typeName = iface?.childForFieldName('interface_name'); + const modport = iface?.childForFieldName('modport_name'); + if (created && typeName) ctx.addUnresolvedReference({ + fromNodeId: created.id, referenceName: getNodeText(typeName, ctx.source) + (modport ? `.${getNodeText(modport, ctx.source)}` : ''), + referenceKind: 'type_of', line: node.startPosition.row + 1, column: node.startPosition.column, + }); + } + return true; + } + if (node.type === 'port_declaration') { + for (const decl of children(node)) { + const dir = decl.type.replace(/_declaration$/, ''); + for (const list of children(decl).filter(n => /list_of_.*port_identifiers/.test(n.type))) { + for (const id of children(list).filter(n => ['simple_identifier', 'escaped_identifier'].includes(n.type))) { + ctx.createNode('field', getNodeText(id, ctx.source), id, + { signature: getNodeText(node, ctx.source), decorators: ['hdl:port', `hdl:${dir}`] }); + } + } + } + return true; + } + if (node.type === 'net_declaration' || node.type === 'data_declaration') { + const lists = children(node).filter(n => ['list_of_net_decl_assignments', 'list_of_variable_decl_assignments'].includes(n.type)); + if (!lists.length) return false; // e.g. typedef: let its structural handler run. + for (const list of lists) for (const decl of children(list)) { + const name = decl.childForFieldName('name') ?? identifier(decl); + if (name) { + const text = getNodeText(name, ctx.source); + const parent = ctx.nodes.find(n => n.id === ctx.nodeStack[ctx.nodeStack.length - 1]); + const qualified = parent ? `${parent.qualifiedName}::${text}` : text; + // A non-ANSI output may legally be redeclared as reg in the body. + const port = ctx.nodes.find(n => n.qualifiedName === qualified && n.decorators?.includes('hdl:port')); + const created = port ?? ctx.createNode('field', text, decl, + { signature: getNodeText(node, ctx.source), decorators: ['hdl:signal'] }); + if (created) { + // The '=' token proves a source initialization; dimensions and bare + // declarations are dependencies/declarations, not writes to the name. + if (!decl.hasError && !hasIncompleteProceduralScope(decl) && decl.children.some(n => n.type === '=')) { + ctx.addUnresolvedReference({ fromNodeId: created.id, referenceName: `hdl:signal:${text}`, + referenceKind: 'references', line: name.startPosition.row + 1, column: name.startPosition.column, + candidates: ['hdl:access:write'] }); + } + for (const child of children(decl)) { + if (child.id !== name.id) addVerilogSignalReferences(child, ctx, created.id); + } + } + } + } + walkChildren(node, ctx); + return true; + } + if (['always_construct', 'initial_construct', 'final_construct', 'continuous_assign'].includes(node.type)) { + const keyword = children(node).find(n => n.type === 'always_keyword'); + const kind = node.type === 'always_construct' + ? (keyword ? getNodeText(keyword, ctx.source) : 'always') + : node.type === 'continuous_assign' ? 'assign' : node.type.replace(/_construct$/, ''); + const name = `${kind}@${node.startPosition.row + 1}:${node.startPosition.column}`; + const created = ctx.createNode('function', name, node, + { signature: getNodeText(node, ctx.source), decorators: ['hdl:process', `hdl:${kind}`] }); + if (created) { + ctx.pushScope(created.id); + addVerilogSignalReferences(node, ctx, created.id, true); + } + walkChildren(node, ctx); + if (created) ctx.popScope(); + return true; + } + return false; +} diff --git a/src/extraction/languages/verilog.ts b/src/extraction/languages/verilog.ts new file mode 100644 index 0000000000..4d0ba7a735 --- /dev/null +++ b/src/extraction/languages/verilog.ts @@ -0,0 +1,455 @@ +import { initializeVerilogGenerateScope, visitVerilogGenvarDeclaration } from './verilog-generate'; +import { getVerilogPortOrder, visitVerilogSignals, addVerilogSignalReferences, isVerilogGenerateBinding } from './verilog-signals'; +import { handleVerilogPackageNode, getVerilogCallName } from './verilog-packages'; +import type { Node as SyntaxNode } from 'web-tree-sitter'; +import { getNodeText, getChildByField } from '../tree-sitter-helpers'; +import type { LanguageExtractor, ExtractorContext } from '../tree-sitter-types'; + +/** + * Verilog / SystemVerilog extractor (grammar: tree-sitter-systemverilog, ABI 15). + * + * The SystemVerilog grammar nests declared names deep inside header/body + * wrapper nodes (`module_declaration → module_ansi_header → name`, + * `function_declaration → function_body_declaration → name`), so the generic + * field/first-identifier name resolution can't reach them. Everything + * structural is therefore handled in a custom `visitNode` hook that resolves + * names explicitly, manages scope, and drives its own child recursion. + * + * What the hook emits: + * - module / interface / program / package / class → container nodes (kind + * `class`/`interface`) that scope their members. + * - function / task → `function` nodes (their bodies are walked for calls). + * - module instances → named `variable` nodes with original connections and + * `instantiates` edges (parent module and instance → module type). + * - generate blocks → source scopes; arrays and loops are not elaborated. + * - parameter / localparam → `constant` nodes. + * - `define macro → `constant` node (a `.vh` header is mostly macros, and a + * `WIDTH` written as `` `DATA_W `` has to land somewhere). + * - typedef → `type_alias` node. + * - function/task subroutine calls (`tf_call`) → `calls` references. + * + * Package imports and `` `include `` directives go through the generic import + * path (`importTypes` + `extractImport`): each `package_import_item` so every + * package in a multi-import (`import a::*, b::*;`) gets its own node, and each + * `include_compiler_directive` with the quoted path as its module name — the + * file-path matcher resolves `"defs.vh"` / `"axi/typedef.svh"` to the indexed + * header closest to the including file, the same way a C `#include` lands. + * Ports, signals and executable blocks are handled by verilog-signals; source + * occurrences are distinct from elaborated connectivity or drive direction. + */ + +// Header wrappers that carry a module/interface/program name. +const HEADER_TYPES = [ + 'module_ansi_header', + 'module_nonansi_header', + 'module_header', + 'interface_ansi_header', + 'interface_nonansi_header', + 'program_ansi_header', + 'program_nonansi_header', +]; + +function firstChildOfType(node: SyntaxNode, types: string[]): SyntaxNode | null { + for (let i = 0; i < node.namedChildCount; i++) { + const c = node.namedChild(i); + if (c && types.includes(c.type)) return c; + } + return null; +} + +function firstSimpleIdentifier(node: SyntaxNode): SyntaxNode | null { + for (let i = 0; i < node.namedChildCount; i++) { + const c = node.namedChild(i); + if (c && c.type === 'simple_identifier') return c; + } + return null; +} + +/** + * Module type names are resolved against module declarations. Function calls + * retain their package or hierarchy qualification in verilog-packages. + */ +function trailingSegment(name: string): string { + const sep = Math.max(name.lastIndexOf('::'), name.lastIndexOf('.')); + return sep >= 0 ? name.slice(sep).replace(/^[:.]+/, '') : name; +} + +function visitNamedChildren(node: SyntaxNode, ctx: ExtractorContext): void { + for (let i = 0; i < node.namedChildCount; i++) { + const c = node.namedChild(i); + if (c) ctx.visitNode(c); + } +} + +/** + * Resolve a declared name that may sit directly on the node (package/class), + * on a header child (module/interface/program), or as the first identifier. + */ +function declName(node: SyntaxNode, source: string): string | undefined { + const direct = getChildByField(node, 'name'); + if (direct) return getNodeText(direct, source); + const header = firstChildOfType(node, HEADER_TYPES); + if (header) { + const hn = getChildByField(header, 'name') ?? firstSimpleIdentifier(header); + if (hn) return getNodeText(hn, source); + } + const si = firstSimpleIdentifier(node); + return si ? getNodeText(si, source) : undefined; +} + +function handleContainer( + node: SyntaxNode, + ctx: ExtractorContext, + kind: 'class' | 'interface' +): boolean { + const name = declName(node, ctx.source); + if (!name) { + visitNamedChildren(node, ctx); + return true; + } + const order = getVerilogPortOrder(node, ctx.source); + const created = ctx.createNode(kind, name, node, order === null ? undefined + : { decorators: [`hdl:port-order:${JSON.stringify(order)}`] }); + if (created) ctx.pushScope(created.id); + visitNamedChildren(node, ctx); + if (created) ctx.popScope(); + return true; +} + +function handleSubroutine(node: SyntaxNode, ctx: ExtractorContext): boolean { + const bodyDecl = firstChildOfType(node, [ + 'function_body_declaration', + 'task_body_declaration', + ]); + const nameNode = bodyDecl + ? getChildByField(bodyDecl, 'name') ?? firstSimpleIdentifier(bodyDecl) + : getChildByField(node, 'name'); + const name = nameNode ? getNodeText(nameNode, ctx.source) : undefined; + if (!name) { + visitNamedChildren(node, ctx); + return true; + } + + let signature: string | undefined; + if (bodyDecl) { + const ports = firstChildOfType(bodyDecl, ['tf_port_list']); + const ret = firstChildOfType(bodyDecl, ['data_type_or_void']); + const portText = ports ? getNodeText(ports, ctx.source) : ''; + const retText = ret ? getNodeText(ret, ctx.source).trim() : ''; + signature = `${retText} (${portText})`.trim(); + } + + const created = ctx.createNode('function', name, node, { signature }); + if (created) ctx.pushScope(created.id); + if (created && bodyDecl) addVerilogSignalReferences(bodyDecl, ctx, created.id, true); + visitNamedChildren(node, ctx); + if (created) ctx.popScope(); + return true; +} + +function handleGenerateBlock(node: SyntaxNode, ctx: ExtractorContext): boolean { + const nameNode = getChildByField(node, 'name'); + // Anonymous labels identify source scopes, not elaborated genblk numbers. + const name = nameNode ? getNodeText(nameNode, ctx.source) + : `generate@${node.startPosition.row + 1}:${node.startPosition.column}`; + const loop = node.parent?.type === 'loop_generate_construct' ? node.parent : null; + const created = ctx.createNode('namespace', name, node, { + decorators: ['hdl:generate-scope'], + ...(loop ? { signature: ctx.source.slice(loop.startIndex, node.startIndex).trim() } : {}), + }); + if (created) { + ctx.pushScope(created.id); + initializeVerilogGenerateScope(node, ctx); + } + visitNamedChildren(node, ctx); + if (created) ctx.popScope(); + return true; +} + +/** Direct AST commas delimit slots; nested expression commas stay inside their node. */ +function positionalConnections(node: SyntaxNode, ctx: ExtractorContext, leading: SyntaxNode[] = []): Array<{ position: SyntaxNode; text: string }> | null { + if (node.hasError || node.namedChildren.some(n => !['ordered_port_connection', 'comment', 'block_comment', 'one_line_comment'].includes(n.type))) return null; + const slots: Array<{ position: SyntaxNode; text: string }> = []; + let start = leading[0]?.startIndex ?? node.startIndex; + let position: SyntaxNode = leading[0] ?? node; + for (const child of [...leading.flatMap(n => n.children), ...node.children]) { + if (child.type === ',') { + slots.push({ position, text: ctx.source.slice(start, child.startIndex).trim() }); + start = child.endIndex; + position = child; + } else if (child.type === 'ordered_port_connection') position = child; + } + slots.push({ position, text: ctx.source.slice(start, node.endIndex).trim() }); + return slots; +} + +function handleInstantiation(node: SyntaxNode, ctx: ExtractorContext): boolean { + const typeNode = getChildByField(node, 'instance_type') ?? firstSimpleIdentifier(node); + if (!typeNode) { + visitNamedChildren(node, ctx); + return true; + } + const typeName = getNodeText(typeNode, ctx.source).trim(); + const moduleName = trailingSegment(typeName); + const addReference = (fromId: string, position: SyntaxNode): void => { + ctx.addUnresolvedReference({ + fromNodeId: fromId, + referenceName: moduleName, + referenceKind: 'instantiates', + line: position.startPosition.row + 1, + column: position.startPosition.column, + }); + }; + // Keep the module-to-module flow even when a generate scope contains this + // declaration. Individual instances additionally provide navigable evidence. + for (let i = ctx.nodeStack.length - 1; i >= 0; i--) { + const parent = ctx.nodes.find(n => n.id === ctx.nodeStack[i]); + if (parent && (parent.kind === 'class' || parent.kind === 'interface')) { + addReference(parent.id, node); + break; + } + } + const parameters = firstChildOfType(node, ['parameter_value_assignment']); + const prefix = `${typeName}${parameters ? ' ' + getNodeText(parameters, ctx.source) : ''}`; + for (let i = 0; i < node.namedChildCount; i++) { + const child = node.namedChild(i); + if (!child) continue; + if (child.type !== 'hierarchical_instance') { + ctx.visitNode(child); + continue; + } + const nameOfInstance = firstChildOfType(child, ['name_of_instance']); + const nameNode = nameOfInstance && (getChildByField(nameOfInstance, 'instance_name') + ?? firstSimpleIdentifier(nameOfInstance)); + const name = nameNode ? getNodeText(nameNode, ctx.source) : undefined; + const created = name ? ctx.createNode('variable', name, child, { + // Preserve arrays, named/ordered/shorthand connections and parameter + // overrides as source evidence; do not infer elaborated wire endpoints. + signature: `${prefix} ${getNodeText(child, ctx.source)}`, + }) : null; + if (created) { + addReference(created.id, child); + addVerilogSignalReferences(child, ctx, created.id); + ctx.pushScope(created.id); + const connections = firstChildOfType(child, ['list_of_port_connections']); + // This grammar wraps leading empty-slot commas in ERROR. Accept only + // those literal separators; all other parse errors/mixed lists stay opaque. + const leadingCommas = child.namedChildren.filter(n => n.type === 'ERROR' + && connections && n.endIndex <= connections.startIndex && /^\s*,+\s*$/.test(getNodeText(n, ctx.source))); + const allowedErrors = new Set(leadingCommas.map(n => n.id)); + const soundTree = (part: SyntaxNode): boolean => !part.isMissing && (part.type === 'ERROR' + ? allowedErrors.has(part.id) : part.namedChildren.every(soundTree)); + const validConnections = soundTree(node) + && child.namedChildren.filter(n => n.type === 'list_of_port_connections').length === 1; + const slots = validConnections && connections ? positionalConnections(connections, ctx, leadingCommas) : null; + if (slots) slots.forEach((slot, index) => { + const binding = ctx.createNode('property', `port[${index}]`, slot.position, { + signature: slot.text, decorators: ['hdl:connection', `hdl:position:${index}`], + }); + if (binding) { + ctx.addUnresolvedReference({ fromNodeId: binding.id, + referenceName: `hdl:port-position:${JSON.stringify([moduleName, index, slots.length, created.id])}`, + referenceKind: 'references', line: slot.position.startPosition.row + 1, + column: slot.position.startPosition.column }); + if (slot.position.type === 'ordered_port_connection') addVerilogSignalReferences(slot.position, ctx, binding.id); + } + }); + for (const connection of validConnections ? connections?.namedChildren ?? [] : []) { + if (connection.type !== 'named_port_connection') continue; + const port = getChildByField(connection, 'port_name'); + if (!port) { + if (!getNodeText(connection, ctx.source).includes('.*')) continue; + const excluded = (connections?.namedChildren ?? []).flatMap(n => { + const formal = getChildByField(n, 'port_name'); + return formal ? [getNodeText(formal, ctx.source)] : []; + }); + // An implicit generate parameter shadows an outer same-name signal. + for (let ancestor = connection.parent; ancestor; ancestor = ancestor.parent) { + if (ancestor.type === 'loop_generate_construct') { + const init = firstChildOfType(ancestor, ['genvar_initialization']); + const id = init && firstSimpleIdentifier(init); + if (id) excluded.push(getNodeText(id, ctx.source)); + } + if (['module_declaration', 'interface_declaration', 'program_declaration'].includes(ancestor.type)) break; + } + const wildcard = ctx.createNode('property', '*', connection, { + signature: getNodeText(connection, ctx.source), decorators: ['hdl:connection', 'hdl:wildcard'], + }); + if (wildcard) ctx.addUnresolvedReference({ fromNodeId: wildcard.id, + referenceName: `hdl:wildcard:${JSON.stringify([moduleName, created.id, excluded])}`, + referenceKind: 'references', line: connection.startPosition.row + 1, column: connection.startPosition.column }); + continue; + } + const portName = getNodeText(port, ctx.source); + const binding = ctx.createNode('property', portName, connection, { + signature: getNodeText(connection, ctx.source), decorators: ['hdl:connection', 'hdl:named'], + }); + if (binding) { + ctx.addUnresolvedReference({ fromNodeId: binding.id, + referenceName: `hdl:port:${JSON.stringify([moduleName, portName, created.id])}`, + referenceKind: 'references', line: port.startPosition.row + 1, column: port.startPosition.column }); + addVerilogSignalReferences(connection, ctx, binding.id); + } + // Shorthand .port connects the same local name. Empty .port() has a + // formal endpoint but no local expression; no drive direction inferred. + if (!isVerilogGenerateBinding(connection, ctx, portName) + && /^\s*\.\s*[^()\s]+\s*$/.test(getNodeText(connection, ctx.source))) { + for (const owner of [created, ...(binding ? [binding] : [])]) { + ctx.addUnresolvedReference({ fromNodeId: owner.id, + referenceName: `hdl:signal:${portName}`, referenceKind: 'references', + line: port.startPosition.row + 1, column: port.startPosition.column }); + } + } + } + } + visitNamedChildren(child, ctx); + if (created) ctx.popScope(); + } + return true; +} + +function handleParam(node: SyntaxNode, ctx: ExtractorContext): boolean { + const list = firstChildOfType(node, ['list_of_param_assignments']); + if (list) { + for (let i = 0; i < list.namedChildCount; i++) { + const assign = list.namedChild(i); + if (!assign || assign.type !== 'param_assignment') continue; + const id = firstSimpleIdentifier(assign); + if (!id) continue; + const created = ctx.createNode('constant', getNodeText(id, ctx.source), assign); + if (!created) continue; + // `localparam K = (N + 1) * 2;` depends on N: the value expression's + // identifiers become scoped references FROM the parameter, so impact on + // N follows the derived-parameter chain (N -> K -> M). The declared name + // itself is skipped; the shared walk already leaves out callee names and + // package qualifiers, and the resolver binds only in the lexical scope. + for (let j = 0; j < assign.namedChildCount; j++) { + const part = assign.namedChild(j); + if (part && part.id !== id.id) addVerilogSignalReferences(part, ctx, created.id); + } + } + } + return true; +} + +function handleMacro(node: SyntaxNode, ctx: ExtractorContext): boolean { + const nameNode = firstChildOfType(node, ['text_macro_name']); + const id = nameNode ? firstSimpleIdentifier(nameNode) : null; + if (id) ctx.createNode('constant', getNodeText(id, ctx.source), node); + return true; +} + +function handleTypedef(node: SyntaxNode, ctx: ExtractorContext): boolean { + const nameNode = + getChildByField(node, 'type_name') ?? + getChildByField(node, 'name') ?? + firstSimpleIdentifier(node); + const name = nameNode ? getNodeText(nameNode, ctx.source) : undefined; + if (name) ctx.createNode('type_alias', name, node); + return true; +} + +function handleCall(node: SyntaxNode, ctx: ExtractorContext): boolean { + if (ctx.nodeStack.length > 0) { + const lexical = ctx.nodes.find(n => n.id === ctx.nodeStack[ctx.nodeStack.length - 1]); + const callable = [...ctx.nodeStack].reverse().map(id => ctx.nodes.find(n => n.id === id)) + .find(n => n && (n.kind === 'function' || n.kind === 'method')); + const fromId = callable?.id ?? lexical?.id; + const callee = firstChildOfType(node, ['hierarchical_identifier']) ?? firstSimpleIdentifier(node); + if (fromId && callee) { + const name = getVerilogCallName(node, ctx.source); + if (name) { + ctx.addUnresolvedReference({ + fromNodeId: fromId, + referenceName: name, + referenceKind: 'calls', + ...(lexical?.kind === 'namespace' && lexical.id !== fromId + ? { candidates: [`hdl:scope:${lexical.qualifiedName}`] } : {}), + line: node.startPosition.row + 1, + column: node.startPosition.column, + }); + } + } + } + // Walk arguments so nested calls (`caller(helper(x))`) get their own refs. + visitNamedChildren(node, ctx); + return true; +} + +export const verilogExtractor: LanguageExtractor = { + // The visitNode hook below owns dispatch for all structural constructs; + // these generic arrays stay empty except imports, which reuse the core path. + functionTypes: [], + classTypes: [], + methodTypes: [], + interfaceTypes: [], + structTypes: [], + enumTypes: [], + typeAliasTypes: [], + // Target the per-item node, not the whole declaration, so every package in a + // multi-import (`import a::*, b::*;`) is indexed — one import node per item. + importTypes: ['package_import_item', 'include_compiler_directive'], + callTypes: [], + variableTypes: [], + nameField: 'name', + bodyField: 'body', + paramsField: 'tf_port_list', + + visitNode: (node, ctx) => { + if (visitVerilogGenvarDeclaration(node, ctx) || visitVerilogSignals(node, ctx) || handleVerilogPackageNode(node, ctx)) return true; + switch (node.type) { + case 'module_declaration': + case 'program_declaration': + case 'package_declaration': + case 'class_declaration': + return handleContainer(node, ctx, 'class'); + case 'interface_declaration': + return handleContainer(node, ctx, 'interface'); + case 'function_declaration': + case 'task_declaration': + return handleSubroutine(node, ctx); + case 'generate_block': + return handleGenerateBlock(node, ctx); + case 'module_instantiation': + return handleInstantiation(node, ctx); + case 'parameter_declaration': + case 'local_parameter_declaration': + return handleParam(node, ctx); + case 'text_macro_definition': + return handleMacro(node, ctx); + case 'type_declaration': + return handleTypedef(node, ctx); + case 'tf_call': + return handleCall(node, ctx); + default: + return false; + } + }, + + extractImport: (node, source) => { + if (node.type === 'include_compiler_directive') { + // `include "defs.vh"` → the quoted path; `include ` → the + // bracketed one. The path is the module name, so the generic import ref + // resolves it to the header FILE by path suffix (name-matcher's + // file-path strategy), never to a symbol. + const quoted = firstChildOfType(node, ['quoted_string']); + const item = quoted ? firstChildOfType(quoted, ['quoted_string_item']) : null; + const system = firstChildOfType(node, ['system_lib_string']); + const target = item + ? getNodeText(item, source) + : system + ? getNodeText(system, source).replace(/^<|>$/g, '') + : ''; + if (!target.trim()) return null; + return { moduleName: target.trim(), signature: getNodeText(node, source).trim() }; + } + // `node` is a package_import_item (`pkg::*` / `pkg::name`); the package name + // is its first simple_identifier. One item → one import node. + const id = firstSimpleIdentifier(node); + if (!id) return null; + return { + moduleName: getNodeText(id, source), + signature: getNodeText(node, source).trim(), + }; + }, +}; diff --git a/src/extraction/wasm/tree-sitter-systemverilog.wasm b/src/extraction/wasm/tree-sitter-systemverilog.wasm new file mode 100644 index 0000000000..ad8201395e Binary files /dev/null and b/src/extraction/wasm/tree-sitter-systemverilog.wasm differ diff --git a/src/graph/named-symbol-flow.ts b/src/graph/named-symbol-flow.ts index 34ae8adf56..c1078cd4ff 100644 --- a/src/graph/named-symbol-flow.ts +++ b/src/graph/named-symbol-flow.ts @@ -134,6 +134,15 @@ export const FLOW_CALLABLE_KINDS: ReadonlySet = new Set([ */ export const FLOW_EDGE_KINDS: ReadonlySet = new Set(['calls', 'navigates']); +function canSitOnFlow(n: { kind: string; language?: string }): boolean { + return FLOW_CALLABLE_KINDS.has(n.kind) || (n.kind === 'class' && n.language === 'verilog'); +} + +function carriesFlow(edge: { kind: string }, target: { language?: string }): boolean { + return FLOW_EDGE_KINDS.has(edge.kind) || (edge.kind === 'instantiates' && target.language === 'verilog'); +} + + /** * Node kinds that can be an endpoint of a SYNTHESIZED edge without being * callable. An RTK thunk is `const X = createAsyncThunk(...)`, so a thunk → @@ -311,7 +320,7 @@ export function resolveNamedTokens( for (const t of tokens) { const hits = findAllSymbols(cg, t).nodes; - const cands = hits.filter((n) => FLOW_CALLABLE_KINDS.has(n.kind)); + const cands = hits.filter((n) => canSitOnFlow(n)); out.tokenFamily.set(t, cands); // A qualified or otherwise-specific name (<=3 hits) keeps all of them. const specific = cands.length <= 3; @@ -345,7 +354,7 @@ export function resolveNamedTokens( if (out.dynNamed.size < 12) { let tokenDyn = 0; for (const n of hits) { - if (FLOW_CALLABLE_KINDS.has(n.kind) || !DYN_KINDS.has(n.kind) || out.dynNamed.has(n.id)) { + if (canSitOnFlow(n) || !DYN_KINDS.has(n.kind) || out.dynNamed.has(n.id)) { continue; } if (hasHeuristicEdge(n.id)) { @@ -444,7 +453,7 @@ function walkCalls( if (id !== seed.id && named.has(id)) reached.push(id); if (at.depth >= maxHops - 1) continue; for (const c of cg.getCallees(id)) { - if (!FLOW_EDGE_KINDS.has(c.edge.kind)) continue; + if (!carriesFlow(c.edge, c.node)) continue; const isNamed = named.has(c.node.id); // A route node is a connector, not a symbol the reader would have named: // crossing one costs no bridge budget. @@ -528,7 +537,7 @@ function walkBidirectional( const next: Node[] = []; for (const node of frontF) { for (const c of cg.getCallees(node.id)) { - if (!FLOW_EDGE_KINDS.has(c.edge.kind) || forward.has(c.node.id)) continue; + if (!carriesFlow(c.edge, c.node) || forward.has(c.node.id)) continue; forward.set(c.node.id, { prev: node.id, edge: c.edge, node: c.node }); next.push(c.node); } @@ -540,7 +549,7 @@ function walkBidirectional( const next: Node[] = []; for (const node of frontB) { for (const c of cg.getCallers(node.id)) { - if (!FLOW_EDGE_KINDS.has(c.edge.kind) || backward.has(c.node.id)) continue; + if (!carriesFlow(c.edge, c.node) || backward.has(c.node.id)) continue; backward.set(c.node.id, { next: node.id, edge: c.edge }); backNodes.set(c.node.id, c.node); next.push(c.node); diff --git a/src/hdl/index-context.ts b/src/hdl/index-context.ts new file mode 100644 index 0000000000..d3e9acdda9 --- /dev/null +++ b/src/hdl/index-context.ts @@ -0,0 +1,132 @@ +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import { createHash } from 'node:crypto'; +import { loadHdlProfile, type ActiveHdlProfile } from './profile'; +import { EXTRACTION_VERSION } from '../extraction/extraction-version'; +import { preprocessVerilog, type HdlPreprocessDiagnostic } from './preprocess'; +import { detectLanguage } from '../extraction/grammars'; +import { loadExtensionOverrides } from '../project-config'; + +export interface HdlDependency { path: string; sha256: string | null } +const hash = (text: string) => createHash('sha256').update(text).digest('hex'); +function inside(root: string, file: string): string { + const relative = path.relative(root, file); + if (relative.startsWith('..' + path.sep) || relative === '..' || path.isAbsolute(relative)) throw new Error('HDL path escapes project root'); + return relative.split(path.sep).join('/'); +} + +/** Immutable per-operation conditional source view. Original bytes remain file hashes/snippets. */ +export class HdlIndexContext { + readonly profile: ActiveHdlProfile | null; + readonly diagnostics: HdlPreprocessDiagnostic[] = []; + readonly dependencies: HdlDependency[]; + readonly fingerprint: string; + readonly sources = new Map(); + readonly unitDiagnostics = new Map(); + private readonly root: string; + private readonly overrides: ReturnType; + + constructor(root: string, previouslyProfiled: boolean) { + this.root = path.resolve(root); + this.overrides = loadExtensionOverrides(root); + const loaded = loadHdlProfile(root); + if (loaded.status === 'invalid-profile' || (loaded.status === 'invalid-config' && previouslyProfiled)) { + throw new Error(`Invalid HDL profile: ${loaded.diagnostics.join('; ')}`); + } + this.profile = loaded.profile; + if (!this.profile) { this.dependencies = []; this.fingerprint = 'raw'; return; } + this.root = fs.realpathSync(root); + const dependency = new Map(); + const bytes = new Map(); + const snapshots = new Map(); + let total = 0; + const read = (relative: string): string | null => { + const lexical = path.resolve(this.root, relative); + const normalized = inside(this.root, lexical); + if (normalized.split('/').some(p => ['.git', '.jj', '.codegraph'].includes(p) || p.startsWith('.codegraph-'))) throw new Error('HDL profile cannot read VCS/index storage'); + if (bytes.has(normalized)) return bytes.get(normalized)!; + try { + inside(this.root, fs.realpathSync(lexical)); + const stat = fs.statSync(lexical); + if (!stat.isFile() || stat.size > 1024 * 1024) throw new Error(`HDL input is not a regular file of at most 1 MiB: ${normalized}`); + const text = fs.readFileSync(lexical, 'utf8'); + const after = fs.statSync(lexical); + if (stat.size !== after.size || stat.mtimeMs !== after.mtimeMs || stat.ino !== after.ino) { + throw new Error(`HDL input changed while preparing snapshot; retry: ${normalized}`); + } + snapshots.set(normalized, after); + total += Buffer.byteLength(text); + if (total > 64 * 1024 * 1024) throw new Error('HDL profile source/dependency snapshot exceeds 64 MiB'); + bytes.set(normalized, text); + return text; + } catch (error) { + if ((error as NodeJS.ErrnoException).code === 'ENOENT') return null; + throw error; + } + }; + for (const file of this.profile.files) { + if (detectLanguage(file, undefined, this.overrides) !== 'verilog') throw new Error(`HDL profile lists a non-HDL source: ${file}`); + const source = read(file); + if (source === null) throw new Error(`HDL source missing: ${file}`); + const result = preprocessVerilog(source, { filePath: file, defines: this.profile.defines, includeDirs: this.profile.includeDirs, + readInclude: (request, from, dirs) => { + if (path.isAbsolute(request) || /^[A-Za-z]:[\\/]/.test(request)) throw new Error('Absolute HDL include paths are outside the supported project-relative profile'); + const candidates = [path.posix.join(path.posix.dirname(from), request), ...dirs.map(dir => path.posix.join(dir, request))]; + for (const candidate of [...new Set(candidates)]) { + const relative = inside(this.root, path.resolve(this.root, candidate)); + const text = read(relative); + dependency.set(relative, text === null ? null : hash(text)); + if (text !== null) return { filePath: relative, source: text }; + } + return null; + } }); + if (result.source.length !== source.length) throw new Error('HDL conditional selection changed source offsets'); + this.sources.set(file, { original: source, selected: result.source, stats: snapshots.get(file)! }); + const diagnostics = result.diagnostics.slice(0, 32); + if (result.diagnostics.length > diagnostics.length) diagnostics.push({ filePath: file, line: 1, + code: 'diagnostic-limit', message: `${result.diagnostics.length - diagnostics.length} additional preprocessing diagnostics omitted for this unit.` }); + this.unitDiagnostics.set(file, diagnostics); + this.diagnostics.push(...diagnostics.slice(0, Math.max(0, 200 - this.diagnostics.length))); + } + this.dependencies = [...dependency].sort(([a], [b]) => a.localeCompare(b)).map(([path, sha256]) => ({ path, sha256 })); + this.fingerprint = hash(JSON.stringify([EXTRACTION_VERSION, this.profile.fingerprint, this.dependencies])); + } + + isHdl(file: string): boolean { return detectLanguage(file, undefined, this.overrides) === 'verilog'; } + accepts(file: string): boolean { return !this.profile || !this.isHdl(file) || this.sources.has(file); } + files(scanned: string[]): string[] { + return this.profile ? [...scanned.filter(file => !this.isHdl(file)), ...this.profile.files] : scanned; + } + source(file: string, original: string): string { + if (!this.profile || !this.isHdl(file)) return original; + const cached = this.sources.get(file); + if (!cached) throw new Error(`HDL source is outside active profile: ${file}`); + if (cached.original !== original) throw new Error(`HDL source changed during profile preparation; retry indexing: ${file}`); + return cached.selected; + } + context(): string { + return JSON.stringify(this.profile ? { mode: 'profile', configurationFingerprint: this.profile.fingerprint, + files: this.profile.files, includeDirs: this.profile.includeDirs, defines: this.profile.defines, + topModules: this.profile.topModules, languageMode: this.profile.languageMode, + dependencies: this.dependencies, unitDiagnostics: Object.fromEntries(this.unitDiagnostics), diagnostics: this.diagnostics.slice(0, 200), + diagnosticCount: this.diagnostics.length, incomplete: this.diagnostics.length > 0, + } : { mode: 'raw' }); + } +} + +/** Include changes can invalidate a profile even when codegraph.json is unchanged. */ +export function hdlDependenciesChanged(root: string, context: string | null): boolean { + let value: unknown; + try { value = context ? JSON.parse(context) : null; } catch { return true; } + if (!value || typeof value !== 'object' || !('dependencies' in value) || !Array.isArray(value.dependencies)) return false; + const canonicalRoot = fs.realpathSync(root); + return value.dependencies.some((entry: HdlDependency) => { + if (!entry || typeof entry.path !== 'string') return true; + try { + const full = path.resolve(canonicalRoot, entry.path); + inside(canonicalRoot, full); inside(canonicalRoot, fs.realpathSync(full)); + const stat = fs.statSync(full); + return !stat.isFile() || stat.size > 1024 * 1024 || hash(fs.readFileSync(full, 'utf8')) !== entry.sha256; + } catch { return entry.sha256 !== null; } + }); +} diff --git a/src/hdl/preprocess.ts b/src/hdl/preprocess.ts new file mode 100644 index 0000000000..538854a3cc --- /dev/null +++ b/src/hdl/preprocess.ts @@ -0,0 +1,237 @@ +/** Bounded, source-preserving HDL conditional selection. No macro expansion. */ +export interface HdlPreprocessDiagnostic { + filePath: string; + line: number; + code: string; + message: string; +} +export interface HdlIncludeSource { filePath: string; source: string } +export interface HdlPreprocessOptions { + filePath: string; + defines?: Record; + includeDirs?: string[]; + /** The caller owns filesystem access and must enforce its allowed roots. */ + readInclude?: (request: string, fromFile: string, includeDirs: readonly string[]) => HdlIncludeSource | null; + maxIncludeDepth?: number; +} +export interface HdlPreprocessResult { + source: string; + dependencies: string[]; + diagnostics: HdlPreprocessDiagnostic[]; + /** All encountered constructs were handled within this bounded mode. */ + complete: boolean; +} +interface Conditional { + parent: boolean; + active: boolean; + taken: boolean; + uncertain: boolean; + elseSeen: boolean; +} +const IDENTIFIER = /^[A-Za-z_][A-Za-z0-9_$]*/; +const OTHER_DIRECTIVES = new Set(['timescale', 'default_nettype', 'celldefine', 'endcelldefine', + 'resetall', 'unconnected_drive', 'nounconnected_drive', 'line', 'begin_keywords', 'end_keywords', 'pragma']); + +/** Each call is a separate translation unit. Includes affect macro definedness, + * but are never pasted into the returned text. All offsets are original UTF-16 + * offsets; inactive text is replaced with spaces while CR/LF bytes stay put. */ +export function preprocessVerilog(source: string, options: HdlPreprocessOptions): HdlPreprocessResult { + const diagnostics: HdlPreprocessDiagnostic[] = []; + const record = (diagnostic: HdlPreprocessDiagnostic): void => { + if (diagnostics.length < 512) diagnostics.push(diagnostic); + else if (diagnostics.length === 512) diagnostics.push({ ...diagnostic, code: 'diagnostic-limit', + message: 'Additional preprocessing diagnostics were omitted after 512 entries.' }); + }; + const dependencies = new Set(); + const macros = new Map(); + let unknownMacros = false; + const trace = new Set(); + const requestedDepth = options.maxIncludeDepth; + const maxDepth = Number.isSafeInteger(requestedDepth) && requestedDepth! >= 0 ? Math.min(requestedDepth!, 256) : 64; + const invalidateMacros = (): void => { macros.clear(); unknownMacros = true; }; + const mask = (text: string): string => text.replace(/[^\r\n]/g, ' '); + for (const name of Object.keys(options.defines ?? {})) { + if (IDENTIFIER.exec(name)?.[0] === name) macros.set(name, true); + else record({ filePath: options.filePath, line: 1, code: 'invalid-profile-define', message: `Invalid macro name: ${name}` }); + } + + const process = (text: string, filePath: string, depth: number): string => { + const output = text.split(''); + const stack: Conditional[] = []; + let fatal = false; + const lineStarts = [0]; + for (let at = text.indexOf('\n'); at >= 0; at = text.indexOf('\n', at + 1)) lineStarts.push(at + 1); + const active = (): boolean => stack.length ? stack[stack.length - 1]!.active : true; + const report = (offset: number, code: string, message: string): void => { + let lo = 0, hi = lineStarts.length; + while (lo + 1 < hi) { const mid = (lo + hi) >>> 1; if (lineStarts[mid]! <= offset) lo = mid; else hi = mid; } + record({ filePath, line: lo + 1, code, message }); + }; + const blank = (start: number, end: number): void => { + for (let n = start; n < end; n++) if (text[n] !== '\n' && text[n] !== '\r') output[n] = ' '; + }; + const lineEnd = (start: number): number => { + let end = text.indexOf('\n', start); + if (end < 0) return text.length; + while (end > 0 && text[end - (text[end - 1] === '\r' ? 2 : 1)] === '\\') { + const next = text.indexOf('\n', end + 1); + if (next < 0) return text.length; + end = next; + } + return end; + }; + const nameAfter = (offset: number): { name?: string; end: number } => { + let start = offset; + while (text[start] === ' ' || text[start] === '\t') start++; + const name = IDENTIFIER.exec(text.slice(start))?.[0]; + return { name, end: name ? start + name.length : lineEnd(start) }; + }; + const defined = (name: string): boolean | undefined => macros.get(name) ?? (unknownMacros ? undefined : false); + const condition = (name: string | undefined, invert: boolean, offset: number): boolean | undefined => { + const value = name ? defined(name) : undefined; + if (value === undefined) { + report(offset, name ? 'unknown-conditional' : 'unsupported-conditional', name + ? `Cannot determine whether ${name} is defined; no branch was selected.` + : 'Only a simple macro identifier is supported in a conditional directive.'); + // Either unknown arm could define or undefine names used later. + invalidateMacros(); + return undefined; + } + return invert ? !value : value; + }; + + let i = 0; + while (i < text.length) { + const start = i; + // Directives inside comments, strings and escaped identifiers are text. + if (text.startsWith('//', i)) { + const end = text.indexOf('\n', i + 2); + i = end < 0 ? text.length : end; + } else if (text.startsWith('/*', i)) { + const end = text.indexOf('*/', i + 2); + i = end < 0 ? text.length : end + 2; + } else if (text[i] === '"') { + i++; + while (i < text.length) { + if (text[i] === '\\') { i += Math.min(2, text.length - i); continue; } + if (text[i] === '"') { i++; break; } + if (text[i] === '\n' || text[i] === '\r') break; + i++; + } + } else if (text[i] === '\\') { + i++; + while (i < text.length && !/\s/.test(text[i]!)) i++; + } else if (text[i] === '`') { + const directive = IDENTIFIER.exec(text.slice(i + 1))?.[0]; + if (!directive) { + if (active()) report(i, 'unsupported-macro-form', 'Unrecognized backtick form was preserved.'); + if (!active()) blank(i, i + 1); + i++; + continue; + } + const tokenEnd = i + 1 + directive.length; + if (['ifdef', 'ifndef', 'elsif'].includes(directive)) { + const argument = nameAfter(tokenEnd); + blank(i, argument.end); + if (directive !== 'elsif') { + const parent = active(); + const value = parent ? condition(argument.name, directive === 'ifndef', i) : false; + stack.push({ parent, active: parent && value === true, taken: value === true, + uncertain: value === undefined, elseSeen: false }); + } else { + const frame = stack[stack.length - 1]; + if (!frame || frame.elseSeen) { + report(i, 'malformed-conditional', '`elsif without a matching open conditional, or after `else.'); fatal = true; + } else if (!frame.parent || frame.taken || frame.uncertain) frame.active = false; + else { + const value = condition(argument.name, false, i); + frame.active = value === true; + frame.taken = value === true; + frame.uncertain = value === undefined; + } + } + i = argument.end; + continue; + } + if (directive === 'else' || directive === 'endif') { + blank(i, tokenEnd); + const frame = stack[stack.length - 1]; + if (!frame || (directive === 'else' && frame.elseSeen)) { + report(i, 'malformed-conditional', `Unexpected or duplicate \`${directive}.`); fatal = true; + } else if (directive === 'endif') stack.pop(); + else { frame.active = frame.parent && !frame.taken && !frame.uncertain; frame.taken ||= frame.active; frame.elseSeen = true; } + i = tokenEnd; + continue; + } + if (directive === 'define') { + const end = lineEnd(tokenEnd); + const argument = nameAfter(tokenEnd); + if (active()) { + if (argument.name) macros.set(argument.name, true); + else { report(i, 'unsupported-definition', 'Macro definition without a supported simple name was preserved.'); invalidateMacros(); } + } else blank(i, end); + i = end; + continue; + } + if (directive === 'undef') { + const argument = nameAfter(tokenEnd); + if (active()) { + if (argument.name) macros.set(argument.name, false); + else { report(i, 'unsupported-definition', 'Macro undefinition without a supported simple name was preserved.'); invalidateMacros(); } + } else blank(i, argument.end); + i = argument.end; + continue; + } + if (directive === 'undefineall') { + if (active()) { macros.clear(); unknownMacros = false; } else blank(i, tokenEnd); + i = tokenEnd; + continue; + } + if (directive === 'include') { + let cursor = tokenEnd; + while (text[cursor] === ' ' || text[cursor] === '\t') cursor++; + const opening = text[cursor]; + const closing = opening === '"' ? '"' : opening === '<' ? '>' : ''; + const end = closing ? text.indexOf(closing, cursor + 1) : -1; + const supported = end >= 0 && !/[\r\n]/.test(text.slice(cursor, end)); + const includeEnd = supported ? end + 1 : lineEnd(cursor); + if (active()) { + if (!supported) { + report(i, 'unsupported-include', 'Only literal quoted or bracketed include paths are supported.'); invalidateMacros(); + } else { + const request = text.slice(cursor + 1, end); + let included: HdlIncludeSource | null = null; + try { included = options.readInclude?.(request, filePath, options.includeDirs ?? []) ?? null; } + catch { report(i, 'include-read-failed', `Cannot read include ${request}.`); } + if (!included) { report(i, 'unresolved-include', `Include ${request} was not resolved; subsequent macro state is uncertain.`); invalidateMacros(); } + else { + dependencies.add(included.filePath); + const key = JSON.stringify([included.filePath, unknownMacros, [...macros].sort(([a], [b]) => a.localeCompare(b))]); + if (depth >= maxDepth || trace.has(key)) { + report(i, 'include-cycle-or-depth', `Include recursion cannot be resolved at ${included.filePath}.`); invalidateMacros(); + } else { trace.add(key); process(included.source, included.filePath, depth + 1); trace.delete(key); } + } + } + } else blank(i, includeEnd); + i = includeEnd; + continue; + } + if (active()) { + report(i, OTHER_DIRECTIVES.has(directive) ? 'unsupported-directive' : 'unsupported-macro-expansion', + `\`${directive} was preserved; ${OTHER_DIRECTIVES.has(directive) ? 'this compiler directive is not interpreted' : 'macro expansion is not supported'}.`); + // A replacement can itself contain `define/`undef/`include. Without + // expanding it, later macro definedness cannot be assumed unchanged. + if (!OTHER_DIRECTIVES.has(directive)) invalidateMacros(); + } else blank(i, tokenEnd); + i = tokenEnd; + continue; + } else i++; + if (!active()) blank(start, i); + } + if (stack.length) { report(text.length, 'malformed-conditional', 'Conditional directive has no matching `endif.'); fatal = true; } + if (fatal) { invalidateMacros(); return mask(text); } + return output.join(''); + }; + const selected = process(source, options.filePath, 0); + return { source: selected, dependencies: [...dependencies], diagnostics, complete: diagnostics.length === 0 }; +} diff --git a/src/hdl/profile.ts b/src/hdl/profile.ts new file mode 100644 index 0000000000..4d3babb926 --- /dev/null +++ b/src/hdl/profile.ts @@ -0,0 +1,213 @@ +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import { createHash } from 'node:crypto'; + +export interface ActiveHdlProfile { + name: string; + files: string[]; + includeDirs: string[]; + defines: Record; + topModules: string[]; + languageMode: 'verilog' | 'systemverilog'; + /** Filelist contents participate in the fingerprint; include contents belong to preprocessing dependencies. */ + dependencies: Array<{ path: string; sha256: string }>; + fingerprint: string; +} +export interface HdlProfileLoadResult { + status: 'none' | 'active' | 'invalid-config' | 'invalid-profile'; + profile: ActiveHdlProfile | null; + diagnostics: string[]; + configInvalid: boolean; + configPresent: boolean; +} +interface ProfileSpec { + files: string[]; + filelists: string[]; + includeDirs: string[]; + defines: Record; + topModules: string[]; + languageMode: 'verilog' | 'systemverilog'; +} +export function isReservedHdlPath(relativePath: string): boolean { + return relativePath.split(/[\\/]/).some(part => part === '.git' || part === '.jj' || part === '.codegraph' || part.startsWith('.codegraph-')); +} +const MAX_BYTES = 1024 * 1024, MAX_LISTS = 256, MAX_DEPTH = 32, MAX_TOKENS = 100000; +const hash = (text: string) => createHash('sha256').update(text).digest('hex'); +const object = (value: unknown): value is Record => !!value && typeof value === 'object' && !Array.isArray(value); +function fail(message: string): never { throw new Error(message); } +function strings(value: unknown, label: string): string[] { + if (value === undefined) return []; + if (!Array.isArray(value) || !value.every(v => typeof v === 'string' && v.trim().length > 0)) fail(`${label}: expected an array of non-empty strings`); + return value; +} +function strictKeys(value: Record, keys: string[], label: string): void { + for (const key of Object.keys(value)) if (!keys.includes(key)) fail(`${label}: unsupported field ${key}`); +} +function spec(value: unknown, label: string): ProfileSpec { + if (!object(value)) fail(`${label}: expected an object`); + strictKeys(value, ['files', 'filelists', 'includeDirs', 'defines', 'topModules', 'languageMode'], label); + const defines: Record = Object.create(null); + if (value.defines !== undefined) { + if (!object(value.defines)) fail(`${label}.defines: expected an object with string values`); + for (const [name, val] of Object.entries(value.defines)) { + if (!/^[A-Za-z_][A-Za-z0-9_$]*$/.test(name) || typeof val !== 'string') fail(`${label}.defines: invalid macro name or non-string value`); + defines[name] = val; + } + } + const languageMode = value.languageMode ?? 'systemverilog'; + if (languageMode !== 'verilog' && languageMode !== 'systemverilog') fail(`${label}.languageMode: expected verilog or systemverilog`); + return { files: strings(value.files, `${label}.files`), filelists: strings(value.filelists, `${label}.filelists`), + includeDirs: strings(value.includeDirs, `${label}.includeDirs`), defines, + topModules: strings(value.topModules, `${label}.topModules`), languageMode }; +} + +/** Preserve quoting so '+' inside a quoted define value or directory is not a separator. */ +function tokens(text: string): string[] { + const out: string[] = []; + let token = '', quote = ''; + const flush = () => { if (token) { out.push(token); token = ''; } }; + for (let i = 0; i < text.length; i++) { + const c = text[i]!, next = text[i + 1]; + if (c === '\\' && next !== undefined) { + if (next === '\n') { i++; continue; } + if (next === '\r' && text[i + 2] === '\n') { i += 2; continue; } + token += c + next; i++; continue; + } + if (quote) { token += c; if (c === quote) quote = ''; continue; } + // An apostrophe in 1'b1 / '0 is a Verilog literal, not a path quote. + if (c === '"' || (c === "'" && (!token || token.endsWith('+')))) { quote = c; token += c; continue; } + if ((c === '/' && next === '/') || (c === '#' && !token)) { + flush(); while (i < text.length && text[i] !== '\n') i++; continue; + } + if (c === '/' && next === '*') { + flush(); const end = text.indexOf('*/', i + 2); if (end < 0) fail('Unterminated filelist block comment'); i = end + 1; continue; + } + if (/\s/.test(c)) { flush(); continue; } + token += c; + } + if (quote) fail('Unterminated filelist quote'); + flush(); + return out; +} +function splitPlus(token: string): string[] { + const parts: string[] = []; + let part = '', quote = ''; + for (let i = 0; i < token.length; i++) { + const c = token[i]!; + if (c === '\\' && i + 1 < token.length) { part += c + token[++i]; continue; } + if (quote) { part += c; if (c === quote) quote = ''; continue; } + if (c === '"' || (c === "'" && !part)) { quote = c; part += c; continue; } + if (c === '+') { parts.push(part); part = ''; } else part += c; + } + parts.push(part); return parts; +} +function unquote(token: string): string { + let out = '', quote = ''; + for (let i = 0; i < token.length; i++) { + const c = token[i]!; + if (c === '\\' && i + 1 < token.length && /[\\"'\s]/.test(token[i + 1]!)) { out += token[++i]; continue; } + if (quote) { if (c === quote) quote = ''; else out += c; continue; } + if (c === '"' || (c === "'" && i === 0)) { quote = c; continue; } + out += c; + } + if (quote) fail('Unterminated filelist quote'); + return out; +} + +/** Load only a selected, explicit source profile. Invalid HDL never means "scan everything". + * No HDL section preserves legacy behavior. Malformed overall JSON is reported separately + * so the existing config loader can retain its policy and an indexed-profile guard can fail safely. + */ +export function loadHdlProfile(projectRoot: string): HdlProfileLoadResult { + const result = (status: HdlProfileLoadResult['status'], diagnostics: string[] = [], profile: ActiveHdlProfile | null = null): HdlProfileLoadResult => + ({ status, profile, diagnostics, configInvalid: status === 'invalid-config', configPresent: present }); + let present = false; + let root: string; + try { root = fs.realpathSync(projectRoot); } catch { return result('invalid-config', ['HDL project root is unavailable']); } + const contained = (candidate: string): string => { + if (isReservedHdlPath(path.relative(root, candidate))) fail('Reserved VCS/CodeGraph paths are unsupported in HDL profiles'); + const real = fs.realpathSync(candidate); + const rel = path.relative(root, real); + if (rel === '..' || rel.startsWith(`..${path.sep}`) || path.isAbsolute(rel)) fail('Paths outside the project are unsupported in HDL profiles'); + if (isReservedHdlPath(rel)) fail('Reserved VCS/CodeGraph paths are unsupported in HDL profiles'); + return real; + }; + const relative = (absolute: string) => path.relative(root, absolute).split(path.sep).join('/') || '.'; + const read = (absolute: string) => { + const stat = fs.statSync(absolute); + if (!stat.isFile() || stat.size > MAX_BYTES) fail('Config/filelist must be a regular file of at most 1 MiB'); + return fs.readFileSync(absolute, 'utf8'); + }; + let config: unknown; + try { + const file = path.join(root, 'codegraph.json'); + if (!fs.existsSync(file)) return result('none'); + present = true; + config = JSON.parse(read(contained(file))); + if (!object(config)) return result('invalid-config', ['codegraph.json: expected a JSON object']); + } catch (error) { return result('invalid-config', [`codegraph.json could not be read: ${(error as Error).message}`]); } + if (!Object.prototype.hasOwnProperty.call(config, 'hdl')) return result('none'); + try { + const hdl = config.hdl; + if (!object(hdl)) fail('codegraph.json hdl: expected an object'); + strictKeys(hdl, ['activeProfile', 'profiles'], 'hdl'); + if (typeof hdl.activeProfile !== 'string' || !hdl.activeProfile.trim()) fail('hdl.activeProfile: expected a non-empty profile name'); + if (!object(hdl.profiles) || !Object.prototype.hasOwnProperty.call(hdl.profiles, hdl.activeProfile)) fail('hdl.activeProfile does not select an existing profile'); + const selected = spec(hdl.profiles[hdl.activeProfile], `hdl.profiles.${hdl.activeProfile}`); + const files = new Set(), includeDirs = new Set(), dependencies = new Map(); + const defines: Record = Object.assign(Object.create(null), selected.defines); + const resolve = (name: string, base: string, directory: boolean) => { + const real = contained(path.resolve(base, name)); + const stat = fs.statSync(real); + if (directory ? !stat.isDirectory() : !stat.isFile()) fail(`HDL path has the wrong file type: ${relative(real)}`); + return real; + }; + const addFile = (name: string, base: string) => files.add(relative(resolve(name, base, false))); + const addInclude = (name: string, base: string) => includeDirs.add(relative(resolve(name, base, true))); + for (const file of selected.files) addFile(file, root); + for (const directory of selected.includeDirs) addInclude(directory, root); + let tokenCount = 0, listCount = 0; + const stack = new Set(); + const visited = new Set(); + const visit = (name: string, lookupBase: string, localPaths: boolean, depth: number): void => { + if (depth > MAX_DEPTH) fail('HDL filelist nesting exceeds 32 levels'); + const file = resolve(name, lookupBase, false), key = `${file}\0${localPaths}`; + if (stack.has(file)) fail(`HDL filelist cycle: ${relative(file)}`); + if (visited.has(key)) return; + if (++listCount > MAX_LISTS) fail('HDL profile exceeds 256 filelists'); + stack.add(file); visited.add(key); + const text = read(file), entries = tokens(text); + dependencies.set(relative(file), hash(text)); + tokenCount += entries.length; + if (tokenCount > MAX_TOKENS) fail('HDL profile exceeds 100000 filelist tokens'); + const base = localPaths ? path.dirname(file) : root; + for (let i = 0; i < entries.length; i++) { + const raw = entries[i]!, token = unquote(raw); + if (token === '-f' || token === '-F') { + const next = entries[++i]; if (!next) fail(`Missing argument for ${token}`); + visit(unquote(next), base, token === '-F', depth + 1); + } else if (raw.startsWith('+incdir+')) { + const dirs = splitPlus(raw.slice('+incdir+'.length)); + if (dirs.some(d => !d)) fail('Empty +incdir+ entry'); + for (const directory of dirs) addInclude(unquote(directory), base); + } else if (raw.startsWith('+define+')) { + for (const definition of splitPlus(raw.slice('+define+'.length))) { + const decoded = unquote(definition), eq = decoded.indexOf('='); + const macro = eq < 0 ? decoded : decoded.slice(0, eq), value = eq < 0 ? '1' : decoded.slice(eq + 1); + if (!/^[A-Za-z_][A-Za-z0-9_$]*$/.test(macro)) fail('Invalid +define+ macro name'); + if (Object.prototype.hasOwnProperty.call(defines, macro) && defines[macro] !== value) fail(`Conflicting values for HDL macro ${macro}`); + defines[macro] = value; + } + } else if (/^[+-]/.test(token)) fail(`Unsupported HDL filelist option: ${token.split('+').slice(0, 2).join('+')}`); + else addFile(token, base); + } + stack.delete(file); + }; + for (const filelist of selected.filelists) visit(filelist, root, path.extname(filelist) === '.F', 1); + const resolved = { name: hdl.activeProfile, files: [...files], includeDirs: [...includeDirs], + defines: Object.fromEntries(Object.entries(defines).sort(([a], [b]) => a.localeCompare(b))), + topModules: [...new Set(selected.topModules)], languageMode: selected.languageMode, + dependencies: [...dependencies].map(([file, sha256]) => ({ path: file, sha256 })).sort((a,b) => a.path.localeCompare(b.path)) }; + return result('active', [], { ...resolved, fingerprint: hash(JSON.stringify(resolved)) }); + } catch (error) { return result('invalid-profile', [(error as Error).message]); } +} diff --git a/src/hdl/semantic-import.ts b/src/hdl/semantic-import.ts new file mode 100644 index 0000000000..bab8abcb85 --- /dev/null +++ b/src/hdl/semantic-import.ts @@ -0,0 +1,234 @@ +import * as path from 'node:path'; + +export interface HdlMacroPoint { file: string; line: number; column: number; byteOffset: number } +export interface HdlMacroFrame { name: string | null; argument: boolean; spelling?: HdlMacroPoint; invocation?: { start: HdlMacroPoint; end: HdlMacroPoint } } + +export type HdlExpressionRole = 'initializer' | 'declared-initializer' | 'type'; +export interface HdlExpressionOrigin { + role: HdlExpressionRole; + sourceOrigin: 'macro'; + source?: { file: string; line: number; column: number | null }; + macroExpansion: HdlMacroFrame[]; + macroExpansionComplete: boolean; +} +export interface HdlExpressionCoverage { + initializer: 'checked' | 'not-applicable' | 'unavailable' | 'command-line'; + declaredInitializer: 'checked' | 'not-applicable' | 'unavailable'; + type: 'checked' | 'not-applicable' | 'unavailable'; + truncated: boolean; +} + +export interface HdlSemanticFact { + kind: 'parameter' | 'port' | 'type'; + name: string; + instancePath: string; + value?: string; + type?: string; + width?: number; + direction?: string; + source?: { file: string; line: number; column: number | null }; + sourceOrigin?: 'direct' | 'macro'; + macroExpansion?: HdlMacroFrame[]; + macroExpansionComplete?: boolean; + expressionOrigins?: HdlExpressionOrigin[]; + expressionOriginCoverage?: HdlExpressionCoverage; +} + +type AstObject = Record; +const object = (value: unknown): value is AstObject => typeof value === 'object' && value !== null && !Array.isArray(value); +function fail(message: string): never { throw new Error(`Invalid or unsupported slang AST: ${message}`); } +const MAX_DEPTH = 128; +const MAX_NODES = 1_000_000; +const MAX_FACTS = 100_000; +const MAX_TEXT = 64 * 1024 * 1024; +const INTEGER_WIDTHS = new Map(Object.entries({ bit: 1, logic: 1, reg: 1, byte: 8, shortint: 16, int: 32, integer: 32, longint: 64, time: 64 })); + +/** Import facts from successful frontend output; the caller must check compiler + * exit/diagnostics first (slang can emit JSON even when compilation failed). + * Addresses are temporary reference keys only, never public identities. */ +export function importSlangSemantics(ast: unknown, sourceRoot: string): HdlSemanticFact[] { + if (!object(ast) || !object(ast.design) || ast.design.kind !== 'Root' || !Array.isArray(ast.design.members)) fail('expected design Root with members'); + const design = ast.design as AstObject; + const root = path.resolve(sourceRoot); + const catalog = new Map(); + const active = new Set(); + const seen = new Set(); + const pending: Array<{ value: unknown; depth: number; exit?: boolean }> = [{ value: ast, depth: 0 }]; + let nodeCount = 0, textSize = 0; + while (pending.length) { + const { value, depth, exit } = pending.pop()!; + if (exit) { active.delete(value as object); continue; } + if (++nodeCount > MAX_NODES || depth > MAX_DEPTH) fail('input size/depth limit exceeded'); + if (typeof value === 'string') { textSize += value.length; if (textSize > MAX_TEXT) fail('input text limit exceeded'); } + if (!object(value) && !Array.isArray(value)) continue; + if (active.has(value)) fail('cyclic input object'); + if (seen.has(value)) continue; + seen.add(value); active.add(value); + pending.push({ value, depth, exit: true }); + if (object(value) && value.addr !== undefined) { + const addr = typeof value.addr === 'number' && Number.isSafeInteger(value.addr) && value.addr >= 0 + ? String(value.addr) : typeof value.addr === 'string' && /^\d+$/.test(value.addr) ? value.addr : fail('malformed address'); + const existing = catalog.get(addr); + if (existing && (existing.kind !== value.kind || existing.name !== value.name)) fail('conflicting address definitions'); + if (!existing || Object.keys(value).length > Object.keys(existing).length) catalog.set(addr, value); + } + const values = Array.isArray(value) ? value : Object.values(value); + if (values.length > MAX_NODES - nodeCount - pending.length) fail('input node limit exceeded'); + for (let i = values.length - 1; i >= 0; i--) pending.push({ value: values[i], depth: depth + 1 }); + } + const resolve = (value: unknown): AstObject => { + if (object(value)) return value; + if (typeof value === 'string') { + const match = /^(\d+)\s/.exec(value); + if (match && catalog.has(match[1]!)) return catalog.get(match[1]!)!; + } + return fail('unresolved symbol/body reference'); + }; + const members = (scope: AstObject): unknown[] => { + if (scope.members === undefined) return []; + if (!Array.isArray(scope.members)) return fail('members must be an array'); + return scope.members; + }; + const nameOf = (node: AstObject): string => { + if (typeof node.name !== 'string' || !node.name || node.name.length > 4096) return fail('missing or oversized symbol name'); + return node.name; + }; + const segment = (name: string): string => /^[A-Za-z_][A-Za-z0-9_$]*$/.test(name) ? name : `\\${name} `; + const join = (parent: string, name: string): string => parent ? `${parent}.${segment(name)}` : segment(name); + const range = (value: unknown): [number, number] => { + const match = typeof value === 'string' && /^\[\s*(-?\d+)\s*:\s*(-?\d+)\s*\]$/.exec(value); + if (!match) return fail('expected a constant numeric range'); + const left = Number(match[1]), right = Number(match[2]); + if (!Number.isSafeInteger(left) || !Number.isSafeInteger(right) || !Number.isSafeInteger(Math.abs(left - right) + 1)) return fail('range exceeds exact numeric bounds'); + return [left, right]; + }; + const stripAddresses = (text: string): string | undefined => { + let unknown = false; + const cleaned = text.replace(/\b(\d+)\s+(?=[A-Za-z_$\\])/g, (match, addr: string) => { + if (catalog.has(addr)) return ''; + unknown = true; return match; + }); + return unknown ? undefined : cleaned; + }; + const typeInfo = (value: unknown, ancestors = new Set()): { type?: string; width?: number } => { + if (typeof value === 'string') { + const ref = /^(\d+)\s/.exec(value); + if (ref) { + const target = catalog.get(ref[1]!); + if (!target) return {}; + return { ...typeInfo(target, ancestors), type: stripAddresses(value) }; + } + // Composite pretty-printed strings do not prove packed vs unpacked shape. + const base = value.replace(/\s+(signed|unsigned)$/, ''); + return { type: stripAddresses(value), ...(INTEGER_WIDTHS.has(base) ? { width: INTEGER_WIDTHS.get(base)! } : {}) }; + } + if (!object(value)) return {}; + if (ancestors.size >= MAX_DEPTH || ancestors.has(value)) return fail('cyclic or overly deep type'); + const next = new Set(ancestors); next.add(value); + const name = typeof value.name === 'string' && value.name ? stripAddresses(value.name) : undefined; + if (value.kind === 'TypeAlias') return { ...typeInfo(value.target, next), ...(name ? { type: name } : {}) }; + if (value.kind === 'ScalarType' || value.kind === 'PredefinedIntegerType') { + const base = name?.replace(/\s+(signed|unsigned)$/, '') ?? ''; + return { type: name, ...(INTEGER_WIDTHS.has(base) ? { width: INTEGER_WIDTHS.get(base)! } : {}) }; + } + if (value.kind === 'PackedArrayType') { + const [left, right] = range(value.range); + const element = typeInfo(value.elementType, next); + const width = element.width === undefined ? undefined : element.width * (Math.abs(left - right) + 1); + if (width !== undefined && !Number.isSafeInteger(width)) fail('packed width exceeds exact numeric bounds'); + return { ...(element.type ? { type: `${element.type}[${left}:${right}]` } : {}), ...(width !== undefined ? { width } : {}) }; + } + return name ? { type: name } : {}; + }; + const sourceOf = (node: AstObject): HdlSemanticFact['source'] => { + const file = node.source_file; + if (file === undefined) return undefined; + if (typeof file !== 'string') return fail('invalid source filename'); + if (!file || file.startsWith('<') || (!path.isAbsolute(file) && path.win32.isAbsolute(file))) return undefined; + const full = path.resolve(root, file.replace(/\\/g, '/')); + const relative = path.relative(root, full); + if (!relative || relative === '..' || relative.startsWith(`..${path.sep}`) || path.isAbsolute(relative)) return undefined; + const line = node.source_line, column = node.source_column; + if (!Number.isSafeInteger(line) || (line as number) < 0 || (column !== undefined && (!Number.isSafeInteger(column) || (column as number) < 0))) return fail('invalid source coordinates'); + if (line === 0) return undefined; + return { file: relative.split(path.sep).join('/'), line: line as number, column: column === undefined || column === 0 ? null : column as number }; + }; + const indexValue = (value: unknown): string => { + if (typeof value !== 'string' || !/^-?\d+$/.test(value) || !Number.isSafeInteger(Number(value))) return fail('generate index is not a known decimal integer'); + return String(Number(value)); + }; + + const facts: HdlSemanticFact[] = []; + const keys = new Set(); + let visits = 0, factText = 0; + const visit = (node: AstObject, scopePath: string, ancestry: Set, depth: number): void => { + if (++visits > MAX_FACTS || depth > MAX_DEPTH || scopePath.length > 16384) fail('reachable hierarchy size/depth limit exceeded'); + if (ancestry.has(node)) fail('cyclic instance hierarchy'); + const chain = new Set(ancestry); chain.add(node); + if (node.kind === 'Parameter' || node.kind === 'Port') { + const name = nameOf(node); + const fact: HdlSemanticFact = { kind: node.kind === 'Parameter' ? 'parameter' : 'port', name, instancePath: scopePath, ...typeInfo(node.type) }; + if (node.kind === 'Parameter' && ['string', 'number', 'boolean'].includes(typeof node.value)) fact.value = String(node.value); + if (node.kind === 'Port' && typeof node.direction === 'string') fact.direction = node.direction; + const source = sourceOf(node); if (source) fact.source = source; + const key = JSON.stringify([fact.kind, scopePath, name]); + if (keys.has(key)) fail('duplicate fact identity'); + keys.add(key); factText += JSON.stringify(fact).length; + if (facts.length >= MAX_FACTS || factText > MAX_TEXT) fail('fact output size limit exceeded'); + facts.push(fact); return; + } + if (node.kind === 'Instance') { + const body = resolve(node.body); + if (body.kind !== 'InstanceBody') fail('instance body has wrong kind'); + visit(body, scopePath, chain, depth + 1); return; + } + if (node.kind === 'GenerateBlock' && node.isUninstantiated === true) return; + if (node.kind === 'GenerateBlockArray') { + // Slang's loopVariable can refer to an evaluator-only symbol absent + // from the serialized catalog. Its explicit name still identifies the + // matching implicit local Parameter in every materialized iteration. + const variableName = object(node.loopVariable) ? nameOf(node.loopVariable) + : typeof node.loopVariable === 'string' ? /^\d+ (.+)$/.exec(node.loopVariable)?.[1] : undefined; + if (!variableName) fail('missing generate loop variable name'); + const indices = new Set(); + for (const entry of members(node)) { + const block = resolve(entry); + if (block.kind === 'Genvar') continue; + if (block.kind !== 'GenerateBlock') fail('invalid generate array member'); + if (block.isUninstantiated === true) continue; + const parameters = members(block).map(resolve).filter(n => n.kind === 'Parameter' && n.name === variableName); + if (parameters.length !== 1) fail('missing or ambiguous generate index parameter'); + const index = indexValue(parameters[0]!.value); + if (indices.has(index)) fail('duplicate generate index'); + indices.add(index); visit(block, `${scopePath}[${index}]`, chain, depth + 1); + } + return; + } + if (node.kind === 'InstanceArray') { + const [left, right] = range(node.range); + const entries = members(node); + if (entries.length !== Math.abs(left - right) + 1) fail('instance array length disagrees with range'); + // Slang serializes elements in ascending numeric index order, including + // descending declarations; verified with distinct per-index defparams. + for (let i = 0; i < entries.length; i++) { + const child = resolve(entries[i]); + if (!['Instance', 'InstanceArray'].includes(String(child.kind))) fail('invalid instance array member'); + visit(child, `${scopePath}[${Math.min(left, right) + i}]`, chain, depth + 1); + } + return; + } + if (!['InstanceBody', 'GenerateBlock', 'Root'].includes(String(node.kind))) return; + for (const entry of members(node)) { + const child = resolve(entry); + if (['Instance', 'InstanceArray', 'GenerateBlockArray', 'GenerateBlock'].includes(String(child.kind))) { + if (child.kind === 'GenerateBlock' && child.isUninstantiated === true) continue; + visit(child, join(scopePath, nameOf(child)), chain, depth + 1); + } else if (child.kind === 'Parameter' || child.kind === 'Port') visit(child, scopePath, chain, depth + 1); + } + }; + for (const entry of members(design)) { + const node = resolve(entry); + if (node.kind === 'Instance' || node.kind === 'InstanceArray') visit(node, segment(nameOf(node)), new Set(), 0); + } + return facts; +} diff --git a/src/hdl/semantic-runner.ts b/src/hdl/semantic-runner.ts new file mode 100644 index 0000000000..78a68af0a4 --- /dev/null +++ b/src/hdl/semantic-runner.ts @@ -0,0 +1,263 @@ +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { createHash } from 'node:crypto'; +import { spawn } from 'node:child_process'; +import { loadHdlProfile, isReservedHdlPath, type ActiveHdlProfile } from './profile'; + +export interface RunSlangOptions { + executable?: string; + pythonExecutable?: string; + top?: string; + parameters?: Record; + allowUseBeforeDeclare?: boolean; + signal?: AbortSignal; + timeoutMs?: number; +} +export interface RunSlangResult { + ast: unknown; + /** The deleted immutable snapshot root; use only to remap AST paths to sources keys. */ + sourceRoot: string; + profileName: string; + configurationFingerprint: string; + fingerprint: string; + version: string; + executableSha256: string; + frontend: 'slang' | 'pyslang'; + exporterSha256?: string; + librarySha256?: string; + diagnostics: string[]; + sources: Record; + top: string; + parameters: Record; + allowUseBeforeDeclare: boolean; + languageStandard: '1364-2005' | '1800-2023'; + compilationUnitMode: 'separate'; + runnerVersion: string; + compilerLimits: readonly string[]; +} +const RUNNER_VERSION = 'slang-snapshot-v2'; +const COMPILER_LIMITS = Object.freeze(['--max-hierarchy-depth=128','--max-generate-steps=10000','--max-instance-array=10000','--max-constexpr-steps=1000000','--max-constant-size=1048576']); +const FILE_LIMIT = 1024 * 1024, TOTAL_LIMIT = 64 * 1024 * 1024, AST_LIMIT = 64 * 1024 * 1024; +const OUTPUT_LIMIT = 1024 * 1024, FILE_COUNT_LIMIT = 20000; +const digest = (content: Buffer | string) => createHash('sha256').update(content).digest('hex'); +function abort(signal?: AbortSignal): void { if (signal?.aborted) throw new Error('slang semantic run cancelled'); } +function resolveExecutable(name: string, preserveLauncherPath = false): string { + if (!name || name.includes('\0')) throw new Error('A slang executable must be specified'); + const candidates = name.includes('/') ? [path.resolve(name)] : (process.env.PATH ?? '').split(path.delimiter).map(dir => path.join(dir, name)); + for (const candidate of candidates) { + try { fs.accessSync(candidate, fs.constants.X_OK); const real = fs.realpathSync(candidate); if (fs.statSync(real).isFile()) return preserveLauncherPath ? path.resolve(candidate) : real; } catch { /* try next PATH entry */ } + } + throw new Error(`slang executable is unavailable: ${name}`); +} +async function executableHash(file: string): Promise { + const hash = createHash('sha256'); + for await (const chunk of fs.createReadStream(file)) hash.update(chunk); + return hash.digest('hex'); +} + +/** Literal includes only. Macro-computed includes cannot be snapshotted without a compiler filesystem API. */ +function includes(source: string): Array<{ name: string; local: boolean }> { + const result: Array<{name: string; local: boolean}> = []; + for (let i = 0; i < source.length;) { + if (source.startsWith('//', i)) { const end = source.indexOf('\n', i); i = end < 0 ? source.length : end; continue; } + if (source.startsWith('/*', i)) { const end = source.indexOf('*/', i + 2); i = end < 0 ? source.length : end + 2; continue; } + if (source[i] === '"') { i++; while (i < source.length) { if (source[i] === '\\') i += 2; else if (source[i++] === '"') break; } continue; } + if (source[i] === '\\') { i++; while (i < source.length && !/\s/.test(source[i]!)) i++; continue; } + const match = source.slice(i).match(/^`include\b[ \t]*/); + if (!match) { i++; continue; } + i += match[0].length; + const opening = source[i], closing = opening === '"' ? '"' : opening === '<' ? '>' : ''; + if (!closing) throw new Error('Macro-computed include paths are unsupported by the slang snapshot runner'); + const end = source.indexOf(closing, i + 1); + if (end < 0 || /[\r\n\\`]/.test(source.slice(i + 1, end))) throw new Error('Unsupported literal include path in slang snapshot'); + result.push({name: source.slice(i + 1, end), local: opening === '"'}); i = end + 1; + } + return result; +} +interface Snapshot { + sources: Record; + hashes: Record; + missing: string[]; +} +function collect(root: string, profile: ActiveHdlProfile, signal?: AbortSignal, deadline?: number): Snapshot { + const check = () => { abort(signal); if(deadline && Date.now() >= deadline)throw new Error('slang semantic run timed out'); }; + const macroIncludes = Object.values(profile.defines).flatMap(includes); + const sources: Record = Object.create(null), hashes: Record = Object.create(null); + const missing = new Set(), visitedDirs = new Set(); + const queue: string[] = []; + let bytes = 0, includeCandidates = 0; + const relative = (file: string) => path.relative(root, file).split(path.sep).join('/'); + const confined = (file: string): string => { + const rel = relative(file); + if (!rel || rel === '..' || rel.startsWith('../') || path.isAbsolute(rel) || isReservedHdlPath(rel)) throw new Error('slang snapshot input escapes the project or uses a reserved path'); + let current = root; + for (const part of rel.split('/')) { + current = path.join(current, part); + try { if (fs.lstatSync(current).isSymbolicLink()) throw new Error('Symbolic links are unsupported in slang snapshot inputs'); } + catch (error) { if ((error as NodeJS.ErrnoException).code === 'ENOENT') break; throw error; } + } + return rel; + }; + const add = (file: string, required: boolean): void => { + check(); + if(!required && ++includeCandidates > 100000)throw new Error('slang snapshot exceeds the include candidate limit'); + const rel = confined(file); + if (Object.prototype.hasOwnProperty.call(hashes, rel)) return; + let stat: fs.Stats; + try { stat = fs.statSync(file); } catch (error) { + if (!required && (error as NodeJS.ErrnoException).code === 'ENOENT') { missing.add(rel); return; } + throw error; + } + if (!stat.isFile()) throw new Error('slang snapshot input is not a regular file'); + if (stat.size > FILE_LIMIT || bytes + stat.size > TOTAL_LIMIT || queue.length >= FILE_COUNT_LIMIT) throw new Error('slang snapshot exceeds the input size/file count limit'); + const content = fs.readFileSync(file); + if (content.length > FILE_LIMIT || bytes + content.length > TOTAL_LIMIT) throw new Error('slang snapshot exceeds the input size limit'); + let text: string; + try { text = new TextDecoder('utf-8', {fatal:true,ignoreBOM:true}).decode(content); } catch { throw new Error(`Non-UTF-8 file in slang snapshot: ${rel}`); } + sources[rel] = text; hashes[rel] = digest(content); bytes += content.length; queue.push(rel); + }; + const tree = (dir: string, depth = 0): void => { + check(); + if(depth > 128 || visitedDirs.size >= FILE_COUNT_LIMIT)throw new Error('slang snapshot exceeds the directory traversal limit'); + if (visitedDirs.has(dir)) return; visitedDirs.add(dir); + for (const entry of fs.readdirSync(dir, {withFileTypes:true})) { + const file = path.join(dir, entry.name), rel = relative(file); + if (isReservedHdlPath(rel)) continue; + confined(file); + if (entry.isDirectory()) tree(file,depth+1); else if (entry.isFile()) add(file, true); else throw new Error('Special files are unsupported in slang snapshot inputs'); + } + }; + for (const file of profile.files) add(path.resolve(root,file), true); + for (const dir of profile.includeDirs) { + const absolute = path.resolve(root,dir); + if (absolute !== root) confined(absolute); + tree(absolute); + } + // Read every literal candidate, including inactive branches and lower-priority + // shadow candidates. Missing candidates are fingerprinted and checked again. + for (let index = 0; index < queue.length; index++) { + const file = queue[index]!; + for (const include of [...includes(sources[file]!), ...macroIncludes]) { + if (!include.name || path.isAbsolute(include.name) || /^[A-Za-z]:/.test(include.name)) throw new Error('Absolute include paths are unsupported by the slang snapshot runner'); + const bases = [...(include.local ? [path.dirname(path.join(root,file))] : []), ...profile.includeDirs.map(dir => path.join(root,dir))]; + for (const base of bases) add(path.resolve(base,include.name), false); + } + } + return {sources,hashes:Object.fromEntries(Object.entries(hashes).sort(([a],[b])=>a.localeCompare(b))),missing:[...missing].sort()}; +} + +function run(executable: string, args: string[], cwd: string, timeout: number, signal?: AbortSignal, astFile?: string): Promise<{stdout:string;stderr:string}> { + abort(signal); + return new Promise((resolve,reject) => { + const child = spawn(executable,args,{cwd,detached:true,shell:false,stdio:['ignore','pipe','pipe'],windowsHide:true}); + let stdout = '', stderr = '', size = 0, failure: Error | undefined, finished = false; + const stop = (message: string): void => { + failure ??= new Error(message); + if (child.pid) { try { process.kill(-child.pid,'SIGKILL'); } catch { try { child.kill('SIGKILL'); } catch { /* already exited */ } } } + }; + const timer = setTimeout(()=>stop('slang semantic run timed out'),timeout); + const onAbort = () => stop('slang semantic run cancelled'); + signal?.addEventListener('abort',onAbort,{once:true}); + const monitor = astFile ? setInterval(()=>{ + try { if (fs.statSync(astFile).size > AST_LIMIT) stop('slang AST output exceeds 64 MiB'); } catch { /* not created yet */ } + },25) : undefined; + const finish = (error?: Error): void => { + if (finished) return; finished=true; clearTimeout(timer); if(monitor)clearInterval(monitor); signal?.removeEventListener('abort',onAbort); + if(error)reject(error); else resolve({stdout,stderr}); + }; + child.stdout.on('data',(chunk:Buffer)=>{size+=chunk.length;if(size>OUTPUT_LIMIT)stop('slang output exceeds 1 MiB');else stdout+=chunk.toString('utf8');}); + child.stderr.on('data',(chunk:Buffer)=>{size+=chunk.length;if(size>OUTPUT_LIMIT)stop('slang output exceeds 1 MiB');else stderr+=chunk.toString('utf8');}); + child.on('error',error=>finish(error)); + child.on('close',(code)=>{ + // Do not leave detached descendants alive after the owned frontend exits. + if(child.pid)try{process.kill(-child.pid,'SIGKILL');}catch{/* group already gone */} + finish(failure ?? (code===0 ? undefined : new Error(`slang failed (${code}): ${stderr.slice(0,4000) || stdout.slice(0,4000)}`))); + }); + if(signal?.aborted)onAbort(); + }); +} + +/** Optional on-demand frontend. Snapshot confinement is not an adversarial OS compiler sandbox. */ +export async function runSlangSemantics(projectRoot: string, options: RunSlangOptions): Promise { + if(process.platform==='win32')throw new Error('slang semantic runner is not supported on Windows until process-tree cancellation is verified'); + const requestedSlang=options.executable, requestedPython=options.pythonExecutable; + if((requestedSlang!==undefined)===(requestedPython!==undefined))throw new Error('Specify exactly one of executable or pythonExecutable'); + const frontend:RunSlangResult['frontend']=requestedPython!==undefined?'pyslang':'slang'; + const requestedExecutable=requestedPython??requestedSlang; + if(typeof requestedExecutable!=='string'||!requestedExecutable.trim())throw new Error('A frontend executable must be specified'); + const signal=options.signal, allowUseBeforeDeclare=!!options.allowUseBeforeDeclare; + abort(signal); + const timeout=options.timeoutMs??30000; + if(!Number.isSafeInteger(timeout)||timeout<1||timeout>300000)throw new Error('slang timeoutMs must be between 1 and 300000'); + const deadline=Date.now()+timeout, remaining=()=>Math.max(1,deadline-Date.now()); + const root=fs.realpathSync(projectRoot), loaded=loadHdlProfile(root); + if(loaded.status!=='active'||!loaded.profile)throw new Error(`slang semantics requires a valid active HDL profile: ${loaded.diagnostics.join('; ')||loaded.status}`); + const profile=loaded.profile; + const top=options.top??(profile.topModules.length===1?profile.topModules[0]:undefined); + if(!top||!/^[A-Za-z_][A-Za-z0-9_$]*$/.test(top))throw new Error('slang semantics requires one explicit simple top module'); + if(!profile.files.length)throw new Error('slang semantics requires at least one source unit'); + const parameters:Record=Object.create(null); + for(const [name,value] of Object.entries(options.parameters??{}).sort(([a],[b])=>a.localeCompare(b))) { + if(!/^[A-Za-z_][A-Za-z0-9_$]*$/.test(name)||typeof value!=='string'||value.includes('\0'))throw new Error('Invalid slang parameter override'); + parameters[name]=value; + } + // Preserve the venv launch path: realpath'ing bin/python before exec would + // select the base interpreter's site-packages instead of the configured venv. + const executable=resolveExecutable(requestedExecutable,frontend==='pyslang'); + const exporter=frontend==='pyslang'?path.join(__dirname,'source-map-export.py'):undefined; + if(exporter) { + try { const stat=fs.statSync(exporter); if(!stat.isFile()||stat.size>FILE_LIMIT)throw new Error('invalid exporter'); } + catch { throw new Error('Shipped pyslang exporter is unavailable or exceeds 1 MiB'); } + } + const commandPrefix=exporter?['-I',exporter]:[]; + const executableSha256=await executableHash(executable); + const exporterSha256=exporter?await executableHash(exporter):undefined; + const before=collect(root,profile,signal,deadline); + const base=process.env.CODEGRAPH_SEMANTIC_TMPDIR||path.join(os.tmpdir(),'codegraph-semantic-runs'); + fs.mkdirSync(base,{recursive:true,mode:0o700}); + const baseRelative=path.relative(root,fs.realpathSync(base)); + if(!baseRelative||(!baseRelative.startsWith(`..${path.sep}`)&&baseRelative!=='..'&&!path.isAbsolute(baseRelative)))throw new Error('Semantic snapshot directory must be outside the source project'); + const temp=fs.mkdtempSync(path.join(base,'run-')),sourceRoot=path.join(temp,'sources'),astFile=path.join(temp,'ast.json'); + try { + fs.mkdirSync(sourceRoot,{mode:0o700}); + for(const dir of profile.includeDirs) fs.mkdirSync(path.join(sourceRoot,dir),{recursive:true,mode:0o700}); + for(const [file,source] of Object.entries(before.sources)) {const dest=path.join(sourceRoot,file);fs.mkdirSync(path.dirname(dest),{recursive:true,mode:0o700});fs.writeFileSync(dest,source,{mode:0o600});} + const version=(await run(executable,[...commandPrefix,'--version'],sourceRoot,Math.min(remaining(),5000),signal)).stdout.trim(); + const pythonVersion=version.match(/^codegraph pyslang 11\.0\.0 exporter 1 native ([a-f0-9]{64})$/); + const supportedVersion=frontend==='pyslang'?!!pythonVersion:/^slang version 11\.0(?:\.\d+)?(?:\+[A-Za-z0-9]+)?(?:\s.*)?$/.test(version); + if(!supportedVersion)throw new Error(`Unsupported ${frontend} schema family: ${version.slice(0,200)}`); + const librarySha256=pythonVersion?.[1]; + const languageStandard=profile.languageMode==='verilog'?'1364-2005':'1800-2023'; + const args=['--ast-json',astFile,'--ast-json-source-info','--ast-json-detailed-types','--threads=1','--error-limit=20',`--std=${languageStandard}`,'--top',top]; + args.push(...COMPILER_LIMITS); + if(allowUseBeforeDeclare)args.push('--allow-use-before-declare'); + for(const dir of profile.includeDirs)args.push('-I',path.join(sourceRoot,dir)); + for(const [name,value] of Object.entries(profile.defines))args.push(`-D${name}=${value}`); + for(const [name,value] of Object.entries(parameters).sort(([a],[b])=>a.localeCompare(b)))args.push('-G',`${name}=${value}`); + args.push('--',...profile.files); + const output=await run(executable,[...commandPrefix,...args],sourceRoot,remaining(),signal,astFile); + const stat=fs.lstatSync(astFile); + if(!stat.isFile()||stat.isSymbolicLink()||stat.size>AST_LIMIT)throw new Error('Invalid or oversized slang AST output'); + const astText=fs.readFileSync(astFile,'utf8');if(Buffer.byteLength(astText)>AST_LIMIT)throw new Error('slang AST output exceeds 64 MiB'); + const ast:unknown=JSON.parse(astText); + if(!ast||typeof ast!=='object'||Array.isArray(ast))throw new Error('slang AST root must be an object'); + if(frontend==='pyslang'&&((ast as Record).codegraphSemanticVersion!==1||!Array.isArray((ast as Record).facts)))throw new Error('Invalid pyslang semantic exporter envelope'); + if(await executableHash(executable)!==executableSha256)throw new Error('slang executable changed during the run; result discarded'); + if(exporter&&await executableHash(exporter)!==exporterSha256)throw new Error('pyslang exporter changed during the run; result discarded'); + if(frontend==='pyslang') { + const verifiedVersion=(await run(executable,[...commandPrefix,'--version'],sourceRoot,Math.min(remaining(),5000),signal)).stdout.trim(); + if(verifiedVersion!==version)throw new Error('pyslang library changed during the run; result discarded'); + } + const snapshotAfter=collect(sourceRoot,profile,signal,deadline); + if(JSON.stringify(before.hashes)!==JSON.stringify(snapshotAfter.hashes)||JSON.stringify(before.missing)!==JSON.stringify(snapshotAfter.missing))throw new Error('slang modified its source snapshot; result discarded'); + const afterConfig=loadHdlProfile(root); + const after=collect(root,profile,signal,deadline); + if(afterConfig.status!=='active'||afterConfig.profile?.fingerprint!==profile.fingerprint||JSON.stringify(before.hashes)!==JSON.stringify(after.hashes)||JSON.stringify(before.missing)!==JSON.stringify(after.missing))throw new Error('HDL sources or configuration changed during the slang run; result discarded'); + abort(signal); + if(Date.now()>=deadline)throw new Error('slang semantic run timed out'); + const provenance={configurationFingerprint:profile.fingerprint,hashes:before.hashes,missing:before.missing,version,executableSha256,frontend,exporterSha256,librarySha256,top,parameters,allowUseBeforeDeclare,languageStandard,compilationUnitMode:'separate' as const,runnerVersion:RUNNER_VERSION,compilerLimits:COMPILER_LIMITS}; + return {ast,sourceRoot,profileName:profile.name,configurationFingerprint:profile.fingerprint,fingerprint:digest(JSON.stringify(provenance)),version,executableSha256,frontend,...(exporterSha256?{exporterSha256}:{}),...(librarySha256?{librarySha256}:{}), + diagnostics:[output.stdout,output.stderr].map(s=>s.trim()).filter(Boolean),sources:before.sources,top,parameters,allowUseBeforeDeclare,languageStandard,compilationUnitMode:'separate' as const,runnerVersion:RUNNER_VERSION,compilerLimits:COMPILER_LIMITS}; + } finally {fs.rmSync(temp,{recursive:true,force:true});} +} diff --git a/src/hdl/semantics.ts b/src/hdl/semantics.ts new file mode 100644 index 0000000000..ee7e27c5bf --- /dev/null +++ b/src/hdl/semantics.ts @@ -0,0 +1,76 @@ +import { createHash } from 'node:crypto'; +import { runSlangSemantics, type RunSlangOptions } from './semantic-runner'; +import { importSlangSemantics, type HdlSemanticFact } from './semantic-import'; +import { importMappedSemantics } from './source-map-import'; +import type { HdlProfileStatus } from './status'; +import type { Node, FileRecord } from '../types'; + +export interface HdlSemanticOptions extends RunSlangOptions { + /** Exact fact name, instance path, or instance-qualified fact name. */ + query?: string; + limit?: number; +} +export interface HdlSemanticResult { + schemaVersion: 1; + provenance: { frontend: 'slang' | 'pyslang'; exporterSha256?: string; librarySha256?: string; version: string; executableSha256: string; + profile: string; configurationFingerprint: string; fingerprint: string; top: string; + parameters: Record; allowUseBeforeDeclare: boolean; + languageStandard: string; compilationUnitMode: 'separate'; runnerVersion: string; compilerLimits: readonly string[] }; + sourceGraphProfileMatches: boolean; + facts: Array; + totalMatches: number; + truncated: boolean; + diagnostics: string[]; + limitations: string[]; +} +export interface HdlSemanticSourceGraph { + profile(): HdlProfileStatus | null; + stale(): boolean; + file(file: string): FileRecord | null | undefined; + nodes(file: string): Node[]; +} + +/** On-demand semantic view; no graph writes or persistent compiler cache. */ +export async function analyzeHdlSemantics(root: string, options: HdlSemanticOptions, + graph: HdlSemanticSourceGraph): Promise { + const limit = options.limit ?? 100; + const query = options.query; + if (!Number.isInteger(limit) || limit < 1 || limit > 1000) throw new Error('HDL semantic limit must be 1..1000'); + const result = await runSlangSemantics(root, options); + const facts = result.frontend === 'pyslang' ? importMappedSemantics(result.ast, result.sources) : importSlangSemantics(result.ast, result.sourceRoot); + const profile = graph.profile(); + const linked = !graph.stale() && profile?.state === 'matches' && + profile.indexed.configurationFingerprint === result.configurationFingerprint; + const byFile = new Map(); + const matches = facts.filter(fact => !query || fact.name === query || + fact.instancePath === query || `${fact.instancePath}.${fact.name}` === query); + const selected = matches.slice(0, limit).map(fact => { + const source = fact.source; + if (!source) return fact; + const text = result.sources[source.file]; + const line = text?.split(/\r?\n/)[source.line - 1]; + // Column 0 / macro invocation remains line-only; never fabricate a source node link. + if (text === undefined || line === undefined) { const { source: _source, ...rest } = fact; return rest; } + if (fact.sourceOrigin === 'macro' || !linked || source.column === null || source.column < 1 || source.column > Buffer.byteLength(line) + 1 || !line.includes(fact.name)) return fact; + if (!byFile.has(source.file)) { + const hash = createHash('sha256').update(text).digest('hex'); + byFile.set(source.file, graph.file(source.file)?.contentHash === hash ? graph.nodes(source.file) : []); + } + const nodes = byFile.get(source.file)!.filter(node => node.name === fact.name && node.startLine === source.line && + (fact.kind === 'parameter' ? node.kind === 'constant' : fact.kind === 'type' ? node.kind === 'type_alias' : node.kind === 'field' && !!node.decorators?.includes('hdl:port'))); + return nodes.length === 1 ? { ...fact, sourceNodeId: nodes[0]!.id } : fact; + }); + return { schemaVersion: 1, provenance: { frontend: result.frontend, exporterSha256: result.exporterSha256, librarySha256: result.librarySha256, version: result.version, + executableSha256: result.executableSha256, profile: result.profileName, + configurationFingerprint: result.configurationFingerprint, fingerprint: result.fingerprint, + top: result.top, parameters: { ...result.parameters }, allowUseBeforeDeclare: result.allowUseBeforeDeclare, + languageStandard: result.languageStandard, compilationUnitMode: result.compilationUnitMode, + runnerVersion: result.runnerVersion, compilerLimits: result.compilerLimits }, + sourceGraphProfileMatches: !!linked, facts: selected, totalMatches: matches.length, + truncated: matches.length > selected.length, diagnostics: result.diagnostics, + limitations: ['Computed on demand from a source snapshot; no persistent semantic cache.', + 'Source columns retain frontend coordinates; null means unavailable, including macro-generated declarations.', + 'Expression origins cover directly visited compiler syntax tokens; coverage/truncation is explicit, not a transitive constant-dependency proof.', + 'Only supported parameter/port facts are returned; absent or unknown widths are not zero.', + 'Compiler semantics are not simulation, synthesis, timing or hardware validation.'] }; +} diff --git a/src/hdl/source-map-export.py b/src/hdl/source-map-export.py new file mode 100644 index 0000000000..a0d448cd41 --- /dev/null +++ b/src/hdl/source-map-export.py @@ -0,0 +1,407 @@ +#!/usr/bin/env python3 +"""Compiler-backed, bounded pyslang export; no textual macro-position recovery. + +macroExpansion contains a set of provenance frames reached through BOTH original +and expansion parent relations. It is not an ordered invocation stack. Points +are physical UTF-8 byte positions in root-confined source files. A pasted token's +spelling point can identify its prefix argument; it is never a name range. +Expression origins describe only macro tokens in the selected direct syntax, +not transitive value/type dependencies. 'checked' plus truncated=true is a +bounded partial scan; command-line initializers remain separate from defaults. +""" +import hashlib +import json +from pathlib import Path +import shlex +import sys + +MAX_FACTS = 100_000 +MAX_DEPTH = 128 +MAX_BYTES = 64 * 1024 * 1024 +MAX_DIAGNOSTICS = 256 * 1024 +MAX_EXPRESSION_ORIGINS = 32 +MAX_PROVENANCE_FRAMES = 100_000 +MAX_SYNTAX_VISITS = 1_000_000 + + +def native_version(pyslang): + if pyslang.__version__ != '11.0.0': + raise RuntimeError('This exporter requires explicitly installed pyslang 11.0.0') + native = Path(pyslang.pyslang.__file__).resolve(strict=True) + with native.open('rb') as stream: + digest = hashlib.file_digest(stream, 'sha256').hexdigest() + return f'codegraph pyslang 11.0.0 exporter 1 native {digest}' + + +class Exporter: + def __init__(self, manager, root): + self.manager = manager + self.root = root.resolve(strict=True) + self.sources = {} + self.source_bytes = 0 + self.origins = {} + self.facts = [] + self.identities = set() + self.visits = 0 + self.output_bytes = 0 + self.syntax_cache = {} + self.syntax_visits = 0 + self.provenance_frames = 0 + + def point(self, location): + try: + if not location.buffer.id or not self.manager.isFileLoc(location): + return None + file = Path(self.manager.getFullPath(location.buffer)).resolve(strict=True) + relative = file.relative_to(self.root).as_posix() + if file not in self.sources: + if not file.is_file() or file.stat().st_size > 8 * 1024 * 1024: + return None + data = file.read_bytes() + self.source_bytes += len(data) + if self.source_bytes > MAX_BYTES: + raise RuntimeError('Source mapping byte limit exceeded') + self.sources[file] = data + data = self.sources[file] + offset = location.offset + if offset < 0 or offset > len(data): + return None + # Do not use logical `line remappings or display/Unicode columns. + line = data.count(b'\n', 0, offset) + 1 + column = offset - data.rfind(b'\n', 0, offset) + return {'file': relative, 'line': line, 'column': column, 'byteOffset': offset} + except (OSError, ValueError): + return None + + def provenance(self, location): + key = (location.buffer.id, location.offset) + if key in self.origins: + return self.origins[key] + sm = self.manager + macro = bool(location.buffer.id and sm.isMacroLoc(location)) + result = {'sourceOrigin': 'macro' if macro else 'direct'} + source = self.point(sm.getFullyOriginalLoc(location)) if location.buffer.id else None + if source is None and location.buffer.id: + source = self.point(sm.getFullyExpandedLoc(location)) + if source: + result['source'] = {k: source[k] for k in ['file', 'line', 'column']} + if macro: + pending, visited, frames = [location], set(), [] + complete = True + while pending: + current = pending.pop() + current_key = (current.buffer.id, current.offset) + if current_key in visited or not sm.isMacroLoc(current): + continue + if len(visited) >= MAX_DEPTH: + complete = False + break + visited.add(current_key) + original = sm.getOriginalLoc(current) + expanded = sm.getExpansionLoc(current) + expansion = sm.getExpansionRange(current) + frame = {'name': sm.getMacroName(current) or None, 'argument': sm.isMacroArgLoc(current)} + spelling = self.point(sm.getFullyOriginalLoc(original)) + if spelling: + frame['spelling'] = spelling + start = self.point(sm.getFullyExpandedLoc(expansion.start)) + end = self.point(sm.getFullyExpandedLoc(expansion.end)) + if start and end and start['file'] == end['file'] and end['byteOffset'] >= start['byteOffset']: + frame['invocation'] = {'start': start, 'end': end} + if not spelling or 'invocation' not in frame or start['byteOffset'] == end['byteOffset']: + complete = False + frames.append(frame) + # Original links preserve argument provenance; expansion links + # preserve nested replacement provenance. Neither alone is full. + for parent in [original, expanded]: + if parent.buffer.id and sm.isMacroLoc(parent): + pending.append(parent) + result['macroExpansion'] = frames + result['macroExpansionComplete'] = complete + self.origins[key] = result + return result + + def bounded_origin(self, origin): + result = dict(origin) + frames = origin.get('macroExpansion', []) + remaining = max(0, MAX_PROVENANCE_FRAMES - self.provenance_frames) + truncated = len(frames) > remaining + if 'macroExpansion' in origin: + result['macroExpansion'] = frames[:remaining] + self.provenance_frames += len(result['macroExpansion']) + if truncated: + result['macroExpansionComplete'] = False + return result, truncated + + @staticmethod + def syntax_key(syntax): + if syntax is None: + return None + span = syntax.sourceRange + return (syntax.kind.name, span.start.buffer.id, span.start.offset, + span.end.buffer.id, span.end.offset) + + def command_line_syntax(self, syntax): + if syntax is None: + return False + location = self.manager.getFullyExpandedLoc(syntax.getFirstToken().location) + if not location.buffer.id: + return False + # Driver's override buffer has an explicit command-line line mapping + # and an unnamed synthetic path, unlike a physical `line spoof. + return (self.manager.getFileName(location) == '' + and self.manager.getRawFileName(location.buffer).startswith(' 20_000 or self.syntax_visits > MAX_SYNTAX_VISITS: + truncated = True + return VisitAction.Interrupt + if isinstance(item, Token) and item.location.buffer.id and self.manager.isMacroLoc(item.location): + token_key = (item.location.buffer.id, item.location.offset) + if token_key not in seen: + if len(locations) >= MAX_EXPRESSION_ORIGINS: + truncated = True + return VisitAction.Interrupt + seen.add(token_key) + locations.append(item.location) + return VisitAction.Advance + + syntax.visit(visit) + # Retain the syntax wrapper: a bare Python id could otherwise be reused + # after collection, yielding origins for an unrelated syntax node. + self.syntax_cache[key] = (syntax, locations, truncated) + return locations, truncated + + def expression_origins(self, symbol): + coverage = {'initializer': 'not-applicable', 'declaredInitializer': 'not-applicable', + 'type': 'unavailable', 'truncated': False} + plans = [] + declared_type = getattr(symbol, 'declaredType', None) + declaration = symbol.syntax + type_declarations = [declaration] if declaration is not None else [] + if symbol.kind.name == 'Port': + internal = symbol.internalSymbol + if declared_type is None: + declared_type = getattr(internal, 'declaredType', None) if internal else None + # A non-ANSI PortReference is just the header name. The actual + # declarator (including unpacked dimensions) belongs to its net/var. + internal_syntax = getattr(internal, 'syntax', None) if internal else None + if internal_syntax is not None: + type_declarations.insert(0, internal_syntax) + if symbol.kind.name == 'Parameter': + expression = symbol.initializer + effective = expression.syntax if expression is not None else None + if effective is None and expression is not None and declared_type is not None: + effective = declared_type.initializerSyntax + clause = getattr(declaration, 'initializer', None) if declaration is not None else None + default = getattr(clause, 'expr', None) if clause is not None else None + if effective is None: + coverage['initializer'] = 'unavailable' + else: + coverage['initializer'] = 'command-line' if self.command_line_syntax(effective) else 'checked' + plans.append(('initializer', effective)) + # isOverridden alone is insufficient: Driver -G overrides currently + # leave it false, but their effective syntax is a distinct buffer. + overridden = symbol.isOverridden or (effective is not None and default is not None + and self.syntax_key(effective) != self.syntax_key(default)) + if overridden and default is not None: + coverage['declaredInitializer'] = 'checked' + plans.append(('declared-initializer', default)) + type_syntax = declared_type.typeSyntax if declared_type is not None else None + dimensions = [] + dimensions_known = symbol.kind.name != 'Port' + for candidate in type_declarations: + declared_dimensions = getattr(candidate, 'dimensions', None) + if declared_dimensions is not None: + dimensions_known = True + dimensions.extend(declared_dimensions) + type_inputs = ([type_syntax] if type_syntax is not None else []) + dimensions + if type_inputs and dimensions_known: + coverage['type'] = 'checked' + seen_types = set() + for node in type_inputs: + key = self.syntax_key(node) + if key not in seen_types: + seen_types.add(key) + plans.append(('type', node)) + origins, seen, normalized_seen = [], set(), set() + for role, syntax in plans: + locations, truncated = self.scan_macro_tokens(syntax, role) + coverage['truncated'] |= truncated + for location in locations: + key = (role, location.buffer.id, location.offset) + if key in seen: + continue + seen.add(key) + if len(origins) >= MAX_EXPRESSION_ORIGINS: + coverage['truncated'] = True + break + origin, limited = self.bounded_origin(self.provenance(location)) + coverage['truncated'] |= limited + entry = {'role': role, **origin} + normalized = json.dumps(entry, sort_keys=True, ensure_ascii=False, separators=(',', ':')) + if normalized in normalized_seen: + continue + normalized_seen.add(normalized) + origins.append(entry) + return {'expressionOrigins': origins, 'expressionOriginCoverage': coverage} + + @staticmethod + def segment(name): + import re + if not name or len(name) > 4096: + raise RuntimeError('Unsupported empty/oversized hierarchy name') + return name if re.fullmatch(r'[A-Za-z_][A-Za-z0-9_$]*', name) else '\\' + name + ' ' + + def append(self, symbol, instance_path): + kind = symbol.kind.name + semantic_type = symbol.canonicalType if kind == 'TypeAlias' else symbol.type + origin, _ = self.bounded_origin(self.provenance(symbol.location)) + fact = {'kind': {'Parameter': 'parameter', 'Port': 'port', 'TypeAlias': 'type'}[kind], + 'name': symbol.name, 'instancePath': instance_path, + 'type': str(semantic_type), **origin, **self.expression_origins(symbol)} + if semantic_type.isIntegral: + width = semantic_type.bitWidth + if 0 < width <= 2**53 - 1: + fact['width'] = width + if kind == 'Parameter': + value = symbol.value + if bool(value) and not value.isContainer(): + fact['value'] = str(value) # Preserve SV width/base and X/Z text. + elif kind == 'Port': + fact['direction'] = symbol.direction.name + identity = (fact['kind'], instance_path, symbol.name) + if identity in self.identities: + raise RuntimeError('Duplicate semantic fact identity') + self.identities.add(identity) + self.output_bytes += len(json.dumps(fact, ensure_ascii=False).encode('utf8')) + if len(self.facts) >= MAX_FACTS or self.output_bytes > MAX_BYTES: + raise RuntimeError('Semantic fact output limit exceeded') + self.facts.append(fact) + + def scope(self, scope, instance_path, depth=0, active=None): + if depth > MAX_DEPTH or len(instance_path) > 16384: + raise RuntimeError('Semantic hierarchy depth/path limit exceeded') + active = set() if active is None else active + key = id(scope) + if key in active: + raise RuntimeError('Cyclic instance hierarchy') + active = active | {key} + for symbol in scope: + self.visits += 1 + if self.visits > 1_000_000: + raise RuntimeError('Semantic traversal size limit exceeded') + kind = symbol.kind.name + if kind in ['Parameter', 'Port', 'TypeAlias']: + self.append(symbol, instance_path) + elif kind == 'Instance': + self.scope(symbol.body, instance_path + '.' + self.segment(symbol.name), depth + 1, active) + elif kind == 'GenerateBlock': + if not symbol.isUninstantiated: + self.scope(symbol, instance_path + '.' + self.segment(symbol.externalName), depth + 1, active) + elif kind == 'GenerateBlockArray': + name = instance_path + '.' + self.segment(symbol.externalName) + for block in symbol.entries: + if block.isUninstantiated: + continue + index = block.arrayIndex + if index is None or index.hasUnknown: + raise RuntimeError('Unknown generate iteration index') + self.scope(block, name + '[' + str(int(index)) + ']', depth + 1, active) + elif kind == 'InstanceArray': + self.array(symbol, instance_path + '.' + self.segment(symbol.name), depth + 1, active) + + def array(self, array, array_path, depth, active): + if depth > MAX_DEPTH: + raise RuntimeError('Instance array depth limit exceeded') + for entry in array.elements: + if entry.kind.name == 'InstanceArray': + self.array(entry, array_path, depth + 1, active) + elif entry.kind.name == 'Instance': + indices = ''.join('[' + str(index) + ']' for index in entry.arrayPath) + self.scope(entry.body, array_path + indices, depth + 1, active) + else: + raise RuntimeError('Unsupported instance array member') + + +def main(argv): + import pyslang + from pyslang.driver import Driver, CommandLineOptions + version = native_version(pyslang) + if argv == ['--version']: + print(version) + return 0 + output = None + compiler_argv = [] + index = 0 + while index < len(argv): + arg = argv[index] + if arg == '--': + compiler_argv.extend(argv[index:]) + break + if arg == '--ast-json': + if index + 1 >= len(argv) or output is not None: + raise RuntimeError('Expected one --ast-json output path') + output = Path(argv[index + 1]) + index += 2 + continue + if arg.startswith('--ast-json='): + if output is not None: + raise RuntimeError('Expected one --ast-json output path') + output = Path(arg.split('=', 1)[1]) + elif arg not in ['--ast-json-source-info', '--ast-json-detailed-types']: + compiler_argv.append(arg) + index += 1 + if output is None or str(output) == '-': + raise RuntimeError('A file --ast-json output path is required') + driver = Driver() + driver.addStandardArgs() + options = CommandLineOptions() + options.ignoreProgramName = True + options.expandEnvVars = False + # Binding accepts one command-line string; shlex.join quotes every original + # argv atom. No shell is invoked, no environment interpolation is enabled. + if not driver.parseCommandLine(shlex.join(compiler_argv), options) or not driver.processOptions() or not driver.parseAllSources(): + driver.reportDiagnostics(True) + return 1 + compilation = driver.createCompilation() + driver.reportCompilation(compilation, True) + analysis = driver.runAnalysis(compilation) # Keep native analysis alive through export. + diagnostics = driver.textDiagClient.getString() + if diagnostics: + sys.stderr.write(diagnostics[:MAX_DIAGNOSTICS]) + if len(diagnostics) > MAX_DIAGNOSTICS: + sys.stderr.write('\nAdditional compiler diagnostics truncated.\n') + if driver.diagEngine.numErrors or compilation.hasFatalErrors or compilation.hasIssuedErrors: + return 1 + exporter = Exporter(driver.sourceManager, Path.cwd()) + for instance in compilation.getRoot().topInstances: + exporter.scope(instance.body, exporter.segment(instance.name)) + payload = {'codegraphSemanticVersion': 1, 'facts': exporter.facts} + encoded = json.dumps(payload, ensure_ascii=False, separators=(',', ':')).encode('utf8') + if len(encoded) > MAX_BYTES: + raise RuntimeError('Semantic JSON size limit exceeded') + output.write_bytes(encoded) + return 0 + + +if __name__ == '__main__': + try: + sys.exit(main(sys.argv[1:])) + except (Exception, ImportError) as error: + print('CodeGraph pyslang exporter: ' + str(error), file=sys.stderr) + sys.exit(2) diff --git a/src/hdl/source-map-import.ts b/src/hdl/source-map-import.ts new file mode 100644 index 0000000000..67f7571187 --- /dev/null +++ b/src/hdl/source-map-import.ts @@ -0,0 +1,99 @@ +import * as path from 'node:path'; +import type { HdlSemanticFact, HdlMacroPoint, HdlMacroFrame, HdlExpressionCoverage, HdlExpressionOrigin } from './semantic-import'; +const object = (v: unknown): v is Record => !!v && typeof v === 'object' && !Array.isArray(v); +function fail(message: string): never { throw new Error('Invalid pyslang semantic export: ' + message); } + +/** Validate the separately versioned exporter protocol against immutable source bytes. */ +export function importMappedSemantics(value: unknown, sources: Record): HdlSemanticFact[] { + if (!object(value) || value.codegraphSemanticVersion !== 1 || !Array.isArray(value.facts) || value.facts.length > 100000) fail('unsupported schema or fact count'); + const cache = new Map(); + const fileData = (file: unknown) => { + if (typeof file !== 'string' || !file || file.includes('\\') || file.includes('\0') || path.posix.isAbsolute(file) || /^[A-Za-z]:/.test(file) || path.posix.normalize(file) !== file || file.split('/').includes('..') || !Object.prototype.hasOwnProperty.call(sources, file)) fail('source outside snapshot'); + if (!cache.has(file)) { + const bytes = Buffer.from(sources[file]!, 'utf8'), starts = [0]; + for (let i = 0; i < bytes.length; i++) if (bytes[i] === 10) starts.push(i + 1); + cache.set(file, { bytes, starts }); + } + return { file, ...cache.get(file)! }; + }; + const source = (v: unknown): NonNullable => { + if (!object(v)) fail('source must be an object'); + const data = fileData(v.file), line = v.line, column = v.column; + if (!Number.isSafeInteger(line) || (line as number) < 1 || (line as number) > data.starts.length || (column !== null && (!Number.isSafeInteger(column) || (column as number) < 1))) fail('invalid source coordinates'); + if (column !== null) { + const offset = data.starts[(line as number) - 1]! + (column as number) - 1; + const end = data.starts[line as number] ?? data.bytes.length; + if (((line as number) < data.starts.length && offset >= end) || offset > end || offset > data.bytes.length || (offset < data.bytes.length && (data.bytes[offset]! & 0xc0) === 0x80)) fail('column outside source line or inside UTF-8 code point'); + } + return { file: data.file, line: line as number, column: column as number | null }; + }; + const point = (v: unknown): HdlMacroPoint => { + const location = source(v); + if (!object(v) || location.column === null || !Number.isSafeInteger(v.byteOffset) || (v.byteOffset as number) < 0) fail('invalid source byte offset'); + const data = fileData(location.file); + if (data.starts[location.line - 1]! + location.column! - 1 !== v.byteOffset) fail('source coordinates disagree with byte offset'); + return { ...location, column: location.column!, byteOffset: v.byteOffset as number }; + }; + const text = (v: unknown, label: string, max = 4096): string => { + if (typeof v !== 'string' || !v || v.length > max || v.includes('\0')) fail('invalid ' + label); + return v; + }; + const result: HdlSemanticFact[] = []; + const identities = new Set(); + let frames = 0; + const macro = (item: Record): Omit => { + if (!Array.isArray(item.macroExpansion) || item.macroExpansion.length > 128 || typeof item.macroExpansionComplete !== 'boolean') fail('invalid macro chain'); + frames += item.macroExpansion.length; + if (frames > 100000) fail('macro frame count limit'); + const macroExpansion = item.macroExpansion.map((frame: unknown): HdlMacroFrame => { + if (!object(frame) || typeof frame.argument !== 'boolean') fail('invalid macro frame'); + const out: HdlMacroFrame = { name: frame.name === null ? null : text(frame.name, 'macro name'), argument: frame.argument }; + if (frame.spelling !== undefined) out.spelling = point(frame.spelling); + if (frame.invocation !== undefined) { + if (!object(frame.invocation)) fail('invalid invocation range'); + out.invocation = { start: point(frame.invocation.start), end: point(frame.invocation.end) }; + if (out.invocation.start.file !== out.invocation.end.file || out.invocation.start.byteOffset > out.invocation.end.byteOffset) fail('invalid invocation order'); + } + return out; + }); + const macroExpansionComplete = item.macroExpansionComplete; + if (macroExpansionComplete && (!macroExpansion.length || macroExpansion.some(f => !f.spelling || !f.invocation || f.invocation.start.byteOffset === f.invocation.end.byteOffset))) fail('incomplete chain marked complete'); + return { macroExpansion, macroExpansionComplete, ...(item.source !== undefined ? { source: source(item.source) } : {}) }; + }; + for (const item of value.facts) { + if (!object(item) || (item.kind !== 'parameter' && item.kind !== 'port' && item.kind !== 'type') || (item.sourceOrigin !== 'direct' && item.sourceOrigin !== 'macro')) fail('invalid fact kind/origin'); + const fact: HdlSemanticFact = { kind: item.kind, name: text(item.name, 'name'), instancePath: text(item.instancePath, 'instance path'), sourceOrigin: item.sourceOrigin }; + const id = JSON.stringify([fact.instancePath, fact.kind, fact.name]); + if (identities.has(id)) fail('duplicate instance fact'); identities.add(id); + if (item.value !== undefined) fact.value = text(item.value, 'value', 65536); + if (item.type !== undefined) fact.type = text(item.type, 'type', 65536); + if (item.width !== undefined) { + if (!Number.isSafeInteger(item.width) || (item.width as number) < 1) fail('invalid width'); + fact.width = item.width as number; + } + if (item.direction !== undefined) fact.direction = text(item.direction, 'direction'); + if (item.source !== undefined) fact.source = source(item.source); + if (item.sourceOrigin === 'macro') Object.assign(fact, macro(item)); + else if (item.macroExpansion !== undefined || item.macroExpansionComplete !== undefined) fail('direct fact carries macro chain'); + if (item.expressionOrigins !== undefined || item.expressionOriginCoverage !== undefined) { + if (!Array.isArray(item.expressionOrigins) || item.expressionOrigins.length > 32 || !object(item.expressionOriginCoverage)) fail('invalid expression origins'); + const coverage = item.expressionOriginCoverage; + const statuses = ['checked', 'not-applicable', 'unavailable']; + if (typeof coverage.initializer !== 'string' || typeof coverage.declaredInitializer !== 'string' || typeof coverage.type !== 'string' || ![...statuses, 'command-line'].includes(coverage.initializer) || !statuses.includes(coverage.declaredInitializer) || !statuses.includes(coverage.type) || typeof coverage.truncated !== 'boolean') fail('invalid expression coverage'); + fact.expressionOriginCoverage = { initializer: coverage.initializer, declaredInitializer: coverage.declaredInitializer, type: coverage.type, truncated: coverage.truncated } as HdlExpressionCoverage; + const keys = { initializer: 'initializer', 'declared-initializer': 'declaredInitializer', type: 'type' } as const; + const seen = new Set(); + fact.expressionOrigins = item.expressionOrigins.map((entry: unknown): HdlExpressionOrigin => { + if (!object(entry) || (entry.role !== 'initializer' && entry.role !== 'declared-initializer' && entry.role !== 'type') || entry.sourceOrigin !== 'macro') fail('invalid expression origin role'); + const status = coverage[keys[entry.role]]; + if (status !== 'checked' && status !== 'command-line') fail('expression origin contradicts coverage'); + const normalized: HdlExpressionOrigin = { role: entry.role, sourceOrigin: 'macro', ...macro(entry) }; + const key = JSON.stringify(normalized); + if (seen.has(key)) fail('duplicate expression origin'); seen.add(key); + return normalized; + }); + } + result.push(fact); + } + return result; +} diff --git a/src/hdl/status.ts b/src/hdl/status.ts new file mode 100644 index 0000000000..7383b12005 --- /dev/null +++ b/src/hdl/status.ts @@ -0,0 +1,129 @@ +/** Public profile state is deliberately separate from the currently configured + * profile: queries describe the context stored with the index until rebuilt. */ +export interface HdlProfileDisplayIdentity { + name: string; + fingerprint: string; + files?: readonly string[]; + includeDirs?: readonly string[]; + defines?: Readonly>; + topModules?: readonly string[]; + languageMode?: 'verilog' | 'systemverilog'; +} +export interface HdlProfileSettingsSummary { + fileCount?: number; + includeDirs?: string[]; + defineNames?: string[]; + topModules?: string[]; + languageMode?: 'verilog' | 'systemverilog'; +} +export interface HdlProfileConfiguration { + status: 'none' | 'active' | 'invalid-config' | 'invalid-profile'; + profile: HdlProfileDisplayIdentity | null; + diagnostics: string[]; +} +export interface HdlIndexedProfileMetadata { + name: string | null; + fingerprint: string | null; + context: string | null; +} +export interface HdlProfileStatus { + configured: { mode: 'raw' | 'profile' | 'invalid'; name: string | null; fingerprint: string | null } & HdlProfileSettingsSummary; + indexed: { mode: 'raw' | 'profile' | 'unknown'; name: string | null; fingerprint: string | null; configurationFingerprint: string | null } & HdlProfileSettingsSummary; + state: 'matches' | 'mismatch' | 'unknown' | 'configuration-error'; + mismatch: boolean | null; + reindexRecommended: boolean; + diagnostics: string[]; + indexedDiagnostics: string[]; + incomplete: boolean; + limitations: string[]; +} +const RAW_LIMIT = 'Raw source may contain multiple conditional alternatives; no HDL build profile was applied.'; +const PROFILE_LIMIT = 'HDL preprocessing selects conditional branches and include macro state; macro invocations are not fully expanded. Displayed snippets are original source and may include inactive branches; indexed symbols follow the selected profile.'; +const DIALECT_LIMIT = 'languageMode records the intended dialect; the SystemVerilog grammar does not enforce Verilog-only conformance. topModules are recorded roots, not an elaborated hierarchy.'; +const SOURCE_LIMIT = 'Source relationships are not elaboration, synthesis, simulation or timing validation.'; + +function settingsSummary(value: object | null): HdlProfileSettingsSummary { + const record = value as Record | null; + if (!record) return {}; + const strings = (candidate: unknown): candidate is string[] => Array.isArray(candidate) && candidate.every(n => typeof n === 'string'); + return { + ...(strings(record.files) ? {fileCount:record.files.length} : {}), + ...(strings(record.includeDirs) ? {includeDirs:[...record.includeDirs]} : {}), + ...(record.defines && typeof record.defines === 'object' && !Array.isArray(record.defines) + ? {defineNames:Object.keys(record.defines).sort()} : {}), + ...(strings(record.topModules) ? {topModules:[...record.topModules]} : {}), + ...(record.languageMode === 'verilog' || record.languageMode === 'systemverilog' ? {languageMode:record.languageMode} : {}), + }; +} + +export function buildHdlProfileStatus( + configuration: HdlProfileConfiguration, + metadata: HdlIndexedProfileMetadata, + hasHdlFiles: boolean, +): HdlProfileStatus | null { + if (!hasHdlFiles && configuration.status === 'none' && !metadata.context && !metadata.fingerprint) return null; + let context: unknown; + try { context = metadata.context ? JSON.parse(metadata.context) : null; } catch { context = null; } + const record = context && typeof context === 'object' && !Array.isArray(context) + ? context as Record : null; + const raw = record?.mode === 'raw'; + if (!hasHdlFiles && configuration.status === 'none' && (raw || !metadata.name)) return null; + const indexed: HdlProfileStatus['indexed'] = raw + ? {mode:'raw',name:null,fingerprint:metadata.fingerprint,configurationFingerprint:null} + : metadata.name && metadata.fingerprint && record?.mode === 'profile' && typeof record.configurationFingerprint === 'string' + ? {mode:'profile',name:metadata.name,fingerprint:metadata.fingerprint,configurationFingerprint:record.configurationFingerprint,...settingsSummary(record)} + : {mode:'unknown',name:metadata.name,fingerprint:metadata.fingerprint,configurationFingerprint:null}; + const indexedDiagnostics = Array.isArray(record?.diagnostics) ? record.diagnostics.flatMap(value => { + if (typeof value === 'string') return [value]; + if (value && typeof value === 'object' && typeof value.message === 'string') { + const location = typeof value.filePath === 'string' ? `${value.filePath}${typeof value.line === 'number' ? ':' + value.line : ''}: ` : ''; + return [location + value.message]; + } + return []; + }) : []; + const invalid = configuration.status === 'invalid-config' || configuration.status === 'invalid-profile' + || (configuration.status === 'active' && !configuration.profile); + const configured: HdlProfileStatus['configured'] = invalid + ? {mode:'invalid',name:configuration.profile?.name ?? null,fingerprint:null} + : configuration.status === 'none' + ? {mode:'raw',name:null,fingerprint:null} + : {mode:'profile',name:configuration.profile!.name,fingerprint:configuration.profile!.fingerprint,...settingsSummary(configuration.profile)}; + const unknown = indexed.mode === 'unknown'; + const mismatch = invalid || unknown ? null : configured.mode !== indexed.mode + || (configured.mode === 'profile' && (configured.fingerprint !== indexed.configurationFingerprint || configured.name !== indexed.name)); + return {configured,indexed,state:invalid ? 'configuration-error' : unknown ? 'unknown' : mismatch ? 'mismatch' : 'matches', + mismatch,reindexRecommended:!invalid && (unknown || mismatch === true),diagnostics:[...configuration.diagnostics], + indexedDiagnostics,incomplete:record?.incomplete === true || indexedDiagnostics.length > 0, + limitations:[...(indexed.mode === 'raw' ? [RAW_LIMIT] : indexed.mode === 'profile' ? [PROFILE_LIMIT,DIALECT_LIMIT] : []), SOURCE_LIMIT]}; +} + +/** Shared compact note for CLI status and MCP explore responses. */ +export function formatHdlProfileStatus(status: HdlProfileStatus | null): string { + if (!status) return ''; + const label = (identity: HdlProfileStatus['indexed'] | HdlProfileStatus['configured']): string => { + if (identity.mode === 'raw') return 'raw source (no profile)'; + if (identity.mode === 'unknown') return 'unknown'; + if (identity.mode === 'invalid') return 'invalid configuration'; + return `${JSON.stringify(identity.name)} (${identity.fingerprint?.slice(0,12)})`; + }; + const lines = [`HDL context: indexed ${label(status.indexed)}; configured ${label(status.configured)}.`]; + if (status.indexed.mode === 'profile') { + const summary = status.indexed; + const parts = [summary.fileCount == null ? undefined : `${summary.fileCount} file(s)`, + summary.includeDirs ? `${summary.includeDirs.length} include path(s)` : undefined, + summary.defineNames ? `${summary.defineNames.length} define(s)` : undefined, + summary.topModules ? `${summary.topModules.length} recorded top module(s)` : undefined, + summary.languageMode ? `intended dialect ${summary.languageMode}` : undefined].filter(Boolean); + if (parts.length) lines.push(`Indexed HDL settings: ${parts.join('; ')}.`); + } + if (status.state === 'mismatch') lines.push('Profile mismatch: rebuild the index before interpreting results as the configured profile.'); + if (status.state === 'unknown') lines.push('Indexed profile provenance is unknown; rebuild the index to establish its HDL context.'); + if (status.state === 'configuration-error') lines.push('Fix the HDL profile configuration before rebuilding; results retain the indexed context.'); + if (status.incomplete) lines.push('Indexed HDL preprocessing is incomplete; unresolved regions are not evidence of an inactive branch.'); + for (const [label, diagnostics] of [['HDL configuration', status.diagnostics], ['Indexed HDL', status.indexedDiagnostics]] as const) { + lines.push(...diagnostics.slice(0,3).map(d=>`${label}: ${d.length > 240 ? d.slice(0,237) + '...' : d}`)); + if (diagnostics.length > 3) lines.push(`${label}: ${diagnostics.length - 3} additional diagnostics; inspect codegraph status --json.`); + } + lines.push(...status.limitations); + return lines.join('\n'); +} diff --git a/src/index.ts b/src/index.ts index ae27c17aaf..7309714867 100644 --- a/src/index.ts +++ b/src/index.ts @@ -32,6 +32,11 @@ import { DatabaseConnection, getDatabasePath, removeDatabaseFiles } from './db'; import { WalCheckpointValve, resolveWalValveMb } from './db/wal-valve'; import { QueryBuilder, type DeprioritizedPathMatcher } from './db/queries'; import { importPathKeys, moduleReferenceKeys } from './db/reference-tail'; +import { loadHdlProfile } from './hdl/profile'; +import { hdlDependenciesChanged } from './hdl/index-context'; +import { buildHdlProfileStatus, type HdlProfileStatus } from './hdl/status'; +import { analyzeHdlSemantics, type HdlSemanticOptions, type HdlSemanticResult } from './hdl/semantics'; +export type { HdlSemanticOptions, HdlSemanticResult } from './hdl/semantics'; import { isInitialized, createDirectory, @@ -862,6 +867,8 @@ export class CodeGraph { } } catch { /* metadata is advisory — never fail an index over it */ } + if (result.success && result.filesErrored === 0) this.orchestrator.commitHdlProfile(); + // The time covers the whole run, like the totals above. The // orchestrator's covers extraction alone, and the summary printed it // as the run's, leaving out resolution and linking (#2334). @@ -910,7 +917,9 @@ export class CodeGraph { // are stored, as indexAll and sync hold it: returned unawaited, the // promise left the `try` at once and the `finally` let the lock go as // soon as indexing started, so another process could write meanwhile. - return await this.orchestrator.indexFiles(filePaths).finally(() => this.resolver.clearCaches()); + const result = await this.orchestrator.indexFiles(filePaths).finally(() => this.resolver.clearCaches()); + if (result.success && result.filesErrored === 0) this.orchestrator.commitHdlProfile(); + return result; } finally { this.fileLock.release(); } @@ -1452,6 +1461,11 @@ export class CodeGraph { result.changedFilePaths ?? [], edgesBeforeRetry ); + // A deleted duplicate module can make an HDL binding unique again. + // Deletion-only syncs skip the changed-file failed-ref retry above. + const hdlRetry = this.queries.getRetryableFailedReferences(result.definitionDelta) + .filter(ref => ref.referenceName.startsWith('hdl:wildcard:') || ref.referenceName.startsWith('hdl:port-position:')); + if (hdlRetry.length > 0) await this.resolver.resolveAndPersistListYielding(hdlRetry); if (process.env.CODEGRAPH_SYNTH_TIMINGS) { console.error( `[phase-timing] sync-rebind: ${Date.now() - tRebind}ms (${result.definitionDelta.length} changed names, ${rebound} edges re-opened)` @@ -1524,6 +1538,10 @@ export class CodeGraph { if (before !== now) this.stampPythonRootFingerprint(now); } + // Access direction is signature-dependent even if no node name changed. + // Re-open typed and still-unclassified HDL argument sites in untouched files. + this.orchestrator.resurrectHdlCallArgumentEdges(result.changedFilePaths ?? [], result.filesRemoved > 0); + // Orphan sweep (#1187). A resolution pass that dies mid-run — the #850 // daemon liveness watchdog's SIGKILL (#1122), Ctrl-C, a crash — leaves // the refs it never reached in unresolved_refs, and the git-scoped fast @@ -1639,6 +1657,7 @@ export class CodeGraph { try { this.queries.setMetadata('index_state', 'complete'); } catch { /* advisory */ } } + this.orchestrator.commitHdlProfile(); this.orchestrator.finishGitIndexState(gitState, fullReconcile, result.failedFilePaths); if (fullReconcile && result.filesChecked > 0) this.pendingFullReconcile = false; @@ -1717,7 +1736,8 @@ export class CodeGraph { options, process.platform === 'win32' ? (filePath) => this.isIndexedFileStateCurrent(filePath) - : undefined + : undefined, + () => this.getHdlDependencyPaths() ); return this.watcher.start(); @@ -1866,6 +1886,36 @@ export class CodeGraph { * index built before stamping existed (treated as stale). See * `extraction-version.ts` and `isIndexStale()`. */ + getHdlDependencyPaths(): string[] { + try { + const context = JSON.parse(this.queries.getMetadata('hdl_profile_context') ?? 'null'); + return Array.isArray(context?.dependencies) ? context.dependencies.flatMap((d: unknown) => + d && typeof d === 'object' && 'path' in d && typeof d.path === 'string' ? [d.path] : []) : []; + } catch { return []; } + } + + /** Explicit optional compiler query; source graph remains available if compilation fails. */ + async getHdlSemantics(options: HdlSemanticOptions): Promise { + return analyzeHdlSemantics(this.projectRoot, options, { + profile: () => this.getHdlProfileStatus(), stale: () => this.isIndexStale(), + file: file => this.queries.getFileByPath(file), nodes: file => this.queries.getNodesByFile(file), + }); + } + + getHdlProfileStatus(): HdlProfileStatus | null { + const configuration = loadHdlProfile(this.projectRoot); + const metadata = { name: this.queries.getMetadata('hdl_profile_name') || null, + fingerprint: this.queries.getMetadata('hdl_profile_fingerprint'), context: this.queries.getMetadata('hdl_profile_context') }; + const hasHdl = this.queries.hasLanguage('verilog'); + if (!hasHdl && !metadata.name && configuration.status === 'invalid-config') return null; + const status = buildHdlProfileStatus(configuration, metadata, hasHdl); + if (status?.state === 'matches' && status.indexed.mode === 'profile' && hdlDependenciesChanged(this.projectRoot, metadata.context)) { + status.state = 'mismatch'; status.mismatch = true; status.reindexRecommended = true; + status.diagnostics.push('HDL include dependency changed since indexing.'); + } + return status; + } + getIndexBuildInfo(): { version: string | null; extractionVersion: number | null } { const version = this.queries.getMetadata('indexed_with_version'); const ev = this.queries.getMetadata('indexed_with_extraction_version'); diff --git a/src/mcp/hdl-access.ts b/src/mcp/hdl-access.ts new file mode 100644 index 0000000000..9e1713ed2a --- /dev/null +++ b/src/mcp/hdl-access.ts @@ -0,0 +1,102 @@ +import type CodeGraph from '../index'; +import type { Edge, Node } from '../types'; +import { readFileSync, statSync } from 'node:fs'; +import { createHash } from 'node:crypto'; +import { validatePathWithinRoot } from '../utils'; + +export const HDL_ACCESS_FILTERS = ['read', 'write', 'readwrite', 'control', 'event', 'all'] as const; +export type HdlAccessFilter = typeof HDL_ACCESS_FILTERS[number]; +const ACCESS_KINDS = new Set(['read', 'write', 'readwrite', 'control', 'event']); + +export function hdlAccessKinds(edge: Edge): string[] { + const value = edge.metadata?.hdlAccess; + return Array.isArray(value) ? [...new Set(value.filter((v): v is string => typeof v === 'string' && ACCESS_KINDS.has(v)))].sort() : []; +} +function matches(kinds: string[], filter: HdlAccessFilter): boolean { + if (filter === 'all') return kinds.length > 0; + if (filter === 'read') return kinds.some(k => ['read', 'readwrite', 'control', 'event'].includes(k)); + if (filter === 'write') return kinds.some(k => k === 'write' || k === 'readwrite'); + if (filter === 'readwrite') return kinds.includes('readwrite') || (kinds.includes('read') && kinds.includes('write')); + return kinds.includes(filter); +} + +/** Exact HDL signal access sites, using raw edges rather than neighbour-deduped callers. */ +export function formatHdlAccess(cg: CodeGraph, query: string, filter: HdlAccessFilter, maxFiles = 12): string { + const name = query.trim(); + const exact = name.includes('::') ? cg.getNodesByQualifiedName(name) : cg.getNodesByName(name); + const matchesByName = exact.length || !name.includes('.') || name.includes('\\') + ? exact : cg.getNodesByQualifiedName(name.replace(/\./g, '::')); + const targets = matchesByName + .filter(n => n.language === 'verilog' && ['field', 'variable', 'constant'].includes(n.kind)) + .sort((a, b) => a.filePath.localeCompare(b.filePath) || a.startLine - b.startLine || a.qualifiedName.localeCompare(b.qualifiedName)); + if (!targets.length) return `No exact HDL signal matches "${name}". Use one signal name or qualified name (for example top::data).`; + const lines = [`**HDL access: ${name} — ${filter}**`, + 'Source occurrences only: read includes control/event uses; write follows a syntactic assignment/update or a known callable formal\'s argument direction, not an elaborated driver or timing result.']; + const sourceCache = new Map(); + let snippetBudget = 12000, shownSites = 0, totalSites = 0, unclassified = 0; + const sourceLines = (file: string): string[] | null => { + if (sourceCache.has(file)) return sourceCache.get(file)!; + if (sourceCache.size >= maxFiles) return null; + let source: string[] | null = null; + try { + const record = cg.getFile(file); + const absolute = validatePathWithinRoot(cg.getProjectRoot(), file); + if (record && absolute) { + const before = statSync(absolute); + if (before.size === record.size && Math.floor(before.mtimeMs) === Math.floor(record.modifiedAt)) { + const text = readFileSync(absolute, 'utf8'); + const after = statSync(absolute); + // Same SHA-256 over UTF-8 text as extraction's hashContent. Avoid + // importing the heavy extraction module on the MCP tools/list path. + const hash = createHash('sha256').update(text).digest('hex'); + if (before.size === after.size && before.mtimeMs === after.mtimeMs && hash === record.contentHash) source = text.split(/\r?\n/); + } + } + } catch { /* A missing or changed file must never be presented as the indexed occurrence. */ } + sourceCache.set(file, source); + return source; + }; + for (const target of targets) { + const sites: { edge: Edge; owner: Node; kinds: string[] }[] = []; + for (const edge of cg.getIncomingEdges(target.id)) { + if (edge.kind !== 'references') continue; + const owner = cg.getNode(edge.source); + if (!owner || owner.language !== 'verilog') continue; + const kinds = hdlAccessKinds(edge); + if (!kinds.length) { unclassified++; continue; } + if (matches(kinds, filter)) sites.push({ edge, owner, kinds }); + } + sites.sort((a, b) => a.owner.filePath.localeCompare(b.owner.filePath) || (a.edge.line ?? 0) - (b.edge.line ?? 0) || (a.edge.column ?? 0) - (b.edge.column ?? 0) || a.owner.qualifiedName.localeCompare(b.owner.qualifiedName)); + totalSites += sites.length; + const definitionVerified = sourceLines(target.filePath) !== null; + lines.push('', `**${target.qualifiedName}** — ${target.filePath}:${target.startLine}`, `${sites.length} matching access site(s).`); + if (!definitionVerified) lines.push('Definition location is from the index; current source is unverified (changed/unavailable or maxFiles reached).'); + for (const { edge, owner, kinds } of sites) { + if (shownSites >= 60) continue; + shownSites++; + lines.push(`- [${kinds.join(', ')}] ${owner.qualifiedName} — ${owner.filePath}:${edge.line ?? owner.startLine}${typeof edge.column === 'number' ? `:${edge.column + 1}` : ''}`); + const callableId = edge.metadata?.hdlCallTargetId, formalId = edge.metadata?.hdlFormalId; + if (typeof callableId === 'string' && typeof formalId === 'string') { + const callable = cg.getNode(callableId), formal = cg.getNode(formalId); + const direction = formal?.decorators?.find(d => d.startsWith('hdl:direction:'))?.slice('hdl:direction:'.length); + if (callable?.language === 'verilog' && formal?.language === 'verilog' && direction) { + const status = sourceLines(callable.filePath) === null ? ' [indexed definition; source unverified]' : ''; + lines.push(` Formal direction: ${callable.qualifiedName}(${direction} ${formal.name}).${status}`); + } + } + const source = sourceLines(owner.filePath); + if (!source) { lines.push(' Source omitted: changed/unavailable since indexing, or maxFiles reached.'); continue; } + if (!edge.line || edge.line > source.length) { lines.push(' No verified source line for this occurrence.'); continue; } + const start = Math.max(1, edge.line - 1), end = Math.min(source.length, edge.line + 1); + const excerpt = source.slice(start - 1, end).map((s, i) => `${start + i}\t${s}`).join('\n'); + if (excerpt.length > snippetBudget || excerpt.length > 2000) { lines.push(' Source snippet omitted by output budget.'); continue; } + snippetBudget -= excerpt.length; + const fence = '`'.repeat(Math.max(3, ...(excerpt.match(/`+/g) ?? []).map(s => s.length + 1))); + lines.push(`${fence}systemverilog`, excerpt, fence); + } + } + if (shownSites < totalSites) lines.push('', `Showing ${shownSites} of ${totalSites} matching sites; narrow with a qualified signal name.`); + if (unclassified) lines.push('', `${unclassified} unclassified reference(s) excluded. Connections and unknown uses do not imply reads or writes.`); + lines.push('', 'An empty result is not proof of no access outside the supported source patterns.'); + return lines.join('\n'); +} diff --git a/src/mcp/server-instructions.ts b/src/mcp/server-instructions.ts index 80a6247a80..f25c250bbf 100644 --- a/src/mcp/server-instructions.ts +++ b/src/mcp/server-instructions.ts @@ -59,6 +59,10 @@ calls; a grep/read exploration is dozens. - Named-symbol call paths require exact matches; partial or mistyped names are never silently substituted as flow endpoints. If a graph query reports a missing symbol with did-you-mean suggestions, query the suggested name explicitly. - Explore matches names and indexed code words lexically, not by meaning; an empty result reports word matches and may suggest indexed candidate names to retry with \`codegraph_explore\`. +## Verilog / SystemVerilog + +Modules, instances, ports, signals, packages, interfaces/modports, always/assign blocks and generate scopes are indexed; \`codegraph_explore\` follows instantiation from a top module down. Pass one exact signal name with \`hdlAccess\` (\`read\`, \`write\`, \`readwrite\`, \`control\`, \`event\`, \`all\`) to list where it is read or written; an empty result is not proof of no access. Relationships are syntactic: no elaboration, no macro expansion, no timing. An optional \`hdl\` profile in \`codegraph.json\` selects source units and conditional branches, and the explore header reports the indexed profile versus the configured one — on a mismatch, sync before reading the result as the new profile. Evaluated parameters and port widths come only from the explicit \`codegraph hdl-semantic\` CLI with an installed slang. + ## Anti-patterns - **Trust codegraph's results — don't re-verify them with grep.** They come from a full AST parse; re-checking with grep is slower, less accurate, and wastes context. diff --git a/src/mcp/tools.ts b/src/mcp/tools.ts index 89fa53101f..c268b39011 100644 --- a/src/mcp/tools.ts +++ b/src/mcp/tools.ts @@ -4,7 +4,9 @@ * Defines the tools exposed by the CodeGraph MCP server. */ +import { formatHdlProfileStatus } from '../hdl/status'; import type CodeGraph from '../index'; +import { formatHdlAccess, HDL_ACCESS_FILTERS, type HdlAccessFilter } from './hdl-access'; import type { QueryPool } from './query-pool'; import { findNearestCodeGraphRoot, isSameIndexRoot } from '../directory'; import { WslSharedIndexError } from '../db/wsl-shared-index'; @@ -1838,6 +1840,11 @@ export const tools: ToolDefinition[] = [ type: 'number', description: 'Maximum number of files to return source from, 1–20. Omit it to use a default sized to the project (4 files below 150 indexed files, 5 below 500, 8 otherwise). Total output is also capped by characters (about 13K–24K by project size), so a higher maxFiles spreads that budget over more files rather than returning more text.', }, + hdlAccess: { + type: 'string', + description: 'HDL signal access filter. With this option query must be one exact signal name or qualified name; read includes control/event uses, write includes readwrite. Returns access sites and source, not elaborated drivers.', + enum: [...HDL_ACCESS_FILTERS], + }, projectPath: projectPathProperty, }, required: ['query'], @@ -3164,7 +3171,17 @@ export class ToolHandler { case 'codegraph_callers': return await this.handleCallers(args); case 'codegraph_callees': return await this.handleCallees(args); case 'codegraph_impact': return await this.handleImpact(args); - case 'codegraph_explore': return await this.handleExplore(args); + case 'codegraph_explore': { + const result = await this.handleExplore(args); + if (result.isError) return result; + const profile = this.getCodeGraph(args.projectPath as string | undefined).getHdlProfileStatus?.() ?? null; + const note = formatHdlProfileStatus(profile); + const first = result.content[0]; + if (!note || first?.type !== 'text') return result; + const emission = result[EXPLORE_EMISSION_KEY]; + return { ...result, content: [{type:'text', text:`${note}\n\n${first.text}`}, ...result.content.slice(1)], + ...(emission ? {[EXPLORE_EMISSION_KEY]: {...emission,responseBytes:emission.responseBytes + note.length + 2}} : {}) }; + } case 'codegraph_node': return await this.handleNode(args); case 'codegraph_files': return await this.handleFiles(args); default: return this.errorResult(`Unknown tool: ${toolName}`); @@ -4214,6 +4231,13 @@ export class ToolHandler { const cg = this.getCodeGraph(args.projectPath as string | undefined); const projectRoot = cg.getProjectRoot(); + if (args.hdlAccess !== undefined) { + if (typeof args.hdlAccess !== 'string' || !HDL_ACCESS_FILTERS.includes(args.hdlAccess as HdlAccessFilter)) { + return this.errorResult(`hdlAccess must be one of: ${HDL_ACCESS_FILTERS.join(', ')}`); + } + return this.textResult(this.truncateOutput(formatHdlAccess(cg, rawQuery, args.hdlAccess as HdlAccessFilter, + clamp((args.maxFiles as number) || 12, 1, 20)))); + } // Resolve adaptive output budget from project size. Falls back to the // largest-tier defaults if stats aren't available, which preserves diff --git a/src/resolution/index.ts b/src/resolution/index.ts index 9430776a53..431391897e 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -43,6 +43,9 @@ import { isJsSuperCallLanguage, isJsSuperCallSite, matchJsSuperCall } from './js import { CLASS_EXPRESSION_REF_LANGUAGES, isCommonJsExportedClass, isJsClassExpression, writtenIn, type ClassExpressionIndex } from './js-class-expressions'; import { matchRubyConstantPathRef } from './ruby-constant-paths'; import { resolveViaImport, resolvePhpImportedStaticCall, resolvePhpQualifiedClassRef, resolveTsQualifiedTypeRef, resolveJvmImport, extractImportMappings, extractReExports, loadCppIncludeDirs, isPhpIncludePathRef, isCobolCopybookRef, isNixPathImportRef, isDartImportRef, isLuaRequireRef, isJsPathImportRef, isBoundToOutOfRepoImport, reachesAmbientModule, importBindingTail, clearImportResolverMemos, resolveImportPath, isExternalImport, ESM_IMPORT_LANGUAGES, jsModuleFile, pythonRootFingerprint, pythonReopenScope, pythonPackageImporters, moduleDefaultExport, isJsFamilyFile, isModuleBindingName } from './import-resolver'; +import { isVerilogMemberRef, matchVerilogMember } from './verilog-members'; +import { isVerilogPortRef, matchVerilogPort } from './verilog-ports'; +import { isVerilogWildcardRef, matchVerilogWildcard } from './verilog-wildcard'; import { ResolverPool, minRefsForPool, shouldEngageAdaptively } from './resolver-pool'; import { resolveAliasBinding } from './alias-binding'; import { detectFrameworks, getResolvingFrameworks } from './frameworks'; @@ -1146,6 +1149,7 @@ export class ReferenceResolver { filePath: ref.filePath || this.getFilePathFromNodeId(ref.fromNodeId), language: ref.language || this.getLanguageFromNodeId(ref.fromNodeId), rowId: ref.rowId, + candidates: ref.candidates, })); const total = refs.length; @@ -1566,6 +1570,9 @@ export class ReferenceResolver { const superCall = matchJsSuperCall(ref, this.context); if (superCall !== undefined) return superCall; + if (isVerilogWildcardRef(ref)) return matchVerilogWildcard(ref, this.context, matchReference); + if (isVerilogPortRef(ref)) return matchVerilogPort(ref, this.context, matchReference); + if (isVerilogMemberRef(ref)) return matchVerilogMember(ref, this.context); // A Kotlin call through a receiver chain (`engine.pump.drain()`) resolves // on the chain's declared type, or gets no edge when that type is a // library one. An untyped chain resolves as the bare method name, the ref @@ -2018,6 +2025,8 @@ export class ReferenceResolver { // wrong rebind; edges without refName (pre-#1240, synthesized) are // deliberately NOT resurrected for the same reason. refName: ref.original.referenceName, + ...(ref.original.language === 'verilog' && ref.original.candidates?.length + ? { refCandidates: ref.original.candidates } : {}), ...(ref.original.referenceKind !== kind ? { refKind: ref.original.referenceKind } : {}), // Uniform marker for function-as-value edges (#756), regardless of // which strategy resolved them (import vs matchFunctionRef) — lets @@ -2323,6 +2332,7 @@ export class ReferenceResolver { filePath: raw.filePath || this.getFilePathFromNodeId(raw.fromNodeId), language: raw.language || this.getLanguageFromNodeId(raw.fromNodeId), rowId: raw.rowId, + candidates: raw.candidates, }; const result = this.resolveOneTimed(ref); if (result) { @@ -2443,6 +2453,7 @@ export class ReferenceResolver { filePath: raw.filePath || this.getFilePathFromNodeId(raw.fromNodeId), language: raw.language || this.getLanguageFromNodeId(raw.fromNodeId), rowId: raw.rowId, + candidates: raw.candidates, }; const result = this.resolveOneTimed(ref); if (result) { diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index 9debc1ec29..3752bece78 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -27,6 +27,10 @@ import { inBlock, isDeclarationFile, isMergedAmbientDeclaration, scanGlobalScope import { isMinifiedContent } from '../extraction/generated-detection'; import { getCargoWorkspaceCrateMap } from './frameworks/cargo-workspace'; import { readRustScopes, rustBindingAt, rustInnerBlockAt, rustBoundsOf, rustFnAt, rustImplAt, splitRustTopLevel, type RustBinding, type RustPatternSite, type RustScopes } from './rust-scopes'; +import { isVerilogMemberRef, matchVerilogMember } from './verilog-members'; +import { isVerilogPortRef, matchVerilogPort } from './verilog-ports'; +import { isVerilogWildcardRef, matchVerilogWildcard } from './verilog-wildcard'; + /** * Ceiling on how many same-named definitions a FUZZY name-match strategy will * score. A name defined more times than this is "ubiquitous" — a method/symbol @@ -28595,6 +28599,13 @@ export function dumpNameMatcherProfile(label: string): void { } } +function isVerilogSimPath(filePath: string): boolean { + return ( + /(^|\/)(sim|tb|tests?|testbench|dv)\//i.test(filePath) || + /_(stub|tb|sim)\.s?vh?$/i.test(filePath) + ); +} + export function matchReference( ref: UnresolvedRef, context: ResolutionContext @@ -28799,6 +28810,28 @@ function matchReferenceInner( ref: UnresolvedRef, context: ResolutionContext ): ResolvedRef | null { + if (isVerilogWildcardRef(ref)) return matchVerilogWildcard(ref, context, matchReference); + if (isVerilogPortRef(ref)) return matchVerilogPort(ref, context, matchReference); + if (isVerilogMemberRef(ref)) return matchVerilogMember(ref, context); + if (ref.language === 'verilog' && ref.referenceKind === 'instantiates' && !isVerilogSimPath(ref.filePath)) { + const modules = context + .getNodesByName(ref.referenceName) + .filter((n) => n.language === 'verilog' && (n.kind === 'class' || n.kind === 'interface')); + const synth = modules.filter((n) => !isVerilogSimPath(n.filePath)); + if (synth.length > 0 && synth.length < modules.length) { + const best = synth.length === 1 ? synth[0]! : findBestMatch(ref, synth, context); + if (best) { + const proximity = computePathProximity(ref.filePath, best.filePath); + return { + original: ref, + targetNodeId: best.id, + confidence: synth.length === 1 || proximity >= 30 ? 0.7 : 0.4, + resolvedBy: 'exact-match', + }; + } + } + } + // Function-as-value refs (#756) resolve ONLY through the dedicated matcher — // never the fuzzy/qualified fallthrough below (a wrong callback edge is // worse than none). diff --git a/src/resolution/resurrect-ref.ts b/src/resolution/resurrect-ref.ts index 52cb8bc4de..fa502d2022 100644 --- a/src/resolution/resurrect-ref.ts +++ b/src/resolution/resurrect-ref.ts @@ -30,6 +30,9 @@ export function resurrectRefFromDroppedEdge( fromNodeId: e.source, referenceName: refName, referenceKind: refKind, + ...(e.sourceLanguage === 'verilog' && Array.isArray(e.metadata?.refCandidates) + && e.metadata.refCandidates.every((c: unknown) => typeof c === 'string') + ? { candidates: e.metadata.refCandidates as string[] } : {}), line: e.line ?? 0, column: e.column ?? 0, filePath: e.sourceFilePath, diff --git a/src/resolution/verilog-call-access.ts b/src/resolution/verilog-call-access.ts new file mode 100644 index 0000000000..4faedce822 --- /dev/null +++ b/src/resolution/verilog-call-access.ts @@ -0,0 +1,55 @@ +import { VERILOG_CALL_ARGUMENT_PREFIX, type VerilogCallArgument } from '../extraction/languages/verilog-call-access'; +import type { UnresolvedRef, ResolutionContext, ResolvedRef } from './types'; + +export interface VerilogCallAccess { + access: 'read' | 'write' | 'readwrite'; + callableId: string; + formalId: string; +} + +/** Classify a call actual only after a precise callable and complete formal list are known. */ +export function resolveVerilogCallArgumentAccess( + ref: UnresolvedRef, + context: ResolutionContext, + resolveCallable: (ref: UnresolvedRef, context: ResolutionContext) => ResolvedRef | null, +): VerilogCallAccess | undefined { + const token = ref.candidates?.find(c => c.startsWith(VERILOG_CALL_ARGUMENT_PREFIX)); + if (!token) return undefined; + let info: VerilogCallArgument; + try { info = JSON.parse(token.slice(VERILOG_CALL_ARGUMENT_PREFIX.length)); } catch { return undefined; } + if (!info || typeof info.callee !== 'string' || typeof info.scope !== 'string' || typeof info.callableId !== 'string' + || !Array.isArray(info.slots) || !['value', 'index'].includes(info.role) || typeof info.writeTarget !== 'boolean' + || typeof info.referenceTarget !== 'boolean') return undefined; + const caller = context.getNodeById?.(info.callableId); + if (!caller || caller.filePath !== ref.filePath || caller.language !== 'verilog') return undefined; + const target = resolveCallable({ ...ref, fromNodeId: caller.id, referenceName: info.callee, + referenceKind: 'calls', candidates: [`hdl:scope:${info.scope}`] }, context); + const callable = target && context.getNodeById?.(target.targetNodeId); + if (!callable || callable.language !== 'verilog' || !['function', 'method'].includes(callable.kind)) return undefined; + const formals = context.getNodesInFile(callable.filePath).filter(n => n.decorators?.includes('hdl:formal') + && n.qualifiedName === `${callable.qualifiedName}::${n.name}` && n.startLine >= callable.startLine && n.endLine <= callable.endLine); + const indexed = formals.map(node => ({node, index: Number(node.decorators?.find(d => d.startsWith('hdl:formal-index:'))?.slice('hdl:formal-index:'.length))})).sort((a,b) => a.index-b.index); + if (!indexed.length || indexed.some((f,i) => f.index !== i)) return undefined; + const canonical = (name: string) => name.replace(/^\\/, ''); + if (new Set(indexed.map(f => canonical(f.node.name))).size !== indexed.length) return undefined; + const assigned = new Set(); + let selected: typeof indexed[number] | undefined; + for (const slot of info.slots) { + if (!slot || typeof slot.omitted !== 'boolean') return undefined; + const formal = typeof slot.key === 'number' && Number.isSafeInteger(slot.key) ? indexed[slot.key] + : typeof slot.key === 'string' ? indexed.find(f => canonical(f.node.name) === canonical(slot.key as string)) : undefined; + if (!formal || assigned.has(formal.index)) return undefined; + assigned.add(formal.index); + if (slot.omitted && !formal.node.decorators?.includes('hdl:default')) return undefined; + if (slot.key === info.argument && !slot.omitted) selected = formal; + } + if (!selected || indexed.some(f => !assigned.has(f.index) && !f.node.decorators?.includes('hdl:default'))) return undefined; + const direction = selected.node.decorators?.find(d => d.startsWith('hdl:direction:'))?.slice('hdl:direction:'.length); + let access: VerilogCallAccess['access']; + if (info.role === 'index' || direction === 'input' || direction === 'const-ref') access = 'read'; + else if (!info.writeTarget) return undefined; + else if (direction === 'output') access = 'write'; + else if (direction === 'inout' || (direction === 'ref' && info.referenceTarget)) access = 'readwrite'; + else return undefined; + return {access, callableId: callable.id, formalId: selected.node.id}; +} diff --git a/src/resolution/verilog-members.ts b/src/resolution/verilog-members.ts new file mode 100644 index 0000000000..9ba8ae93c7 --- /dev/null +++ b/src/resolution/verilog-members.ts @@ -0,0 +1,74 @@ +import type { Node } from '../types'; +import { resolveVerilogCallArgumentAccess } from './verilog-call-access'; +import type { ResolvedRef, ResolutionContext, UnresolvedRef } from './types'; + +export function isVerilogMemberRef(ref: UnresolvedRef): boolean { + return ref.language === 'verilog' && (ref.referenceKind === 'calls' || ref.referenceKind === 'type_of' || + (ref.referenceKind === 'references' && (ref.referenceName.includes('::') || ref.referenceName.startsWith('hdl:signal:')))); +} + +/** Resolve only explicit HDL identity or visible lexical/import scope, never a trailing-name guess. */ +export function matchVerilogMember(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { + const exact = (name: string, kinds: Node['kind'][]) => context.getNodesByQualifiedName(name) + .filter(n => n.language === 'verilog' && kinds.includes(n.kind)); + const result = (nodes: Node[]): ResolvedRef | null => { + const unique = [...new Map(nodes.map(n => [n.id, n])).values()]; + if (unique.length !== 1) return null; + let metadata: Record | undefined; + if (ref.referenceKind === 'references' && ref.referenceName.startsWith('hdl:signal:')) { + const access = [...new Set((ref.candidates ?? []).filter(c => c.startsWith('hdl:access:')) + .map(c => c.slice('hdl:access:'.length)).filter(c => ['read', 'write', 'readwrite', 'control', 'event'].includes(c)))]; + const argument = access.length ? undefined : resolveVerilogCallArgumentAccess(ref, context, matchVerilogMember); + if (argument) access.push(argument.access); + const events = [...new Set((ref.candidates ?? []).filter(c => c.startsWith('hdl:event:')) + .map(c => c.slice('hdl:event:'.length)).filter(c => ['posedge', 'negedge'].includes(c)))]; + if (access.length) metadata = { hdlAccess: access, ...(events.length ? { hdlEvent: events } : {}), + ...(argument ? { hdlCallTargetId: argument.callableId, hdlFormalId: argument.formalId } : {}) }; + } + return { original: ref, targetNodeId: unique[0]!.id, confidence: 0.95, + resolvedBy: 'qualified-name', ...(metadata ? { metadata } : {}) }; + }; + if (ref.referenceKind === 'references' && ref.referenceName.startsWith('hdl:signal:')) { + const name = ref.referenceName.slice('hdl:signal:'.length); + const caller = context.getNodeById?.(ref.fromNodeId); + const scopes = caller?.qualifiedName.split('::') ?? []; + for (let i = scopes.length; i > 0; i--) { + const matches = exact(`${scopes.slice(0, i).join('::')}::${name}`, ['field', 'variable', 'constant']) + .filter(n => n.filePath === ref.filePath); + if (matches.length) return result(matches); + } + return null; + } + if (ref.referenceKind === 'references') return result(exact(ref.referenceName, ['field', 'variable', 'constant']) + .filter(n => n.filePath === ref.filePath)); + if (ref.referenceKind === 'type_of') return result(exact(ref.referenceName.replace(/\./g, '::'), ['interface', 'class', 'type_alias'])); + if (ref.referenceName.includes('::')) return result(exact(ref.referenceName, ['function', 'method'])); + // Hierarchical instance calls require elaborated receiver identity, which this pass does not infer. + if (ref.referenceName.includes('.')) return null; + const caller = context.getNodeById?.(ref.fromNodeId); + const hint = ref.candidates?.find(c => c.startsWith('hdl:scope:'))?.slice('hdl:scope:'.length); + const lexical = hint && caller && context.getNodesByQualifiedName(hint).some(n => + n.language === 'verilog' && n.filePath === ref.filePath && n.kind === 'namespace' + && n.qualifiedName.startsWith(`${caller.qualifiedName}::`) && n.startLine <= ref.line && n.endLine >= ref.line) + ? hint : caller?.qualifiedName; + const scopes = lexical?.split('::') ?? []; + const imports = context.getNodesInFile(ref.filePath).filter(n => n.kind === 'import' && n.startLine <= ref.line); + // Resolve one lexical level at a time: an inner package import can shadow + // an outer module function, just like a nearer declaration can. + for (let i = scopes.length; i >= 0; i--) { + const scope = scopes.slice(0, i).join('::'); + const matches = exact(scope ? `${scope}::${ref.referenceName}` : ref.referenceName, ['function', 'method']) + .filter(n => n.filePath === ref.filePath); + if (matches.length) return result(matches); + const imported: Node[] = []; + for (const item of imports) { + if (item.qualifiedName.split('::').slice(0, -1).join('::') !== scope) continue; + const match = item.signature?.trim().match(/^([\w$]+)::([\w$]+|\*)$/); + if (match && (match[2] === '*' || match[2] === ref.referenceName)) { + imported.push(...exact(`${match[1]}::${ref.referenceName}`, ['function', 'method'])); + } + } + if (imported.length) return result(imported); + } + return null; +} diff --git a/src/resolution/verilog-ports.ts b/src/resolution/verilog-ports.ts new file mode 100644 index 0000000000..d6dbf5fe91 --- /dev/null +++ b/src/resolution/verilog-ports.ts @@ -0,0 +1,93 @@ +import type { Node } from '../types'; +import type { ResolutionContext, ResolvedRef, UnresolvedRef } from './types'; + +const PREFIX = 'hdl:port:'; +const POSITION_PREFIX = 'hdl:port-position:'; + +export function isVerilogPortRef(ref: UnresolvedRef): boolean { + return ref.language === 'verilog' && ref.referenceKind === 'references' && (ref.referenceName.startsWith(PREFIX) || ref.referenceName.startsWith(POSITION_PREFIX)); +} + +/** Bind a named source connection to the selected module's own declared port. + * Reuse the instantiation matcher so simulation-file selection cannot diverge. + * This is source correspondence, not an elaborated net or drive-direction edge. + */ +export function matchVerilogPort( + ref: UnresolvedRef, + context: ResolutionContext, + resolveModule: (ref: UnresolvedRef, context: ResolutionContext) => ResolvedRef | null, +): ResolvedRef | null { + const positional = ref.referenceName.startsWith(POSITION_PREFIX); + let parts: unknown; + try { parts = JSON.parse(ref.referenceName.slice(positional ? POSITION_PREFIX.length : PREFIX.length)); } catch { return null; } + if (!Array.isArray(parts)) return null; + let moduleName: string, instanceId: string, portName: string; + let position = -1, count = -1; + if (positional) { + if (parts.length !== 4 || typeof parts[0] !== 'string' || !parts[0] || typeof parts[3] !== 'string' || !parts[3] + || !Number.isSafeInteger(parts[1]) || !Number.isSafeInteger(parts[2]) || parts[1] < 0 || parts[2] <= parts[1]) return null; + [moduleName, position, count, instanceId] = parts as [string, number, number, string]; + portName = ''; + } else { + if (parts.length !== 3 || !parts.every(p => typeof p === 'string' && p.length > 0)) return null; + [moduleName, portName, instanceId] = parts as [string, string, string]; + } + const selected = selectVerilogModule(ref, context, resolveModule, moduleName, instanceId); + if (!selected) return null; + const module = selected.module; + if (positional) { + const tag = module.decorators?.find(d => d.startsWith('hdl:port-order:')); + let order: unknown; + try { order = tag && JSON.parse(tag.slice('hdl:port-order:'.length)); } catch { return null; } + if (!Array.isArray(order) || !order.every(n => typeof n === 'string') || count > order.length + || new Set(order.map(n => n.replace(/^\\/, ''))).size !== order.length) return null; + portName = order[position]!; + } + const canonicalPort = portName.replace(/^\\/, ''); + const names = [canonicalPort, `\\${canonicalPort}`]; + const ports = names.flatMap(name => context.getNodesByQualifiedName(`${module.qualifiedName}::${name}`)) + .filter(n => n.language === 'verilog' && n.filePath === module.filePath && n.kind === 'field' + && n.decorators?.includes('hdl:port') && n.startLine >= module.startLine && n.endLine <= module.endLine); + if (ports.length !== 1) return null; + return { original: ref, targetNodeId: ports[0]!.id, confidence: selected!.confidence, + resolvedBy: 'qualified-name', ...(positional ? { alsoTargets: [{ targetNodeId: module.id, metadata: { binding: 'hdl-positional-dependency', endpoint: 'module' } }] } : {}), + metadata: { binding: positional ? 'hdl-positional-port' : 'hdl-named-port', moduleId: module.id, instanceId, ...(positional ? { position, moduleName } : {}) } }; +} + +/** Shared selection for named, positional and wildcard source bindings. */ +export function selectVerilogModule( + ref: UnresolvedRef, + context: ResolutionContext, + resolveModule: (ref: UnresolvedRef, context: ResolutionContext) => ResolvedRef | null, + moduleName: string, + instanceId: string, +): { module: Node; confidence: number } | null { + const instance = context.getNodeById?.(instanceId); + if (!instance || instance.language !== 'verilog' || instance.kind !== 'variable' || instance.filePath !== ref.filePath) return null; + const selected = resolveModule({ ...ref, fromNodeId: instance.id, referenceName: moduleName, + referenceKind: 'instantiates', line: instance.startLine, column: instance.startColumn }, context); + const module = selected && context.getNodeById?.(selected.targetNodeId); + if (!module || module.name !== moduleName || module.language !== 'verilog' + || !['class', 'interface'].includes(module.kind)) return null; + let candidates = context.getNodesByName(moduleName) + .filter(n => n.language === 'verilog' && ['class', 'interface'].includes(n.kind)); + const simulation = (file: string) => /(^|\/)(sim|tb|tests?|testbench|dv)\//i.test(file) + || /_(stub|tb|sim)\.s?vh?$/i.test(file); + if (!simulation(ref.filePath)) { + const synthesis = candidates.filter(n => !simulation(n.filePath)); + if (synthesis.length) candidates = synthesis; + } + // The module matcher may choose the first equal candidate. Port correspondence + // is stricter: a tie in source-file/directory proximity stays unresolved. + const dirs = ref.filePath.split('/').slice(0, -1); + const score = (file: string): number => { + const other = file.split('/').slice(0, -1); + let shared = 0; + while (shared < Math.min(dirs.length, other.length) && dirs[shared] === other[shared]) shared++; + return (file === ref.filePath ? 100 : 0) + Math.min(shared * 15, 80); + }; + if (candidates.some(n => n.id !== module.id && score(n.filePath) >= score(module.filePath))) return null; + if (context.getNodesByQualifiedName(module.qualifiedName).filter(n => n.filePath === module.filePath + && n.language === 'verilog' && ['class', 'interface'].includes(n.kind)).length !== 1) return null; + return { module, confidence: selected!.confidence }; +} diff --git a/src/resolution/verilog-wildcard.ts b/src/resolution/verilog-wildcard.ts new file mode 100644 index 0000000000..fcce42c339 --- /dev/null +++ b/src/resolution/verilog-wildcard.ts @@ -0,0 +1,70 @@ +import type { Node } from '../types'; +import type { ResolutionContext, ResolvedRef, UnresolvedRef } from './types'; +import { selectVerilogModule } from './verilog-ports'; + +const PREFIX = 'hdl:wildcard:'; +export function isVerilogWildcardRef(ref: UnresolvedRef): boolean { + return ref.language === 'verilog' && ref.referenceKind === 'references' && ref.referenceName.startsWith(PREFIX); +} + +/** One source .* clause, with paired endpoints for only unambiguous source names. + * This records source correspondence, not elaborated nets or signal direction. */ +export function matchVerilogWildcard( + ref: UnresolvedRef, + context: ResolutionContext, + resolveModule: (ref: UnresolvedRef, context: ResolutionContext) => ResolvedRef | null, +): ResolvedRef | null { + let parts: unknown; + try { parts = JSON.parse(ref.referenceName.slice(PREFIX.length)); } catch { return null; } + if (!Array.isArray(parts) || parts.length !== 3 || typeof parts[0] !== 'string' || !parts[0] + || typeof parts[1] !== 'string' || !parts[1] || !Array.isArray(parts[2]) || !parts[2].every(n => typeof n === 'string')) return null; + const [moduleName, instanceId, exclusions] = parts as [string, string, string[]]; + const selected = selectVerilogModule(ref, context, resolveModule, moduleName, instanceId); + const instance = context.getNodeById?.(instanceId); + if (!selected || !instance) return null; + const module = selected.module; + const canonical = (name: string) => name.replace(/^\\/, ''); + const excluded = new Set(exclusions.map(canonical)); + const tag = module.decorators?.find(d => d.startsWith('hdl:port-order:')); + let order: unknown; + try { order = tag && JSON.parse(tag.slice('hdl:port-order:'.length)); } catch { order = null; } + // Non-ANSI alias/concatenation headers and unknown macro headers do not prove + // that an internal input declaration is an externally named formal port. + const names: string[] = Array.isArray(order) && order.every(n => typeof n === 'string' && canonical(n)) + && new Set(order.map(canonical)).size === order.length ? order.map(canonical) : []; + const declared = context.getNodesInFile(module.filePath).filter(n => n.language === 'verilog' + && n.kind === 'field' && n.decorators?.includes('hdl:port') + && n.qualifiedName === `${module.qualifiedName}::${n.name}` + && n.startLine >= module.startLine && n.endLine <= module.endLine); + const formals = names.flatMap(name => { + const matches = declared.filter(n => canonical(n.name) === name); + return matches.length === 1 ? matches : []; + }); + const scopes = instance.qualifiedName.split('::').slice(0, -1); + const localSignal = (name: string): Node | null => { + for (let i = scopes.length; i > 0; i--) { + const matches = [name, `\\${name}`].flatMap(n => context.getNodesByQualifiedName(`${scopes.slice(0, i).join('::')}::${n}`)) + .filter(n => n.language === 'verilog' && n.filePath === ref.filePath); + if (!matches.length) continue; + // Any nearer declaration shadows the outer scope; unsupported declarations + // and duplicates must not silently become the outer signal. + return matches.length === 1 && matches[0]!.kind === 'field' + && matches[0]!.decorators?.some(d => d === 'hdl:port' || d === 'hdl:signal') ? matches[0]! : null; + } + return null; + }; + const targets: { targetNodeId: string; metadata: Record }[] = []; + for (const formal of formals) { + const name = canonical(formal.name); + if (excluded.has(name) || formals.filter(n => canonical(n.name) === name).length !== 1) continue; + const local = localSignal(name); + if (!local) continue; + const metadata = { binding: 'hdl-wildcard-port', moduleId: module.id, instanceId, + formalNodeId: formal.id, portName: formal.name }; + targets.push({ targetNodeId: formal.id, metadata: { ...metadata, endpoint: 'formal' } }, + { targetNodeId: local.id, metadata: { ...metadata, endpoint: 'actual' } }); + } + return { original: ref, targetNodeId: module.id, confidence: selected.confidence, + resolvedBy: 'qualified-name', metadata: { binding: 'hdl-wildcard-dependency', endpoint: 'module', moduleId: module.id, instanceId }, + alsoTargets: targets }; +} diff --git a/src/sync/hdl-watch-scope.ts b/src/sync/hdl-watch-scope.ts new file mode 100644 index 0000000000..009def0983 --- /dev/null +++ b/src/sync/hdl-watch-scope.ts @@ -0,0 +1,47 @@ +import * as path from 'node:path'; +import { loadHdlProfile, isReservedHdlPath, type ActiveHdlProfile } from '../hdl/profile'; + +/** Watch inputs to a selected HDL profile, including files outside the ordinary source filter. + * Retain the last valid scope after a transient deletion/invalid config so a repair is observable. */ +export class HdlWatchScope { + private profile: ActiveHdlProfile | null = null; + private files = new Set(); + private filelists = new Set(); + private includeDirs: string[] = []; + private sourceDirs = new Set(); + private identity = ''; + constructor(private readonly root: string, private readonly getDependencies?: () => string[]) {} + refresh(): boolean { + const loaded = loadHdlProfile(this.root); + if (loaded.status === 'active') this.profile = loaded.profile; + else if (loaded.status === 'none') this.profile = null; + const profile = this.profile; + const files = new Set(profile?.files ?? []); + this.filelists = new Set(profile?.dependencies.map(d => d.path) ?? []); + for (const file of this.filelists) files.add(file); + const dependencies = profile ? this.getDependencies?.() : []; + if (Array.isArray(dependencies)) for (const file of dependencies) { + if (typeof file !== 'string') continue; + const rel = path.relative(this.root, path.resolve(this.root, file)).split(path.sep).join('/'); + if (rel && rel !== '..' && !rel.startsWith('../') && !path.isAbsolute(rel) && !isReservedHdlPath(rel)) files.add(rel); + } + this.files = files; + this.includeDirs = profile?.includeDirs ?? []; + this.sourceDirs = new Set(profile?.files.map(file => path.posix.dirname(file)) ?? []); + const identity = JSON.stringify([[...files].sort(), [...this.filelists].sort(), this.includeDirs, [...this.sourceDirs].sort()]); + const changed = identity !== this.identity; + this.identity = identity; + return changed; + } + isFilelist(rel: string): boolean { return this.filelists.has(rel); } + matchesFile(rel: string): boolean { + if (isReservedHdlPath(rel)) return false; + return this.files.has(rel) || this.sourceDirs.has(path.posix.dirname(rel)) + || this.includeDirs.some(dir => dir === '.' || rel.startsWith(`${dir}/`)); + } + matchesDirectory(rel: string): boolean { + if (!this.profile || isReservedHdlPath(rel)) return false; + return [...this.files].some(file => file.startsWith(`${rel}/`)) + || this.includeDirs.some(dir => dir === '.' || dir === rel || dir.startsWith(`${rel}/`) || rel.startsWith(`${dir}/`)); + } +} diff --git a/src/sync/watcher.ts b/src/sync/watcher.ts index 612c8596d5..e9447e36a9 100644 --- a/src/sync/watcher.ts +++ b/src/sync/watcher.ts @@ -41,6 +41,7 @@ import { logDebug, logWarn } from '../errors'; import { normalizePath } from '../utils'; import { isCodeGraphDataDir } from '../directory'; import { watchDisabledReason } from './watch-policy'; +import { HdlWatchScope } from './hdl-watch-scope'; /** * Number of consecutive lock-contention retries the watcher tolerates before @@ -385,6 +386,7 @@ export class FileWatcher { // scope (#1728). An embedded repo created after start() joins the scope on // the next scope refresh / watcher restart / re-index. private ignoreMatcher: ScopeIgnore | null = null; + private readonly hdlScope: HdlWatchScope; private readonly projectRoot: string; private readonly debounceMs: number; @@ -405,9 +407,11 @@ export class FileWatcher { projectRoot: string, syncFn: (paths?: string[]) => Promise<{ filesChanged: number; durationMs: number }>, options: WatchOptions = {}, - isFileStateCurrent?: (relativePath: string) => boolean + isFileStateCurrent?: (relativePath: string) => boolean, + getHdlDependencyPaths?: () => string[] ) { this.projectRoot = projectRoot; + this.hdlScope = new HdlWatchScope(projectRoot, getHdlDependencyPaths); this.syncFn = syncFn; this.debounceMs = options.debounceMs ?? 2000; this.onSyncComplete = options.onSyncComplete; @@ -443,6 +447,7 @@ export class FileWatcher { // Reuse the indexer's ignore set so the watcher and indexer agree on scope. this.ignoreMatcher = buildScopeIgnore(this.projectRoot); + this.hdlScope.refresh(); try { if (this.inertForTests) { @@ -676,7 +681,12 @@ export class FileWatcher { this.refreshScope(rel); return; } - if (this.ignoreMatcher && this.ignoreMatcher.ignores(rel)) return; + const hdlChange = this.hdlScope.matchesFile(rel) || this.hdlScope.matchesDirectory(rel); + if (this.hdlScope.isFilelist(rel)) { + this.refreshHdlScope(); + this.needsFullScan = true; + } + if (!hdlChange && this.ignoreMatcher && this.ignoreMatcher.ignores(rel)) return; // A nested `.gitignore` (an embedded child repo's own rules, #514, or a // subdirectory rule the git-backed full scan honors) is only a scope // change when it sits INSIDE the current scope — checked after the matcher @@ -687,7 +697,7 @@ export class FileWatcher { this.refreshScope(rel); return; } - if (!isSourceFile(rel, loadExtensionOverrides(this.projectRoot), this.projectRoot)) { + if (!hdlChange && !isSourceFile(rel, loadExtensionOverrides(this.projectRoot), this.projectRoot)) { this.maybeScheduleForRemovedDir(rel); return; } @@ -742,11 +752,18 @@ export class FileWatcher { private refreshScope(rel: string): void { logDebug('Scope config changed; rebuilding watcher scope', { file: rel }); this.ignoreMatcher = buildScopeIgnore(this.projectRoot); + this.hdlScope.refresh(); this.scheduleTreeRefresh(); this.needsFullScan = true; this.scheduleSync(); } + private refreshHdlScope(): void { + // A profile switch can admit a formerly ignored directory; the walk adds + // its watches (and drops ones no longer in scope). + if (this.hdlScope.refresh()) this.refreshWatchTree(); + } + /** * A deleted DIRECTORY arrives as one event on the directory's own path — * no source extension, so the source-file filter drops it, and the files @@ -849,6 +866,7 @@ export class FileWatcher { const rel = normalizePath(path.relative(this.projectRoot, dirPath)); if (!rel || rel === '.' || rel.startsWith('..')) return false; // root / outside if (this.isAlwaysIgnored(rel)) return true; + if (this.hdlScope.matchesDirectory(rel)) return false; if (!this.ignoreMatcher) return false; return this.ignoreMatcher.ignores(rel + '/'); } @@ -1096,6 +1114,7 @@ export class FileWatcher { if (!scoped) this.needsFullScan = false; try { const result = await this.syncFn(scoped); + this.refreshHdlScope(); if (this.recoveringFromLock && !scoped && !this.needsFullScan) { this.degradedReason = null; this.recoveringFromLock = false; @@ -1177,7 +1196,8 @@ export class FileWatcher { // above already set `stopped`, so this won't reschedule a watcher that // has given up. A directory removal whose full sync failed adds no // pending file, only `needsFullScan` — it still owes the full reconcile - // it asked for (#1964). + // it asked for (#1964). So does a scope change (`codegraph.json`, an HDL + // filelist) that landed while a SCOPED sync was in flight. if ((this.pendingFiles.size > 0 || this.needsFullScan) && !this.stopped) { const retryCount = Math.max(this.lockRetryCount, this.syncFailureRetryCount); if (retryCount > 0) { diff --git a/src/types.ts b/src/types.ts index 9c4c4461f9..6723259689 100644 --- a/src/types.ts +++ b/src/types.ts @@ -109,6 +109,7 @@ export const LANGUAGES = [ 'pascal', 'scala', 'lua', + 'verilog', 'luau', 'objc', 'r',