Repository navigation
perf(depscan): skip a scheduled rescan when OSV answers the same - #2685
Conversation
Every 12 h each active repository gets a scheduled dependency scan at its tracked head. About 97% hit the commit already scanned 12 h before, and each one still resolved the head twice, listed the tree, asked OSV, read the records and wrote a new scan with its matches. A completed scan now stores a fingerprint of the OSV answer it matched against (sorted package -> id@modified, plus the parser version). A scheduled scan of the same commit loads the commit's lockfile plan from the edge cache, reads the cached parses, asks OSV, and returns Unchanged when the fingerprint matches. A different answer runs the full scan with the answer already fetched. OSV is still asked on every scheduled scan of a repository with packages. Manual and push scans are unchanged.
This comment has been minimized.
This comment has been minimized.
|
No actionable comments were generated in the recent review. Walkthrough
Merge Risk:
|
There was a problem hiding this comment.
Devin Review found 1 potential issue.
1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)
The repository list shows the newest head scan of any status. The precheck read only the latest completed scan, so a transient failure after it stayed on the repository until the commit changed. The precheck now reads the newest head scan whatever its status and skips only when that scan completed.
Maple review🟢 Confidence 10/10 · safe to merge. Skip, fallbacks and the failed-scan regression are each pinned by call-counting tests; the residual assumption is OSV bumping A scheduled dependency rescan now returns
Before merge
|
|
Merged with 2 steps from "Before merge" still open. If they are done, tick them on the review comment; if not, now is the time.
|
Problem
The scheduled dependency scan runs every 12 h (
VcsScheduledSyncService). It queues onedependency-scan(triggerschedule) per active repository on its tracked or default branch. Almost every one of them repeats work it did 12 h earlier.Production, burst of 2026-11-15 11:30-13:30Z: 3,784 scheduled scans across 3,784 repositories in 33 orgs. p50 1.72 s, 6,979 s of
DependencyScanner.scanin total. 1,064 of them (28%) found no lockfile and still ran the whole path. That is about 11k scans a day, and about 97% are at the same commit as 12 h before.Where the time went (11:50-12:50Z, 3,228 scans, 5,987 s):
osv.querybatchgetCommit(2 per scan)getRecursiveTreeDependencyScanStore.completeScanCause
DependencyScanner.tsskipped an unchanged scan only fortrigger === "push", by comparing lockfile blob shas. That check cannot be used forschedule: it would hide newly published advisories.commitSha.resolveRefcalls (the head, then the head-moved check), the recursive tree listing, OSVquerybatch, the vulnerability record reads,startScanandcompleteScan.Fix
A vulnerability record never changes for one
id@modified. The samequerybatchanswer for the same packages therefore means the same matches and the same findings.dependency_scans.advisory_fingerprint(nullable text, migration20261116082407_dependency_scan_advisory_fingerprint).completeScanstores it only when the scan has no incomplete reason, so a scan that never reconciled is never skipped on.PackageAdvisoryCatalogis split intoquery(querybatch: id and modified per package) andresolve(record reads).advisoriesisquerythenresolve.fingerprintOfis a SHA-256 over the sortedpackageKey -> [id@modified]of every queried package, plusPARSER_VERSION.provider:externalRepoId:commitSha. A commit is immutable. A truncated tree is never cached.trigger === "schedule"on the default branch head, after the firstresolveRef. It reads the latest completed scan of the head. If that scan has the same commit, no error and a fingerprint, the branch loads the plan (cached), reads the packages (parse cache) and callscatalog.query.Unchangedand annotatesmaple.depscan.outcome=unchanged,maple.depscan.skip_reason=advisories_unchanged(no_lockfileswhen the commit has none). No tree fetch, no secondresolveRef, nostartScanorcompleteScan.Security invariant: OSV is still asked on every scheduled scan of a repository that has packages. A new advisory id, or a new
modifiedtime on an existing one, changes the fingerprint and runs a full scan that stores the match and opens the finding. Tests cover both cases. A commit with no packages has nothing to ask OSV about, so it skips without a call.Possible follow-up: no
checked_atcolumn in this PR. "Last scanned" stays at the time of the commit's last full scan, even though advisories are rechecked every 12 h.Before / After
resolveRef(GitHub getCommit)fetchTreeFiles(getRecursiveTree)querybatchstartScan+completeScan(Postgres)DependencyScanner.scantotalgetRecursiveTreecountgetCommitcountosv.querybatchcountMethod:
DependencyScanner.scanand child spans, 2026-11-15 11:30-13:30Z (the 12:00 UTC burst).DependencyScanner.test.ts. It runs the real scanner wiring on PGlite (createTestDb) and the in-memory edge cache, with a stub provider and a stub OSV that count calls. The "Before" row is the first scheduled scan, which still takes the full path. The "After" row is the second scheduled scan of the same commit.How to verify after deploy
From the second burst after deploy (the first fills the new column):
Expect:
schedulerows areunchanged/advisories_unchanged(orno_lockfiles).sumSdrops from ~7.0k s to ~3.0k s per burst.getRecursiveTreecount drops about 95% andgetCommitabout 50%.osv.querybatchcount stays flat. That proves advisories are still checked every 12 h.Open in [code]smith · Need help on this PR? Tag
@codesmith-botwith what you need.Summary by CodeRabbit