Improve package scan performance #4606

AyanSinhaMahapatra · 2025-11-17T09:49:46Z

This PR improve package scan performance by....

References:

fast-scan: Apply performance improvements to code: batch 2 (file system walking or from benchmark results) #4064
fast-scan: Create scan benchmark for package detection in ScanCode Toolkit #4061

Tasks

Reviewed contribution guidelines
PR is descriptively titled 📑 and links the original issue above 🔗
Tests pass -- look for a green checkbox ✔️ a few minutes after opening your PR
Run tests locally to check for errors.
Commits are in uniquely-named feature branch and has no merge conflicts 📁
Updated documentation pages (if applicable)
Updated CHANGELOG.rst (if applicable)

Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

Reference: https://github.com/Quantco/multiregex Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

Use multiregex to use a cached regex path patterns and datafile handlers mapping to detect package datafiles faster. Reference: #4064 Reference: #4061 Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

src/packagedcode/cache.py

Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

src/packagedcode/cache.py

Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

Introduce a new option --binary-packages which looks for package/dependency data in binaries. Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

We do not need the license index in a --package-only scan as this is designed to do a fast package detection only scan which skips the license detection. As license index loading takes a couple seconds in each case, this makes the package only scan much faster. Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

pombredanne · 2025-12-15T12:57:37Z

docs/source/rst_snippets/basic_options.rst

 --system-package             Scan ``<input>`` for installed system package
                             databases.

+-b, --binary-package         Scan <input> for package and dependency related


What about this:

--package-in-exec Scan compiled executable binaries such as ELF, WinpE and Mach-O files, looking for structured package and dependency metadata as found for example in Go and Rust binaries.

Or

--package-in-compiled Scan compiled executable binaries such as ELF, WinpE and Mach-O files, looking for structured package and dependency metadata as found for example in Go and Rust compiled binaries.

pombredanne · 2025-12-15T13:04:44Z

etc/release/scancode-create-pypi-wheel.sh


 ./configure --dev
 venv/bin/scancode-reindex-licenses
+venv/bin/scancode-cache-package-patterns


What about naming this venv/bin/scancode-reindex-package-patterns to be consistent?

pombredanne · 2025-12-15T13:07:11Z

src/packagedcode/__init__.py


+
+# These handlers are special as they use filetype to
+# detect these binaries instead of datafile path patterns


Suggested change

# detect these binaries instead of datafile path patterns

# detect these compiled executable binaries instead of datafile path patterns

src/packagedcode/cache.py

pombredanne · 2025-12-15T13:15:24Z

src/packagedcode/cache.py

+PACKAGE_INDEX_DIR = 'package_patterns_index'
+PACKAGE_INDEX_FILENAME = 'index_cache'
+PACKAGE_LOCKFILE_NAME = 'scancode_package_index_lockfile'
+PACKAGE_CHECKSUM_FILE = 'scancode_package_index_tree_checksums'


This is not used anymore (also should be dropped from licensing)

Suggested change

PACKAGE_CHECKSUM_FILE = 'scancode_package_index_tree_checksums'

pombredanne

Here are some nits for your consideration!

src/packagedcode/cache.py

src/packagedcode/recognize.py

pombredanne · 2025-12-15T13:36:48Z

docs/source/rst_snippets/basic_options.rst

 --system-package             Scan ``<input>`` for installed system package
                             databases.

+-b, --binary-package         Scan <input> for package and dependency related


Or

--package-in-compiled Scan compiled executable binaries such as ELF, WinpE and Mach-O files, looking for structured package and dependency metadata as found for example in Go and Rust compiled binaries.

AyanSinhaMahapatra added 3 commits November 17, 2025 15:12

Simplify GemfileHandler path patterns

489af1b

Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

Add multiregex as a dependency

8d6fa73

Reference: https://github.com/Quantco/multiregex Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

Add initial multiregex implementation

4fc3af2

Use multiregex to use a cached regex path patterns and datafile handlers mapping to detect package datafiles faster. Reference: #4064 Reference: #4061 Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

AyanSinhaMahapatra marked this pull request as draft November 17, 2025 09:49

AyanSinhaMahapatra changed the title ~~Fast package scan~~ Improve package scan performance Nov 17, 2025

github-advanced-security bot found potential problems Nov 17, 2025

View reviewed changes

src/packagedcode/cache.py Dismissed Show dismissed Hide dismissed

src/packagedcode/cache.py Dismissed Show dismissed Hide dismissed

src/packagedcode/cache.py Fixed Show fixed Hide fixed

src/packagedcode/cache.py Fixed Show fixed Hide fixed

AyanSinhaMahapatra added 2 commits November 17, 2025 19:27

Add minimal tests for package cache

4438526

Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

Cache multiregex matchers instead of patterns

3bbf35e

Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

github-advanced-security bot found potential problems Nov 18, 2025

View reviewed changes

src/packagedcode/cache.py Dismissed Show dismissed Hide dismissed

src/packagedcode/cache.py Dismissed Show dismissed Hide dismissed

src/packagedcode/cache.py Dismissed Show dismissed Hide dismissed

Restore binary package manifest scanning

e0460ef

Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

AyanSinhaMahapatra requested review from JonoYang and pombredanne November 19, 2025 09:47

AyanSinhaMahapatra marked this pull request as ready for review November 19, 2025 09:47

Only scan for bianry packages optionally

dde6bc9

Introduce a new option --binary-packages which looks for package/dependency data in binaries. Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

AyanSinhaMahapatra force-pushed the fast-package-scan branch from a99c4f8 to dde6bc9 Compare November 19, 2025 10:26

AyanSinhaMahapatra added 2 commits November 19, 2025 16:15

Add a new console script to build the package patterns cache

6b6a79b

Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

pombredanne reviewed Dec 15, 2025

View reviewed changes

src/packagedcode/cache.py Show resolved Hide resolved

pombredanne reviewed Dec 15, 2025

View reviewed changes



		# These handlers are special as they use filetype to
		# detect these binaries instead of datafile path patterns

	# detect these binaries instead of datafile path patterns
	# detect these compiled executable binaries instead of datafile path patterns

Uh oh!

Improve package scan performance #4606

Are you sure you want to change the base?

Improve package scan performance #4606

Uh oh!

Conversation

AyanSinhaMahapatra commented Nov 17, 2025 • edited by pombredanne Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tasks

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pombredanne Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

pombredanne Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

pombredanne Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

pombredanne Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pombredanne Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

pombredanne left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pombredanne Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

AyanSinhaMahapatra commented Nov 17, 2025 •

edited by pombredanne

Loading