Skip to content

Support PEP 751 pylock.toml lockfiles #4638 - #5386

Open
aryan1234-l wants to merge 6 commits into
aboutcode-org:developfrom
aryan1234-l:support-pylock-toml
Open

aryan1234-l wants to merge 6 commits into
aboutcode-org:developfrom
aryan1234-l:support-pylock-toml

Conversation

@aryan1234-l

Copy link
Copy Markdown

Summary

This PR adds support for PEP 751 pylock.toml lockfiles in packagedcode.

Details

  • Implemented PylockTomlHandler and BasePylockPythonLayout in src/packagedcode/pypi.py.
  • Supports standard pylock.toml as well as named lockfiles (matching pylock.<name>.toml).
  • Parses packages and extracts dependencies, sdist, wheels, VCS, directory, and archive sources along with hashes (sha256, sha512, md5, sha1), size, and qualifiers.
  • Assembles dependencies and package relationships with pyproject.toml projects.
  • Added unit and end-to-end package assembly tests with regenerated expected test fixtures in tests/packagedcode/.
  • Updated AUTHORS.rst and CHANGELOG.rst.

Closes #4638.

Signed-off-by: Aryan Gupta arynrav11@gmail.com

Add PylockTomlHandler to packagedcode to support PEP 751 pylock.toml
lockfiles and named variants (such as pylock.<name>.toml).

- Support parsing packages, sdist, wheels, vcs, directory, and archive sources
- Extract dependencies, resolved packages, hashes, qualifiers, and metadata
- Assemble packages and dependencies with pyproject.toml projects
- Add unit and end-to-end assembly tests with test fixtures

Resolves: aboutcode-org#4638
Signed-off-by: Aryan Gupta <arynrav11@gmail.com>

@pombredanne pombredanne left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR.
Here are a few comments for you consideration. I feel that you are trying too hard to handle cases that are unlikely to ever exists and makes the code hard to read. You should also break the parsing code in smaller functions. Or find a proper pylock parser like https://packaging.pypa.io/en/latest/pylock.html

Comment thread src/packagedcode/pypi.py Outdated
return bool(file_name and PYLOCK_FILENAME_PATTERN.match(file_name))


def is_pylock_toml(location):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why so much cruft? we do not need that level of checks ahead

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok I shall keep that in mind now.

Comment thread src/packagedcode/pypi.py Outdated
PYLOCK_FILENAME_PATTERN = re.compile(r"^pylock(\.[^.]+)?\.toml$", re.IGNORECASE)


def is_pylock_toml_file_name(file_name):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have a way in a handler to match path patterns, why create something else?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Removed the custom regex, filename helper, and is_datafile override. PylockTomlHandler now relies on standard path_patterns and parses through packaging.pylock.Pylock.

relevance: 100
minimum_coverage: 100
notes: Rule based on the manifest file of the liquidbase source jar.
notes: Rule based on the manifest file of the liquidbase source jar.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not mix unrelated rules changes in that PR.

Comment thread src/packagedcode/pypi.py Outdated
"""

@classmethod
def assemble(cls, package_data, resource, codebase, package_adder):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like a very complex assembly. What is so special about lock files? And can they even be used to assemble (e.g., collect file pats of the packages) anything unless a package is editable?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okk, Removed BasePylockPythonLayout entirely. PylockTomlHandler now inherits directly from models.DatafileHandler, treating lockfiles as standard datafiles that yield locked dependencies.

Comment thread src/packagedcode/pypi.py Outdated
datasource_id = "pypi_pylock_toml"
path_patterns = (
"*pylock.toml",
"*pylock.*.toml",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes the is_pylock_toml_file_name() function redundant.

Comment thread src/packagedcode/pypi.py Outdated
dep_version = dep.get("version")
dep_marker = dep.get("marker")

extracted_requirement = None

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is an empty requirement None or ""?

Comment thread src/packagedcode/pypi.py Outdated
elif dep_version:
extracted_requirement = f"=={dep_version}"
elif dep_marker:
extracted_requirement = dep_marker

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this really a possibility?

Comment thread src/packagedcode/pypi.py Outdated
size = None

sdist = package.get("sdist")
if isinstance(sdist, dict):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this ever possible that it is not a dict?

Comment thread src/packagedcode/pypi.py Outdated
wheel_entries = []
if isinstance(wheels, list):
wheel_entries = [w for w in wheels if isinstance(w, dict)]
elif isinstance(wheels, dict):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this possible?

Comment thread src/packagedcode/pypi.py Outdated
wheels = package.get("wheels")
wheel_entries = []
if isinstance(wheels, list):
wheel_entries = [w for w in wheels if isinstance(w, dict)]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do understand why you are doing this

Comment thread src/packagedcode/pypi.py Outdated
yield models.PackageData.from_data(package_data, package_only)


PYLOCK_FILENAME_PATTERN = re.compile(r"^pylock(\.[^.]+)?\.toml$", re.IGNORECASE)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why ignore case? Also Use the packaging library and the PyLock object instead of your own custom parsing

…edback

- Use packaging.pylock.Pylock and Package objects for spec-compliant parsing.
- Break down parsing into small, modular helper functions.
- Simplify PylockTomlHandler to inherit directly from DatafileHandler using standard path_patterns without custom regex or ahead-of-time datafile overrides.
- Do not skip editable or directory packages; parse them as first-class locked dependencies.
- Revert unrelated license rule edits.
- Add packaging >= 26.1 dependency to pyproject.toml.

Signed-off-by: Aryan Gupta <arynrav11@gmail.com>
Signed-off-by: Aryan Gupta <arynrav11@gmail.com>
@aryan1234-l

Copy link
Copy Markdown
Author

Hi @pombredanne,

Thank you very much for the review and the pointer to packaging.pylock!

I have refactored the implementation to address all of your feedback:

  1. Adopted standard packaging.pylock parser: Replaced custom TOML dictionary parsing with packaging.pylock.Pylock.from_dict() and standard packaging.pylock.Package data structures, providing robust and spec-compliant PEP 751 lockfile parsing.
  2. Simplified Handler Structure:
    • Removed custom filename regexes, is_pylock_toml_file_name(), and custom is_datafile() overrides.
    • Removed BasePylockPythonLayout.
    • PylockTomlHandler now cleanly subclasses DatafileHandler using standard path_patterns = ('*/pylock.toml', 'pylock.toml', '*/pylock.*.toml', 'pylock.*.toml').
  3. Decomposed parsing into modular helpers:
    • format_pylock_requirement(): builds clean requirement strings.
    • build_pylock_package_sources(): maps hashes, download URLs, and source metadata.
    • build_pylock_dependencies_for_resolved(): extracts and formats runtime and extra dependencies.
    • build_pylock_lockfile_extra_data(): captures lockfile-level metadata.
  4. Supported all package types: Editable and local directory packages are now parsed as first-class locked dependencies rather than skipped.
  5. Reverted unintended rule edits: Removed extraneous modifications to fsl-1.1_1.RULE and fsl-1.1_2.RULE.
  6. Dependency & CI Constraints:
    • Added packaging >= 26.1 to pyproject.toml.
    • Updated requirements.txt to packaging==26.3 to resolve pip constraint conflicts in CI environments.
  7. Regenerated Test Expectations: Updated test expectation fixtures in tests/packagedcode/data/pypi/pylock/ and verified full test suite passes locally.

Could you please take another look when you have a moment? Thank you!

Signed-off-by: Aryan Gupta arynrav11@gmail.com

Signed-off-by: Aryan Gupta <arynrav11@gmail.com>
Signed-off-by: Aryan Gupta <arynrav11@gmail.com>
Signed-off-by: Aryan Gupta <arynrav11@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support pylock.toml

2 participants