Repository navigation
Support PEP 751 pylock.toml lockfiles #4638 - #5386
aryan1234-l wants to merge 6 commits into
Conversation
Add PylockTomlHandler to packagedcode to support PEP 751 pylock.toml lockfiles and named variants (such as pylock.<name>.toml). - Support parsing packages, sdist, wheels, vcs, directory, and archive sources - Extract dependencies, resolved packages, hashes, qualifiers, and metadata - Assemble packages and dependencies with pyproject.toml projects - Add unit and end-to-end assembly tests with test fixtures Resolves: aboutcode-org#4638 Signed-off-by: Aryan Gupta <arynrav11@gmail.com>
d79228f to
fc93150
Compare
pombredanne
left a comment
There was a problem hiding this comment.
Thanks for the PR.
Here are a few comments for you consideration. I feel that you are trying too hard to handle cases that are unlikely to ever exists and makes the code hard to read. You should also break the parsing code in smaller functions. Or find a proper pylock parser like https://packaging.pypa.io/en/latest/pylock.html
| return bool(file_name and PYLOCK_FILENAME_PATTERN.match(file_name)) | ||
|
|
||
|
|
||
| def is_pylock_toml(location): |
There was a problem hiding this comment.
Why so much cruft? we do not need that level of checks ahead
There was a problem hiding this comment.
Ok I shall keep that in mind now.
| PYLOCK_FILENAME_PATTERN = re.compile(r"^pylock(\.[^.]+)?\.toml$", re.IGNORECASE) | ||
|
|
||
|
|
||
| def is_pylock_toml_file_name(file_name): |
There was a problem hiding this comment.
We have a way in a handler to match path patterns, why create something else?
There was a problem hiding this comment.
Agreed. Removed the custom regex, filename helper, and is_datafile override. PylockTomlHandler now relies on standard path_patterns and parses through packaging.pylock.Pylock.
| relevance: 100 | ||
| minimum_coverage: 100 | ||
| notes: Rule based on the manifest file of the liquidbase source jar. | ||
| notes: Rule based on the manifest file of the liquidbase source jar. |
There was a problem hiding this comment.
Do not mix unrelated rules changes in that PR.
| """ | ||
|
|
||
| @classmethod | ||
| def assemble(cls, package_data, resource, codebase, package_adder): |
There was a problem hiding this comment.
This looks like a very complex assembly. What is so special about lock files? And can they even be used to assemble (e.g., collect file pats of the packages) anything unless a package is editable?
There was a problem hiding this comment.
Okk, Removed BasePylockPythonLayout entirely. PylockTomlHandler now inherits directly from models.DatafileHandler, treating lockfiles as standard datafiles that yield locked dependencies.
| datasource_id = "pypi_pylock_toml" | ||
| path_patterns = ( | ||
| "*pylock.toml", | ||
| "*pylock.*.toml", |
There was a problem hiding this comment.
This makes the is_pylock_toml_file_name() function redundant.
| dep_version = dep.get("version") | ||
| dep_marker = dep.get("marker") | ||
|
|
||
| extracted_requirement = None |
There was a problem hiding this comment.
Is an empty requirement None or ""?
| elif dep_version: | ||
| extracted_requirement = f"=={dep_version}" | ||
| elif dep_marker: | ||
| extracted_requirement = dep_marker |
There was a problem hiding this comment.
Is this really a possibility?
| size = None | ||
|
|
||
| sdist = package.get("sdist") | ||
| if isinstance(sdist, dict): |
There was a problem hiding this comment.
Is this ever possible that it is not a dict?
| wheel_entries = [] | ||
| if isinstance(wheels, list): | ||
| wheel_entries = [w for w in wheels if isinstance(w, dict)] | ||
| elif isinstance(wheels, dict): |
| wheels = package.get("wheels") | ||
| wheel_entries = [] | ||
| if isinstance(wheels, list): | ||
| wheel_entries = [w for w in wheels if isinstance(w, dict)] |
There was a problem hiding this comment.
I do understand why you are doing this
| yield models.PackageData.from_data(package_data, package_only) | ||
|
|
||
|
|
||
| PYLOCK_FILENAME_PATTERN = re.compile(r"^pylock(\.[^.]+)?\.toml$", re.IGNORECASE) |
There was a problem hiding this comment.
Why ignore case? Also Use the packaging library and the PyLock object instead of your own custom parsing
…edback - Use packaging.pylock.Pylock and Package objects for spec-compliant parsing. - Break down parsing into small, modular helper functions. - Simplify PylockTomlHandler to inherit directly from DatafileHandler using standard path_patterns without custom regex or ahead-of-time datafile overrides. - Do not skip editable or directory packages; parse them as first-class locked dependencies. - Revert unrelated license rule edits. - Add packaging >= 26.1 dependency to pyproject.toml. Signed-off-by: Aryan Gupta <arynrav11@gmail.com>
Signed-off-by: Aryan Gupta <arynrav11@gmail.com>
|
Hi @pombredanne, Thank you very much for the review and the pointer to I have refactored the implementation to address all of your feedback:
Could you please take another look when you have a moment? Thank you! Signed-off-by: Aryan Gupta arynrav11@gmail.com |
Signed-off-by: Aryan Gupta <arynrav11@gmail.com>
Signed-off-by: Aryan Gupta <arynrav11@gmail.com>
Signed-off-by: Aryan Gupta <arynrav11@gmail.com>
Summary
This PR adds support for PEP 751
pylock.tomllockfiles inpackagedcode.Details
PylockTomlHandlerandBasePylockPythonLayoutinsrc/packagedcode/pypi.py.pylock.tomlas well as named lockfiles (matchingpylock.<name>.toml).sha256,sha512,md5,sha1), size, and qualifiers.pyproject.tomlprojects.tests/packagedcode/.AUTHORS.rstandCHANGELOG.rst.Closes #4638.
Signed-off-by: Aryan Gupta arynrav11@gmail.com