`py_binary`: Broken venv symlinks with `venvs_site_packages=yes`

[hartikainen]

🐞 bug report

Affected Rule

py_binary when used with py_image_layer rule from aspect_rules_py.

Is this a regression?

I don't know.

I'm not sure if this is a rules_python issue or aspect_rules_py issue, but I know that what I'm doing works with @rules_python//python/config_settings:venvs_site_packages=no and fails with @rules_python//python/config_settings:venvs_site_packages=yes, which is why I'm reporting it here.

Description

When using aspect_rules_py and rules_oci to package a py_binary into a container image, the build fails when @rules_python//python/config_settings:venvs_site_packages=yes, even though it works with @rules_python//python/config_settings:venvs_site_packages=no. More specifically, I'm using py_image_layer rule from aspect_rules_py to package the python binary, and it fails to create the tarball. The error suggests that a file inside the virtual environment created by py_binary cannot be found. The symlink clearly exists yet its target does not (see the errors below).

🔬 Minimal Reproduction

The reproduction is in the examples/oci-container directory of commit 2c596f17cf08 (main...hartikainen:rules_python:oci-container-example).

To reproduce, run bazel build //:server_layer from the example folder, with the different flag combinations in .bazelrc as described below.

🔥 Exception or Error

Setting 1 (success ✅)

# .bazelrc
common --@rules_python//python/config_settings:bootstrap_impl=script
common --@rules_python//python/config_settings:venvs_site_packages=no
common --@rules_python//python/config_settings:venvs_use_declare_symlink=no

Details

$ cd ./examples/oci-container
$ bazel build //:server_layer
WARNING: Build options --@@rules_python+//python/config_settings:venvs_site_packages and --@@rules_python+//python/config_settings:venvs_use_declare_symlink have changed, discarding analysis cache (this can be expensive, see https://bazel.build/advanced/performance/iteration-speed).
INFO: Analyzed target //:server_layer (2 packages loaded, 6998 targets configured).
INFO: Found 1 target...
Target //:server_layer up-to-date:
  bazel-bin/server_layer_interpreter.tar.gz
  bazel-bin/server_layer_packages.tar.gz
  bazel-bin/server_layer_default.tar.gz
INFO: Elapsed time: 24.130s, Critical Path: 23.71s
INFO: 41 processes: 5 action cache hit, 10 internal, 31 processwrapper-sandbox.
INFO: Build completed successfully, 41 total actions

Setting 2 (failure ❌)

# .bazelrc
common --@rules_python//python/config_settings:bootstrap_impl=script
common --@rules_python//python/config_settings:venvs_site_packages=yes
common --@rules_python//python/config_settings:venvs_use_declare_symlink=no

Details

$ cd ./examples/oci-container
$ bazel build //:server_layer
INFO: Analyzed target //:server_layer (146 packages loaded, 6968 targets configured).
ERROR: /home/user/bazel-contrib/rules_python/main/examples/oci-container/BUILD.bazel:20:15: Tar server_layer_packages.tar.gz failed: (Exit 1): tar failed: error executing Tar command (from target //:server_layer_packages) external/aspect_bazel_lib++toolchains+bsd_tar_linux_amd64/tar --create --gzip '--options=gzip:!timestamp' --file bazel-out/k8-fastbuild/bin/server_layer_packages.tar.gz ... (remaining 1 argument skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
tar: Error reading archive bazel-out/k8-fastbuild/bin/server_layer.packages.manifest.spec: Can't open bazel-out/k8-fastbuild/bin/_server.venv/lib/python3.12/site-packages/absl
tar: Error exit delayed from previous errors.
Target //:server_layer failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 25.961s, Critical Path: 23.29s
INFO: 62 processes: 56 internal, 6 processwrapper-sandbox.
ERROR: Build did NOT complete successfully

$ readlink -f ./bazel-out/k8-fastbuild/bin/_server.venv/lib/python3.12/site-packages/absl  # empty
$ ls -lah ./bazel-out/k8-fastbuild/bin/_server.venv/lib/python3.12/site-packages/absl
lrwxrwxrwx 1 user user 68 Nov  2 18:19 ./bazel-out/k8-fastbuild/bin/_server.venv/lib/python3.12/site-packages/absl -> ../../../../../rules_python++pip+pypi_312_absl_py/site-packages/absl

Setting 3 (failure ❌)

# .bazelrc
common --@rules_python//python/config_settings:bootstrap_impl=script
common --@rules_python//python/config_settings:venvs_site_packages=yes
common --@rules_python//python/config_settings:venvs_use_declare_symlink=yes

Details

$ cd ./examples/oci-container
$ bazel build //:server_layer
WARNING: Build option --@@rules_python+//python/config_settings:venvs_use_declare_symlink has changed, discarding analysis cache (this can be expensive, see https://bazel.build/advanced/performance/iteration-speed).
INFO: Analyzed target //:server_layer (0 packages loaded, 6948 targets configured).
ERROR: /home/user/bazel-contrib/rules_python/main/examples/oci-container/BUILD.bazel:20:15: Tar server_layer_default.tar.gz failed: (Exit 1): tar failed: error executing Tar command (from target //:server_layer_default) external/aspect_bazel_lib++toolchains+bsd_tar_linux_amd64/tar --create --gzip '--options=gzip:!timestamp' --file bazel-out/k8-fastbuild/bin/server_layer_default.tar.gz ... (remaining 1 argument skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
tar: Error reading archive bazel-out/k8-fastbuild/bin/server_layer.default.manifest.spec: Can't open bazel-out/k8-fastbuild/bin/_server.venv/bin/python3
tar: Error exit delayed from previous errors.
Target //:server_layer failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.836s, Critical Path: 0.35s
INFO: 5 processes: 57 action cache hit, 5 internal.
ERROR: Build did NOT complete successfully

$ readlink -f ./bazel-out/k8-fastbuild/bin/_server.venv/bin/python3  # empty
$ ls -lah ./bazel-out/k8-fastbuild/bin/_server.venv/bin/python3
lrwxrwxrwx 1 user user 78 Nov  2 18:20 ./bazel-out/k8-fastbuild/bin/_server.venv/bin/python3 -> ../../../rules_python++python+python_3_12_x86_64-unknown-linux-gnu/bin/python3

Setting 4 (probably pointless, but included for completeness; failure ❌)

# .bazelrc
common --@rules_python//python/config_settings:bootstrap_impl=script
common --@rules_python//python/config_settings:venvs_site_packages=no
common --@rules_python//python/config_settings:venvs_use_declare_symlink=yes

Details

$ cd ./examples/oci-container
$ bazel build //:server_layer
INFO: Analyzed target //:server_layer (148 packages loaded, 7018 targets configured).
ERROR: /home/user/bazel-contrib/rules_python/main/examples/oci-container/BUILD.bazel:20:15: Tar server_layer_default.tar.gz failed: (Exit 1): tar failed: error executing Tar command (from target //:server_layer_default) external/aspect_bazel_lib++toolchains+bsd_tar_linux_amd64/tar --create --gzip '--options=gzip:!timestamp' --file bazel-out/k8-fastbuild/bin/server_layer_default.tar.gz ... (remaining 1 argument skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
tar: Error reading archive bazel-out/k8-fastbuild/bin/server_layer.default.manifest.spec: Can't open bazel-out/k8-fastbuild/bin/_server.venv/bin/python3
tar: Error exit delayed from previous errors.
Target //:server_layer failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 3.157s, Critical Path: 0.66s
INFO: 46 processes: 18 internal, 28 processwrapper-sandbox.
ERROR: Build did NOT complete successfully

$ readlink -f ./bazel-out/k8-fastbuild/bin/_server.venv/bin/python3  # empty
$ ls -lah ./bazel-out/k8-fastbuild/bin/_server.venv/bin/python3
lrwxrwxrwx 1 user user 78 Nov  2 18:02 bazel-out/k8-fastbuild/bin/_server.venv/bin/python3 -> ../../../rules_python++python+python_3_12_x86_64-unknown-linux-gnu/bin/python3

🌍 Your Environment

Operating System:

$ lsb_release -a

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 24.04.1 LTS
Release:        24.04
Codename:       noble

Output of bazel version:

$ bazel --version
bazel 8.2.1

Rules_python version:

e85728f

Anything else relevant?
The issue seems to be related to how the symlinks are created in the virtual environment when venvs_site_packages is enabled. The tar command is unable to find the python3 binary in the venv. Its symlink exists on disk, as confirmed by the ls command, yet the symlink target does not, as confirmed by readlink command.

[aignas]

This is an upstream issue.

[aignas]

More context (same issues) in #3117

[shayanhoshyari]

I did this for now until upstream is addressed, seems to work ok

Latest version (works with 1.8): https://github.com/shayanhoshyari/issue-reports/tree/main/rules_python/rules_pkg_pyvenv

My first version

"""
A replacement of py_image_layer that does not suffer from the following:

- https://github.com/bazel-contrib/rules_python/issues/3388
- https://github.com/bazel-contrib/tar.bzl/issues/16
"""

load("@rules_python//python:defs.bzl", "PyRuntimeInfo")

def _tar_runfiles(ctx):
    """
    Implementation of tar_runfiles
    """
    py_binary = ctx.attr.py_binary
    compress = ctx.attr.compress
    root = ctx.attr.root
    entrypoint = ctx.attr.entrypoint

    # 1. Write the manifest
    output = []
    default_info = py_binary[DefaultInfo]
    runfiles = {"external": [], "main": []}
    for f in default_info.default_runfiles.files.to_list():
        label = "external" if f.path.startswith("external/") else "main"
        line = "|".join([f.path, f.short_path, str(int(f.is_symlink)), label])
        runfiles[label].append(f)
        output.append(line)

    manifest = ctx.actions.declare_file(ctx.attr.name + ".manifest")
    ctx.actions.write(
        output = manifest,
        content = "\n".join(output),
    )

    # 2. Execute the tar rule
    tarballs = []
    for label in ["main", "external"]:
        tarball = ctx.actions.declare_file(ctx.attr.name + "." + label + ".tar")

        args = [manifest.path, tarball.path, "--label", label, "--compress", compress, "--root", root]
        if entrypoint and label == "main":
            args.extend(["--entrypoint", entrypoint])

        ctx.actions.run(
            outputs = [tarball],
            inputs = [manifest] + runfiles[label],
            executable = ctx.executable._tool,
            arguments = args,
        )
        tarballs.append(tarball)

    return [
        DefaultInfo(files = depset(tarballs)),
    ]

tar_runfiles = rule(
    implementation = _tar_runfiles,
    attrs = {
        "py_binary": attr.label(
            doc = "py_binary target whose runfiles to iterate over.",
            providers = [PyRuntimeInfo],
        ),
        "compress": attr.string(
            default = "none",
            doc = "Compression algorithm to use. Supported: zstd, gzip, bzip2, xz, lzma, none",
        ),
        "root": attr.string(
            default = "root",
            doc = "Default top level folder",
        ),
        "entrypoint": attr.string(
            doc = "If provided, adds a symlink called run.sh to entrypoint for user to see easier how to start.",
        ),
        "_tool": attr.label(
            default = Label(":tar_runfiles"),
            executable = True,
            cfg = "exec",
        ),
    },
    doc = "A rule which iterates over all runfiles of a given py_binary. Replacement for py_image_layer that supports symlinks.",
)

"""
Script to create a tar archive.

The manifest should be a text file with lines formatted as:
    <path> | <short path> | <is_link>  | <label>
    ...

Usage:
    python tar_runfiles.py manifest output [--label ...] [--compress ...]
"""

import argparse
import os
import tarfile
import tempfile
from pathlib import Path
from typing import cast

COMPRESS_FORMATS = {
    None: "w",
    "": "w",
    "none": "w",
    "zstd": "w|zst",
    "gz": "w:gz",
    "gzip": "w:gz",
    "bz2": "w:bz2",
    "bzip2": "w:bz2",
    "xz": "w:xz",
    "lzma": "w:xz",
}


def main() -> None:
    parser = argparse.ArgumentParser(description="Create two tars ('local' and 'external') based on a manifest.")
    parser.add_argument("manifest", help="Path to manifest file")
    parser.add_argument("output", help="Output tar file for 'local' group")
    parser.add_argument("--label", required=False, help="Labels to include in the tar")
    parser.add_argument("--compress", default="", help="Compression to use (zstd, gzip, bz2, xz)")
    parser.add_argument("--root", default="app", help="Default top level folder")
    parser.add_argument("--entrypoint", help="If provided, will add run.sh symlinked to this.")
    args = parser.parse_args()

    root = cast(str, args.root)
    output = cast(str, args.output)
    manifest_path = cast(str, args.manifest)
    target_label = cast(str | None, args.label)
    compress = cast(str, args.compress).lower()
    entrypoint = cast(str | None, args.entrypoint)

    mode = COMPRESS_FORMATS.get(compress)
    if mode is None:
        raise ValueError(f"Unknown compression type: {compress}")

    lines = Path(manifest_path).read_text().strip().splitlines()

    with tarfile.open(output, mode) as tar_stream:  # type: ignore[call-overload]
        tar_stream = cast(tarfile.TarFile, tar_stream)
        for line in lines:
            line = line.strip()
            if not line:
                continue

            path, short, is_link, label = line.split("|")
            if target_label is not None and label != target_label:
                continue

            # Links should be preserved as links
            if int(is_link):
                target = path
            else:
                target = os.path.realpath(path)

            # destination is always _main/ added to short_path, not sure why bazel does this
            arcname = os.path.normpath(os.path.join(root, "_main", short))

            tar_stream.add(target, arcname=arcname, recursive=False)

        if entrypoint is not None:
            with tempfile.NamedTemporaryFile("w") as run_sh:
                run_content = _ENTRYPOINT.format(root=root, entrypoint=os.path.join(root, "_main", entrypoint))
                run_sh.write(run_content)
                run_sh.flush()
                os.chmod(run_sh.name, 0o755)
                # Add it both to root of app, and root of container. Just so that things are more obvious.
                tar_stream.add(run_sh.name, arcname=os.path.join(root, "run.sh"), recursive=False)
                tar_stream.add(run_sh.name, arcname="run.sh", recursive=False)


# /{root} and /{entrypoint} because in the container everything will be absolute.
_ENTRYPOINT = """#!/bin/bash
set -ex
export RUNFILES_DIR=/{root}
exec /{entrypoint} "$@"
"""

if __name__ == "__main__":
    main()