Use khiops-core and khiops-driver-* pip packages containing binaries only

.github/instructions/ci-workflows.instructions.md

@@ -40,8 +40,7 @@ Three job groups:
   JUnit XML via `unittest-xml-reporting`.
 - **`check-khiops-integration-on-linux`**: Runs integration tests on multiple
   Linux containers (ubuntu22.04, rocky8, rocky9, debian13). Validates Khiops
-  status, runs samples, tests major-version mismatch detection with a
-  `py3_khiops10_conda` environment, and runs the integration test suite.
+  status, runs samples and the integration tests suite.
 - **`check-khiops-integration-on-windows`**: Installs Khiops Desktop via NSIS
   installer on Windows 2022 with Python 3.12. Runs integration tests and
   samples outside a Python virtual environment, then installs khiops-python

.github/instructions/docker-changes.instructions.md

@@ -17,23 +17,22 @@ by the `dev-docker.yml` workflow.
 
 ## Image Variants
 
-There are three Dockerfiles, one per OS family:
+There are two Dockerfiles:
 
-| Dockerfile | OS targets | Package manager | Remote file drivers |
-|---|---|---|---|
-| `Dockerfile.ubuntu` | Ubuntu 22.04 | apt | System-wide `.deb` (GCS, S3, Azure) + fakeS3 |
-| `Dockerfile.debian` | Debian 13 | apt | System-wide bookworm `.deb` (GCS, S3, Azure) + fakeS3 |
-| `Dockerfile.rocky` | Rocky 8, Rocky 9 | dnf | None |
+| Dockerfile                 | OS targets       | Package manager | Remote file drivers |
+|----------------------------|------------------|---|---|
+| `Dockerfile.ubuntu-debian` | Ubuntu 22.04 or Debian 13 | apt | System-wide `.deb` (GCS, S3, Azure) + fakeS3 |
+| `Dockerfile.rocky`         | Rocky 8, Rocky 9 | dnf | None |
 
 All images are published to `ghcr.io/khiopsml/khiops-python/khiopspydev-<os>`.
 
 ### Debian and Ubuntu
 
-`Dockerfile.debian` and `Dockerfile.ubuntu` are nearly identical. They diverge
-because Debian 13 remote file driver packages are not available, so Debian
-forces the Debian 12 (bookworm) builds for the GCS, S3, and Azure driver `.deb`
-packages. Any shared change should be applied to both files. There is an open
-TODO to unify them (see the comment at the top of `Dockerfile.debian`).
+`Dockerfile.ubuntu-debian` is a unique Dockerfile to build either a Ubuntu or a Debian target image 
+because there is no difference except for the base image used.
+
+The only reason why the system-wide native Khiops binary (from a `.deb` package) is still installed in the target image 
+is to allow the automated integration tests (located in the `tests/test_remote_access.py` test file) run against a `KhiopsDockerRunner`. 
 
 ### Rocky
 
@@ -52,18 +51,14 @@ All Dockerfiles accept these `ARG` values, supplied by `dev-docker.yml`:
 | `KHIOPS_REVISION` | Khiops native release tag to install | `11.0.0` |
 | `SERVER_REVISION` | Git ref for the `khiops-server` image (copied into the final stage) | `main` |
 | `PYTHON_VERSIONS` | Space-separated Python versions for Conda environments | `3.10 3.11 3.12 3.13 3.14` |
-| `KHIOPS_GCS_DRIVER_REVISION` | GCS remote file driver version (Debian/Ubuntu only) | `0.0.16` |
-| `KHIOPS_S3_DRIVER_REVISION` | S3 remote file driver version (Debian/Ubuntu only) | `0.0.15` |
-| `KHIOPS_AZURE_DRIVER_REVISION` | Azure remote file driver version (Debian/Ubuntu only) | `0.0.6` |
 
 ## Multi-Stage Build Structure
 
 Each Dockerfile uses a multi-stage build:
 
 1. **`khiopsdev`** — Based on `ghcr.io/khiopsml/khiops/khiopsdev-<os>:latest`.
-   Installs dev tools (git, pip, pandoc, wget), the Khiops native binary, and
-   Miniforge for Conda. On Debian/Ubuntu, also installs system-wide remote file
-   drivers (GCS, S3, Azure `.deb` packages) and `ruby-dev` (needed for fakeS3).
+   Installs dev tools (git, pip, pandoc, wget, zip, unzip), the Khiops native binary, and
+   Miniforge for Conda. On Debian/Ubuntu, also installs `ruby-dev` (needed for fakeS3).
 2. **`server`** — Pulls the `khiops-server` binary from
    `ghcr.io/khiopsml/khiops-server:<SERVER_REVISION>`.
 3. **`base`** (final) — Copies the server binary into the `khiopsdev` stage. On
@@ -75,10 +70,6 @@ For each Python version in `PYTHON_VERSIONS`, a Conda environment is created:
 
 - **`py<version>`** — Bare Python (for pip-based test installs).
 
-A special **`py3_khiops10_conda`** environment is always created with
-`khiops-core==10.3.2` to test backward compatibility with Khiops major
-version 10.
-
 ## Helper Scripts
 
 | Script | Purpose |
@@ -100,8 +91,7 @@ version 10.
   optionally also `latest`.
 - **Build context**: `./packaging/docker/khiopspydev/`.
 - **Build args**: Passes `KHIOPS_REVISION`, `KHIOPSDEV_OS`, `SERVER_REVISION`,
-  `PYTHON_VERSIONS`, `KHIOPS_GCS_DRIVER_REVISION`, `KHIOPS_S3_DRIVER_REVISION`,
-  and `KHIOPS_AZURE_DRIVER_REVISION`.
+  `PYTHON_VERSIONS`.
 - **Important**: The `add-hosts: s3-bucket.localhost:127.0.0.1` input is required
   because buildx mounts `/etc/hosts` read-only, so the fakeS3 hostname cannot
   be added inside the Dockerfile.
@@ -133,12 +123,6 @@ Builds the Sphinx documentation inside the `khiopspydev-ubuntu22.04` container.
   in consumer workflows (`tests.yml`, etc.).
 - **Adding system dependencies**: Install them in the appropriate `RUN` block
   (apt for Debian/Ubuntu, dnf for Rocky).
-- **Remote file drivers**: Version bumps go in the
-  `DEFAULT_KHIOPS_GCS_DRIVER_REVISION` / `DEFAULT_KHIOPS_S3_DRIVER_REVISION` /
-  `DEFAULT_KHIOPS_AZURE_DRIVER_REVISION` env vars in `dev-docker.yml`. Note:
-  there is a known workaround in the Ubuntu and Debian Dockerfiles for a release
-  tag typo in the Azure driver repository (the download URL hard-codes tag
-  `0.0.7` regardless of the revision ARG — see the `XXX` comment).
 - **fakeS3**: Pinned to version `1.2.1` because `>= 1.3` requires a license key.
   If fakeS3 becomes incompatible, consider alternatives (the Dockerfile comments
   mention `s3rver` as a candidate).

.github/workflows/api-docs.yml

@@ -60,6 +60,16 @@ jobs:
         run: |
           # Install package itself to install the samples datasets
           python -m pip install --upgrade pip
+          # Add homogeneous TOML support (Python >= 3.12 has standard tomllib)
+          python -m pip install tomli
+          # First, install all dependencies except khiops-core and khiops-drivers-*
+          python scripts/extract_dependencies_from_pyproject_toml.py -f "pyproject.toml" --exclude-khiops-family > requires-no-khiops.txt          
+          python -m pip install --user `cat requires-no-khiops.txt`
+          # khiops-core and khiops-drivers-* must always be installed from TestPyPI in order to avoid distorting usage statistics
+          python scripts/extract_dependencies_from_pyproject_toml.py -f "pyproject.toml" --khiops-family-only > requires-khiops.txt
+          python -m pip install --user --index-url https://test.pypi.org/simple `cat requires-khiops.txt`
+          rm -f requires-khiops.txt requires-no-khiops.txt
+          # Lastly, install khiops-python
           python -m pip install --user .
           kh-download-datasets --force-overwrite --version ${{ inputs.khiops-samples-revision || env.DEFAULT_KHIOPS_SAMPLES_REVISION }}
           kh-status

.github/workflows/dev-docker.yml

@@ -5,9 +5,6 @@ env:
   DEFAULT_IMAGE_INCREMENT: 0
   DEFAULT_SERVER_REVISION: main
   DEFAULT_PYTHON_VERSIONS: 3.10 3.11 3.12 3.13 3.14
-  DEFAULT_KHIOPS_GCS_DRIVER_REVISION: 0.0.16
-  DEFAULT_KHIOPS_S3_DRIVER_REVISION: 0.0.15
-  DEFAULT_KHIOPS_AZURE_DRIVER_REVISION: 0.0.6  # XXX : to modify soon
 on:
   pull_request:
     paths: [packaging/docker/khiopspydev/Dockerfile.*, .github/workflows/dev-docker.yml]
@@ -16,7 +13,7 @@ on:
       khiops-revision:
         type: string
         default: 11.0.0
-        description: Khiops Revision
+        description: Khiops Revision (for tests against KhiopsDockerRunner)
       image-increment:
         type: number
         default: 0
@@ -37,18 +34,6 @@ on:
         type: string
         default: main
         description: Khiops Server Revision
-      khiops-gcs-driver-revision:
-        type: string
-        default: 0.0.16
-        description: Driver version for Google Cloud Storage remote files
-      khiops-s3-driver-revision:
-        type: string
-        default: 0.0.15
-        description: Driver version for AWS-S3 remote files
-      khiops-azure-driver-revision:
-        type: string
-        default: 0.0.6  # XXX : to modify soon
-        description: Driver version for Azure remote files and blobs
 concurrency:
   group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
   cancel-in-progress: true
@@ -63,16 +48,28 @@ jobs:
       packages: write  # to write in the Github package registry
     steps:
       - name: Set input parameters as env or output
+        shell: bash
         run: |
           set -x
           echo "KHIOPS_REVISION=${{ inputs.khiops-revision || env.DEFAULT_KHIOPS_REVISION }}" >> "$GITHUB_ENV"
           echo "IMAGE_INCREMENT=${{ inputs.image-increment || env.DEFAULT_IMAGE_INCREMENT }}" >> "$GITHUB_ENV"
-          echo "KHIOPSDEV_OS_CODENAME=$(echo '${{ matrix.khiopsdev-os }}' | tr -d '0-9.')" >> "$GITHUB_ENV"
+          KHIOPSDEV_OS_CODENAME=$(echo '${{ matrix.khiopsdev-os }}' | tr -d '0-9.')
+          echo "KHIOPSDEV_OS_CODENAME=${KHIOPSDEV_OS_CODENAME}" >> "$GITHUB_ENV"
+          case ${KHIOPSDEV_OS_CODENAME} in
+            ubuntu | debian)
+              # same Dockerfile for the whole family
+              echo "DOCKER_FILE_NAME=Dockerfile.ubuntu-debian" >> "$GITHUB_ENV"
+              ;;
+            rocky)              
+              echo "DOCKER_FILE_NAME=Dockerfile.rocky" >> "$GITHUB_ENV"
+              ;;
+            *)
+              echo "::error::Status error: '${KHIOPSDEV_OS_CODENAME}' is an unexpected OS codename."
+              exit 1
+              ;;
+          esac        
           echo "SERVER_REVISION=${{ inputs.server-revision || env.DEFAULT_SERVER_REVISION }}" >> "$GITHUB_ENV"
           echo "IMAGE_URL=ghcr.io/khiopsml/khiops-python/khiopspydev-${{ matrix.khiopsdev-os }}" >> "$GITHUB_ENV"
-          echo "KHIOPS_GCS_DRIVER_REVISION=${{ inputs.khiops-gcs-driver-revision || env.DEFAULT_KHIOPS_GCS_DRIVER_REVISION }}" >> "$GITHUB_ENV"
-          echo "KHIOPS_S3_DRIVER_REVISION=${{ inputs.khiops-s3-driver-revision || env.DEFAULT_KHIOPS_S3_DRIVER_REVISION }}" >> "$GITHUB_ENV"
-          echo "KHIOPS_AZURE_DRIVER_REVISION=${{ inputs.khiops-azure-driver-revision || env.DEFAULT_KHIOPS_AZURE_DRIVER_REVISION }}" >> "$GITHUB_ENV"
       - name: Checkout khiops-python sources
         uses: actions/checkout@v4
       - name: Set up Docker Buildx
@@ -103,15 +100,12 @@ jobs:
           # added using inputs because /etc/hosts is read-only for alternate builders (buildx via moby buildkit)
           add-hosts: s3-bucket.localhost:127.0.0.1
           context: ./packaging/docker/khiopspydev/
-          file: ./packaging/docker/khiopspydev/Dockerfile.${{ env.KHIOPSDEV_OS_CODENAME }}
+          file: ./packaging/docker/khiopspydev/${{ env.DOCKER_FILE_NAME }}
           build-args: |
             "KHIOPS_REVISION=${{ env.KHIOPS_REVISION }}"
             "KHIOPSDEV_OS=${{ matrix.khiopsdev-os }}"
             "SERVER_REVISION=${{ env.SERVER_REVISION }}"
             "PYTHON_VERSIONS=${{ inputs.python-versions || env.DEFAULT_PYTHON_VERSIONS }}"
-            "KHIOPS_GCS_DRIVER_REVISION=${{ env.KHIOPS_GCS_DRIVER_REVISION }}"
-            "KHIOPS_S3_DRIVER_REVISION=${{ env.KHIOPS_S3_DRIVER_REVISION }}"
-            "KHIOPS_AZURE_DRIVER_REVISION=${{ env.KHIOPS_AZURE_DRIVER_REVISION }}"
           tags: ${{ env.DOCKER_IMAGE_TAGS }}
           # Push only on manual request
           push: ${{ inputs.push || false }}

.github/workflows/pip.yml

@@ -77,6 +77,12 @@ jobs:
         run: |
           SAMPLES_REVISION=${{ inputs.samples-revision || env.DEFAULT_SAMPLES_REVISION }}
           echo "SAMPLES_REVISION=$SAMPLES_REVISION" >> "$GITHUB_ENV"
+      - name: Checkout sources  # Checkout the sources to be able to extract the dependencies list
+        uses: actions/checkout@v4
+        with:
+          # Get Git tags so that versioneer can function correctly
+          # See issue https://github.com/actions/checkout/issues/701
+          fetch-depth: 0
       - name: Checkout Khiops samples
         uses: actions/checkout@v4
         with:
@@ -88,12 +94,13 @@ jobs:
         uses: actions/download-artifact@v4
         with:
           name: pip-package
-      - name: Install package
+      - name: Install the package
         shell: bash
         run: |
           # Allow Pip to write to its cache
           mkdir -p /github/home/.cache/pip
           chown -R $(whoami) /github/home/.cache/pip
+
           # Install the Khiops Python library
 
           # A virtual env is mandatory under debian
@@ -102,6 +109,16 @@ jobs:
             source khiops-debian-venv/bin/activate
           fi
           pip install --upgrade pip
+          # Add homogeneous TOML support (Python >= 3.12 has standard tomllib)
+          pip install tomli
+          # First, install all dependencies except khiops-core and khiops-drivers-*
+          python scripts/extract_dependencies_from_pyproject_toml.py -f "pyproject.toml" --exclude-khiops-family > requires-no-khiops.txt
+          pip install `cat requires-no-khiops.txt`
+          # khiops-core and khiops-drivers-* must always be installed from TestPyPI in order to avoid distorting usage statistics
+          python scripts/extract_dependencies_from_pyproject_toml.py -f "pyproject.toml" --khiops-family-only > requires-khiops.txt
+          pip install --index-url https://test.pypi.org/simple `cat requires-khiops.txt`
+          rm -f requires-khiops.txt requires-no-khiops.txt
+          # Lastly, install khiops-python
           pip install $(ls khiops*.tar.gz)
           if [[ "${{ matrix.container }}" == "debian13" ]]; then            
             deactivate

.github/workflows/quick-checks.yml

@@ -19,11 +19,15 @@ jobs:
       - name: Install khiops-python dev dependencies
         run: |
           # Extract and install package dependency requirements from metadata
-          pip install pip-tools
-          python -m piptools compile -o requirements.txt
-
-          # Install dev dependencies
-          pip install -r requirements.txt
+          # Add homogeneous TOML support (Python >= 3.12 has standard tomllib)
+          pip install tomli
+          # First, install all dependencies except khiops-core and khiops-drivers-*
+          python scripts/extract_dependencies_from_pyproject_toml.py -f "pyproject.toml" --exclude-khiops-family > requires-no-khiops.txt
+          pip install `cat requires-no-khiops.txt`
+          # khiops-core and khiops-drivers-* must always be installed from TestPyPI in order to avoid distorting usage statistics
+          python scripts/extract_dependencies_from_pyproject_toml.py -f "pyproject.toml" --khiops-family-only > requires-khiops.txt
+          pip install --index-url https://test.pypi.org/simple `cat requires-khiops.txt`
+          rm -f requires-khiops.txt requires-no-khiops.txt
 
           # Install black for the samples-generation script
           pip install black