[ashtongraves]

What

New image osg-htc/nrp-provisioner: the demand-driven HTCondor provisioner that launches osg-htc/nrp-ospool-ep worker pods on the NRP Nautilus Kubernetes cluster. One provisioner per namespace reads idle jobs from whitelisted schedds, groups them by resource shape, and creates Kubernetes Jobs whose pods join the OSPool (or IGWN) with one static slot each.

It currently runs in four NRP namespaces (osg-opportunistic, osg-icecube, osg-nrao, osg-ligo) from a personal registry project. Building it here puts it next to its worker image, osg-htc/nrp-ospool-ep.

Contents

• Dockerfile: Rocky 10, OSG 25, HTCondor 25 with the htcondor2 bindings, Python 3.12, kubernetes client pinned at 37.0.1.
• src/: the provisioner library; provisioner_main.py and the startup scripts.
• README.md: how it works, configuration, authentication, troubleshooting, building and tests.
• tests/: unit tests, an image test (tests/image.sh <image>) and a sample config.
• LICENSES/: the code comes from sfiligoi/prp-osg-pool (MIT) and the prp-htcondor-portal provisioner (BSD-2-Clause), and keeps those copyright headers.

Notes

• build-config.json is el10 / OSG 25 / release, so CI publishes nrp-provisioner:25-release. The Dockerfile installs OSG 25 on Rocky 10 itself and doesn't use the BASE_* build arguments, so Docker may warn that they're unused.
• Tested on an amd64 build of this directory: tests/image.sh passes (32 unit tests inside the image; startup with a fake service account reaches the provisioning loop and reads the OSPool collector and a schedd).
• The deployment configs, operations notes and live checks stay in ashtongraves/nrp-provisioner, whose provisioner/ directory is kept identical to this one (changes land here first).

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

Scheduling, resource accounting, and failure-isolation defects can stall provisioning, repeatedly create workers, or terminate running workloads.

5 open issues

• Preserve GPU, disk, and label fields in demand cluster keys · New
• Prevent one worker failure from terminating sibling pods · New
• Omit empty node affinity to avoid matching no nodes · New
• Propagate background process failures from the worker script · New
• Use Ki units for HTCondor disk resource limits · New

What changed in this PR

Adds osg-htc/nrp-provisioner alongside its worker image, bringing demand-driven HTCondor worker provisioning for NRP Nautilus into this repository.

Changes:

• Adds the container build, credential setup, and runtime entry points.
• Implements demand queries, resource grouping, Kubernetes Job creation, trial pods, and back-off.
• Includes unit and image checks, configuration examples, operational documentation, and upstream licenses.

File	Description
osg-htc/​nrp-provisioner/​tests/​test_schedd_query.py	Tests schedd filtering, query constraints, and failures.
osg-htc/​nrp-provisioner/​tests/​test_logging.py	Tests file logging and stdout mirroring.
osg-htc/​nrp-provisioner/​tests/​test_k8s_submit.py	Checks Job submission and resource quantities.
osg-htc/​nrp-provisioner/​tests/​test_htcondor_ads.py	Tests ad conversion and resource grouping.
osg-htc/​nrp-provisioner/​tests/​test_backoff.py	Tests back-off, trial pods, and loop integration.
osg-htc/​nrp-provisioner/​tests/​sample_osg_provisioner.conf	Supplies configuration for tests.
osg-htc/​nrp-provisioner/​tests/​image.sh	Checks image dependencies, environment, and startup.
osg-htc/​nrp-provisioner/​src/​provisioner_logging.py	Provides buffered file and stdout logging.
osg-htc/​nrp-provisioner/​src/​provisioner_k8s.py	Configures, queries, and creates worker Jobs.
osg-htc/​nrp-provisioner/​src/​provisioner_htcondor.py	Queries schedds and collector ads using htcondor2.
osg-htc/​nrp-provisioner/​src/​provisioner_config_parser.py	Parses configuration values.
osg-htc/​nrp-provisioner/​src/​provisioner_clustering.py	Groups demand and workers by resource shape.
osg-htc/​nrp-provisioner/​src/​provisioner_backoff.py	Pauses provisioning after repeated unclaimed exits.
osg-htc/​nrp-provisioner/​src/​event_loop.py	Coordinates queries, accounting, and provisioning.
osg-htc/​nrp-provisioner/​src/​__init__.py	Marks the provisioner library package.
osg-htc/​nrp-provisioner/​setup_k8s_creds.sh	Preserves Kubernetes settings for the runtime user.
osg-htc/​nrp-provisioner/​setup_htcondor_creds.sh	Copies tokens and configures optional authentication.
osg-htc/​nrp-provisioner/​README.md	Documents operation, configuration, troubleshooting, and tests.
osg-htc/​nrp-provisioner/​provisioner_startup.sh	Prepares runtime settings and launches the provisioner.
osg-htc/​nrp-provisioner/​provisioner_main.py	Initializes clients and runs the provisioning loop.
osg-htc/​nrp-provisioner/​provisioner_info.py	Reports demand and worker counts by shape.
osg-htc/​nrp-provisioner/​LICENSES/​MIT-prp-osg-pool.txt	Includes the upstream MIT license.
osg-htc/​nrp-provisioner/​LICENSES/​BSD-2-Clause-prp-htcondor-portal.txt	Includes the upstream BSD license.
osg-htc/​nrp-provisioner/​Dockerfile	Builds the provisioner image and runtime environment.
osg-htc/​nrp-provisioner/​build-config.json	Registers the EL10/OSG 25 release build.
osg-htc/​nrp-provisioner/​03_condor_sec.config	Configures HTCondor to use OSG certificates.

_{Review effort: Balanced}