Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions .github/workflows/CI-test_image_action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
name: CI | test-image action

on:
pull_request:
paths:
- "test-image/**"
- ".github/workflows/CI-test_image_action.yml"
push:
branches:
- main
paths:
- "test-image/**"

jobs:
unit-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v6
- name: Run unit tests
working-directory: test-image
run: uv run -p 3.12 --no-project --with pytest --with runpod==2.1.0 pytest -q tests
93 changes: 0 additions & 93 deletions .github/workflows/TEMPLATE-run_worker_tests.yml

This file was deleted.

4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,10 @@ This repository hosts the code for deploying a GitHub Actions worker on the Runp

5. Deploy your worker to the Runpod platform.

## Testing worker images in CI

The [`test-image`](./test-image) action runs a worker image's `.runpod/tests.json` cases on a temporary endpoint and deletes it afterwards. See [docs/usage.md](./docs/usage.md).

## Documentation

For more details on how this GitHub Actions worker functions and how to integrate it into your CI/CD pipeline, please refer to the [Documentation](./docs).
Expand Down
137 changes: 86 additions & 51 deletions docs/usage.md
Original file line number Diff line number Diff line change
@@ -1,53 +1,88 @@
# Documentation

## CI/CD with GitHub Actions and Runpod

This repository uses GitHub Actions to run a series of tests on the Runpod serverless platform. The workflow is defined in the `.github/workflows/test-worker.yml` file, with the primary purpose to deploy a worker to the Runpod platform, execute the tests, and then terminate the worker.

Here is a step-by-step breakdown of the `test-worker.yml` file:

### Triggers

This workflow runs whenever:

- A push is made to the main branch.
- A pull request is opened to the main branch.
- The workflow is manually triggered (workflow_dispatch).

### Jobs

#### `initialize_worker`

This job deploys a worker to the Runpod platform. It:

- Sends a POST request to the Runpod API to start a new job (uses `fjogeleit/http-request-action@v1`).
- Extracts the job ID from the response and sets it as a job output.

#### `run_tests`

This job runs the tests in the worker environment. It:

- Checks out the repository code.
- Sets up Python 3.11 and installs dependencies.
- Executes tests by running `handler.py` with a test input.

#### `terminate_worker`

This job terminates the worker in the Runpod platform. It:

- Sends a POST request to the Runpod API to cancel the job (uses `fjogeleit/http-request-action@v1`).

It only runs if the previous job (`initialize_worker` or `run_tests`) was cancelled or failed.

### Secrets

The workflow requires the following secrets to be set in the repository settings:

- `RUNPOD_API_KEY`: The API key to authenticate requests to the Runpod API.
- `RUNPOD_ENDPOINT`: The endpoint of the Runpod API (not including the base URL).

### Customizing the Workflow

This workflow can be customized by editing the `.github/workflows/test-worker.yml` file. For example, to change the Python version, edit the `python-version` input in the `Set up Python 3.11 & install dependencies` step. To add or modify tests, edit the `Execute Tests` step.

**Note:** Always ensure that the necessary permissions are granted when modifying GitHub workflows. It's also a good practice to validate your changes with a trusted colleague or through a pull request to prevent accidental disruptions to the workflow.
## Testing a worker image in CI

The `test-image` action runs a worker image's test cases on a temporary serverless endpoint and deletes the endpoint afterwards. Each run gets its own endpoint, so CI does not depend on a long-lived endpoint that can be paused for being idle.

It uses Flash from runpod-python (`rp flash dev --once`):

1. The CLI creates a live endpoint from your image, with at most one worker.
2. For each test in your tests file, it sends a job to that endpoint. What the job does on the worker depends on the mode:
- **Handler mode** (default, queue workers): load your handler file, capture the handler it registers with `runpod.serverless.start(...)`, and call it with the test input.
- **Server mode** (load-balancer workers, selected by `start-command`): start your server once with `PORT` set, wait until its health path returns 200, and send the test's HTTP request to it.
3. A handler-mode test fails if the job fails, times out, returns no output, or returns a dict with an `error` key. A server-mode test fails if the server exits or never becomes healthy, or the response status differs from the expected one.
4. The endpoint is deleted when the tests finish, fail, or the run is cancelled.

### Usage

Build and push the image first, then:

```yaml
- uses: runpod-workers/worker-github_runner/test-image@main
with:
image: runpod/worker-example:ci-pr-${{ github.event.number }}
runpod-api-key: ${{ secrets.RUNPOD_API_KEY }}
handler: /handler.py
```

| Input | Default | Description |
|---|---|---|
| `image` | (required) | Worker image to test |
| `runpod-api-key` | (required) | API key used to create and delete the endpoint |
| `tests-file` | `.runpod/tests.json` | Hub-format tests file in your checkout |
| `handler` | `/handler.py` | Path of the handler file inside the image (handler mode) |
| `start-command` | (none) | Command that starts your HTTP server; selects server mode. Runs on the worker through `sh -c` |
| `port` | `8000` | Port your server listens on, passed to it as `PORT` (server mode) |
| `health-path` | `/ping` | Path that returns 200 once the server is ready (server mode) |
| `gpu` | `config.gpuTypeId` from the tests file | GPU type for the endpoint |
| `min-cuda` | lowest `config.allowedCudaVersions` | Minimum host CUDA version for the endpoint |
| `startup-timeout` | `900` | Seconds allowed for the worker to pull the image and start, and in server mode for the server to become healthy |
| `runpod-version` | `2.1.0` | runpod-python version for the CLI and the worker |
| `runtime-package-spec` | pinned `runpod-sdk-runtime` | Flash worker runtime installed into the image |
Comment thread
deanq marked this conversation as resolved.
Resolved

### Tests file

The action reads the same format as the Hub:

```json
{
"tests": [
{"name": "basic", "input": {"prompt": "hello"}, "timeout": 10000}
],
"config": {
"gpuTypeId": "NVIDIA GeForce RTX 4090",
"env": [{"key": "MODEL", "value": "base"}]
}
}
```

`timeout` is in milliseconds and also sets the job's execution timeout, together with `startup-timeout`. Entries in `config.env` are set on the endpoint. The lowest entry of `config.allowedCudaVersions` becomes the endpoint's minimum CUDA version.

In server mode, each test's `input` is an HTTP request. `method` defaults to `GET` and `status` (the expected response status) to `200`:

```json
{
"tests": [
{
"name": "completion",
"input": {"method": "POST", "path": "/v1/completions", "body": {"prompt": "Hello", "max_tokens": 8}},
"timeout": 120000
}
]
}
```

```yaml
- uses: runpod-workers/worker-github_runner/test-image@main
with:
image: runpod/worker-example-lb:ci-pr-${{ github.event.number }}
runpod-api-key: ${{ secrets.RUNPOD_API_KEY }}
start-command: cd /src && python3 handler.py
```

### What can go wrong

- **The image's start command does not run.** A live endpoint starts the Flash runtime instead of the image's start command. In handler mode the action calls your handler directly, so code that runs only in the start command (for example, a shell script that runs before `python handler.py`) is not exercised. In server mode, `start-command` should match the image's start command.
- **The worker runs `runpod==<runpod-version>`.** The endpoint installs that version into the image before starting, even if the image pins another one. A handler that only works with an older SDK will fail here.
- **The image needs Python 3.10 or newer, with `pip` on its `python3`,** so the runtime can be installed. Images built on a uv-managed venv without pip need `uv pip install pip` in their Dockerfile.
- **Pull requests from forks** do not get repository secrets, so the action cannot create an endpoint for them. Skip the job for forks.
92 changes: 92 additions & 0 deletions test-image/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
name: "Test a Runpod worker image"
description: >
Run a worker image's Hub test cases on a temporary Flash live endpoint,
then delete the endpoint.
inputs:
image:
description: "Worker image to test, e.g. runpod/worker-faster_whisper:ci-pr-12"
required: true
runpod-api-key:
description: "Runpod API key used to create and delete the endpoint"
required: true
tests-file:
description: "Hub-format tests file in the caller's checkout"
required: false
default: ".runpod/tests.json"
handler:
description: "Path of the handler file inside the image (handler mode)"
required: false
default: "/handler.py"
start-command:
description: >
Command that starts the image's HTTP server, e.g. "cd /src && python3 handler.py".
Setting it selects server mode, for load-balancer workers. It runs on the
worker through `sh -c`, so shell syntax such as `cd` and `&&` works.
required: false
default: ""
port:
description: "Port the server listens on; passed to it as PORT (server mode)"
required: false
default: "8000"
health-path:
description: "Path that returns 200 once the server is ready (server mode)"
required: false
default: "/ping"
gpu:
description: "GPU type; defaults to config.gpuTypeId in the tests file"
required: false
default: ""
min-cuda:
description: "Minimum host CUDA version; defaults to the lowest config.allowedCudaVersions"
required: false
default: ""
startup-timeout:
description: "Seconds allowed for the worker to pull the image and start (and, in server mode, for the server to become healthy)"
required: false
default: "900"
runpod-version:
description: "runpod-python version used by the CLI and installed into the worker"
required: false
default: "2.1.0"
runtime-package-spec:
description: "pip spec of the Flash worker runtime installed into the image"
required: false
# runpod-sdk-runtime v0.2.0; not published to PyPI
default: "https://github.com/runpod-workers/sdk-runtime/archive/14f70227008c1ec317e3fe1865a4688b14f5d432.tar.gz"
runs:
using: "composite"
steps:
- uses: actions/setup-python@v5
with:
python-version: "3.12"

- name: Install runpod-python
shell: bash
env:
RUNPOD_VERSION: ${{ inputs.runpod-version }}
run: python -m pip install --quiet "runpod==${RUNPOD_VERSION}"

- name: Run tests on a temporary endpoint
shell: bash
env:
RUNPOD_API_KEY: ${{ inputs.runpod-api-key }}
RP_TEST_IMAGE: ${{ inputs.image }}
RP_TEST_FILE: ${{ inputs.tests-file }}
RP_TEST_HANDLER: ${{ inputs.handler }}
RP_TEST_START_COMMAND: ${{ inputs.start-command }}
RP_TEST_PORT: ${{ inputs.port }}
RP_TEST_HEALTH_PATH: ${{ inputs.health-path }}
RP_TEST_GPU: ${{ inputs.gpu }}
RP_TEST_MIN_CUDA: ${{ inputs.min-cuda }}
RP_TEST_STARTUP_S: ${{ inputs.startup-timeout }}
# The live endpoint's launcher force-installs these into the image,
# so the worker runs the same SDK as the CLI.
RUNPOD_PACKAGE_SPEC: runpod==${{ inputs.runpod-version }}
RUNPOD_RUNTIME_PACKAGE_SPEC: ${{ inputs.runtime-package-spec }}
run: |
export RP_TEST_FILE="$(realpath "$RP_TEST_FILE")"
export RP_TEST_APP_NAME="$(python -c 'import os, sys; sys.path.insert(0, sys.argv[1]); from cases import app_name; print(app_name(os.environ["GITHUB_REPOSITORY"], os.environ["GITHUB_RUN_ID"], os.environ["GITHUB_RUN_ATTEMPT"]))' "$GITHUB_ACTION_PATH")"
echo "Flash app: $RP_TEST_APP_NAME"
# --once runs main(), deletes the endpoint, and exits non-zero if
# main() raised. stdin from /dev/null keeps the session non-interactive.
rp flash dev --once "$GITHUB_ACTION_PATH/flash_app.py" < /dev/null
Loading
Loading