Repository navigation
feat(ci): add test-image action for per-run endpoint tests #13
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
44f823c
feat(ci): add test-image action for per-run endpoint tests
deanq 757ffa2
feat(ci): add test-image action for per-run endpoint tests
deanq 0f06aab
feat(ci): add server mode, CUDA minimum and job timeout to test-image
deanq 4976704
fix(ci): run async handlers inside the runtime's event loop
deanq 92fbfd5
fix(ci): validate config.env and tolerate a bad pid file
deanq File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| name: CI | test-image action | ||
|
|
||
| on: | ||
| pull_request: | ||
| paths: | ||
| - "test-image/**" | ||
| - ".github/workflows/CI-test_image_action.yml" | ||
| push: | ||
| branches: | ||
| - main | ||
| paths: | ||
| - "test-image/**" | ||
|
|
||
| jobs: | ||
| unit-tests: | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - uses: astral-sh/setup-uv@v6 | ||
| - name: Run unit tests | ||
| working-directory: test-image | ||
| run: uv run -p 3.12 --no-project --with pytest --with runpod==2.1.0 pytest -q tests |
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,53 +1,88 @@ | ||
| # Documentation | ||
|
|
||
| ## CI/CD with GitHub Actions and Runpod | ||
|
|
||
| This repository uses GitHub Actions to run a series of tests on the Runpod serverless platform. The workflow is defined in the `.github/workflows/test-worker.yml` file, with the primary purpose to deploy a worker to the Runpod platform, execute the tests, and then terminate the worker. | ||
|
|
||
| Here is a step-by-step breakdown of the `test-worker.yml` file: | ||
|
|
||
| ### Triggers | ||
|
|
||
| This workflow runs whenever: | ||
|
|
||
| - A push is made to the main branch. | ||
| - A pull request is opened to the main branch. | ||
| - The workflow is manually triggered (workflow_dispatch). | ||
|
|
||
| ### Jobs | ||
|
|
||
| #### `initialize_worker` | ||
|
|
||
| This job deploys a worker to the Runpod platform. It: | ||
|
|
||
| - Sends a POST request to the Runpod API to start a new job (uses `fjogeleit/http-request-action@v1`). | ||
| - Extracts the job ID from the response and sets it as a job output. | ||
|
|
||
| #### `run_tests` | ||
|
|
||
| This job runs the tests in the worker environment. It: | ||
|
|
||
| - Checks out the repository code. | ||
| - Sets up Python 3.11 and installs dependencies. | ||
| - Executes tests by running `handler.py` with a test input. | ||
|
|
||
| #### `terminate_worker` | ||
|
|
||
| This job terminates the worker in the Runpod platform. It: | ||
|
|
||
| - Sends a POST request to the Runpod API to cancel the job (uses `fjogeleit/http-request-action@v1`). | ||
|
|
||
| It only runs if the previous job (`initialize_worker` or `run_tests`) was cancelled or failed. | ||
|
|
||
| ### Secrets | ||
|
|
||
| The workflow requires the following secrets to be set in the repository settings: | ||
|
|
||
| - `RUNPOD_API_KEY`: The API key to authenticate requests to the Runpod API. | ||
| - `RUNPOD_ENDPOINT`: The endpoint of the Runpod API (not including the base URL). | ||
|
|
||
| ### Customizing the Workflow | ||
|
|
||
| This workflow can be customized by editing the `.github/workflows/test-worker.yml` file. For example, to change the Python version, edit the `python-version` input in the `Set up Python 3.11 & install dependencies` step. To add or modify tests, edit the `Execute Tests` step. | ||
|
|
||
| **Note:** Always ensure that the necessary permissions are granted when modifying GitHub workflows. It's also a good practice to validate your changes with a trusted colleague or through a pull request to prevent accidental disruptions to the workflow. | ||
| ## Testing a worker image in CI | ||
|
|
||
| The `test-image` action runs a worker image's test cases on a temporary serverless endpoint and deletes the endpoint afterwards. Each run gets its own endpoint, so CI does not depend on a long-lived endpoint that can be paused for being idle. | ||
|
|
||
| It uses Flash from runpod-python (`rp flash dev --once`): | ||
|
|
||
| 1. The CLI creates a live endpoint from your image, with at most one worker. | ||
| 2. For each test in your tests file, it sends a job to that endpoint. What the job does on the worker depends on the mode: | ||
| - **Handler mode** (default, queue workers): load your handler file, capture the handler it registers with `runpod.serverless.start(...)`, and call it with the test input. | ||
| - **Server mode** (load-balancer workers, selected by `start-command`): start your server once with `PORT` set, wait until its health path returns 200, and send the test's HTTP request to it. | ||
| 3. A handler-mode test fails if the job fails, times out, returns no output, or returns a dict with an `error` key. A server-mode test fails if the server exits or never becomes healthy, or the response status differs from the expected one. | ||
| 4. The endpoint is deleted when the tests finish, fail, or the run is cancelled. | ||
|
|
||
| ### Usage | ||
|
|
||
| Build and push the image first, then: | ||
|
|
||
| ```yaml | ||
| - uses: runpod-workers/worker-github_runner/test-image@main | ||
| with: | ||
| image: runpod/worker-example:ci-pr-${{ github.event.number }} | ||
| runpod-api-key: ${{ secrets.RUNPOD_API_KEY }} | ||
| handler: /handler.py | ||
| ``` | ||
|
|
||
| | Input | Default | Description | | ||
| |---|---|---| | ||
| | `image` | (required) | Worker image to test | | ||
| | `runpod-api-key` | (required) | API key used to create and delete the endpoint | | ||
| | `tests-file` | `.runpod/tests.json` | Hub-format tests file in your checkout | | ||
| | `handler` | `/handler.py` | Path of the handler file inside the image (handler mode) | | ||
| | `start-command` | (none) | Command that starts your HTTP server; selects server mode. Runs on the worker through `sh -c` | | ||
| | `port` | `8000` | Port your server listens on, passed to it as `PORT` (server mode) | | ||
| | `health-path` | `/ping` | Path that returns 200 once the server is ready (server mode) | | ||
| | `gpu` | `config.gpuTypeId` from the tests file | GPU type for the endpoint | | ||
| | `min-cuda` | lowest `config.allowedCudaVersions` | Minimum host CUDA version for the endpoint | | ||
| | `startup-timeout` | `900` | Seconds allowed for the worker to pull the image and start, and in server mode for the server to become healthy | | ||
| | `runpod-version` | `2.1.0` | runpod-python version for the CLI and the worker | | ||
| | `runtime-package-spec` | pinned `runpod-sdk-runtime` | Flash worker runtime installed into the image | | ||
|
|
||
| ### Tests file | ||
|
|
||
| The action reads the same format as the Hub: | ||
|
|
||
| ```json | ||
| { | ||
| "tests": [ | ||
| {"name": "basic", "input": {"prompt": "hello"}, "timeout": 10000} | ||
| ], | ||
| "config": { | ||
| "gpuTypeId": "NVIDIA GeForce RTX 4090", | ||
| "env": [{"key": "MODEL", "value": "base"}] | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| `timeout` is in milliseconds and also sets the job's execution timeout, together with `startup-timeout`. Entries in `config.env` are set on the endpoint. The lowest entry of `config.allowedCudaVersions` becomes the endpoint's minimum CUDA version. | ||
|
|
||
| In server mode, each test's `input` is an HTTP request. `method` defaults to `GET` and `status` (the expected response status) to `200`: | ||
|
|
||
| ```json | ||
| { | ||
| "tests": [ | ||
| { | ||
| "name": "completion", | ||
| "input": {"method": "POST", "path": "/v1/completions", "body": {"prompt": "Hello", "max_tokens": 8}}, | ||
| "timeout": 120000 | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| ```yaml | ||
| - uses: runpod-workers/worker-github_runner/test-image@main | ||
| with: | ||
| image: runpod/worker-example-lb:ci-pr-${{ github.event.number }} | ||
| runpod-api-key: ${{ secrets.RUNPOD_API_KEY }} | ||
| start-command: cd /src && python3 handler.py | ||
| ``` | ||
|
|
||
| ### What can go wrong | ||
|
|
||
| - **The image's start command does not run.** A live endpoint starts the Flash runtime instead of the image's start command. In handler mode the action calls your handler directly, so code that runs only in the start command (for example, a shell script that runs before `python handler.py`) is not exercised. In server mode, `start-command` should match the image's start command. | ||
| - **The worker runs `runpod==<runpod-version>`.** The endpoint installs that version into the image before starting, even if the image pins another one. A handler that only works with an older SDK will fail here. | ||
| - **The image needs Python 3.10 or newer, with `pip` on its `python3`,** so the runtime can be installed. Images built on a uv-managed venv without pip need `uv pip install pip` in their Dockerfile. | ||
| - **Pull requests from forks** do not get repository secrets, so the action cannot create an endpoint for them. Skip the job for forks. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,92 @@ | ||
| name: "Test a Runpod worker image" | ||
| description: > | ||
| Run a worker image's Hub test cases on a temporary Flash live endpoint, | ||
| then delete the endpoint. | ||
| inputs: | ||
| image: | ||
| description: "Worker image to test, e.g. runpod/worker-faster_whisper:ci-pr-12" | ||
| required: true | ||
| runpod-api-key: | ||
| description: "Runpod API key used to create and delete the endpoint" | ||
| required: true | ||
| tests-file: | ||
| description: "Hub-format tests file in the caller's checkout" | ||
| required: false | ||
| default: ".runpod/tests.json" | ||
| handler: | ||
| description: "Path of the handler file inside the image (handler mode)" | ||
| required: false | ||
| default: "/handler.py" | ||
| start-command: | ||
| description: > | ||
| Command that starts the image's HTTP server, e.g. "cd /src && python3 handler.py". | ||
| Setting it selects server mode, for load-balancer workers. It runs on the | ||
| worker through `sh -c`, so shell syntax such as `cd` and `&&` works. | ||
| required: false | ||
| default: "" | ||
| port: | ||
| description: "Port the server listens on; passed to it as PORT (server mode)" | ||
| required: false | ||
| default: "8000" | ||
| health-path: | ||
| description: "Path that returns 200 once the server is ready (server mode)" | ||
| required: false | ||
| default: "/ping" | ||
| gpu: | ||
| description: "GPU type; defaults to config.gpuTypeId in the tests file" | ||
| required: false | ||
| default: "" | ||
| min-cuda: | ||
| description: "Minimum host CUDA version; defaults to the lowest config.allowedCudaVersions" | ||
| required: false | ||
| default: "" | ||
| startup-timeout: | ||
| description: "Seconds allowed for the worker to pull the image and start (and, in server mode, for the server to become healthy)" | ||
| required: false | ||
| default: "900" | ||
| runpod-version: | ||
| description: "runpod-python version used by the CLI and installed into the worker" | ||
| required: false | ||
| default: "2.1.0" | ||
| runtime-package-spec: | ||
| description: "pip spec of the Flash worker runtime installed into the image" | ||
| required: false | ||
| # runpod-sdk-runtime v0.2.0; not published to PyPI | ||
| default: "https://github.com/runpod-workers/sdk-runtime/archive/14f70227008c1ec317e3fe1865a4688b14f5d432.tar.gz" | ||
| runs: | ||
| using: "composite" | ||
| steps: | ||
| - uses: actions/setup-python@v5 | ||
| with: | ||
| python-version: "3.12" | ||
|
|
||
| - name: Install runpod-python | ||
| shell: bash | ||
| env: | ||
| RUNPOD_VERSION: ${{ inputs.runpod-version }} | ||
| run: python -m pip install --quiet "runpod==${RUNPOD_VERSION}" | ||
|
|
||
| - name: Run tests on a temporary endpoint | ||
| shell: bash | ||
| env: | ||
| RUNPOD_API_KEY: ${{ inputs.runpod-api-key }} | ||
| RP_TEST_IMAGE: ${{ inputs.image }} | ||
| RP_TEST_FILE: ${{ inputs.tests-file }} | ||
| RP_TEST_HANDLER: ${{ inputs.handler }} | ||
| RP_TEST_START_COMMAND: ${{ inputs.start-command }} | ||
| RP_TEST_PORT: ${{ inputs.port }} | ||
| RP_TEST_HEALTH_PATH: ${{ inputs.health-path }} | ||
| RP_TEST_GPU: ${{ inputs.gpu }} | ||
| RP_TEST_MIN_CUDA: ${{ inputs.min-cuda }} | ||
| RP_TEST_STARTUP_S: ${{ inputs.startup-timeout }} | ||
| # The live endpoint's launcher force-installs these into the image, | ||
| # so the worker runs the same SDK as the CLI. | ||
| RUNPOD_PACKAGE_SPEC: runpod==${{ inputs.runpod-version }} | ||
| RUNPOD_RUNTIME_PACKAGE_SPEC: ${{ inputs.runtime-package-spec }} | ||
| run: | | ||
| export RP_TEST_FILE="$(realpath "$RP_TEST_FILE")" | ||
| export RP_TEST_APP_NAME="$(python -c 'import os, sys; sys.path.insert(0, sys.argv[1]); from cases import app_name; print(app_name(os.environ["GITHUB_REPOSITORY"], os.environ["GITHUB_RUN_ID"], os.environ["GITHUB_RUN_ATTEMPT"]))' "$GITHUB_ACTION_PATH")" | ||
| echo "Flash app: $RP_TEST_APP_NAME" | ||
| # --once runs main(), deletes the endpoint, and exits non-zero if | ||
| # main() raised. stdin from /dev/null keeps the session non-interactive. | ||
| rp flash dev --once "$GITHUB_ACTION_PATH/flash_app.py" < /dev/null |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.