From bf759773115d038a9436d123e17147180f0f4317 Mon Sep 17 00:00:00 2001 From: Gin Biak Date: Wed, 8 Jul 2026 13:39:25 +0530 Subject: [PATCH 1/5] docs(cli,mcp): document evalhub server and mcp command groups (#81, #84) Add documentation for the new `evalhub server run|start|stop|status` commands, `server_config_file` config key, `--unfold` flag on `config get`, and `config unset`. Convert CLI guide from .md to .mdx for Since component support. Update MCP installation docs with CLI-managed `mcp_config_file` configuration flow, `evalhub mcp` subcommands, and breaking change notice for the old `evalhub mcp` stdio command removed in 0.4.3. Co-Authored-By: Claude --- src/content/docs/guides/{cli.md => cli.mdx} | 145 ++++++++++++++++++++ src/content/docs/mcp/installation.mdx | 116 ++++++++++++++-- 2 files changed, 253 insertions(+), 8 deletions(-) rename src/content/docs/guides/{cli.md => cli.mdx} (79%) diff --git a/src/content/docs/guides/cli.md b/src/content/docs/guides/cli.mdx similarity index 79% rename from src/content/docs/guides/cli.md rename to src/content/docs/guides/cli.mdx index aa8f315..f38b7ad 100644 --- a/src/content/docs/guides/cli.md +++ b/src/content/docs/guides/cli.mdx @@ -2,6 +2,8 @@ title: "CLI" --- +import Since from '../../../components/Since.astro'; + The `evalhub` command-line tool lets you submit evaluation jobs, check status, retrieve results, and manage collections and configuration — all from a terminal or shell script. EvalHub is assumed to be running on your OpenShift cluster. If it is not, see [Installation](/getting-started/installation/) first. @@ -48,6 +50,34 @@ evalhub config use staging Switch back with `evalhub config use default`. Any command accepts `--profile ` to override at runtime without changing the active profile. +### Remove a config value + + + +```bash +evalhub config unset token +``` + +For file-based keys (`server_config_file`, `mcp_config_file`), `unset` also removes the stored file copy from the profile directory. + +### Inspect file-based config values + + + +Some config keys (`server_config_file`, `mcp_config_file`) point to YAML files. Use `--unfold` to print the file contents instead of just the stored path: + +```bash +evalhub config get server_config_file --unfold +``` + +Sensitive values (such as tokens) are masked by default. Combine with `--unmask` to reveal them: + +```bash +evalhub config get server_config_file --unfold --unmask +``` + +`--unfold` and `--unmask` are mutually exclusive unless used together — `--unmask` alone shows the raw path value without masking, while `--unfold` alone shows the file contents with sensitive values masked. + ## I want to see what providers and benchmarks are available ```bash @@ -350,6 +380,110 @@ evalhub collections delete bias-and-fairness-a1b2 Pass `--yes` to skip the confirmation prompt in scripts. +## I want to manage the local server + + + +The `evalhub server` commands manage the `eval-hub-server` binary on your local machine. The binary must be on your `PATH` (or set `EVALHUB_SERVER_BIN` to its location). + +### Configure the server + +Before starting the server, provide a configuration file. The CLI validates the file (must exist, be valid YAML, and be a mapping) and stores a profile-scoped copy under `~/.config/evalhub/server//config.yaml`: + +```bash +evalhub config set server_config_file server-config.yaml +``` + +Example `server-config.yaml`: + +```yaml +service: + port: 8080 +``` + +Verify the stored config: + +```bash +evalhub config get server_config_file --unfold +``` + +To remove the server configuration: + +```bash +evalhub config unset server_config_file +``` + +### Run in the foreground + +```bash +evalhub server run +``` + +The server runs in the current terminal session. Press `Ctrl+C` to stop it. + +### Start as a background daemon + +```bash +evalhub server start +# Server started (PID 12345). +# URL: http://localhost:8080 +# Logs: ~/.config/evalhub/server/server.log +``` + +The CLI writes a PID file to `~/.config/evalhub/server/pid` and logs to `~/.config/evalhub/server/server.log`. It waits up to 30 seconds for the health endpoint (`/api/v1/health`) to respond before reporting success. + +### Stop the background server + +```bash +evalhub server stop +# Server stopped. +``` + +Sends `SIGTERM` for a graceful shutdown. If the process does not exit within 5 seconds, it is force-killed. + +### Check server status + +```bash +evalhub server status +# Server is running (PID 12345). +# Health: healthy +# URL: http://localhost:8080 +# Logs: ~/.config/evalhub/server/server.log +``` + +### TLS + +If the server config includes `tls_cert_file` and `tls_key_file` under the `service` key, the server runs with TLS enabled. The CLI detects this automatically and reports `https://` URLs: + +```yaml +service: + port: 8443 + tls_cert_file: /path/to/cert.pem + tls_key_file: /path/to/key.pem +``` + +```bash +evalhub server start +# Server started (PID 12345). +# URL: https://localhost:8443 +# Logs: ~/.config/evalhub/server/server.log +``` + +## I want to manage the MCP server + + + +The `evalhub mcp` commands manage the `evalhub-mcp` Go binary. Like the server commands, the binary must be on your `PATH`. + +```bash +evalhub mcp run # run in foreground (defaults to stdio transport) +evalhub mcp start # start as background daemon (defaults to http transport) +evalhub mcp stop # stop the background daemon +evalhub mcp status # check if running (reports server name, version, and git hash) +``` + +MCP configuration is managed through `evalhub config set mcp_config_file `. See the [MCP documentation](/mcp/installation/) for full configuration and usage details. + ## Using in CI/CD The CLI is designed for scripted use. All commands return standard exit codes (0 on success, non-zero on failure), and every command that produces data supports `--format json` for machine-readable output. @@ -426,9 +560,20 @@ ANSI colour codes are stripped automatically when stdout is not a TTY, so piped | **providers** | | | `evalhub providers list` | List registered providers | | `evalhub providers describe ` | Show provider details | +| **server** | | +| `evalhub server run` | Run eval-hub-server in the foreground | +| `evalhub server start` | Start eval-hub-server as a background daemon | +| `evalhub server stop` | Stop the background server | +| `evalhub server status` | Check server status and health | +| **mcp** | | +| `evalhub mcp run` | Run MCP server in the foreground | +| `evalhub mcp start` | Start MCP server as a background daemon | +| `evalhub mcp stop` | Stop the background MCP server | +| `evalhub mcp status` | Check MCP server status | | **config** | | | `evalhub config set ` | Set a config value | | `evalhub config get ` | Read a config value | +| `evalhub config unset ` | Remove a config value | | `evalhub config list` | Show active profile | | `evalhub config use ` | Switch profile | diff --git a/src/content/docs/mcp/installation.mdx b/src/content/docs/mcp/installation.mdx index 8a7d70e..17df6af 100644 --- a/src/content/docs/mcp/installation.mdx +++ b/src/content/docs/mcp/installation.mdx @@ -96,26 +96,52 @@ evalhub-mcp --version ``` - + - + -```text +If you already use the EvalHub Python SDK or CLI, the MCP server lifecycle is managed through the `evalhub mcp` commands: + +```bash +pip install "eval-hub-sdk[mcp]" ``` -If you already use the EvalHub Python SDK or CLI, the MCP server is included: +Run the MCP server in the foreground (defaults to `stdio` transport): ```bash -pip install "eval-hub-sdk[mcp]" +evalhub mcp run ``` -Run the MCP server: +Or start it as a background daemon (defaults to `http` transport): ```bash -evalhub mcp +evalhub mcp start +# MCP server started (PID 12345). +# Transport: http +# URL: http://localhost:3001 +# Logs: ~/.config/evalhub/mcp/mcp.log ``` -The Python SDK MCP server uses stdio transport only and reads connection settings from `~/.config/evalhub/config.yaml` (managed by `evalhub config set`). +Manage the background daemon: + +```bash +evalhub mcp status # check if running (reports server name, version, git hash) +evalhub mcp stop # stop the daemon +``` + +See [CLI-managed configuration](#cli-managed-configuration-evalhub-mcp) below for how to configure the MCP connection through the CLI. + +:::caution[Breaking change in 0.4.3 eval-hub-sdk] +Between versions 0.1.5 and 0.4.3, running `evalhub mcp` started a stdio MCP server directly. This no longer works — `evalhub mcp` is now a command group. Use `evalhub mcp run` instead: + +```bash +# 0.1.5 – 0.4.2 +evalhub mcp + +# 0.4.3 and later +evalhub mcp run +``` +::: @@ -197,6 +223,80 @@ Settings are resolved in this order (highest priority first): The `http` transport includes a health endpoint at `GET /health` returning `{"status":"ok"}`. +## CLI-managed configuration (`evalhub mcp`) — Python SDK CLI only + + + +This section applies when managing the Go MCP binary through the Python SDK CLI (`evalhub mcp` commands). If you run the `evalhub-mcp` binary directly, use the [configuration file](#configuration-file), [environment variables](#environment-variables), or [CLI flags](#cli-flags) above instead. + +Configuration is managed with a file-based config key instead of individual profile keys. + +### Set the MCP config file + +Provide a YAML config file with MCP-specific settings. The CLI validates the file and stores a profile-scoped copy: + +```bash +evalhub config set mcp_config_file mcp-config.yaml +``` + +Example `mcp-config.yaml`: + +```yaml +transport: http +host: localhost +port: 3001 +``` + +### Config merging + +The CLI merges connection keys from the active profile (`base_url`, `token`, `tenant`, `insecure`) with the MCP config file. MCP config values take precedence. The merged result is written to `~/.config/evalhub/mcp//mcp-config.yaml` and passed to the `evalhub-mcp` binary automatically. + +Resolution priority (highest first): + +1. Values in `mcp_config_file` +2. Root profile values (`base_url`, `token`, `tenant`, `insecure`) + +If neither source provides a value, the key is omitted. + +### Inspect the config + +```bash +evalhub config get mcp_config_file # show stored path +evalhub config get mcp_config_file --unfold # show file contents (tokens masked) +evalhub config get mcp_config_file --unfold --unmask # show file contents with tokens revealed +``` + +### Remove the MCP config + +```bash +evalhub config unset mcp_config_file +``` + +This removes the stored config file and cleans up the profile directory. + +### Default transports + +| Command | Default transport | Use case | +|---|---|---| +| `evalhub mcp run` | `stdio` | Local clients (Claude Code, VS Code) | +| `evalhub mcp start` | `http` | Background daemon for remote clients | + +`evalhub mcp start` rejects `stdio` transport — use `evalhub mcp run` for stdio. + +### Checking MCP server status + +`evalhub mcp status` reports whether the background MCP server is running. If running, it performs a full MCP JSON-RPC handshake to report the server name, version, and git commit hash: + +```bash +evalhub mcp status +# MCP server is running (PID 12345). +# Name: evalhub-mcp +# Version: 0.4.3 +# Commit: abc1234 +# URL: http://localhost:3001 +# Logs: ~/.config/evalhub/mcp/mcp.log +``` + ## OpenShift multi-tenant setup When using EvalHub in a multi-tenant OpenShift deployment, create a dedicated ServiceAccount for your AI agent: From 13af338d624eae094264bdc43c3c584b75da1303 Mon Sep 17 00:00:00 2001 From: Gin Biak Date: Wed, 8 Jul 2026 20:56:37 +0530 Subject: [PATCH 2/5] docs(local-mode): update guides to use evalhub CLI for server management Replace eval-hub-server --local --configdir with evalhub server start/stop workflow. Add inline server config YAML, BYOF provider template with MLFLOW_TRACKING_URI, sample job.yaml, and OCI credentials guidance. Remove deprecated --local flag references, job.json schema, and environment variables section. Co-Authored-By: Claude --- .../docs/guides/local-mode-tutorial.mdx | 41 +++-- src/content/docs/guides/local-mode.mdx | 168 +++++++++++------- 2 files changed, 125 insertions(+), 84 deletions(-) diff --git a/src/content/docs/guides/local-mode-tutorial.mdx b/src/content/docs/guides/local-mode-tutorial.mdx index 7df917b..f3b6d55 100644 --- a/src/content/docs/guides/local-mode-tutorial.mdx +++ b/src/content/docs/guides/local-mode-tutorial.mdx @@ -32,7 +32,7 @@ Install the following tools before starting: 2. **Install dependencies** - Install eval-hub-server and the evalhub CLI: + Install the EvalHub server and CLI: ```bash uv pip install "eval-hub-sdk[server,cli]" @@ -118,29 +118,40 @@ Install the following tools before starting: }' ``` -7. **Start eval-hub-server** +7. **Start the EvalHub server** - Download the template `config.yaml` from the eval-hub repository: + Create a server configuration file `my-config.yaml`: - ```bash - mkdir -p config - curl -o config/config.yaml https://raw.githubusercontent.com/eval-hub/eval-hub/main/config/config.yaml - ``` + ```yaml + service: + port: 8080 - In another terminal, start the server: + database: + driver: sqlite + url: file::eval_hub:?mode=memory&cache=shared - ```bash - eval-hub-server --local --configdir ./config + mlflow: + tracking_uri: http://localhost:5000 ``` - If MLflow is running, pass the tracking URI: + If you skipped the MLflow step, omit the `mlflow` section. + + Set the configuration and start the server: ```bash - MLFLOW_TRACKING_URI=http://localhost:5000 \ - eval-hub-server --local --configdir ./config + evalhub config set server_config_file my-config.yaml + evalhub server start ``` - Verify from another terminal: + ```text + Server started (PID 12345). + URL: http://localhost:8080 + Logs: ~/.config/evalhub/server/server.log + ``` + + To stop the server later, run `evalhub server stop`. + + Verify the server is healthy: ```bash evalhub config set base_url http://localhost:8080 @@ -330,7 +341,7 @@ After completing setup, you have four services on localhost: | Service | URL | Purpose | |---|---|---| -| **eval-hub-server** | `http://localhost:8080` | Evaluation orchestration | +| **EvalHub server** (`evalhub server`) | `http://localhost:8080` | Evaluation orchestration | | **MLflow** | `http://localhost:5000` | Experiment tracking dashboard | | **OCI registry** | `http://localhost:5001` | Artifact storage | | **Ollama** | `http://localhost:11434` | LLM inference | diff --git a/src/content/docs/guides/local-mode.mdx b/src/content/docs/guides/local-mode.mdx index 2f6349f..3d3b94b 100644 --- a/src/content/docs/guides/local-mode.mdx +++ b/src/content/docs/guides/local-mode.mdx @@ -4,9 +4,9 @@ title: "Local Mode" import Since from '../../../components/Since.astro'; - + -Local mode runs the full EvalHub evaluation pipeline on your workstation without a Kubernetes cluster. Activate it with the `--local` flag. The REST API is identical to cluster mode — the same endpoints, request bodies, and response schemas apply. +Local mode runs the full EvalHub evaluation pipeline on your workstation without a Kubernetes cluster. The REST API is identical to cluster mode — the same endpoints, request bodies, and response schemas apply. Local mode is useful for: @@ -19,31 +19,58 @@ For a hands-on walkthrough, see the [Local Mode Tutorial](/guides/local-mode-tut ## Starting the Server in Local Mode +Install the SDK with the `server` and `cli` extras: + ```bash -pip install eval-hub-server # Standalone -# or -pip install eval-hub-sdk[server] # Via the SDK extra -eval-hub-server --version +pip install "eval-hub-sdk[server,cli]" ``` -The server requires a `--configdir` pointing to a directory containing a server `config.yaml` (see the [Local Mode Tutorial](/guides/local-mode-tutorial/) for a step-by-step setup). The `--local` flag disables authentication and enables CORS. The server starts at `http://localhost:8080` with SQLite in-memory storage. +Create a server configuration file (e.g. `my-config.yaml`): -```bash -eval-hub-server --local --configdir ./config +```yaml +service: + port: 8080 + +database: + driver: sqlite + url: file::eval_hub:?mode=memory&cache=shared + +mlflow: + tracking_uri: http://localhost:5000 ``` -**Custom port:** +Set the configuration and start the server: ```bash -PORT=8090 eval-hub-server --local --configdir ./config +evalhub config set server_config_file my-config.yaml +evalhub server start +``` + +```text +Server started (PID 12345). + URL: http://localhost:8080 + Logs: ~/.config/evalhub/server/server.log ``` -**With MLflow tracking:** +Stop the server when you're done: ```bash -MLFLOW_TRACKING_URI=http://localhost:5000 eval-hub-server --local --configdir ./config +evalhub server stop ``` +The server runs evaluation jobs as subprocesses in local mode. Authentication is disabled and CORS is enabled automatically. + +:::note[MLflow tracking URI] +The MLflow tracking URI appears in two places with distinct purposes: + +- **Server config YAML** (`mlflow.tracking_uri`) — tells the EvalHub server where MLflow is, so it can create experiments and link evaluation results to MLflow runs. +- **Provider YAML** (`runtime.local.env`) — passed to the adapter subprocess as the `MLFLOW_TRACKING_URI` environment variable, so the adapter can log metrics and artifacts to MLflow directly. + +Both should point to the same MLflow instance. +::: + +For full details on `evalhub server` commands, see the [CLI guide — "I want to manage the local server"](/guides/cli/#i-want-to-manage-the-local-server). + ## Differences from Cluster Mode | Aspect | Cluster mode | Local mode | @@ -70,7 +97,7 @@ When an evaluation job is submitted in local mode, for each benchmark the server ```mermaid flowchart TD - A["REST API\nPOST /api/v1/evaluations/jobs"] --> B["eval-hub-server\n(--local flag)"] + A["REST API\nPOST /api/v1/evaluations/jobs"] --> B["evalhub server"] B --> C["Write job.json"] C --> D["Spawn adapter process"] D --> E["Adapter Process\n(e.g. python main.py)"] @@ -95,86 +122,83 @@ flowchart TD └── jobrun.log # Stdout/stderr from the adapter process ``` -### Job specification (job.json) - -The job specification is the same structure used in cluster mode (where it is mounted into the container as a ConfigMap). It contains all the information the adapter needs to run the benchmark: - -```json -{ - "id": "", - "provider_id": "", - "benchmark_id": "", - "benchmark_index": 0, - "model": { - "url": "http://localhost:11434/v1", - "name": "llama3.2:3b-instruct-q4_K_M" - }, - "num_examples": 10, - "parameters": {}, - "experiment_name": "my-experiment", - "tags": [ - { "key": "model", "value": "llama3.2:3b-instruct-q4_K_M" } - ], - "callback_url": "http://localhost:8080", - "exports": { - "oci": { - "coordinates": { - "oci_host": "localhost:5001", - "oci_repository": "eval-results" - } - } - } -} -``` - -:::note[OCI registry credentials] -In local mode, the OCI persister reads credentials from `~/.docker/config.json`. If your OCI registry requires authentication, run [`docker login`](https://docs.docker.com/reference/cli/docker/login/) (or `podman login`) before starting an evaluation. For local registries without authentication, set `OCI_INSECURE=true` in the provider's `runtime.local.env` instead. -::: - ## Provider Configuration for Local Mode Each provider must have a `runtime.local` section specifying the adapter command and optional environment variables. The `runtime.local.command` is executed via `sh -c ""`. ```yaml -id: my-provider -name: My Evaluation Provider -description: Custom evaluation framework adapter +name: my-provider +title: My Provider +description: My BYOF runtime: local: command: "python main.py" env: + - name: MLFLOW_TRACKING_URI + value: http://localhost:5000 - name: OCI_INSECURE value: "true" benchmarks: - - id: my_benchmark - name: My benchmark - description: Description of what this benchmark measures - category: reasoning + - id: my-benchmark + name: My Benchmark + description: |- + Mock evaluation benchmark with accuracy and exact_match metrics. + category: general metrics: - - acc + - accuracy + - exact_match + num_few_shot: 0 + dataset_size: 500 + tags: + - general + - custom + - byof primary_score: - metric: acc + metric: accuracy lower_is_better: false pass_criteria: threshold: 0.25 ``` +The `OCI_INSECURE` variable allows pushing artifacts to a local OCI registry running without TLS. For registries that require authentication, the OCI persister reads credentials from `~/.docker/config.json` — run `docker login` (or `podman login`) before starting an evaluation. + A provider configuration can include both `runtime.local` and `runtime.k8s` sections, allowing the same definition to work in both modes. :::tip[Registering providers] -Providers can be registered using the `evalhub` CLI (`evalhub providers create --file .yaml`) or via the [REST API](/reference/server-api/) (`POST /api/v1/evaluations/providers`) after the server is running. See the [Local Mode Tutorial](/guides/local-mode-tutorial/) for a complete example. +Register providers with `evalhub providers create --file .yaml` after the server is running. See the [Local Mode Tutorial](/guides/local-mode-tutorial/) for a complete example. ::: -### Environment variables +## Running an Evaluation -The adapter process receives the following environment variables: +Once the server is running and a provider is registered, submit a job using a config file or CLI flags. Here is a sample `job.yaml`: -| Variable | Description | -|---|---| -| `EVALHUB_JOB_SPEC_PATH` | Absolute path to the `job.json` file | -| Custom env vars from `runtime.local.env` | Any additional variables defined in the provider config | +```yaml +name: my-job +model: + url: http://localhost:11434/v1 + name: llama3.2:3b-instruct-q4_K_M +benchmarks: + - id: my-benchmark + provider_id: + parameters: + num_examples: 10 + num_few_shot: 0 +experiment: + name: my-experiment +exports: + oci: + coordinates: + oci_host: localhost:5001 + oci_repository: myorg/eval-results +``` -See [System Overview — AdapterSettings](/architecture/system-overview/#adaptersettings) for the complete list of adapter environment variables. +Replace `` with the ID returned by `evalhub providers create`. The `experiment` and `exports` sections are optional — omit them if you are not using MLflow or an OCI registry. + +```bash +evalhub eval run --config job.yaml --wait +``` + +See the [CLI guide](/guides/cli/#i-want-to-run-an-evaluation) for the full set of `eval run` options, including inline flags and output formats. ## Writing Adapters for Both Modes @@ -201,7 +225,13 @@ cat /tmp/evalhub-jobs/////jo ### Server logs -The server logs structured JSON to stderr. Look for `local runtime` messages: +The server log file is managed by `evalhub server` and stored at `~/.config/evalhub/server/server.log`. Check its location with: + +```bash +evalhub server status +``` + +Look for `local runtime` messages in the log: - `local runtime job spec written` — job spec was created successfully - `local runtime process started` — adapter process was launched with the logged PID and command @@ -214,5 +244,5 @@ The server logs structured JSON to stderr. Look for `local runtime` messages: | Job fails immediately | Adapter command not found | Verify `runtime.local.command` path and that dependencies are installed | | Job stays in `running` state | Adapter is not reporting back | Check the adapter logs in `jobrun.log`; verify the callback URL is reachable | | `provider has no local runtime configured` | Missing `runtime.local` in provider YAML | Add a `runtime.local.command` to the provider configuration | -| MLflow experiment not created | MLflow not configured | Set `MLFLOW_TRACKING_URI` or `mlflow.tracking_uri` in `config.yaml` | +| MLflow experiment not created | MLflow not configured | Set `mlflow.tracking_uri` in the server config YAML (`server_config_file`) | | OCI push fails | Registry not reachable or requires auth | Verify the registry is running; run `docker login`/`podman login` for authenticated registries or set `OCI_INSECURE=true` for local ones | From 396eea7b408f8d26d95340da81b5ce9aefc89f9a Mon Sep 17 00:00:00 2001 From: Gin Biak Date: Wed, 8 Jul 2026 21:20:29 +0530 Subject: [PATCH 3/5] docs(local-mode): refine tutorial steps and update version tags Bump Since tags to 0.4.3 eval-hub-sdk, reorder tutorial steps to download adapter before installing dependencies, pin SDK to >=0.4.3, add MLFLOW_TRACKING_URI to tutorial provider YAML, update cluster mode multi-tenancy description, and clean up next steps. Co-Authored-By: Claude --- .../docs/guides/local-mode-tutorial.mdx | 29 +++++++++---------- src/content/docs/guides/local-mode.mdx | 2 +- 2 files changed, 15 insertions(+), 16 deletions(-) diff --git a/src/content/docs/guides/local-mode-tutorial.mdx b/src/content/docs/guides/local-mode-tutorial.mdx index f3b6d55..9dc7eab 100644 --- a/src/content/docs/guides/local-mode-tutorial.mdx +++ b/src/content/docs/guides/local-mode-tutorial.mdx @@ -5,7 +5,7 @@ title: "Local Mode Tutorial" import { Steps } from '@astrojs/starlight/components'; import Since from '../../../components/Since.astro'; - + End-to-end walkthrough: run a LightEval evaluation locally with EvalHub, MLflow experiment tracking, and OCI artifact storage — no Kubernetes required. @@ -30,26 +30,26 @@ Install the following tools before starting: uv venv --python 3.12 ``` -2. **Install dependencies** - - Install the EvalHub server and CLI: - - ```bash - uv pip install "eval-hub-sdk[server,cli]" - ``` - :::tip - Whenever the tutorial asks you to open or use another terminal, activate the virtual environment first with `source .venv/bin/activate` — even when not explicitly mentioned. + Whenever the tutorial asks you to open or use another terminal, activate the virtual environment first with `source .venv/bin/activate`, or prefix commands with `uv run` if not activating the venv — even when not explicitly mentioned. ::: -3. **Download the LightEval adapter** +2. **Download the LightEval adapter** - Download the adapter driver and its requirements from eval-hub-contrib, then install them: + Download the adapter driver and its requirements from eval-hub-contrib: ```bash curl -o main.py https://raw.githubusercontent.com/eval-hub/eval-hub-contrib/main/adapters/lighteval/main.py curl -o requirements.txt https://raw.githubusercontent.com/eval-hub/eval-hub-contrib/main/adapters/lighteval/requirements.txt + ``` + +3. **Install dependencies** + + Install the EvalHub server, CLI, and adapter requirements: + + ```bash uv pip install -r requirements.txt + uv pip install "eval-hub-sdk[server,cli]>=0.4.3" ``` 4. **Start MLflow (optional)** @@ -149,8 +149,6 @@ Install the following tools before starting: Logs: ~/.config/evalhub/server/server.log ``` - To stop the server later, run `evalhub server stop`. - Verify the server is healthy: ```bash @@ -174,6 +172,8 @@ Install the following tools before starting: local: command: "python main.py" env: + - name: MLFLOW_TRACKING_URI + value: http://localhost:5000 - name: OCI_INSECURE value: "true" benchmarks: @@ -350,4 +350,3 @@ After completing setup, you have four services on localhost: - Browse the MLflow UI at `http://localhost:5000` to see experiment metrics - Read the [Local Mode guide](/guides/local-mode/) for provider configuration details and troubleshooting -- Try other adapters by registering additional providers with `evalhub providers create --file .yaml` diff --git a/src/content/docs/guides/local-mode.mdx b/src/content/docs/guides/local-mode.mdx index 3d3b94b..919d65c 100644 --- a/src/content/docs/guides/local-mode.mdx +++ b/src/content/docs/guides/local-mode.mdx @@ -77,7 +77,7 @@ For full details on `evalhub server` commands, see the [CLI guide — "I want to |---|---|---| | **Job execution** | Kubernetes Jobs (containers) | Host subprocesses (`sh -c ""`) | | **Authentication** | Enabled (configurable) | Disabled automatically | -| **Multi-tenancy** | Tenant isolation via `X-Tenant` header | Single-tenant only | +| **Multi-tenancy** | Single-tenant or multi-tenant (`X-Tenant` header) | Single-tenant only | | **CORS** | Disabled by default | Enabled | | **Sidecar proxy** | Injected into job pods | Not used; adapters call services directly | | **Init container** | Downloads test data to `/test_data` | Not used | From 915e22ce552b0cda6ec9e8d4483f8ba7f3e87a72 Mon Sep 17 00:00:00 2001 From: Gin Biak Date: Thu, 9 Jul 2026 15:18:23 +0530 Subject: [PATCH 4/5] docs(local-mode): improve tutorial with shell variables and real CLI output Replace cat heredoc with file-creation instruction for consistency, add PROVIDER_ID/BENCHMARK_ID/JOB_ID shell variables from CLI output to use in subsequent commands, and update example results with actual evalhub output. Pin minimum SDK version in local-mode guide. Co-Authored-By: Claude --- .../docs/guides/local-mode-tutorial.mdx | 85 +++++++++++++------ src/content/docs/guides/local-mode.mdx | 2 +- 2 files changed, 61 insertions(+), 26 deletions(-) diff --git a/src/content/docs/guides/local-mode-tutorial.mdx b/src/content/docs/guides/local-mode-tutorial.mdx index 9dc7eab..0e348fa 100644 --- a/src/content/docs/guides/local-mode-tutorial.mdx +++ b/src/content/docs/guides/local-mode-tutorial.mdx @@ -162,10 +162,9 @@ Install the following tools before starting: 8. **Register the provider** - Create a provider definition file and register it with the CLI: + Create a provider definition file `lighteval.yaml`: - ```bash - cat > lighteval.yaml << 'EOF' + ```yaml name: lighteval description: LightEval adapter for evaluation framework runtime: @@ -197,11 +196,24 @@ Install the following tools before starting: lower_is_better: false pass_criteria: threshold: 0.25 - EOF + ``` + Register it with the CLI: + + ```bash evalhub providers create --file lighteval.yaml ``` + ```text + Provider created: a578921e-053a-496a-889b-ecd97b8dbd1a + ``` + + Save the provider ID for subsequent commands (your ID will differ): + + ```bash + PROVIDER_ID="a578921e-053a-496a-889b-ecd97b8dbd1a" + ``` + See the [Local Mode guide — Provider Configuration](/guides/local-mode/#provider-configuration-for-local-mode) for details on the `runtime.local` section. Verify the provider was registered: @@ -214,19 +226,19 @@ Install the following tools before starting: ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓ ┃ ID ┃ NAME ┃ DESCRIPTION ┃ BENCHMARKS ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩ - │ 11031769-2ee5-4433-a129-8c6ad84d7185 │ lighteval │ LightEval adapter for evaluation framework │ 1 │ + │ a578921e-053a-496a-889b-ecd97b8dbd1a │ lighteval │ LightEval adapter for evaluation framework │ 1 │ └──────────────────────────────────────┴───────────┴────────────────────────────────────────────┴────────────┘ ``` - Use the provider ID from the output above to query its available benchmarks (your ID will differ): + Query the provider's available benchmarks: ```bash - evalhub providers describe + evalhub providers describe $PROVIDER_ID ``` ```text Provider: lighteval - ID: 11031769-2ee5-4433-a129-8c6ad84d7185 + ID: a578921e-053a-496a-889b-ecd97b8dbd1a Description: LightEval adapter for evaluation framework Benchmarks (1): @@ -237,7 +249,11 @@ Install the following tools before starting: └───────┴─────────────────────────────────┴──────────┴──────────────────┘ ``` - The benchmark ID `gsm8k` is defined in the provider YAML above. Use the provider ID and benchmark ID from this output as the `--provider` and `--benchmark` values when running evaluations. + Save the benchmark ID for subsequent commands: + + ```bash + BENCHMARK_ID="gsm8k" + ``` :::note Providers can also be registered via the [REST API](/reference/server-api/) (`POST /api/v1/evaluations/providers`). @@ -254,17 +270,27 @@ evalhub eval run \ --name my-eval-job \ --model-url http://localhost:11434/v1 \ --model-name llama3.2:3b-instruct-q4_K_M \ - --provider \ - --benchmark \ + --provider $PROVIDER_ID \ + --benchmark $BENCHMARK_ID \ --param num_examples=10 \ --param num_few_shot=0 \ --wait ``` -Check results: +Save the job ID from the output and check results: ```bash -evalhub eval results +JOB_ID="" +evalhub eval results $JOB_ID +``` + +```text +┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┓ +┃ BENCHMARK ┃ PROVIDER ┃ METRIC ┃ VALUE ┃ +┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━┩ +│ gsm8k │ a578921e-053a-496a-889b-ecd97b8dbd1a │ all.extractive_match │ 0.7 │ +│ gsm8k │ a578921e-053a-496a-889b-ecd97b8dbd1a │ gsm8k.extractive_match │ 0.7 │ +└───────────┴──────────────────────────────────────┴────────────────────────┴───────┘ ``` ### With MLflow experiments and OCI artifact storage @@ -276,8 +302,8 @@ evalhub eval run \ --name my-eval-job \ --model-url http://localhost:11434/v1 \ --model-name llama3.2:3b-instruct-q4_K_M \ - --provider \ - --benchmark \ + --provider $PROVIDER_ID \ + --benchmark $BENCHMARK_ID \ --param num_examples=10 \ --param num_few_shot=0 \ --experiment my-local-experiment \ @@ -286,8 +312,17 @@ evalhub eval run \ --wait ``` +```text +Job submitted: ef42ad16-57b6-4bf4-a7ff-eb907e848bb1 +Waiting for job ef42ad16-57b6-4bf4-a7ff-eb907e848bb1 to complete... +Job ef42ad16-57b6-4bf4-a7ff-eb907e848bb1 finished with state: completed +``` + +Save the job ID and check results: + ```bash -evalhub eval results --format json +JOB_ID="ef42ad16-57b6-4bf4-a7ff-eb907e848bb1" +evalhub eval results $JOB_ID --format json ``` Example output: @@ -296,11 +331,11 @@ Example output: [ { "id": "gsm8k", - "provider_id": "11031769-2ee5-4433-a129-8c6ad84d7185", + "provider_id": "a578921e-053a-496a-889b-ecd97b8dbd1a", "benchmark_index": 0, "metrics": { "all.extractive_match": 0.7, - "gsm8k|0.extractive_match": 0.7 + "gsm8k.extractive_match": 0.7 }, "artifacts": { "evalhub.env_card": { @@ -313,19 +348,19 @@ Example output: "k8s_pod_labels": {}, "k8s_resource_limits": {}, "key_packages": { - "mlflow": "3.12.0", - "torch": "2.12.0", - "transformers": "5.9.0" + "mlflow": "3.14.0", + "torch": "2.13.0", + "transformers": "5.13.0" }, - "os_info": "macOS-26.5-arm64-arm-64bit", + "os_info": "macOS-26.5.2-arm64-arm-64bit", "per_task_results": {}, "python_version": "3.12.12", "scorer_ids": [] }, - "oci_digest": "sha256:31f4da6f8aca49f50d97da92775eef691875701fae0bb81cf72c5b60fb7f9f20", - "oci_reference": "localhost:5001/local-eval-results:evalhub-bdf6096c95aff921cc5067a2c679829db76541f07ee8d607211666129bbcfef7@sha256:31f4da6f8aca49f50d97da92775eef691875701fae0bb81cf72c5b60fb7f9f20" + "oci_digest": "sha256:5f853d7fa2c67247ca35b511deaec78a56d64c98734974e274504c039bedc558", + "oci_reference": "localhost:5001/local-eval-results:evalhub-9129f2cac2db213158f8c106e1b658e0e05ecd9db54630dfe6bc2edf2cdb413b@sha256:5f853d7fa2c67247ca35b511deaec78a56d64c98734974e274504c039bedc558" }, - "mlflow_run_id": "0a3505d2c1d24025b13f05d4ec96cb01", + "mlflow_run_id": "3c03bb8128a349fe822e793637b7ffe2", "logs_path": null } ] diff --git a/src/content/docs/guides/local-mode.mdx b/src/content/docs/guides/local-mode.mdx index 919d65c..b9aac81 100644 --- a/src/content/docs/guides/local-mode.mdx +++ b/src/content/docs/guides/local-mode.mdx @@ -22,7 +22,7 @@ For a hands-on walkthrough, see the [Local Mode Tutorial](/guides/local-mode-tut Install the SDK with the `server` and `cli` extras: ```bash -pip install "eval-hub-sdk[server,cli]" +pip install "eval-hub-sdk[server,cli]>=0.4.3" ``` Create a server configuration file (e.g. `my-config.yaml`): From 62fecf568e6f794581e6965a33beaee36a5a7209 Mon Sep 17 00:00:00 2001 From: Gin Biak Date: Thu, 9 Jul 2026 15:38:11 +0530 Subject: [PATCH 5/5] docs(cli): clarify --unfold and --unmask flag interaction MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Remove contradictory "mutually exclusive" language — the flags can be used independently or combined. Co-Authored-By: Claude --- src/content/docs/guides/cli.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/content/docs/guides/cli.mdx b/src/content/docs/guides/cli.mdx index f38b7ad..ec55ea0 100644 --- a/src/content/docs/guides/cli.mdx +++ b/src/content/docs/guides/cli.mdx @@ -76,7 +76,7 @@ Sensitive values (such as tokens) are masked by default. Combine with `--unmask` evalhub config get server_config_file --unfold --unmask ``` -`--unfold` and `--unmask` are mutually exclusive unless used together — `--unmask` alone shows the raw path value without masking, while `--unfold` alone shows the file contents with sensitive values masked. +`--unmask` alone shows the raw path value without masking, while `--unfold` alone shows the file contents with sensitive values masked. Both flags can be combined to show unmasked file contents. ## I want to see what providers and benchmarks are available