diff --git a/astro.config.mjs b/astro.config.mjs index 3afe63f..b0de2bb 100644 --- a/astro.config.mjs +++ b/astro.config.mjs @@ -107,7 +107,15 @@ export default defineConfig({ label: 'Architecture', items: [ { label: 'System Overview', slug: 'architecture/system-overview' }, - { label: 'Multi-Tenancy', slug: 'architecture/multi-tenancy' }, + { + label: 'Tenancy', + collapsed: false, + items: [ + { label: 'Overview', slug: 'architecture/tenancy' }, + { label: 'Multi-Tenancy', slug: 'architecture/multi-tenancy' }, + { label: 'Single-Tenancy', slug: 'architecture/single-tenancy' }, + ], + }, ], }, { diff --git a/src/content/docs/architecture/multi-tenancy.mdx b/src/content/docs/architecture/multi-tenancy.mdx index 4370bd8..8ceae79 100644 --- a/src/content/docs/architecture/multi-tenancy.mdx +++ b/src/content/docs/architecture/multi-tenancy.mdx @@ -4,47 +4,9 @@ title: "Multi-Tenancy" import { Steps, Tabs, TabItem } from '@astrojs/starlight/components'; -Guide for deploying EvalHub with namespace-based multi-tenancy on OpenShift. +Step-by-step guide for deploying EvalHub with `spec.tenancy: multi` (the default) on OpenShift. One EvalHub instance in a control-plane namespace serves multiple tenant namespaces. -## Overview - -EvalHub uses Kubernetes namespaces as tenant boundaries. Each tenant operates in its own namespace, and access is enforced through Kubernetes RBAC via SubjectAccessReview (SAR) checks. - -The core principle is: **namespace = tenant = MLFlow workspace**. - -```mermaid -flowchart LR - subgraph "Control Plane (opendatahub)" - Operator[TrustyAI Operator] - EvalHub[EvalHub API] - end - - subgraph "Tenant A (team-a)" - SA_A[Job SA] - Job_A[Eval Job] - end - - subgraph "Tenant B (team-b)" - SA_B[Job SA] - Job_B[Eval Job] - end - - Operator -->|creates RBAC| SA_A - Operator -->|creates RBAC| SA_B - EvalHub -->|schedules jobs| Job_A - EvalHub -->|schedules jobs| Job_B -``` - -### How tenant isolation works - -1. **Authentication** -- Bearer tokens are validated via the Kubernetes TokenReview API. The token can belong to a ServiceAccount, an OpenShift User, or a member of an OpenShift Group — EvalHub treats all identically at this stage. -2. **Authorisation** -- Every API request includes an `X-Tenant` header specifying the target namespace. EvalHub runs a SAR check: _"can this principal perform this action in that namespace?"_ -3. **Data isolation** -- All database queries are filtered by `tenant_id` -4. **Job isolation** -- Evaluation jobs run in the tenant namespace with a scoped ServiceAccount - -:::note[Tenant ID and user identity are independent] -The tenant is always determined by the `X-Tenant` request header — **not** by the identity in the token. A ServiceAccount has a home namespace, but EvalHub does not use it. A real OpenShift User or Group member has no namespace at all. In all cases the caller must supply `X-Tenant` explicitly, and a matching RoleBinding (for the User, ServiceAccount, or Group) must exist in that namespace. -::: +For shared concepts — tenant isolation, SAR authorisation, ClusterRoles, and common troubleshooting — see the [Tenancy overview](/architecture/tenancy/). ## Prerequisites @@ -60,7 +22,7 @@ Before setting up multi-tenancy, ensure you have: 1. **Deploy EvalHub** - Create an EvalHub instance in the control-plane namespace. For multi-tenancy, a persistent database is recommended: + Create an EvalHub instance in the control-plane namespace. `spec.tenancy: multi` is the default and can be omitted. For multi-tenancy, a persistent database is recommended: ```yaml apiVersion: trustyai.opendatahub.io/v1alpha1 @@ -295,10 +257,6 @@ Before setting up multi-tenancy, ensure you have: - :::note[Virtual resources] - The resources in the Role (`evaluations`, `collections`, `providers`, `status-events`) are virtual -- they don't correspond to actual Kubernetes API resources. EvalHub uses them as SAR targets to enforce fine-grained access control via the Kubernetes authorisation API. - ::: - 4. **Access the API with tenant scoping** All evaluation API requests must include the `X-Tenant` header set to the target namespace. The header is the sole source of the tenant ID — it is not derived from the token. @@ -488,121 +446,7 @@ Before setting up multi-tenancy, ensure you have: -## Authorisation model - -EvalHub uses an embedded SAR authoriser. The auth config (`config/auth.yaml`) maps API endpoints to Kubernetes resource attributes: - -| Endpoint | Resource | Verb | Namespace source | -|----------|----------|------|------------------| -| `POST /api/v1/evaluations/jobs` | `evaluations` | `create` | `X-Tenant` header | -| `GET /api/v1/evaluations/jobs` | `evaluations` | `get` | `X-Tenant` header | -| `POST /api/v1/evaluations/jobs/*/events` | `status-events` | `create` | `X-Tenant` header | -| `* /api/v1/evaluations/collections` | `collections` | _(from HTTP method)_ | `X-Tenant` header | -| `* /api/v1/evaluations/providers` | `providers` | _(from HTTP method)_ | `X-Tenant` header | - -For `POST /api/v1/evaluations/jobs`, two additional SAR checks are performed for MLFlow access (`mlflow.kubeflow.org/experiments` with `create` and `get` verbs). - -### Request flow - -The flow is identical for both principal types. The only difference is what the TokenReview returns and which subject kind is matched in the RoleBinding. - - - - -```mermaid -sequenceDiagram - participant User as Tenant SA - participant EH as EvalHub API - participant K8s as Kubernetes API - participant Job as Eval Job Pod - - User->>EH: POST /jobs (Bearer token + X-Tenant: team-a) - EH->>K8s: TokenReview (validate token) - K8s-->>EH: user=system:serviceaccount:team-a:team-a-user - EH->>K8s: SAR (can SA create evaluations in team-a?) - K8s-->>EH: Allowed (kind:ServiceAccount RoleBinding matched) - EH->>K8s: Create Job in team-a (SA: evalhub-opendatahub-job) - K8s-->>EH: Job created - EH-->>User: 202 Accepted - - Job->>EH: POST /jobs/{id}/events (job SA token + X-Tenant: team-a) - EH->>K8s: SAR (can job SA create status-events in team-a?) - K8s-->>EH: Allowed - EH-->>Job: 200 OK -``` - - - - - -```mermaid -sequenceDiagram - participant User as OpenShift User (alice) - participant EH as EvalHub API - participant K8s as Kubernetes API - participant Job as Eval Job Pod - - User->>EH: POST /jobs (OAuth token + X-Tenant: team-a) - EH->>K8s: TokenReview (validate token) - K8s-->>EH: user=alice - EH->>K8s: SAR (can alice create evaluations in team-a?) - K8s-->>EH: Allowed (kind:User RoleBinding matched) - EH->>K8s: Create Job in team-a (SA: evalhub-opendatahub-job) - K8s-->>EH: Job created - EH-->>User: 202 Accepted - - Job->>EH: POST /jobs/{id}/events (job SA token + X-Tenant: team-a) - EH->>K8s: SAR (can job SA create status-events in team-a?) - K8s-->>EH: Allowed - EH-->>Job: 200 OK -``` - - - - - -```mermaid -sequenceDiagram - participant User as Group Member (alice) - participant EH as EvalHub API - participant K8s as Kubernetes API - participant Job as Eval Job Pod - - User->>EH: POST /jobs (OAuth token + X-Tenant: team-a) - EH->>K8s: TokenReview (validate token) - K8s-->>EH: user=alice, groups=[team-a-evaluators, ...] - EH->>K8s: SAR (can alice create evaluations in team-a?) - K8s-->>EH: Allowed (kind:Group RoleBinding matched via group membership) - EH->>K8s: Create Job in team-a (SA: evalhub-opendatahub-job) - K8s-->>EH: Job created - EH-->>User: 202 Accepted - - Job->>EH: POST /jobs/{id}/events (job SA token + X-Tenant: team-a) - EH->>K8s: SAR (can job SA create status-events in team-a?) - K8s-->>EH: Allowed - EH-->>Job: 200 OK -``` - - - - -Note that the job pod always runs as the operator-provisioned job ServiceAccount (`evalhub-opendatahub-job`), regardless of whether the submitter was a User, Group member, or a ServiceAccount. - -## RBAC reference - -### ClusterRoles (installed by operator) - -| ClusterRole | Purpose | Key permissions | -|-------------|---------|-----------------| -| `evalhub-auth-reviewer-role` | Token and SAR validation | `tokenreviews`, `subjectaccessreviews` | -| `evalhub-jobs-writer` | Create evaluation jobs | `batch/jobs` (create, delete) | -| `evalhub-job-config` | Manage job config | `configmaps` (create, get, update, delete) | -| `evalhub-providers-access` | Providers endpoint SAR | `providers` (get) | -| `evalhub-collections-access` | Collections endpoint SAR | `collections` (get) | -| `evalhub-mlflow-access` | API server MLFlow access | `experiments` (create, get, list, update, delete) | -| `evalhub-mlflow-jobs-access` | Job pod MLFlow access | `experiments` (create, get, list) | - -### Per-tenant resources (operator-provisioned) +## RBAC reference — per-tenant resources For each namespace labelled with `evalhub.trustyai.opendatahub.io/tenant`, the operator creates: @@ -716,66 +560,7 @@ oc get pod -n team-a -l app.kubernetes.io/part-of=eval-hub \ ## Troubleshooting -### 403 Forbidden on API calls - -The SAR check is failing. Verify: - -1. The `X-Tenant` header matches a namespace where the principal has a RoleBinding -2. The Role includes the correct resources and verbs -3. The RoleBinding `subjects[].kind` matches the principal type (`ServiceAccount`, `User`, or `Group`) - - - - -```bash -oc auth can-i create evaluations.trustyai.opendatahub.io \ - -n team-a \ - --as=system:serviceaccount:team-a:team-a-user -v=6 -``` - - - - - -```bash -oc auth can-i create evaluations.trustyai.opendatahub.io \ - -n team-a \ - --as=alice -v=6 -``` - -Also confirm the RoleBinding subject kind is `User` and has no `namespace` field: - -```bash -oc get rolebinding evalhub-evaluator-binding -n team-a -o yaml | grep -A5 subjects -``` - - - - - -```bash -oc auth can-i create evaluations.trustyai.opendatahub.io \ - -n team-a \ - --as=alice --as-group=team-a-evaluators -v=6 -``` - -If the SAR check passes with `--as-group` but real requests still fail: - -1. **Token doesn't include group claims** -- The user must re-login (`oc login`) after being added to the group. Existing OAuth tokens may not include the new group membership. -2. **Verify group membership:** - - ```bash - oc get group team-a-evaluators -o jsonpath='{.users[*]}' - ``` - -3. **Confirm the RoleBinding subject kind is `Group`:** - - ```bash - oc get rolebinding evalhub-evaluator-group-binding -n team-a -o yaml | grep -A5 subjects - ``` - - - +For common issues such as 403 Forbidden responses and missing `X-Tenant` headers, see [Troubleshooting on the Tenancy overview](/architecture/tenancy/#troubleshooting). ### Jobs not created in tenant namespace @@ -794,3 +579,7 @@ oc auth can-i create status-events.trustyai.opendatahub.io \ -n team-a \ --as=system:serviceaccount:team-a:evalhub-opendatahub-job ``` + +--- + +For single-namespace deployments, see [Single-Tenancy](/architecture/single-tenancy/). diff --git a/src/content/docs/architecture/single-tenancy.mdx b/src/content/docs/architecture/single-tenancy.mdx new file mode 100644 index 0000000..842ff6b --- /dev/null +++ b/src/content/docs/architecture/single-tenancy.mdx @@ -0,0 +1,410 @@ +--- +title: "Single-Tenancy" +--- + +import { Steps, Tabs, TabItem } from '@astrojs/starlight/components'; + +Step-by-step guide for deploying EvalHub with `spec.tenancy: single` on OpenShift. EvalHub runs in the same namespace as the workloads it serves — ideal for a single team or project that does not need a shared control-plane instance. + +For shared concepts — tenant isolation, SAR authorisation, ClusterRoles, and common troubleshooting — see the [Tenancy overview](/architecture/tenancy/). + +## Prerequisites + +Before setting up single-tenancy, ensure you have: + +- The TrustyAI Operator installed and reconciling +- Namespace-admin access in the workload namespace (no cluster-admin required for tenant labelling) +- A namespace for your team or project (e.g. `team-a`) + +## Set up + + + +1. **Deploy EvalHub** + + Create an EvalHub instance in the workload namespace: + + ```yaml + apiVersion: trustyai.opendatahub.io/v1alpha1 + kind: EvalHub + metadata: + name: evalhub + namespace: team-a + spec: + tenancy: single + replicas: 1 + database: + type: sqlite + providers: + - lm-evaluation-harness + - garak + - garak-kfp + - guidellm + - lighteval + - ibm-clear + collections: + - leaderboard-v2 + - safety-and-fairness-v1 + - toxicity-and-ethical-principles + ``` + + For production workloads, use PostgreSQL instead of SQLite: + + ```yaml + database: + type: postgresql + secret: evalhub-db-credentials + ``` + + The operator automatically creates in the instance namespace: + + | Resource | Purpose | + |----------|---------| + | `evalhub-service` SA | API server identity | + | `evalhub-team-a-job` SA | Identity for evaluation job pods | + | ClusterRole RoleBindings | jobs-writer, job-config, auth-reviewer, MLFlow | + | `evalhub-tenant-admin` Role | Full CRUD on virtual EvalHub resources; MLflow read-only (get, list) | + | `evalhub-user` Role | Read providers/collections, submit via `evalhubs/proxy`, create status-events, MLflow create/get | + | `evalhub-tenant-admin-binding` | Binds admin Role to `system:serviceaccounts:team-a` | + + :::note[Default admin access and job submission] + The operator binds **all** ServiceAccounts in the namespace to `evalhub-tenant-admin`. That Role grants `evaluations` create but MLflow `experiments` are read-only. `POST /api/v1/evaluations/jobs` also requires MLflow `create` and `get` — bind principals to `evalhub-user` for those permissions (RBAC unions across bindings). For restricted access without admin CRUD, bind `evalhub-user` only. + ::: + +2. **Grant user access** + + Unlike multi-tenancy, you do not need to write a custom `evalhub-evaluator` Role. The operator creates `evalhub-user` and `evalhub-tenant-admin` for you. Bind your principals to `evalhub-user` for standard evaluation access. + + EvalHub supports three principal types: + + | Principal | Token source | RoleBinding `kind` | Has home namespace? | + |-----------|-------------|-------------------|---------------------| + | ServiceAccount | `oc create token` | `ServiceAccount` | Yes (where SA lives) | + | OpenShift User | `oc whoami -t` (OAuth) | `User` | No | + | OpenShift Group | `oc whoami -t` (OAuth, via member) | `Group` | No | + + + + + **Create the ServiceAccount:** + + ```bash + oc apply -f - < + + + + **Bind an OpenShift User to `evalhub-user`:** + + ```bash + oc apply -f - < + + + + **Create the Group and add members:** + + ```bash + oc adm groups new team-a-evaluators + oc adm groups add-users team-a-evaluators alice + oc adm groups add-users team-a-evaluators bob + ``` + + **Bind the group to `evalhub-user`:** + + ```bash + oc apply -f - < + + +3. **Access the API** + + All evaluation API requests must include the `X-Tenant` header set to the EvalHub instance namespace. In single-tenancy mode, this is always the namespace where EvalHub is deployed. + + **Get the EvalHub route:** + + ```bash + EVALHUB_URL=$(oc get route evalhub -n team-a -o jsonpath='{.spec.host}') + ``` + + **Get a token:** + + + + + ```bash + TOKEN=$(oc create token team-a-user -n team-a --duration=1h) + ``` + + + + + + ```bash + oc login --username=alice --password= + TOKEN=$(oc whoami -t) + ``` + + + + + + ```bash + oc login --username=alice --password= + TOKEN=$(oc whoami -t) + ``` + + + + + **List providers:** + + + + + ```bash + evalhub providers list --token $TOKEN --tenant team-a + ``` + + + + + ```python + client = SyncEvalHubClient( + base_url=os.environ["EVALHUB_URL"], + auth_token=TOKEN, + tenant="team-a" + ) + providers = client.providers.list() + ``` + + + + + ```bash + curl -sS -k \ + -H "Authorization: Bearer $TOKEN" \ + -H "X-Tenant: team-a" \ + "https://$EVALHUB_URL/api/v1/evaluations/providers" | jq . + ``` + + + + + **Submit an evaluation job:** + + + + + ```bash + evalhub eval run \ + --token $TOKEN \ + --tenant team-a \ + --name mmlu-eval \ + --model-url http://vllm-server.team-a.svc.cluster.local:8000/v1 \ + --model-name meta-llama/Llama-3.2-1B-Instruct \ + --provider lm_evaluation_harness \ + --benchmark mmlu + ``` + + + + + ```python + job = client.jobs.submit(JobSubmissionRequest( + name="mmlu-eval", + model=ModelConfig( + url="http://vllm-server.team-a.svc.cluster.local:8000/v1", + name="meta-llama/Llama-3.2-1B-Instruct" + ), + benchmarks=[ + BenchmarkConfig(id="mmlu", provider_id="lm_evaluation_harness") + ] + )) + ``` + + + + + ```bash + curl -sS -k -X POST \ + -H "Authorization: Bearer $TOKEN" \ + -H "X-Tenant: team-a" \ + -H "Content-Type: application/json" \ + -d '{ + "model": { + "url": "http://vllm-server.team-a.svc.cluster.local:8000/v1", + "name": "meta-llama/Llama-3.2-1B-Instruct" + }, + "benchmarks": [ + { + "id": "mmlu", + "provider_id": "lm_evaluation_harness" + } + ] + }' \ + "https://$EVALHUB_URL/api/v1/evaluations/jobs" | jq . + ``` + + + + + The job pod is created in `team-a` using the `evalhub-team-a-job` ServiceAccount. + + + +## RBAC reference — convenience Roles + +In single-tenancy mode, the operator creates these namespace-scoped Roles (garbage-collected when the EvalHub CR is deleted): + +### `evalhub-tenant-admin` + +| API group | Resources | Verbs | +|-----------|-----------|-------| +| `trustyai.opendatahub.io` | `evaluations`, `collections`, `providers`, `status-events` | get, list, watch, create, update, patch, delete | +| `trustyai.opendatahub.io` | `evalhubs/proxy` (scoped to instance name) | get, create | +| `mlflow.kubeflow.org` | `experiments` | get, list | + +MLflow access is read-only — no `create` verb. Although this Role includes `evaluations` create, `POST /api/v1/evaluations/jobs` also performs SAR checks for MLflow `experiments` with `create` and `get`. Bind principals to [`evalhub-user`](#evalhub-user) for MLflow submission permissions. + +Bound to `system:serviceaccounts:{namespace}` via `evalhub-tenant-admin-binding`, so all ServiceAccounts in the namespace receive admin access by default. + +### `evalhub-user` + +| API group | Resources | Verbs | +|-----------|-----------|-------| +| `trustyai.opendatahub.io` | `collections`, `providers` | get, list, watch | +| `trustyai.opendatahub.io` | `evalhubs/proxy` (scoped to instance name) | get, create | +| `trustyai.opendatahub.io` | `status-events` | create | +| `mlflow.kubeflow.org` | `experiments` | get, list, create | + +Bind individual principals to this Role for standard evaluation access without full admin permissions. + +## Tenant provider and collection ConfigMaps + +You can add custom providers and collections in the instance namespace without modifying the operator namespace. Label ConfigMaps with: + +- Providers: `trustyai.opendatahub.io/evalhub-provider-type=tenant` +- Collections: `trustyai.opendatahub.io/evalhub-collection-type=tenant` + +The operator discovers these in-place and mounts them at `/config/providers/tenant/` and `/config/collections/tenant/` — separate from system configs at `/config/providers/` and `/config/collections/` to avoid shadowing. + +Changes to tenant ConfigMaps trigger reconciliation via the operator's ConfigMap watch; no CR edit is required. + +## What does not happen + +In single-tenancy mode, the operator does **not**: + +- Watch for `evalhub.trustyai.opendatahub.io/tenant` labels on other namespaces +- Provision job ServiceAccounts or RoleBindings in namespaces other than the instance namespace +- Create `evalhub-discovery` ConfigMaps in other namespaces + +## Verifying the setup + +```bash +# Convenience Roles created by the operator +oc get role evalhub-tenant-admin evalhub-user -n team-a +oc get rolebinding evalhub-tenant-admin-binding -n team-a + +# Job SA in the instance namespace +oc get sa evalhub-team-a-job -n team-a + +# Confirm no EvalHub resources in a second namespace +oc get sa,role,rolebinding -n team-b 2>/dev/null | grep evalhub || echo "No cross-namespace resources (expected)" +``` + +Test API access with the correct tenant header: + +```bash +oc auth can-i get providers.trustyai.opendatahub.io \ + -n team-a \ + --as=system:serviceaccount:team-a:team-a-user +``` + +## Troubleshooting + +For common issues such as 403 Forbidden responses, see [Troubleshooting on the Tenancy overview](/architecture/tenancy/#troubleshooting). + +### Wrong `X-Tenant` value + +In single-tenancy mode, `X-Tenant` must be set to the EvalHub instance namespace — not a different namespace. A request with `X-Tenant: other-ns` is rejected unless the caller has a RoleBinding in `other-ns`. + +### Missing `X-Tenant` header + +Requests without the `X-Tenant` header are rejected even though there is only one tenant. Always set `X-Tenant` to the instance namespace. + +### Default admin binding vs `evalhub-user` + +If a ServiceAccount can access the API without an explicit `evalhub-user` RoleBinding, it is likely receiving permissions through `evalhub-tenant-admin-binding` (which binds all namespace ServiceAccounts to admin). To enforce least privilege, remove or avoid relying on the default binding and bind principals explicitly to `evalhub-user`. + +--- + +For shared control-plane deployments serving multiple teams, see [Multi-Tenancy](/architecture/multi-tenancy/). diff --git a/src/content/docs/architecture/tenancy.mdx b/src/content/docs/architecture/tenancy.mdx new file mode 100644 index 0000000..3c38476 --- /dev/null +++ b/src/content/docs/architecture/tenancy.mdx @@ -0,0 +1,291 @@ +--- +title: "Tenancy" +--- + +import { LinkCard, CardGrid, Tabs, TabItem } from '@astrojs/starlight/components'; + +EvalHub uses Kubernetes namespaces as tenant boundaries. Each tenant operates in its own namespace, and access is enforced through Kubernetes RBAC via SubjectAccessReview (SAR) checks. + +The TrustyAI Operator controls deployment behaviour via `spec.tenancy` on the EvalHub custom resource. Two modes are available: + +| | `multi` (default) | `single` | +|---|---|---| +| EvalHub placement | Control-plane namespace (e.g. `opendatahub`) | Workload namespace | +| Tenant discovery | Label `evalhub.trustyai.opendatahub.io/tenant` on namespaces | None — serves own namespace only | +| Cross-namespace RBAC | Operator provisions job SA + RoleBindings per tenant | Not performed | +| Convenience Roles | Not created | `evalhub-tenant-admin`, `evalhub-user`, admin binding | +| InvalidPlacement guard | Yes — cannot deploy in tenant-labelled NS | No — may deploy anywhere | +| Database | PostgreSQL recommended | SQLite acceptable for dev/small teams | + +The core principle in both modes is: **namespace = tenant = MLFlow workspace**. + +## Topologies + + + + +One EvalHub instance in a control-plane namespace serves multiple tenant namespaces. + +```mermaid +flowchart LR + subgraph "Control Plane (opendatahub)" + Operator[TrustyAI Operator] + EvalHub[EvalHub API] + end + + subgraph "Tenant A (team-a)" + SA_A[Job SA] + Job_A[Eval Job] + end + + subgraph "Tenant B (team-b)" + SA_B[Job SA] + Job_B[Eval Job] + end + + Operator -->|creates RBAC| SA_A + Operator -->|creates RBAC| SA_B + EvalHub -->|schedules jobs| Job_A + EvalHub -->|schedules jobs| Job_B +``` + + + + + +EvalHub runs in the same namespace as the workloads it serves. + +```mermaid +flowchart LR + subgraph "Workload Namespace (team-a)" + Operator[TrustyAI Operator] + EvalHub[EvalHub API] + SA[Job SA] + Job[Eval Job] + User[User SA] + end + + Operator -->|creates RBAC| SA + Operator -->|creates convenience Roles| User + EvalHub -->|schedules jobs| Job + User -->|API calls| EvalHub +``` + + + + +## How tenant isolation works + +1. **Authentication** -- Bearer tokens are validated via the Kubernetes TokenReview API. The token can belong to a ServiceAccount, an OpenShift User, or a member of an OpenShift Group — EvalHub treats all identically at this stage. +2. **Authorisation** -- Every API request includes an `X-Tenant` header specifying the target namespace. EvalHub runs a SAR check: _"can this principal perform this action in that namespace?"_ +3. **Data isolation** -- All database queries are filtered by `tenant_id` +4. **Job isolation** -- Evaluation jobs run in the tenant namespace with a scoped ServiceAccount + +:::note[Tenant ID and user identity are independent] +The tenant is always determined by the `X-Tenant` request header — **not** by the identity in the token. A ServiceAccount has a home namespace, but EvalHub does not use it. A real OpenShift User or Group member has no namespace at all. In all cases the caller must supply `X-Tenant` explicitly, and a matching RoleBinding (for the User, ServiceAccount, or Group) must exist in that namespace. +::: + +## Authorisation model + +EvalHub uses an embedded SAR authoriser. The auth config (`config/auth.yaml`) maps API endpoints to Kubernetes resource attributes: + +| Endpoint | Resource | Verb | Namespace source | +|----------|----------|------|------------------| +| `POST /api/v1/evaluations/jobs` | `evaluations` | `create` | `X-Tenant` header | +| `GET /api/v1/evaluations/jobs` | `evaluations` | `get` | `X-Tenant` header | +| `POST /api/v1/evaluations/jobs/*/events` | `status-events` | `create` | `X-Tenant` header | +| `* /api/v1/evaluations/collections` | `collections` | _(from HTTP method)_ | `X-Tenant` header | +| `* /api/v1/evaluations/providers` | `providers` | _(from HTTP method)_ | `X-Tenant` header | + +For `POST /api/v1/evaluations/jobs`, two additional SAR checks are performed for MLFlow access (`mlflow.kubeflow.org/experiments` with `create` and `get` verbs). + +### Request flow + +The flow is identical for all principal types. The only difference is what the TokenReview returns and which subject kind is matched in the RoleBinding. + + + + +```mermaid +sequenceDiagram + participant User as Tenant SA + participant EH as EvalHub API + participant K8s as Kubernetes API + participant Job as Eval Job Pod + + User->>EH: POST /jobs (Bearer token + X-Tenant: team-a) + EH->>K8s: TokenReview (validate token) + K8s-->>EH: user=system:serviceaccount:team-a:team-a-user + EH->>K8s: SAR (can SA create evaluations in team-a?) + K8s-->>EH: Allowed (kind:ServiceAccount RoleBinding matched) + EH->>K8s: Create Job in team-a (SA: evalhub-opendatahub-job) + K8s-->>EH: Job created + EH-->>User: 202 Accepted + + Job->>EH: POST /jobs/{id}/events (job SA token + X-Tenant: team-a) + EH->>K8s: SAR (can job SA create status-events in team-a?) + K8s-->>EH: Allowed + EH-->>Job: 200 OK +``` + + + + + +```mermaid +sequenceDiagram + participant User as OpenShift User (alice) + participant EH as EvalHub API + participant K8s as Kubernetes API + participant Job as Eval Job Pod + + User->>EH: POST /jobs (OAuth token + X-Tenant: team-a) + EH->>K8s: TokenReview (validate token) + K8s-->>EH: user=alice + EH->>K8s: SAR (can alice create evaluations in team-a?) + K8s-->>EH: Allowed (kind:User RoleBinding matched) + EH->>K8s: Create Job in team-a (SA: evalhub-opendatahub-job) + K8s-->>EH: Job created + EH-->>User: 202 Accepted + + Job->>EH: POST /jobs/{id}/events (job SA token + X-Tenant: team-a) + EH->>K8s: SAR (can job SA create status-events in team-a?) + K8s-->>EH: Allowed + EH-->>Job: 200 OK +``` + + + + + +```mermaid +sequenceDiagram + participant User as Group Member (alice) + participant EH as EvalHub API + participant K8s as Kubernetes API + participant Job as Eval Job Pod + + User->>EH: POST /jobs (OAuth token + X-Tenant: team-a) + EH->>K8s: TokenReview (validate token) + K8s-->>EH: user=alice, groups=[team-a-evaluators, ...] + EH->>K8s: SAR (can alice create evaluations in team-a?) + K8s-->>EH: Allowed (kind:Group RoleBinding matched via group membership) + EH->>K8s: Create Job in team-a (SA: evalhub-opendatahub-job) + K8s-->>EH: Job created + EH-->>User: 202 Accepted + + Job->>EH: POST /jobs/{id}/events (job SA token + X-Tenant: team-a) + EH->>K8s: SAR (can job SA create status-events in team-a?) + K8s-->>EH: Allowed + EH-->>Job: 200 OK +``` + + + + +Note that the job pod always runs as the operator-provisioned job ServiceAccount (e.g. `evalhub-opendatahub-job`), regardless of whether the submitter was a User, Group member, or a ServiceAccount. + +## RBAC reference — ClusterRoles + +The operator installs these ClusterRoles cluster-wide. Mode-specific Role and RoleBinding details are documented on the setup pages linked below. + +| ClusterRole | Purpose | Key permissions | +|-------------|---------|-----------------| +| `evalhub-auth-reviewer-role` | Token and SAR validation | `tokenreviews`, `subjectaccessreviews` | +| `evalhub-jobs-writer` | Create evaluation jobs | `batch/jobs` (create, delete) | +| `evalhub-job-config` | Manage job config | `configmaps` (create, get, update, delete) | +| `evalhub-providers-access` | Providers endpoint SAR | `providers` (get) | +| `evalhub-collections-access` | Collections endpoint SAR | `collections` (get) | +| `evalhub-mlflow-access` | API server MLFlow access | `experiments` (create, get, list, update, delete) | +| `evalhub-mlflow-jobs-access` | Job pod MLFlow access | `experiments` (create, get, list) | + +:::note[Virtual resources] +The SAR targets (`evaluations`, `collections`, `providers`, `status-events`) are virtual — they don't correspond to actual Kubernetes API resources. EvalHub uses them to enforce fine-grained access control via the Kubernetes authorisation API. +::: + +## Mode switching + +You can change `spec.tenancy` on a running EvalHub instance. The operator reconciles the difference automatically: + +| Transition | Operator behaviour | +|---|---| +| `single` → `multi` | Removes convenience Roles; provisions cross-namespace RBAC in tenant-labelled namespaces | +| `multi` → `single` | Cleans up cross-namespace resources; creates convenience Roles in the instance namespace | + +:::caution[InvalidPlacement] +A **multi-tenant** instance placed in a namespace carrying the `evalhub.trustyai.opendatahub.io/tenant` label enters `Error` phase with reason `InvalidPlacement`. Fix by removing the tenant label from that namespace or switching to `spec.tenancy: single`. +::: + +## Troubleshooting + +### 403 Forbidden on API calls + +The SAR check is failing. Verify: + +1. The `X-Tenant` header matches a namespace where the principal has a RoleBinding +2. The Role includes the correct resources and verbs +3. The RoleBinding `subjects[].kind` matches the principal type (`ServiceAccount`, `User`, or `Group`) + + + + +```bash +oc auth can-i create evaluations.trustyai.opendatahub.io \ + -n team-a \ + --as=system:serviceaccount:team-a:team-a-user -v=6 +``` + + + + + +```bash +oc auth can-i create evaluations.trustyai.opendatahub.io \ + -n team-a \ + --as=alice -v=6 +``` + +Also confirm the RoleBinding subject kind is `User` and has no `namespace` field: + +```bash +oc get rolebinding evalhub-evaluator-binding -n team-a -o yaml | grep -A5 subjects +``` + + + + + +```bash +oc auth can-i create evaluations.trustyai.opendatahub.io \ + -n team-a \ + --as=alice --as-group=team-a-evaluators -v=6 +``` + +If the SAR check passes with `--as-group` but real requests still fail: + +1. **Token doesn't include group claims** -- The user must re-login (`oc login`) after being added to the group. Existing OAuth tokens may not include the new group membership. +2. **Verify group membership:** + + ```bash + oc get group team-a-evaluators -o jsonpath='{.users[*]}' + ``` + +3. **Confirm the RoleBinding subject kind is `Group`:** + + ```bash + oc get rolebinding evalhub-evaluator-group-binding -n team-a -o yaml | grep -A5 subjects + ``` + + + + +### Missing `X-Tenant` header + +All evaluation API requests require the `X-Tenant` header, even in single-tenancy mode where the tenant is always the EvalHub instance namespace. Requests without the header are rejected. + +## Setup guides + + + + + diff --git a/src/content/docs/home.mdx b/src/content/docs/home.mdx index 6aa58a1..95af1ff 100644 --- a/src/content/docs/home.mdx +++ b/src/content/docs/home.mdx @@ -29,7 +29,7 @@ EvalHub provides a unified way to evaluate LLMs across multiple frameworks — s - **Provider registry** with benchmark discovery - **OCI artifact persistence** for evaluation results - **Prometheus metrics** and OpenTelemetry tracing -- **Multi-tenancy** with namespace-based isolation and Kubernetes RBAC +- **[Tenancy](/architecture/tenancy/)** with multi- and single-namespace deployment modes and Kubernetes RBAC ## How It Works