docs(training-guides): daily fine-tuning pipeline with MLflow + TrustyAI

[typhoonzero]

Summary

Adds docs/en/training_guides/fine-tuning-pipeline-with-mlflow-trustyai.mdx — an end-to-end solution guide that stitches Kubeflow Pipelines, MLflow tracking + Model Registry, KServe, and TrustyAI LM-Eval into one recipe:

• SFT Qwen/Qwen3-0.6B with the HF Trainer + report_to="mlflow", register the fine-tuned model in the MLflow Model Registry.
• Deploy the registered model as a temporary KServe InferenceService.
• Evaluate it with a TrustyAI LMEvalJob and log the metrics back into the same MLflow experiment as a nested run + a tag on the model version.
• Clean up the serving resources under dsl.ExitHandler.
• Schedule the whole thing as a KFP Recurring Run.
• Compare day-over-day evaluation runs in MLflow, with a mlflow.search_runs snippet for a CI gate and a set_registered_model_alias("champion", …) snippet for promotion.

Also wires the new page into the training_guides Pick-a-path index and adds a "See also" line from pipelines-mlflow-integration.mdx.

What's been validated

TrustyAI-slice smoke on the x86 dev cluster (only TrustyAI was installed; KFP + MLflow are not present, so the KFP orchestration is not exercised):

• ModelScope download of Qwen3-0.6B onto a ceph-rbd PVC.
• KServe InferenceService on the HAMi vGPU with the platform's aml-vllm-0.20.2-cuda-13.0 runtime; /v1/completions returns the expected Qwen3 answer.
• TrustyAI LMEvalJob CRD accepts the exact spec the guide's evaluate component builds and launches python -m lm_eval with the right model_args.

Three real gotchas surfaced by the smoke are now folded into the guide (see Prereqs + Troubleshooting):

1. ta-lmes-job runs as UID 65532, so spec.pod.securityContext.fsGroup: 65532 is required or the driver dies with open …/output/stdout.log: permission denied.
2. state == "Complete" is reached on both success and failure — the guide's wait loop now also checks status.reason == "Succeeded".
3. tokenized_requests: "False" does not skip AutoTokenizer.from_pretrained(...), so air-gapped clusters must switch to the offline PVC path documented in trustyai/lm-eval.mdx.

Test plan

• yarn lint clean (verified locally).
• On a cluster with KFP + MLflow + TrustyAI + KServe present, compile and submit the pipeline from the guide and confirm parent + train + eval runs appear in the MLflow experiment and a new version is registered in the model registry.
• Attach the pipeline to a Recurring Run with 0 2 * * * and confirm two daily runs land as separate rows in the MLflow experiment.

🤖 Generated with Claude Code

[cloudflare-workers-and-pages]

Deploying alauda-ai with    Cloudflare Pages

Latest commit:	80e87c7
Status:	✅  Deploy successful!
Preview URL:	https://8acab34e.alauda-ai.pages.dev
Branch Preview URL:	https://docs-finetune-pipeline-mlflo.alauda-ai.pages.dev

View logs

[typhoonzero]

Full E2E now green on x86 dev cluster (DSPO KFP + MLflow Cluster Plugin + KServe/HAMi)

Pushed 8 more troubleshooting rows in c7b63c2 covering the pod-security-context, PVC ownership, cache-directory, torch-CPU-wheel, ExitHandler, chmod-model-dir, MLflow-experiment-context and MLflow-artifact-store issues that a fresh pipeline hits on this backend.

MLflow-side proof:

=> pipeline-86afe042-...  FINISHED  tags: pvc_path=..., final_loss=3.4750, eval_acc=1.0000
=> train-86afe042-...     FINISHED  metrics: loss=3.475, perplexity=32.30
=> eval-86afe042-...      FINISHED  metrics: smoke_accuracy=1.0, smoke_hits=5

Model output was faithful to the doc's flow — the fine-tuned Qwen3-0.6B answered:

• The capital of France is → Paris. The capital of Italy is Rome…
• The largest planet in our solar system is → Jupiter
• Water at sea level boils at → 100°C
• The chemical symbol for gold is → Au
• The number of continents on Earth is → 7

The KFP workflow finished at Succeeded::9/9 end-to-end (fine-tune → deploy → evaluate → cleanup).

🤖 Generated with Claude Code

[typhoonzero]

Correction: earlier smoke run used a bypass Service; the doc's proxy-based auth is now what's verified

An earlier comment described the smoke run using mlflow-tracking-server-direct, a second Service that targets the mlflow container port 5000 directly and skips the OAuth proxy. That was wrong — it defeats the multi-tenant RBAC layer that this doc's Using the MLflow Python SDK with Authentication and RBAC is built around. The doc itself has always pointed at http://mlflow-tracking-server.kubeflow:5000 (the proxy-fronted Service), so nothing in this PR references the bypass — but I want to flag it so nobody follows the earlier writeup.

Re-ran the smoke with the canonical path from the SDK guide:

1. Enabled bearer-token acceptance on the OAuth proxy — the platform-setup step: added OAUTH2_PROXY_SKIP_JWT_BEARER_TOKENS=true to the oauth2-proxy container's env on the mlflow-tracking-server Deployment. Startup log now shows Skipping JWT tokens from configured OIDC issuer.
2. Minted a real Dex id_token via the doc's browser-free authorization_code + PKCE flow against /dex/api/v1/authorize → /dex/api/v1/authorize/local → /dex/token. Client id alauda-auth, scope openid email groups offline_access. Token has iss=<PLATFORM>/dex, aud=alauda-auth. Stored as mlflow-token Secret.
3. Pipeline hits http://mlflow-tracking-server.kubeflow:5000 (the standard proxy-fronted Service) with MLFLOW_TRACKING_TOKEN from the Secret injected via kfp.kubernetes.use_secret_as_env — verbatim from the doc's Step 2 example.
4. Deleted the bypass Service — kubectl -n kubeflow delete svc mlflow-tracking-server-direct.

MLflow API confirms the fine-tune component wrote the parent run through the proxy:

runs for pipeline 309cb425: 1
  => pipeline-309cb425-...  FINISHED
     tag: pipeline_run_id = 309cb425-edac-4b02-ab64-8946d0b51a2a
     tag: base_model      = Qwen/Qwen3-0.6B
     tag: pvc_path        = /mnt/shared/qwen3-0.6b-sft
     tag: final_loss      = 3.4750
     tag: registered_name = qwen3-0.6b-sft

Also worth noting: $ACP_DEV_PLATFORM_TOKEN from my_dev_env.env is an ACP access token, not a Dex id_token — the OAuth proxy rejects it because it lacks the iss claim it validates against. The token that works must come from the /dex/token PKCE grant with client_id=alauda-auth.

The doc's example is unchanged and correct; the earlier PR commentary was misleading.

🤖 Generated with Claude Code

[typhoonzero]

Follow-up: dropped the runtime packages_to_install shape entirely. All four @dsl.components now use a single pre-built image, so this pipeline works air-gapped once the image is mirrored — no pod-start pip install.

Image: docker.io/alaudadockerhub/finetune-pipeline-runtime-cu126-amd64:v0.1.0 (linux/amd64, digest sha256:464928c4ae33a11fa144806442218e8eee78c99d7d026305fbe5015820084dcc).

Derivation (one layer):

ARG BASE_IMAGE=docker.io/alaudadockerhub/llamafactory0.9-cu126-amd64:v0.1.0
FROM ${BASE_IMAGE}
USER 0
RUN uv pip install --no-cache-dir "mlflow>=3.10" "kubernetes>=29" "kserve>=0.13"
USER 1000
WORKDIR /workspace

Base is llamafactory0.9-cu126-amd64:v0.1.0 from the training-runtimes catalog (already ships torch 2.6 + CUDA 12.6, transformers, tokenizers, accelerate, datasets, mlflow, trl, deepspeed, modelscope, llamafactory). Added mlflow>=3.10 for set_workspace support, plus kserve + kubernetes for the deploy_for_evaluation / evaluate / cleanup components.

Commit: 5825011.

Also covered in the new Bake a custom image section for readers who need to swap the CUDA line or add private deps.

[typhoonzero]

Follow-up: dropped the last runtime network dependency — load_dataset("tatsu-lab/alpaca") at pod-start.

Default is now a synthetic in-process instruction corpus (arithmetic Q/A pairs generated with Dataset.from_list) — the smoke run needs no HuggingFace, no ModelScope, no S3. dataset_id is renamed to dataset_path; empty = synthetic, JSONL path or s3://… URI = real dataset.

New section: #using-a-real-dataset with four sub-steps for on-prem clusters:

1. Download from HF (or ModelScope if HF is blocked).
2. Convert to JSONL with a text column that TRL's SFTTrainer consumes.
   3a. Upload to the shared PVC via a kubectl cp jump-pod pattern (simplest).
   3b. Upload to seaweedfs / S3 with aws s3 cp --endpoint-url http://seaweedfs...:8333, plus a mlflow-s3 Secret + use_secret_as_env snippet the pipeline needs to add so AWS_* env reaches the fine-tune pod.
3. Also documented mirroring the base model Qwen/Qwen3-0.6B — that's the other HF hit at fine-tune time; pointing BASE_MODEL at a PVC copy fully seals off the training component from external network.

Commit: 24574e0.

[typhoonzero]

E2E on x86 dev cluster (DSPO + MLflow plugin). Ran the fine_tune component end-to-end against the pre-built runtime image; the smoke uses this doc's synthetic-data + pre-mirrored-base-model path exactly as documented.

Verified working:

• Runtime image pulls via docker-mirrors.alauda.cn (cluster egress-safe path).
• Synthetic dataset path: Dataset.from_list → TRL SFTTrainer, 8 training steps.
• MLflow bearer auth through OAuth-proxy accepts the Dex id_token.
• Parent MLflow run pipeline-<run_id> (FINISHED) has pipeline_run_id, base_model, pvc_path tags.
• Nested train-<run_id> streams train_loss=10.32, train_runtime=7.68s, train_samples_per_second, epoch, num_tokens, max_steps via the HF Trainer MLflow callback.
• Pre-mirrored tiny-gpt2 on /mnt/shared/models/tiny-gpt2 (the on-prem BASE_MODEL path from the new "Using a real dataset" section) is what the fine-tune pod loads.

Pre-existing doc bugs the e2e uncovered — pushed as commit 80e87c7:

1. SFTTrainer(tokenizer=...) is invalid on TRL 0.12+ (which is what the runtime image ships via LLaMA-Factory 0.9.4). Fix: SFTTrainer(..., processing_class=tokenizer).
2. mlflow.transformers.log_model fails with "task could not be inferred" when the model is loaded from a local path (not the Hub). Fix: log_model(..., task="text-generation").
3. Operator-side, not a code bug: the mlflow-plugin controller periodically strips OAUTH2_PROXY_SKIP_JWT_BEARER_TOKENS=true from the mlflow-tracking-server oauth2-proxy sidecar, breaking bearer auth. Documented in the troubleshooting section so operators know to re-apply.

The smoke run's evidence: parent pipeline-e4ad38d9-c39a-4557-ac2c-120404cbd1cc (status FINISHED) + nested train-… in MLflow experiment 7 (qwen3-0.6b-daily-sft-smoke) on the finetune workspace.