diff --git a/content/patterns/vllm-ray-inference-server/_index.adoc b/content/patterns/vllm-ray-inference-server/_index.adoc new file mode 100644 index 000000000..d8b8a088e --- /dev/null +++ b/content/patterns/vllm-ray-inference-server/_index.adoc @@ -0,0 +1,106 @@ +--- +title: vLLM Ray Inference Server +date: 2026-11-08 +tier: sandbox +summary: This pattern deploys an OpenAI-compatible large language model (LLM) server that runs vLLM on Ray Serve with one GPU, built inside the cluster from upstream packages and managed by OpenShift GitOps. +rh_products: + - Red Hat OpenShift Container Platform + - Red Hat OpenShift GitOps +partners: + - Eugo +industries: + - General +focus_areas: + - AI +aliases: /vllm-ray-inference-server/ +links: + github: https://github.com/eugo-inc/vllm-ray-inference-server + install: getting-started + arch: /images/vllm-ray-inference-server/architecture.svg + bugs: https://github.com/eugo-inc/vllm-ray-inference-server/issues + feedback: https://docs.google.com/forms/d/e/1FAIpQLScI76b6tD1WyPu2-d_9CCVDr3Fu5jYERthqLKJDUGwqBg7Vcg/viewform +--- +:toc: +:imagesdir: /images +:_content-type: ASSEMBLY +include::modules/comm-attributes.adoc[] +:vllm-ray-pattern: vLLM Ray Inference Server pattern + +[id="about-vllm-ray-inference-server-pattern"] += About the {vllm-ray-pattern} + +The {vllm-ray-pattern} deploys a private, OpenAI-compatible large language model (LLM) endpoint on {rh-ocp}. {rh-gitops} deploys every component from your fork of the pattern repository, and the cluster builds the server image itself from upstream packages. Eugo maintains the pattern as community content in the {sandbox-tier-first}. + +[WARNING] +==== +On {AWS}, the default installation creates one billable GPU instance: a `g6.2xlarge` instance with one NVIDIA L4 GPU. AWS bills the instance for as long as it runs. To skip the instance, set `global.gpuMachineSet: none` in the `values-global.yaml` file in your fork, and push the change before you install the pattern. On other platforms, the pattern creates no instance. Uninstalling the pattern does not delete the instance. For the steps to delete it, see link:getting-started/#uninstalling-the-vllm-ray-pattern[Uninstalling the pattern]. +==== + +[id="background-vllm-ray"] +== Background + +The {vllm-ray-pattern} does not replace the Red{nbsp}Hat AI products, such as {rhoai}, and it does not integrate with them. The pattern is a small, community-maintained reference point: the upstream vLLM engine on Ray Serve, built from known versions and measured the same way each time. The pattern does not need a {rhoai} subscription. + +Your fork of the pattern repository describes the whole deployment: GPU enablement, the server, a model cache, a secret for gated models, and a timing notebook. You run the `./pattern.sh make install` command one time. After that, {rh-gitops} syncs each commit on your branch. It does not delete the objects that a commit removes, and it does not change the image of the running server or notebook. A job in the cluster starts new image builds: every 10 minutes, it starts a build when the server or notebook folder changed and no build of that image is running. For details, see link:https://github.com/eugo-inc/vllm-ray-inference-server/blob/main/AGENTS.md#updating-a-running-install[Updating a running install] in the pattern repository. + +The pattern pins every Python package by hash and installs wheels only. It pins the base images of the server and the notebook by digest, and the default model by its Hugging Face commit. Some parts are not pinned, among them the system packages that the server image installs with `dnf`, the Operator versions, and the patch versions of the Validated Patterns Helm charts. Two builds from the same commit therefore install the same Python packages, but their system packages can differ. + +The notebook measures the server in your cluster: the time from pod start to `Ready`, the time to first token, the decode speed of one stream, and the token throughput at 1, 8, and 32 concurrent requests. The notebook chooses its prompts from a fixed seed, and every request uses a temperature of 0 and a fixed number of output tokens. The time to `Ready` is in whole seconds and can trail the server by up to about 2 seconds. + +The server runs Ray Serve LLM and the vLLM engine in one pod of a standard deployment, without KServe or KubeRay. Ray, vLLM, and PyTorch come from PyPI and run on the NVIDIA CUDA image that is based on Red{nbsp}Hat Universal Base Image 9. The pattern creates no KServe `InferenceService` or `ServingRuntime` resource. + +The pattern installs the {rh-nfd} and the NVIDIA GPU Operator itself. On a cluster that already runs these Operators, the pattern can conflict with them, and the pattern has not been tested on a cluster that runs {rhoai}. Install the pattern on a cluster that does not run these Operators. + +[id="architecture-vllm-ray"] +== Architecture + +.Architecture of the {vllm-ray-pattern} +image::/images/vllm-ray-inference-server/architecture.svg[Diagram of the vLLM Ray Inference Server pattern. The Validated Patterns Operator and OpenShift GitOps deploy secrets management, GPU enablement, the inference server, and the notebook from your Git fork.] + +The pattern works in the following way: + +* The {validated-patterns-op} installs {rh-gitops} and points the Argo CD instance at your fork. The `clustergroup` Helm chart then creates every application that the `prod` variant lists. +* {hashicorp-vault} and the {eso-op} deliver an optional Hugging Face token to the server as a Kubernetes secret. The default model is not gated and needs no token. +* The {rh-nfd} and the NVIDIA GPU Operator prepare the GPU node. On AWS, an imperative job creates the GPU node. +* A `BuildConfig` object builds the server image inside the cluster. The server deployment runs one pod with a Ray head, Ray Serve, and the vLLM engine on one GPU. The pod reports `Ready` only after the model loads. +* A persistent volume claim keeps the model weights across pod restarts. +* A JupyterLab notebook runs behind the OpenShift OAuth proxy. Only users who can edit the `vllm-notebook` namespace can open it. The notebook sends requests to the server through the in-cluster service. + +[id="components-vllm-ray"] +== Components that the pattern deploys + +The pattern deploys the following Operators: + +* {rh-gitops} +* The {validated-patterns-op}, a community Operator +* The {eso-op} for Red{nbsp}Hat OpenShift +* The {rh-nfd} +* The NVIDIA GPU Operator, from the certified Operators catalog + +The pattern deploys the following applications: + +* {hashicorp-vault}, from the {solution-name-upstream} Helm chart. +* The inference server. Ray Serve LLM runs the vLLM engine and serves an OpenAI-compatible API on port 8000. The default model is `Qwen/Qwen2.5-0.5B-Instruct`. The server image turns off the usage statistics of Ray and vLLM. +* The notebook. A JupyterLab notebook measures the running server: the time from pod start to `Ready`, the time to first token, the decode speed of one stream, and the token throughput at several concurrency levels. + +The link:https://github.com/eugo-inc/vllm-ray-inference-server#what-is-in-the-stock-stack[pattern README] lists the exact version of each component. + +[id="partner-results-vllm-ray"] +== Partner results: Eugo + +Eugo measured its next vLLM image (release pending) against the upstream `vllm/vllm-openai` v0.33.0 image on one NVIDIA DGX Spark that other jobs kept busy, in paired rounds with an exact sign test. + +* *Time for a new server to serve.* With Qwen2.5-0.5B-Instruct in plain `vllm serve` containers, the software caches empty and the model files already in memory, a new Eugo server served its first token a median of 20.2 seconds (22%) sooner than the upstream image with `--attention-backend FLASHINFER`, in 6 of 6 rounds (p = 0.031). +* *Starts that succeed first time.* The vLLM start-up memory check stopped 4 of 12 upstream starts and none of 28 Eugo starts. Eugo's vLLM carries a fix that mitigates this. + +For details, see the link:https://github.com/eugo-inc/vllm-ray-inference-server#partner-results-eugo[Partner results: Eugo] section of the pattern README. + +[id="support-vllm-ray"] +== Support + +The {vllm-ray-pattern} is community content in the {sandbox-tier-first}. Eugo maintains the pattern and supports it on a best-effort basis through link:https://github.com/eugo-inc/vllm-ray-inference-server/issues[GitHub issues] in the pattern repository. The maintainers do not guarantee a response time. Red{nbsp}Hat does not produce, support, or validate this pattern. For an issue with a product that the pattern deploys, contact the vendor of that product. + +[id="next-steps-vllm-ray"] +== Next steps + +* link:getting-started[Getting started] diff --git a/content/patterns/vllm-ray-inference-server/getting-started.adoc b/content/patterns/vllm-ray-inference-server/getting-started.adoc new file mode 100644 index 000000000..ff220e8a6 --- /dev/null +++ b/content/patterns/vllm-ray-inference-server/getting-started.adoc @@ -0,0 +1,205 @@ +--- +title: Getting started +weight: 10 +aliases: /vllm-ray-inference-server/getting-started/ +--- + +:toc: +:imagesdir: /images +:_content-type: ASSEMBLY +include::modules/comm-attributes.adoc[] +:vllm-ray-pattern: vLLM Ray Inference Server pattern + +[id="deploying-the-vllm-ray-pattern"] +== Deploying the {vllm-ray-pattern} + +You deploy the pattern from your own fork of the pattern repository. {rh-gitops} deploys the content of your fork on GitHub, not the files on your workstation, so commit and push every change before you install the pattern. + +.Prerequisites + +* An {ocp} 4.18 or later cluster, and cluster administrator access. To create a cluster, go to the link:https://console.redhat.com/[{hybrid-console-first}] and select *Services -> Containers -> Create cluster*. +* A cluster that does not already run the {rh-nfd} or the NVIDIA GPU Operator. The pattern installs both Operators itself, with its own `OperatorGroup` objects and settings, and can conflict with existing installations of these Operators. The pattern has not been tested on a cluster that runs {rhoai}. +* A storage class that provisions persistent volumes dynamically. The model cache uses one persistent volume claim. +* The {ocp-registry} in the `Managed` state, and the `Build` and `ImageRegistry` cluster capabilities enabled. The cluster builds the server image and the notebook image itself. On bare metal and on vSphere, the registry starts in the `Removed` state. To check the registry state, run the following command: ++ +[source,terminal] +---- +$ oc get configs.imageregistry.operator.openshift.io cluster -o jsonpath='{.spec.managementState}' +---- +* A GPU node: +** On AWS, the pattern creates one `g6.2xlarge` node with one NVIDIA L4 GPU by default. AWS bills this instance for as long as it runs. +** On an AWS cluster that was installed into existing subnets, the job that creates the GPU node fails. Set `global.gpuMachineSet: none` and add a GPU node yourself. +** On other platforms, add a node with an NVIDIA GPU that the NVIDIA GPU Operator supports. +* The server runs on `amd64` nodes by default. If your GPU node is an `arm64` node, read the prerequisites in the link:https://github.com/eugo-inc/vllm-ray-inference-server/blob/main/AGENTS.md#prerequisites[AGENTS.md file] in the pattern repository before you install. +* On your workstation, Podman 4.3 or later, Git, and the `oc` CLI, logged in to the cluster. If you set the `KUBECONFIG` environment variable, it must point to a file in your home directory. For the other installation tool dependencies, see link:https://validatedpatterns.io/learn/quickstart/[Patterns quick start]. + +.Procedure + +. Fork the link:https://github.com/eugo-inc/vllm-ray-inference-server[vllm-ray-inference-server] repository on GitHub. + +. Clone your fork: ++ +[source,terminal,subs="+quotes"] +---- +$ git clone https://github.com/____/vllm-ray-inference-server.git +---- + +. Go to the root directory of the repository: ++ +[source,terminal] +---- +$ cd vllm-ray-inference-server +---- + +. Edit the `values-global.yaml` file: +** Set `global.gpuMachineSet` to `create` to let the pattern create one AWS `g6.2xlarge` GPU node, or to `none` to use a GPU node that you provide. +** Optional: To serve a different model from Hugging Face, set `global.model.id`, `global.model.source`, and `global.model.revision`. + +. Commit and push the change: ++ +[source,terminal] +---- +$ git commit -am "Configure the pattern" +$ git push +---- + +. Optional: To serve a gated model, create the secrets file for the pattern and add your Hugging Face token: +.. Copy the secrets template: ++ +[source,terminal] +---- +$ mkdir -p ~/.config/validated-patterns +$ cp values-secret.yaml.template \ + ~/.config/validated-patterns/values-secret-vllm-ray-inference-server.yaml +---- +.. In the copied file, set the `value` of the `hftoken` field to your Hugging Face token, which starts with `hf_`. Do not commit this file. + +. Log in to the cluster: ++ +[source,terminal,subs="+quotes"] +---- +$ oc login --token=____ --server=____ +---- + +. Install the pattern: ++ +[source,terminal] +---- +$ ./pattern.sh make install +---- ++ +The installer waits up to 60 minutes for every Argo CD application to report `Synced` and `Healthy`. A first installation can take longer, because the cluster builds a large server image, the GPU node joins the cluster, and the server downloads the model. If the wait times out, the pattern stays installed. Check the progress with the commands in the next step, and then run `./pattern.sh make argo-healthcheck`. + +. Watch the deployment: ++ +[source,terminal] +---- +$ oc get applications.argoproj.io -A +$ oc get builds -n vllm-inference +$ oc get pods -n vllm-inference -w +---- ++ +The server pod stays in the `Pending` state until a node advertises the `nvidia.com/gpu` resource. The pod reports `Ready` after it loads the model. + +[id="verifying-the-vllm-ray-pattern"] +== Verifying the inference server + +The server is reachable inside the cluster at `http://ray-serve-llm.vllm-inference.svc:8000`. To send a test request from your workstation, forward the server port. + +.Procedure + +. Forward the server port to your workstation: ++ +[source,terminal] +---- +$ oc port-forward svc/ray-serve-llm 8000:8000 -n vllm-inference +---- + +. In another terminal, list the models that the server serves: ++ +[source,terminal] +---- +$ curl -s http://localhost:8000/v1/models +---- + +. Send a chat request: ++ +[source,terminal] +---- +$ curl -s http://localhost:8000/v1/chat/completions \ + -H 'Content-Type: application/json' \ + -d '{ + "model": "qwen2.5-0.5b-instruct", + "messages": [{"role": "user", "content": "Say hello in one sentence."}], + "max_tokens": 64 + }' +---- ++ +The `model` field takes the value of `global.model.id`. Any OpenAI-compatible client works with the base URL `http://localhost:8000/v1`. + +[id="opening-the-notebook-vllm-ray"] +== Opening the notebook + +The notebook pod runs JupyterLab behind the OpenShift OAuth proxy. Users who can edit the `vllm-notebook` namespace, and cluster administrators, can open it. + +.Procedure + +. Get the URL of the notebook route: ++ +[source,terminal] +---- +$ oc get route -n vllm-notebook +---- + +. Open the URL in a browser and log in with your OpenShift account. + +. Open the `inference_timings.ipynb` notebook and run all cells. The notebook pod copies the notebook folder to a scratch directory each time that it starts, so download any change that you want to keep. + +. Optional: To give another user access to the notebook, grant the `edit` role in the `vllm-notebook` namespace: ++ +[source,terminal,subs="+quotes"] +---- +$ oc adm policy add-role-to-user edit ____ -n vllm-notebook +---- + +[id="uninstalling-the-vllm-ray-pattern"] +== Uninstalling the pattern + +The {solution-name-upstream} framework marks uninstallation as experimental. Uninstalling the pattern leaves the {validated-patterns-op} and {rh-gitops} installed, and it does not delete the AWS GPU node. AWS bills the GPU instance until you delete its machine set. + +.Procedure + +. Uninstall the pattern: ++ +[source,terminal] +---- +$ ./pattern.sh make uninstall +---- + +. List the machine sets with their instance types: ++ +[source,terminal] +---- +$ oc get machinesets -n openshift-machine-api \ + -o custom-columns=NAME:.metadata.name,TYPE:.spec.template.spec.providerSpec.value.instanceType +---- + +. Delete the machine set of the `g6.2xlarge` instance type. Delete it only after the uninstallation finishes, or after the `global.gpuMachineSet: none` setting syncs. Otherwise, the job creates the machine set again. ++ +[source,terminal,subs="+quotes"] +---- +$ oc delete machineset ____ -n openshift-machine-api +---- + +. Check for a leftover persistent volume from the `vllm-inference/ray-serve-llm-models` claim: ++ +[source,terminal] +---- +$ oc get pv +---- + +[id="next-steps-vllm-ray-getting-started"] +== Next steps + +* For the web console installation, private forks, updates, and troubleshooting, see the link:https://github.com/eugo-inc/vllm-ray-inference-server/blob/main/AGENTS.md[AGENTS.md file] in the pattern repository. +* For the pattern overview, see link:../[{vllm-ray-pattern}]. diff --git a/static/images/vllm-ray-inference-server/architecture.svg b/static/images/vllm-ray-inference-server/architecture.svg new file mode 100644 index 000000000..012e72bf3 --- /dev/null +++ b/static/images/vllm-ray-inference-server/architecture.svg @@ -0,0 +1 @@ +OpenShift clusterNamespacevllm-notebookNamespacevllm-inferenceGPU enablementSecretsGithftokenSecrethuggingface-secretmodel weights at/modelsOpenAI-compatible HTTPYour fork of this repositoryYou, in a browserValidated Patterns OperatorOpenShift GitOps: Argo CD instancevp-gitopsclustergroup app-of-apps, variant prodHashiCorp VaultExternal Secrets OperatorNode Feature DiscoveryNVIDIA GPU OperatorMachineSet job, AWS only: oneg6.2xlarge with one NVIDIA L4BuildConfig ray-serve-llmImageStream ray-serve-llmDeployment ray-serve-llm: Ray head +Ray Serve + vLLM, one GPUPVC ray-serve-llm-modelsService ray-serve-llm, port 8000BuildConfig and ImageStreamnotebookNotebook pod: JupyterLab +oauth-proxyRoute with OpenShift login \ No newline at end of file