From 3ad27d5fef941fdcaa5cb079d70d2f1eda9251ef Mon Sep 17 00:00:00 2001 From: Ben Date: Sun, 8 Nov 2026 19:52:27 +0000 Subject: [PATCH 1/6] docs(patterns): add vLLM Ray Inference Server sandbox pattern Add an overview page and a getting-started page for the rhvp-vllm-ray-inference-server sandbox pattern, plus its architecture diagram. --- .../_index.adoc | 101 +++++++++ .../getting-started.adoc | 206 ++++++++++++++++++ .../architecture.svg | 1 + 3 files changed, 308 insertions(+) create mode 100644 content/patterns/rhvp-vllm-ray-inference-server/_index.adoc create mode 100644 content/patterns/rhvp-vllm-ray-inference-server/getting-started.adoc create mode 100644 static/images/rhvp-vllm-ray-inference-server/architecture.svg diff --git a/content/patterns/rhvp-vllm-ray-inference-server/_index.adoc b/content/patterns/rhvp-vllm-ray-inference-server/_index.adoc new file mode 100644 index 000000000..6059f0c8d --- /dev/null +++ b/content/patterns/rhvp-vllm-ray-inference-server/_index.adoc @@ -0,0 +1,101 @@ +--- +# The directory name, the aliases, and the GitHub links in this page set use the +# current repository name, rhvp-vllm-ray-inference-server. That name can change, +# pending two open questions to the Validated Patterns team: whether the "rhvp-" +# prefix is acceptable under the Red Hat trademark guidelines, and whether the +# repository moves to the validatedpatterns-sandbox organization. If either +# changes, rename the directory, the aliases, the image folder, and every link +# in both pages together. +title: vLLM Ray Inference Server +date: 2026-11-08 +validated: false +tier: sandbox +summary: This pattern deploys an OpenAI-compatible large language model (LLM) server that runs vLLM on Ray Serve with one GPU, built inside the cluster from upstream packages and managed by OpenShift GitOps. +rh_products: + - Red Hat OpenShift Container Platform + - Red Hat OpenShift GitOps +# partners: left out on purpose. The pattern deploys software from NVIDIA (the +# GPU Operator and the CUDA base image) and from the Ray and vLLM projects, but +# no partner organization takes part in the pattern. A partners entry suggests a +# partnership, and the site prints a partner trademark notice for it. Add an +# entry only when a partner organization joins the pattern. +industries: + - General +focus_areas: + - AI +aliases: /rhvp-vllm-ray-inference-server/ +links: + github: https://github.com/eugo-inc/rhvp-vllm-ray-inference-server + install: getting-started + arch: /images/rhvp-vllm-ray-inference-server/architecture.svg + bugs: https://github.com/eugo-inc/rhvp-vllm-ray-inference-server/issues + feedback: https://docs.google.com/forms/d/e/1FAIpQLScI76b6tD1WyPu2-d_9CCVDr3Fu5jYERthqLKJDUGwqBg7Vcg/viewform +--- +:toc: +:imagesdir: /images +:_content-type: ASSEMBLY +include::modules/comm-attributes.adoc[] +:vllm-ray-pattern: vLLM Ray Inference Server pattern + +// The pattern name can change. See the comment at the top of the front matter. + +[id="about-vllm-ray-inference-server-pattern"] += About the {vllm-ray-pattern} + +The {vllm-ray-pattern} deploys a private, OpenAI-compatible large language model (LLM) endpoint on {rh-ocp}. {rh-gitops} deploys every component from your fork of the pattern repository, and the cluster builds the server image itself from upstream packages. Eugo maintains the pattern as community content in the {sandbox-tier-first}. + +[WARNING] +==== +On {AWS}, the default installation creates one billable GPU instance: a `g6.2xlarge` instance with one NVIDIA L4 GPU. AWS bills the instance for as long as it runs. To skip the instance, set `global.gpuMachineSet: none` in the `values-global.yaml` file in your fork, and push the change before you install the pattern. On other platforms, the pattern creates no instance. Uninstalling the pattern does not delete the instance. For the steps to delete it, see link:getting-started/#uninstalling-the-vllm-ray-pattern[Uninstalling the pattern]. +==== + +[id="background-vllm-ray"] +== Background + +A team that wants a private, OpenAI-compatible LLM endpoint on its own {ocp} cluster usually assembles the parts by hand: GPU enablement, a serving runtime, a model cache, a secret for gated models, and a way to rebuild the server image when the software stack changes. This pattern describes all of these parts in one Git repository. You fork the repository, set the model, and push the change, and the cluster converges to a running endpoint. + +The pattern pins every Python package by hash and the base image by digest, so each rebuild installs the same software stack, and each change is a reviewable commit. The pattern uses upstream open source components only: vLLM, Ray Serve, and Python wheels from PyPI, on the NVIDIA CUDA image that is based on Red{nbsp}Hat Universal Base Image 9. + +[id="architecture-vllm-ray"] +== Architecture + +.Architecture of the {vllm-ray-pattern} +image::/images/rhvp-vllm-ray-inference-server/architecture.svg[Diagram of the vLLM Ray Inference Server pattern. The Validated Patterns Operator and OpenShift GitOps deploy secrets management, GPU enablement, the inference server, and the notebook from your Git fork.] + +The pattern works in the following way: + +* The {validated-patterns-op} installs {rh-gitops} and points the Argo CD instance at your fork. The `clustergroup` Helm chart then creates every application that the `prod` variant lists. +* {hashicorp-vault} and the {eso-op} deliver an optional Hugging Face token to the server as a Kubernetes secret. The default model is not gated and needs no token. +* The {rh-nfd} and the NVIDIA GPU Operator prepare the GPU node. On AWS, an imperative job creates the GPU node. +* A `BuildConfig` object builds the server image inside the cluster. The server deployment runs one pod with a Ray head, Ray Serve, and the vLLM engine on one GPU. The pod reports `Ready` only after the model loads. +* A persistent volume claim keeps the model weights across pod restarts. +* A JupyterLab notebook runs behind the OpenShift OAuth proxy. Only users who can edit the `rhvp-notebook` namespace can open it. The notebook sends requests to the server through the in-cluster service. + +[id="components-vllm-ray"] +== Components that the pattern deploys + +The pattern deploys the following Operators: + +* {rh-gitops} +* The {validated-patterns-op}, a community Operator +* The {eso-op} for Red{nbsp}Hat OpenShift +* The {rh-nfd} +* The NVIDIA GPU Operator, from the certified Operators catalog + +The pattern deploys the following applications: + +* {hashicorp-vault}, from the {solution-name-upstream} Helm chart. +* The inference server. Ray Serve LLM runs the vLLM engine and serves an OpenAI-compatible API on port 8000. The default model is `Qwen/Qwen2.5-0.5B-Instruct`. The server image turns off the usage statistics of Ray and vLLM. +* The notebook. A JupyterLab notebook measures the running server: the time from pod start to `Ready`, the time to first token, and the token throughput at several concurrency levels. + +The link:https://github.com/eugo-inc/rhvp-vllm-ray-inference-server#what-is-in-the-stock-stack[pattern README] lists the exact version of each component. + +[id="support-vllm-ray"] +== Support + +The {vllm-ray-pattern} is community content in the {sandbox-tier-first}. Eugo maintains the pattern and supports it on a best-effort basis through link:https://github.com/eugo-inc/rhvp-vllm-ray-inference-server/issues[GitHub issues] in the pattern repository. The maintainers do not guarantee a response time. Red{nbsp}Hat does not produce, support, or validate this pattern. For an issue with a product that the pattern deploys, contact the vendor of that product. + +[id="next-steps-vllm-ray"] +== Next steps + +* link:getting-started[Getting started] diff --git a/content/patterns/rhvp-vllm-ray-inference-server/getting-started.adoc b/content/patterns/rhvp-vllm-ray-inference-server/getting-started.adoc new file mode 100644 index 000000000..9d1592400 --- /dev/null +++ b/content/patterns/rhvp-vllm-ray-inference-server/getting-started.adoc @@ -0,0 +1,206 @@ +--- +# The pattern name in this page can change. See the comment at the top of the +# front matter in _index.adoc in this folder. +title: Getting started +weight: 10 +aliases: /rhvp-vllm-ray-inference-server/getting-started/ +--- + +:toc: +:imagesdir: /images +:_content-type: ASSEMBLY +include::modules/comm-attributes.adoc[] +:vllm-ray-pattern: vLLM Ray Inference Server pattern + +[id="deploying-the-vllm-ray-pattern"] +== Deploying the {vllm-ray-pattern} + +You deploy the pattern from your own fork of the pattern repository. {rh-gitops} deploys the content of your fork on GitHub, not the files on your workstation, so commit and push every change before you install the pattern. + +.Prerequisites + +* An {ocp} 4.18 or later cluster, and cluster administrator access. To create a cluster, go to the link:https://console.redhat.com/[{hybrid-console-first}] and select *Services -> Containers -> Create cluster*. +* A storage class that provisions persistent volumes dynamically. The model cache uses one persistent volume claim. +* The {ocp-registry} in the `Managed` state, and the `Build` and `ImageRegistry` cluster capabilities enabled. The cluster builds the server image and the notebook image itself. On bare metal and on vSphere, the registry starts in the `Removed` state. To check the registry state, run the following command: ++ +[source,terminal] +---- +$ oc get configs.imageregistry.operator.openshift.io cluster -o jsonpath='{.spec.managementState}' +---- +* A GPU node: +** On AWS, the pattern creates one `g6.2xlarge` node with one NVIDIA L4 GPU by default. AWS bills this instance for as long as it runs. +** On an AWS cluster that was installed into existing subnets, the job that creates the GPU node fails. Set `global.gpuMachineSet: none` and add a GPU node yourself. +** On other platforms, add a node with an NVIDIA GPU that the NVIDIA GPU Operator supports. +* The server runs on `amd64` nodes by default. If your GPU node is an `arm64` node, read the prerequisites in the link:https://github.com/eugo-inc/rhvp-vllm-ray-inference-server/blob/main/AGENTS.md#prerequisites[AGENTS.md file] in the pattern repository before you install. +* On your workstation, Podman 4.3 or later, Git, and the `oc` CLI, logged in to the cluster. If you set the `KUBECONFIG` environment variable, it must point to a file in your home directory. For the other installation tool dependencies, see link:https://validatedpatterns.io/learn/quickstart/[Patterns quick start]. + +.Procedure + +. Fork the link:https://github.com/eugo-inc/rhvp-vllm-ray-inference-server[rhvp-vllm-ray-inference-server] repository on GitHub. + +. Clone your fork: ++ +[source,terminal,subs="+quotes"] +---- +$ git clone https://github.com/____/rhvp-vllm-ray-inference-server.git +---- + +. Go to the root directory of the repository: ++ +[source,terminal] +---- +$ cd rhvp-vllm-ray-inference-server +---- + +. Edit the `values-global.yaml` file: +** Set `global.gpuMachineSet` to `create` to let the pattern create one AWS `g6.2xlarge` GPU node, or to `none` to use a GPU node that you provide. +** Optional: To serve a different model from Hugging Face, set `global.model.id`, `global.model.source`, and `global.model.revision`. + +. Commit and push the change: ++ +[source,terminal] +---- +$ git commit -am "Configure the pattern" +$ git push +---- + +. Optional: To serve a gated model, create the secrets file for the pattern and add your Hugging Face token: +.. Copy the secrets template: ++ +[source,terminal] +---- +$ mkdir -p ~/.config/validated-patterns +$ cp values-secret.yaml.template \ + ~/.config/validated-patterns/values-secret-rhvp-vllm-ray-inference-server.yaml +---- +.. In the copied file, set the `value` of the `hftoken` field to your Hugging Face token, which starts with `hf_`. Do not commit this file. + +. Log in to the cluster: ++ +[source,terminal,subs="+quotes"] +---- +$ oc login --token=____ --server=____ +---- + +. Install the pattern: ++ +[source,terminal] +---- +$ ./pattern.sh make install +---- ++ +The installer waits up to 60 minutes for every Argo CD application to report `Synced` and `Healthy`. A first installation can take longer, because the cluster builds a large server image, the GPU node joins the cluster, and the server downloads the model. If the wait times out, the pattern stays installed. Check the progress with the commands in the next step, and then run `./pattern.sh make argo-healthcheck`. + +. Watch the deployment: ++ +[source,terminal] +---- +$ oc get applications.argoproj.io -A +$ oc get builds -n rhvp-inference +$ oc get pods -n rhvp-inference -w +---- ++ +The server pod stays in the `Pending` state until a node advertises the `nvidia.com/gpu` resource. The pod reports `Ready` after it loads the model. + +[id="verifying-the-vllm-ray-pattern"] +== Verifying the inference server + +The server is reachable inside the cluster at `http://ray-serve-llm.rhvp-inference.svc:8000`. To send a test request from your workstation, forward the server port. + +.Procedure + +. Forward the server port to your workstation: ++ +[source,terminal] +---- +$ oc port-forward svc/ray-serve-llm 8000:8000 -n rhvp-inference +---- + +. In another terminal, list the models that the server serves: ++ +[source,terminal] +---- +$ curl -s http://localhost:8000/v1/models +---- + +. Send a chat request: ++ +[source,terminal] +---- +$ curl -s http://localhost:8000/v1/chat/completions \ + -H 'Content-Type: application/json' \ + -d '{ + "model": "qwen2.5-0.5b-instruct", + "messages": [{"role": "user", "content": "Say hello in one sentence."}], + "max_tokens": 64 + }' +---- ++ +The `model` field takes the value of `global.model.id`. Any OpenAI-compatible client works with the base URL `http://localhost:8000/v1`. + +[id="opening-the-notebook-vllm-ray"] +== Opening the notebook + +The notebook pod runs JupyterLab behind the OpenShift OAuth proxy. Users who can edit the `rhvp-notebook` namespace, and cluster administrators, can open it. + +.Procedure + +. Get the URL of the notebook route: ++ +[source,terminal] +---- +$ oc get route -n rhvp-notebook +---- + +. Open the URL in a browser and log in with your OpenShift account. + +. Open the `stock_vs_eugo_inference.ipynb` notebook and run all cells. The notebook pod copies the notebook folder to a scratch directory each time that it starts, so download any change that you want to keep. + +. Optional: To give another user access to the notebook, grant the `edit` role in the `rhvp-notebook` namespace: ++ +[source,terminal,subs="+quotes"] +---- +$ oc adm policy add-role-to-user edit ____ -n rhvp-notebook +---- + +[id="uninstalling-the-vllm-ray-pattern"] +== Uninstalling the pattern + +The {solution-name-upstream} framework marks uninstallation as experimental. Uninstalling the pattern leaves the {validated-patterns-op} and {rh-gitops} installed, and it does not delete the AWS GPU node. AWS bills the GPU instance until you delete its machine set. + +.Procedure + +. Uninstall the pattern: ++ +[source,terminal] +---- +$ ./pattern.sh make uninstall +---- + +. List the machine sets with their instance types: ++ +[source,terminal] +---- +$ oc get machinesets -n openshift-machine-api \ + -o custom-columns=NAME:.metadata.name,TYPE:.spec.template.spec.providerSpec.value.instanceType +---- + +. Delete the machine set of the `g6.2xlarge` instance type. Delete it only after the uninstallation finishes, or after the `global.gpuMachineSet: none` setting syncs. Otherwise, the job creates the machine set again. ++ +[source,terminal,subs="+quotes"] +---- +$ oc delete machineset ____ -n openshift-machine-api +---- + +. Check for a leftover persistent volume from the `rhvp-inference/ray-serve-llm-models` claim: ++ +[source,terminal] +---- +$ oc get pv +---- + +[id="next-steps-vllm-ray-getting-started"] +== Next steps + +* For the web console installation, private forks, updates, and troubleshooting, see the link:https://github.com/eugo-inc/rhvp-vllm-ray-inference-server/blob/main/AGENTS.md[AGENTS.md file] in the pattern repository. +* For the pattern overview, see link:../[{vllm-ray-pattern}]. diff --git a/static/images/rhvp-vllm-ray-inference-server/architecture.svg b/static/images/rhvp-vllm-ray-inference-server/architecture.svg new file mode 100644 index 000000000..9ddcbee4b --- /dev/null +++ b/static/images/rhvp-vllm-ray-inference-server/architecture.svg @@ -0,0 +1 @@ +OpenShift clusterNamespacerhvp-notebookNamespacerhvp-inferenceGPU enablementSecretsGithftokenSecrethuggingface-secretmodel weights at/modelsOpenAI-compatible HTTPYour fork of this repositoryYou, in a browserValidated Patterns OperatorOpenShift GitOps: Argo CD instancevp-gitopsclustergroup app-of-apps, variant prodHashiCorp VaultExternal Secrets OperatorNode Feature DiscoveryNVIDIA GPU OperatorMachineSet job, AWS only: oneg6.2xlarge with one NVIDIA L4BuildConfig ray-serve-llmImageStream ray-serve-llmDeployment ray-serve-llm: Ray head +Ray Serve + vLLM, one GPUPVC ray-serve-llm-modelsService ray-serve-llm, port 8000BuildConfig and ImageStreamrhvp-notebookNotebook pod: JupyterLab +oauth-proxyRoute with OpenShift login \ No newline at end of file From 4faa817797bdb0d8673128d9a40fc1f3799d1a41 Mon Sep 17 00:00:00 2001 From: Ben Date: Wed, 11 Nov 2026 22:46:08 +0000 Subject: [PATCH 2/6] docs(patterns): rename to vllm-ray-inference-server, list Eugo as a partner, apply review notes The pattern repository is now named vllm-ray-inference-server. Move both page folders and the image folder to that name, and update the aliases, the architecture image path, the GitHub links, the clone and cd commands, the secrets file name and the notebook name to match. List Eugo, the maintainer, under partners, so the overview page prints the partner trademark notice. Apply the review notes: drop the front matter comments about a possible rename and the left-out partners entry, and drop validated: false. --- .../_index.adoc | 31 ++++++------------- .../getting-started.adoc | 18 +++++------ .../architecture.svg | 0 3 files changed, 17 insertions(+), 32 deletions(-) rename content/patterns/{rhvp-vllm-ray-inference-server => vllm-ray-inference-server}/_index.adoc (70%) rename content/patterns/{rhvp-vllm-ray-inference-server => vllm-ray-inference-server}/getting-started.adoc (87%) rename static/images/{rhvp-vllm-ray-inference-server => vllm-ray-inference-server}/architecture.svg (100%) diff --git a/content/patterns/rhvp-vllm-ray-inference-server/_index.adoc b/content/patterns/vllm-ray-inference-server/_index.adoc similarity index 70% rename from content/patterns/rhvp-vllm-ray-inference-server/_index.adoc rename to content/patterns/vllm-ray-inference-server/_index.adoc index 6059f0c8d..c1e3b0cd7 100644 --- a/content/patterns/rhvp-vllm-ray-inference-server/_index.adoc +++ b/content/patterns/vllm-ray-inference-server/_index.adoc @@ -1,34 +1,23 @@ --- -# The directory name, the aliases, and the GitHub links in this page set use the -# current repository name, rhvp-vllm-ray-inference-server. That name can change, -# pending two open questions to the Validated Patterns team: whether the "rhvp-" -# prefix is acceptable under the Red Hat trademark guidelines, and whether the -# repository moves to the validatedpatterns-sandbox organization. If either -# changes, rename the directory, the aliases, the image folder, and every link -# in both pages together. title: vLLM Ray Inference Server date: 2026-11-08 -validated: false tier: sandbox summary: This pattern deploys an OpenAI-compatible large language model (LLM) server that runs vLLM on Ray Serve with one GPU, built inside the cluster from upstream packages and managed by OpenShift GitOps. rh_products: - Red Hat OpenShift Container Platform - Red Hat OpenShift GitOps -# partners: left out on purpose. The pattern deploys software from NVIDIA (the -# GPU Operator and the CUDA base image) and from the Ray and vLLM projects, but -# no partner organization takes part in the pattern. A partners entry suggests a -# partnership, and the site prints a partner trademark notice for it. Add an -# entry only when a partner organization joins the pattern. +partners: + - Eugo industries: - General focus_areas: - AI -aliases: /rhvp-vllm-ray-inference-server/ +aliases: /vllm-ray-inference-server/ links: - github: https://github.com/eugo-inc/rhvp-vllm-ray-inference-server + github: https://github.com/eugo-inc/vllm-ray-inference-server install: getting-started - arch: /images/rhvp-vllm-ray-inference-server/architecture.svg - bugs: https://github.com/eugo-inc/rhvp-vllm-ray-inference-server/issues + arch: /images/vllm-ray-inference-server/architecture.svg + bugs: https://github.com/eugo-inc/vllm-ray-inference-server/issues feedback: https://docs.google.com/forms/d/e/1FAIpQLScI76b6tD1WyPu2-d_9CCVDr3Fu5jYERthqLKJDUGwqBg7Vcg/viewform --- :toc: @@ -37,8 +26,6 @@ links: include::modules/comm-attributes.adoc[] :vllm-ray-pattern: vLLM Ray Inference Server pattern -// The pattern name can change. See the comment at the top of the front matter. - [id="about-vllm-ray-inference-server-pattern"] = About the {vllm-ray-pattern} @@ -60,7 +47,7 @@ The pattern pins every Python package by hash and the base image by digest, so e == Architecture .Architecture of the {vllm-ray-pattern} -image::/images/rhvp-vllm-ray-inference-server/architecture.svg[Diagram of the vLLM Ray Inference Server pattern. The Validated Patterns Operator and OpenShift GitOps deploy secrets management, GPU enablement, the inference server, and the notebook from your Git fork.] +image::/images/vllm-ray-inference-server/architecture.svg[Diagram of the vLLM Ray Inference Server pattern. The Validated Patterns Operator and OpenShift GitOps deploy secrets management, GPU enablement, the inference server, and the notebook from your Git fork.] The pattern works in the following way: @@ -88,12 +75,12 @@ The pattern deploys the following applications: * The inference server. Ray Serve LLM runs the vLLM engine and serves an OpenAI-compatible API on port 8000. The default model is `Qwen/Qwen2.5-0.5B-Instruct`. The server image turns off the usage statistics of Ray and vLLM. * The notebook. A JupyterLab notebook measures the running server: the time from pod start to `Ready`, the time to first token, and the token throughput at several concurrency levels. -The link:https://github.com/eugo-inc/rhvp-vllm-ray-inference-server#what-is-in-the-stock-stack[pattern README] lists the exact version of each component. +The link:https://github.com/eugo-inc/vllm-ray-inference-server#what-is-in-the-stock-stack[pattern README] lists the exact version of each component. [id="support-vllm-ray"] == Support -The {vllm-ray-pattern} is community content in the {sandbox-tier-first}. Eugo maintains the pattern and supports it on a best-effort basis through link:https://github.com/eugo-inc/rhvp-vllm-ray-inference-server/issues[GitHub issues] in the pattern repository. The maintainers do not guarantee a response time. Red{nbsp}Hat does not produce, support, or validate this pattern. For an issue with a product that the pattern deploys, contact the vendor of that product. +The {vllm-ray-pattern} is community content in the {sandbox-tier-first}. Eugo maintains the pattern and supports it on a best-effort basis through link:https://github.com/eugo-inc/vllm-ray-inference-server/issues[GitHub issues] in the pattern repository. The maintainers do not guarantee a response time. Red{nbsp}Hat does not produce, support, or validate this pattern. For an issue with a product that the pattern deploys, contact the vendor of that product. [id="next-steps-vllm-ray"] == Next steps diff --git a/content/patterns/rhvp-vllm-ray-inference-server/getting-started.adoc b/content/patterns/vllm-ray-inference-server/getting-started.adoc similarity index 87% rename from content/patterns/rhvp-vllm-ray-inference-server/getting-started.adoc rename to content/patterns/vllm-ray-inference-server/getting-started.adoc index 9d1592400..2e8e07f4c 100644 --- a/content/patterns/rhvp-vllm-ray-inference-server/getting-started.adoc +++ b/content/patterns/vllm-ray-inference-server/getting-started.adoc @@ -1,9 +1,7 @@ --- -# The pattern name in this page can change. See the comment at the top of the -# front matter in _index.adoc in this folder. title: Getting started weight: 10 -aliases: /rhvp-vllm-ray-inference-server/getting-started/ +aliases: /vllm-ray-inference-server/getting-started/ --- :toc: @@ -31,25 +29,25 @@ $ oc get configs.imageregistry.operator.openshift.io cluster -o jsonpath='{.spec ** On AWS, the pattern creates one `g6.2xlarge` node with one NVIDIA L4 GPU by default. AWS bills this instance for as long as it runs. ** On an AWS cluster that was installed into existing subnets, the job that creates the GPU node fails. Set `global.gpuMachineSet: none` and add a GPU node yourself. ** On other platforms, add a node with an NVIDIA GPU that the NVIDIA GPU Operator supports. -* The server runs on `amd64` nodes by default. If your GPU node is an `arm64` node, read the prerequisites in the link:https://github.com/eugo-inc/rhvp-vllm-ray-inference-server/blob/main/AGENTS.md#prerequisites[AGENTS.md file] in the pattern repository before you install. +* The server runs on `amd64` nodes by default. If your GPU node is an `arm64` node, read the prerequisites in the link:https://github.com/eugo-inc/vllm-ray-inference-server/blob/main/AGENTS.md#prerequisites[AGENTS.md file] in the pattern repository before you install. * On your workstation, Podman 4.3 or later, Git, and the `oc` CLI, logged in to the cluster. If you set the `KUBECONFIG` environment variable, it must point to a file in your home directory. For the other installation tool dependencies, see link:https://validatedpatterns.io/learn/quickstart/[Patterns quick start]. .Procedure -. Fork the link:https://github.com/eugo-inc/rhvp-vllm-ray-inference-server[rhvp-vllm-ray-inference-server] repository on GitHub. +. Fork the link:https://github.com/eugo-inc/vllm-ray-inference-server[vllm-ray-inference-server] repository on GitHub. . Clone your fork: + [source,terminal,subs="+quotes"] ---- -$ git clone https://github.com/____/rhvp-vllm-ray-inference-server.git +$ git clone https://github.com/____/vllm-ray-inference-server.git ---- . Go to the root directory of the repository: + [source,terminal] ---- -$ cd rhvp-vllm-ray-inference-server +$ cd vllm-ray-inference-server ---- . Edit the `values-global.yaml` file: @@ -71,7 +69,7 @@ $ git push ---- $ mkdir -p ~/.config/validated-patterns $ cp values-secret.yaml.template \ - ~/.config/validated-patterns/values-secret-rhvp-vllm-ray-inference-server.yaml + ~/.config/validated-patterns/values-secret-vllm-ray-inference-server.yaml ---- .. In the copied file, set the `value` of the `hftoken` field to your Hugging Face token, which starts with `hf_`. Do not commit this file. @@ -154,7 +152,7 @@ $ oc get route -n rhvp-notebook . Open the URL in a browser and log in with your OpenShift account. -. Open the `stock_vs_eugo_inference.ipynb` notebook and run all cells. The notebook pod copies the notebook folder to a scratch directory each time that it starts, so download any change that you want to keep. +. Open the `inference_timings.ipynb` notebook and run all cells. The notebook pod copies the notebook folder to a scratch directory each time that it starts, so download any change that you want to keep. . Optional: To give another user access to the notebook, grant the `edit` role in the `rhvp-notebook` namespace: + @@ -202,5 +200,5 @@ $ oc get pv [id="next-steps-vllm-ray-getting-started"] == Next steps -* For the web console installation, private forks, updates, and troubleshooting, see the link:https://github.com/eugo-inc/rhvp-vllm-ray-inference-server/blob/main/AGENTS.md[AGENTS.md file] in the pattern repository. +* For the web console installation, private forks, updates, and troubleshooting, see the link:https://github.com/eugo-inc/vllm-ray-inference-server/blob/main/AGENTS.md[AGENTS.md file] in the pattern repository. * For the pattern overview, see link:../[{vllm-ray-pattern}]. diff --git a/static/images/rhvp-vllm-ray-inference-server/architecture.svg b/static/images/vllm-ray-inference-server/architecture.svg similarity index 100% rename from static/images/rhvp-vllm-ray-inference-server/architecture.svg rename to static/images/vllm-ray-inference-server/architecture.svg From 60102f448a6ef2745f9187a7ea12146e2f4ebd72 Mon Sep 17 00:00:00 2001 From: Ben Date: Thu, 12 Nov 2026 00:05:57 +0000 Subject: [PATCH 3/6] docs(patterns): follow the pattern's namespace rename and list decode speed The pattern's namespaces are now vllm-inference (the server) and vllm-notebook (the notebook). Update the oc commands, the in-cluster server URL, the notebook access text and the leftover-volume check on both pages, and the namespace labels in the architecture diagram, whose notebook build box now names the BuildConfig and ImageStream notebook. The overview page's notebook entry also lists the single-stream decode speed that the notebook measures. --- .../vllm-ray-inference-server/_index.adoc | 4 ++-- .../getting-started.adoc | 18 +++++++++--------- .../vllm-ray-inference-server/architecture.svg | 2 +- 3 files changed, 12 insertions(+), 12 deletions(-) diff --git a/content/patterns/vllm-ray-inference-server/_index.adoc b/content/patterns/vllm-ray-inference-server/_index.adoc index c1e3b0cd7..e6b637f4c 100644 --- a/content/patterns/vllm-ray-inference-server/_index.adoc +++ b/content/patterns/vllm-ray-inference-server/_index.adoc @@ -56,7 +56,7 @@ The pattern works in the following way: * The {rh-nfd} and the NVIDIA GPU Operator prepare the GPU node. On AWS, an imperative job creates the GPU node. * A `BuildConfig` object builds the server image inside the cluster. The server deployment runs one pod with a Ray head, Ray Serve, and the vLLM engine on one GPU. The pod reports `Ready` only after the model loads. * A persistent volume claim keeps the model weights across pod restarts. -* A JupyterLab notebook runs behind the OpenShift OAuth proxy. Only users who can edit the `rhvp-notebook` namespace can open it. The notebook sends requests to the server through the in-cluster service. +* A JupyterLab notebook runs behind the OpenShift OAuth proxy. Only users who can edit the `vllm-notebook` namespace can open it. The notebook sends requests to the server through the in-cluster service. [id="components-vllm-ray"] == Components that the pattern deploys @@ -73,7 +73,7 @@ The pattern deploys the following applications: * {hashicorp-vault}, from the {solution-name-upstream} Helm chart. * The inference server. Ray Serve LLM runs the vLLM engine and serves an OpenAI-compatible API on port 8000. The default model is `Qwen/Qwen2.5-0.5B-Instruct`. The server image turns off the usage statistics of Ray and vLLM. -* The notebook. A JupyterLab notebook measures the running server: the time from pod start to `Ready`, the time to first token, and the token throughput at several concurrency levels. +* The notebook. A JupyterLab notebook measures the running server: the time from pod start to `Ready`, the time to first token, the decode speed of one stream, and the token throughput at several concurrency levels. The link:https://github.com/eugo-inc/vllm-ray-inference-server#what-is-in-the-stock-stack[pattern README] lists the exact version of each component. diff --git a/content/patterns/vllm-ray-inference-server/getting-started.adoc b/content/patterns/vllm-ray-inference-server/getting-started.adoc index 2e8e07f4c..3a2206316 100644 --- a/content/patterns/vllm-ray-inference-server/getting-started.adoc +++ b/content/patterns/vllm-ray-inference-server/getting-started.adoc @@ -94,8 +94,8 @@ The installer waits up to 60 minutes for every Argo CD application to report `Sy [source,terminal] ---- $ oc get applications.argoproj.io -A -$ oc get builds -n rhvp-inference -$ oc get pods -n rhvp-inference -w +$ oc get builds -n vllm-inference +$ oc get pods -n vllm-inference -w ---- + The server pod stays in the `Pending` state until a node advertises the `nvidia.com/gpu` resource. The pod reports `Ready` after it loads the model. @@ -103,7 +103,7 @@ The server pod stays in the `Pending` state until a node advertises the `nvidia. [id="verifying-the-vllm-ray-pattern"] == Verifying the inference server -The server is reachable inside the cluster at `http://ray-serve-llm.rhvp-inference.svc:8000`. To send a test request from your workstation, forward the server port. +The server is reachable inside the cluster at `http://ray-serve-llm.vllm-inference.svc:8000`. To send a test request from your workstation, forward the server port. .Procedure @@ -111,7 +111,7 @@ The server is reachable inside the cluster at `http://ray-serve-llm.rhvp-inferen + [source,terminal] ---- -$ oc port-forward svc/ray-serve-llm 8000:8000 -n rhvp-inference +$ oc port-forward svc/ray-serve-llm 8000:8000 -n vllm-inference ---- . In another terminal, list the models that the server serves: @@ -139,7 +139,7 @@ The `model` field takes the value of `global.model.id`. Any OpenAI-compatible cl [id="opening-the-notebook-vllm-ray"] == Opening the notebook -The notebook pod runs JupyterLab behind the OpenShift OAuth proxy. Users who can edit the `rhvp-notebook` namespace, and cluster administrators, can open it. +The notebook pod runs JupyterLab behind the OpenShift OAuth proxy. Users who can edit the `vllm-notebook` namespace, and cluster administrators, can open it. .Procedure @@ -147,18 +147,18 @@ The notebook pod runs JupyterLab behind the OpenShift OAuth proxy. Users who can + [source,terminal] ---- -$ oc get route -n rhvp-notebook +$ oc get route -n vllm-notebook ---- . Open the URL in a browser and log in with your OpenShift account. . Open the `inference_timings.ipynb` notebook and run all cells. The notebook pod copies the notebook folder to a scratch directory each time that it starts, so download any change that you want to keep. -. Optional: To give another user access to the notebook, grant the `edit` role in the `rhvp-notebook` namespace: +. Optional: To give another user access to the notebook, grant the `edit` role in the `vllm-notebook` namespace: + [source,terminal,subs="+quotes"] ---- -$ oc adm policy add-role-to-user edit ____ -n rhvp-notebook +$ oc adm policy add-role-to-user edit ____ -n vllm-notebook ---- [id="uninstalling-the-vllm-ray-pattern"] @@ -190,7 +190,7 @@ $ oc get machinesets -n openshift-machine-api \ $ oc delete machineset ____ -n openshift-machine-api ---- -. Check for a leftover persistent volume from the `rhvp-inference/ray-serve-llm-models` claim: +. Check for a leftover persistent volume from the `vllm-inference/ray-serve-llm-models` claim: + [source,terminal] ---- diff --git a/static/images/vllm-ray-inference-server/architecture.svg b/static/images/vllm-ray-inference-server/architecture.svg index 9ddcbee4b..012e72bf3 100644 --- a/static/images/vllm-ray-inference-server/architecture.svg +++ b/static/images/vllm-ray-inference-server/architecture.svg @@ -1 +1 @@ -OpenShift clusterNamespacerhvp-notebookNamespacerhvp-inferenceGPU enablementSecretsGithftokenSecrethuggingface-secretmodel weights at/modelsOpenAI-compatible HTTPYour fork of this repositoryYou, in a browserValidated Patterns OperatorOpenShift GitOps: Argo CD instancevp-gitopsclustergroup app-of-apps, variant prodHashiCorp VaultExternal Secrets OperatorNode Feature DiscoveryNVIDIA GPU OperatorMachineSet job, AWS only: oneg6.2xlarge with one NVIDIA L4BuildConfig ray-serve-llmImageStream ray-serve-llmDeployment ray-serve-llm: Ray head +Ray Serve + vLLM, one GPUPVC ray-serve-llm-modelsService ray-serve-llm, port 8000BuildConfig and ImageStreamrhvp-notebookNotebook pod: JupyterLab +oauth-proxyRoute with OpenShift login \ No newline at end of file +OpenShift clusterNamespacevllm-notebookNamespacevllm-inferenceGPU enablementSecretsGithftokenSecrethuggingface-secretmodel weights at/modelsOpenAI-compatible HTTPYour fork of this repositoryYou, in a browserValidated Patterns OperatorOpenShift GitOps: Argo CD instancevp-gitopsclustergroup app-of-apps, variant prodHashiCorp VaultExternal Secrets OperatorNode Feature DiscoveryNVIDIA GPU OperatorMachineSet job, AWS only: oneg6.2xlarge with one NVIDIA L4BuildConfig ray-serve-llmImageStream ray-serve-llmDeployment ray-serve-llm: Ray head +Ray Serve + vLLM, one GPUPVC ray-serve-llm-modelsService ray-serve-llm, port 8000BuildConfig and ImageStreamnotebookNotebook pod: JupyterLab +oauth-proxyRoute with OpenShift login \ No newline at end of file From 55fd71dc5bece9ae9c5961b87f64122db764052a Mon Sep 17 00:00:00 2001 From: Ben Date: Thu, 12 Nov 2026 04:03:49 +0000 Subject: [PATCH 4/6] docs(patterns): frame the vLLM Ray Inference Server beside Red Hat AI products - Background: say that the pattern does not replace or integrate with the Red Hat AI products, such as Red Hat OpenShift AI, which parts are pinned and which are not, the one-time install step and what OpenShift GitOps does not do after it, and what the notebook measures. Drop the claims that each rebuild installs the same software stack and that the pattern uses upstream open source components only. - Prerequisites: use a cluster that does not already run the Node Feature Discovery Operator or the NVIDIA GPU Operator. The pattern installs both itself and can conflict with existing installations. It has not been tested on a cluster that runs Red Hat OpenShift AI. --- .../patterns/vllm-ray-inference-server/_index.adoc | 12 ++++++++++-- .../vllm-ray-inference-server/getting-started.adoc | 1 + 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/content/patterns/vllm-ray-inference-server/_index.adoc b/content/patterns/vllm-ray-inference-server/_index.adoc index e6b637f4c..d90094b4a 100644 --- a/content/patterns/vllm-ray-inference-server/_index.adoc +++ b/content/patterns/vllm-ray-inference-server/_index.adoc @@ -39,9 +39,17 @@ On {AWS}, the default installation creates one billable GPU instance: a `g6.2xla [id="background-vllm-ray"] == Background -A team that wants a private, OpenAI-compatible LLM endpoint on its own {ocp} cluster usually assembles the parts by hand: GPU enablement, a serving runtime, a model cache, a secret for gated models, and a way to rebuild the server image when the software stack changes. This pattern describes all of these parts in one Git repository. You fork the repository, set the model, and push the change, and the cluster converges to a running endpoint. +The {vllm-ray-pattern} does not replace the Red{nbsp}Hat AI products, such as {rhoai}, and it does not integrate with them. The pattern is a small, community-maintained reference point: the upstream vLLM engine on Ray Serve, built from known versions and measured the same way each time. The pattern does not need a {rhoai} subscription. -The pattern pins every Python package by hash and the base image by digest, so each rebuild installs the same software stack, and each change is a reviewable commit. The pattern uses upstream open source components only: vLLM, Ray Serve, and Python wheels from PyPI, on the NVIDIA CUDA image that is based on Red{nbsp}Hat Universal Base Image 9. +Your fork of the pattern repository describes the whole deployment: GPU enablement, the server, a model cache, a secret for gated models, and a timing notebook. You run the `./pattern.sh make install` command once. After that, {rh-gitops} syncs each commit on your branch. It does not delete the objects that a commit removes, and it does not change the image of the running server or notebook. A job in the cluster starts new image builds: every 10 minutes, it starts a build when the server or notebook folder changed and no build of that image is running. For details, see link:https://github.com/eugo-inc/vllm-ray-inference-server/blob/main/AGENTS.md#updating-a-running-install[Updating a running install] in the pattern repository. + +The pattern pins every Python package by hash and installs wheels only. It pins the base images of the server and the notebook by digest, and the default model by its Hugging Face commit. Some parts are not pinned, among them the system packages that the server image installs with `dnf`, the Operator versions, and the patch versions of the Validated Patterns Helm charts. Two builds from the same commit therefore install the same Python packages, but their system packages can differ. + +The notebook measures the server in your cluster: the time from pod start to `Ready`, the time to first token, the decode speed of one stream, and the token throughput at 1, 8, and 32 concurrent requests. The notebook chooses its prompts from a fixed seed, and every request uses a temperature of 0 and a fixed number of output tokens. The time to `Ready` is in whole seconds and can trail the server by up to about 2 seconds. + +The server runs Ray Serve LLM and the vLLM engine in one pod of a standard deployment, without KServe or KubeRay. Ray, vLLM, and PyTorch come from PyPI and run on the NVIDIA CUDA image that is based on Red{nbsp}Hat Universal Base Image 9. The pattern creates no KServe `InferenceService` or `ServingRuntime` resource. + +The pattern installs the {rh-nfd} and the NVIDIA GPU Operator itself. On a cluster that already runs these Operators, the pattern can conflict with them, and the pattern has not been tested on a cluster that runs {rhoai}. Install the pattern on a cluster that does not run these Operators. [id="architecture-vllm-ray"] == Architecture diff --git a/content/patterns/vllm-ray-inference-server/getting-started.adoc b/content/patterns/vllm-ray-inference-server/getting-started.adoc index 3a2206316..ff220e8a6 100644 --- a/content/patterns/vllm-ray-inference-server/getting-started.adoc +++ b/content/patterns/vllm-ray-inference-server/getting-started.adoc @@ -18,6 +18,7 @@ You deploy the pattern from your own fork of the pattern repository. {rh-gitops} .Prerequisites * An {ocp} 4.18 or later cluster, and cluster administrator access. To create a cluster, go to the link:https://console.redhat.com/[{hybrid-console-first}] and select *Services -> Containers -> Create cluster*. +* A cluster that does not already run the {rh-nfd} or the NVIDIA GPU Operator. The pattern installs both Operators itself, with its own `OperatorGroup` objects and settings, and can conflict with existing installations of these Operators. The pattern has not been tested on a cluster that runs {rhoai}. * A storage class that provisions persistent volumes dynamically. The model cache uses one persistent volume claim. * The {ocp-registry} in the `Managed` state, and the `Build` and `ImageRegistry` cluster capabilities enabled. The cluster builds the server image and the notebook image itself. On bare metal and on vSphere, the registry starts in the `Removed` state. To check the registry state, run the following command: + From d98e6932bc7cc665afee9851ed6b88a47ccfa531 Mon Sep 17 00:00:00 2001 From: Ben Date: Thu, 12 Nov 2026 04:06:30 +0000 Subject: [PATCH 5/6] docs(patterns): say one time instead of once in the vLLM Ray Inference Server background --- content/patterns/vllm-ray-inference-server/_index.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/patterns/vllm-ray-inference-server/_index.adoc b/content/patterns/vllm-ray-inference-server/_index.adoc index d90094b4a..9f03307a5 100644 --- a/content/patterns/vllm-ray-inference-server/_index.adoc +++ b/content/patterns/vllm-ray-inference-server/_index.adoc @@ -41,7 +41,7 @@ On {AWS}, the default installation creates one billable GPU instance: a `g6.2xla The {vllm-ray-pattern} does not replace the Red{nbsp}Hat AI products, such as {rhoai}, and it does not integrate with them. The pattern is a small, community-maintained reference point: the upstream vLLM engine on Ray Serve, built from known versions and measured the same way each time. The pattern does not need a {rhoai} subscription. -Your fork of the pattern repository describes the whole deployment: GPU enablement, the server, a model cache, a secret for gated models, and a timing notebook. You run the `./pattern.sh make install` command once. After that, {rh-gitops} syncs each commit on your branch. It does not delete the objects that a commit removes, and it does not change the image of the running server or notebook. A job in the cluster starts new image builds: every 10 minutes, it starts a build when the server or notebook folder changed and no build of that image is running. For details, see link:https://github.com/eugo-inc/vllm-ray-inference-server/blob/main/AGENTS.md#updating-a-running-install[Updating a running install] in the pattern repository. +Your fork of the pattern repository describes the whole deployment: GPU enablement, the server, a model cache, a secret for gated models, and a timing notebook. You run the `./pattern.sh make install` command one time. After that, {rh-gitops} syncs each commit on your branch. It does not delete the objects that a commit removes, and it does not change the image of the running server or notebook. A job in the cluster starts new image builds: every 10 minutes, it starts a build when the server or notebook folder changed and no build of that image is running. For details, see link:https://github.com/eugo-inc/vllm-ray-inference-server/blob/main/AGENTS.md#updating-a-running-install[Updating a running install] in the pattern repository. The pattern pins every Python package by hash and installs wheels only. It pins the base images of the server and the notebook by digest, and the default model by its Hugging Face commit. Some parts are not pinned, among them the system packages that the server image installs with `dnf`, the Operator versions, and the patch versions of the Validated Patterns Helm charts. Two builds from the same commit therefore install the same Python packages, but their system packages can differ. From ad90260cf9bb4a407c3e5f727f5eeb836f1313c7 Mon Sep 17 00:00:00 2001 From: Ben Date: Sat, 14 Nov 2026 20:18:22 +0000 Subject: [PATCH 6/6] docs(patterns): add Eugo's partner results to the vLLM Ray Inference Server page Add a "Partner results: Eugo" section before Support: the start-up time and start-up reliability of Eugo's next vLLM image (release pending) against the upstream vLLM image, and a link to the matching section of the pattern README. --- content/patterns/vllm-ray-inference-server/_index.adoc | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/content/patterns/vllm-ray-inference-server/_index.adoc b/content/patterns/vllm-ray-inference-server/_index.adoc index 9f03307a5..d8b8a088e 100644 --- a/content/patterns/vllm-ray-inference-server/_index.adoc +++ b/content/patterns/vllm-ray-inference-server/_index.adoc @@ -85,6 +85,16 @@ The pattern deploys the following applications: The link:https://github.com/eugo-inc/vllm-ray-inference-server#what-is-in-the-stock-stack[pattern README] lists the exact version of each component. +[id="partner-results-vllm-ray"] +== Partner results: Eugo + +Eugo measured its next vLLM image (release pending) against the upstream `vllm/vllm-openai` v0.33.0 image on one NVIDIA DGX Spark that other jobs kept busy, in paired rounds with an exact sign test. + +* *Time for a new server to serve.* With Qwen2.5-0.5B-Instruct in plain `vllm serve` containers, the software caches empty and the model files already in memory, a new Eugo server served its first token a median of 20.2 seconds (22%) sooner than the upstream image with `--attention-backend FLASHINFER`, in 6 of 6 rounds (p = 0.031). +* *Starts that succeed first time.* The vLLM start-up memory check stopped 4 of 12 upstream starts and none of 28 Eugo starts. Eugo's vLLM carries a fix that mitigates this. + +For details, see the link:https://github.com/eugo-inc/vllm-ray-inference-server#partner-results-eugo[Partner results: Eugo] section of the pattern README. + [id="support-vllm-ray"] == Support