From bcf77babed2dfbb1cc3a2b02d8ec498d803b208d Mon Sep 17 00:00:00 2001 From: "mintlify[bot]" <109931778+mintlify[bot]@users.noreply.github.com> Date: Tue, 30 Jun 2026 22:46:23 +0000 Subject: [PATCH 1/2] docs: apply style guide fixes to checkpointing page --- performance/checkpointing.mdx | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/performance/checkpointing.mdx b/performance/checkpointing.mdx index 3fb1fee8..395b6d8d 100644 --- a/performance/checkpointing.mdx +++ b/performance/checkpointing.mdx @@ -11,7 +11,7 @@ This is useful for both CPU-only and GPU workloads. For CPU applications, checkp For example, ML and LLM frameworks often load large model weights and compile CUDA kernels at container start time, which can take many seconds or minutes. Loading from a checkpoint that already contains this initialized state can skip most of that delay. -Since this feature is still in beta, please report all issues to the team via our [Discord Community](https://discord.gg/ATj6USmeE2) or via [Email](mailto:support@cerebrium.ai). +Since this feature is still in beta, please report all issues to the team via our [Discord community](https://discord.gg/ATj6USmeE2) or via [email](mailto:support@cerebrium.ai). ## How to use @@ -66,7 +66,7 @@ If checkpoint creation succeeds, subsequent containers restore from that snapsho A checkpoint is tightly coupled to a single deployment. To stop restoring from checkpoints, remove the POST request and redeploy the application. -You can find several implementations in our [Examples repository on Github](https://github.com/CerebriumAI/examples). +You can find several implementations in our [examples repository on GitHub](https://github.com/CerebriumAI/examples). ### vLLM example @@ -111,7 +111,7 @@ engine.wake_up() **Ephemeral filesystem:** Any files written to disk before the checkpoint are not copied to the restored container. Only memory is checkpointed. -**Provider Availability:** Checkpointing is only available on the AWS provider. More coming soon. +**Provider availability:** Checkpointing is only available on the **AWS provider**. More coming soon. ## Platform-specific recommendations @@ -119,4 +119,4 @@ engine.wake_up() vLLM checkpointing support is not complete but is still possible. See [vllm-project/vllm#34303](https://github.com/vllm-project/vllm/issues/34303) and related issues. -The larger the size of the memory checkpoint the slower the restore is. Reduce the size of the snapshot substantially and improve startup times by dropping the KV Cache before checkpoint and recreating it after restore. vLLM has functionality that does this built in as part of [vLLM Sleep Mode](https://docs.vllm.ai/en/latest/features/sleep_mode/). +The larger the memory checkpoint, the slower the restore. To reduce snapshot size and improve startup times, drop the KV cache before checkpointing and recreate it after restore. vLLM has built-in functionality for this as part of [vLLM Sleep Mode](https://docs.vllm.ai/en/latest/features/sleep_mode/). From 94d45a4f484f7b9af07d9ea9b5656d0523e18c80 Mon Sep 17 00:00:00 2001 From: "mintlify[bot]" <109931778+mintlify[bot]@users.noreply.github.com> Date: Wed, 15 Jul 2026 15:50:37 +0000 Subject: [PATCH 2/2] docs: normalize heading case and tighten prose across scoped pages --- container-images/custom-web-servers.mdx | 4 +- .../defining-container-images.mdx | 40 +++++++++---------- deployments/multi-region-deployment.mdx | 10 ++--- endpoints/webhook.mdx | 4 +- endpoints/websockets.mdx | 4 +- migrations/mystic.mdx | 10 ++--- v4/examples/aiVoiceAgents.mdx | 4 +- v4/examples/comfyUI.mdx | 10 ++--- v4/examples/gpt-oss.mdx | 8 ++-- v4/examples/high-throughput-embeddings.mdx | 2 +- v4/examples/mistral-vllm.mdx | 4 +- .../openai-compatible-endpoint-vllm.mdx | 2 +- v4/examples/realtime-voice-agents.mdx | 8 ++-- v4/examples/streaming-falcon-7B.mdx | 6 +-- v4/examples/transcribe-whisper.mdx | 2 +- v4/examples/twilio-voice-agent.mdx | 4 +- 16 files changed, 61 insertions(+), 61 deletions(-) diff --git a/container-images/custom-web-servers.mdx b/container-images/custom-web-servers.mdx index 0eab95e0..e0874989 100644 --- a/container-images/custom-web-servers.mdx +++ b/container-images/custom-web-servers.mdx @@ -5,7 +5,7 @@ description: "Run ASGI/WSGI Python apps on Cerebrium" Cerebrium's default runtime covers most app needs. For more control, use ASGI or WSGI servers through the custom runtime feature - enabling custom authentication, dynamic batching, frontend dashboards, public endpoints, and WebSocket connections. -## Setting Up Custom Servers +## Setting up custom servers A basic FastAPI server running as a custom server on Cerebrium: @@ -58,6 +58,6 @@ The configuration requires three key parameters: The [FastAPI Server Example](https://github.com/CerebriumAI/examples) provides a complete implementation. -## Request Headers +## Request headers Custom web servers receive the Cerebrium run ID in the `X-Request-Id` header on every request. This corresponds to the internal `run_id` and is useful for tracking and debugging. diff --git a/container-images/defining-container-images.mdx b/container-images/defining-container-images.mdx index 1e125f15..5bdd9912 100644 --- a/container-images/defining-container-images.mdx +++ b/container-images/defining-container-images.mdx @@ -12,7 +12,7 @@ Unlike traditional Docker or Kubernetes setups with multiple configuration files Python decorators scatter infrastructure settings throughout code files, making changes risky and reviews difficult. TOML centralizes configuration in one place, making it easier to track changes and maintain consistency. Its hierarchical structure maps naturally to app requirements without the accidental complexity of code-based configuration. -### Getting Started +### Getting started Run `cerebrium init` to create a `cerebrium.toml` file in the project root. Edit it to match the app's requirements. @@ -24,7 +24,7 @@ Run `cerebrium init` to create a `cerebrium.toml` file in the project root. Edit section of your `cerebrium.toml` file. -## Hardware Configuration +## Hardware configuration Configure GPU type and memory allocations in the hardware section: @@ -40,7 +40,7 @@ For detailed hardware specifications see the [toml reference](/toml-reference/to ## Dependency management -### Selecting a Python Version +### Selecting a Python version The Python runtime version forms the foundation of every Cerebrium app. Supported versions: 3.10 to 3.13. Specify the version in the deployment section: @@ -58,7 +58,7 @@ To use a later Python version, please use a [Dockerfile](http://localhost:3000/c the base environment and all Python package installations. -### Adding Python Packages +### Adding Python packages Manage Python dependencies directly in TOML or through requirement files: @@ -83,7 +83,7 @@ pip = "requirements.txt" Cerebrium caches pip packages at the node level - including wheel files and compiled binaries - so subsequent builds only install new or updated packages. This significantly reduces build times. -### Adding APT Packages +### Adding APT packages System-level packages (image-processing libraries, audio codecs, etc.) are declared under `[cerebrium.dependencies.apt]`: @@ -103,7 +103,7 @@ apt = "deps_folder/pkglist.txt" Changes to APT packages trigger a full rebuild of the container image, so builds take longer than when modifying Python packages alone. -### Conda Packages +### Conda packages Conda excels at managing complex system-level Python dependencies, particularly for GPU support and scientific computing: @@ -123,11 +123,11 @@ conda = "conda_pkglist.txt" Like APT packages, Conda packages modify system-level components. Changes trigger a full rebuild. Batch Conda dependency updates together to minimize rebuild time. -## Build Commands +## Build commands The build process includes two command types that execute at different stages during container image creation. -### Pre-build Commands +### Pre-build commands Pre-build commands execute at the start of the build process, before dependency installation. Use them to set up the build environment: @@ -143,7 +143,7 @@ pre_build_commands = [ Common uses: installing build tools, configuring system settings, or preparing the environment for subsequent build steps. -### Shell Commands +### Shell commands Shell commands execute after all dependencies install and the application code copies into the container. This later timing ensures access to the complete environment: @@ -159,11 +159,11 @@ shell_commands = [ Use shell commands for tasks that require the fully configured environment — such as compiling code that depends on installed libraries or downloading resources. -## Custom Docker Base Images +## Custom Docker base images The base image determines the OS foundation for the container. The default Debian slim image works for most Python apps; other validated base images support specific requirements. -### Supported Base Images +### Supported base images Supported base image categories include NVIDIA, Ubuntu, and Python images. @@ -182,7 +182,7 @@ docker_base_image_url = "debian:bookworm-slim" # Default minimal image add only essential components as needed. -#### Public Docker Hub Images with Namespaces +#### Public Docker Hub images with namespaces Public Docker Hub images with a namespace (e.g., `bob/infinity`, `huggingface/transformers`) require a local Docker Hub login, even though the image is public. Cerebrium reads `~/.docker/config.json` to authenticate image pulls. @@ -211,7 +211,7 @@ docker_base_image_url = "bob/infinity:latest" with our build system. -#### Public AWS ECR Images +#### Public AWS ECR images Public ECR images from the `public.ecr.aws` registry work without authentication: @@ -222,11 +222,11 @@ docker_base_image_url = "public.ecr.aws/lambda/python:3.11" However, **private ECR images** require authentication. See [Using Private Docker Registries](/container-images/private-docker-registry) for setup instructions. -## Custom Runtimes +## Custom runtimes Cerebrium's default runtime covers most apps. Custom runtimes provide more control, enabling features like custom authentication, dynamic batching, public endpoints, or WebSocket connections. -### Basic Configuration +### Basic configuration Define a custom runtime by adding the `cerebrium.runtime.custom` section to the configuration: @@ -252,7 +252,7 @@ Key parameters: for a detailed implementation of a FastAPI server that uses a custom runtime. -### Self-Contained Servers +### Self-contained servers Custom runtimes also support apps with built-in servers. For example, deploying a VLLM server requires no Python code: @@ -268,7 +268,7 @@ torch = "latest" vllm = "latest" ``` -### Important Notes +### Important notes - Code is mounted in `/cortex` - adjust paths accordingly. - The port in your entrypoint must match the `port` parameter. @@ -283,11 +283,11 @@ Deploy with `cerebrium deploy -y` - the system automatically detects custom runt The build process follows a sequence that transforms source code into a production-ready container image: -### Stage 1: App Upload +### Stage 1: App upload Code is uploaded to Cerebrium, including all source files, configuration, and additional assets needed for the app. -### Stage 2: Image Creation +### Stage 2: Image creation The system creates a container image through the following sequential steps: @@ -298,6 +298,6 @@ The system creates a container image through the following sequential steps: 5. **Python Code Copy**: The app's source code copies into the container, placing it in the correct directory structure. 6. **Shell Commands Execute**: Finally, any build-time shell commands run to complete the image setup. -### Stage 3: Production Image +### Stage 3: Production image The result is a production-ready container image that contains everything needed to run the app. This image serves as a blueprint for creating individual containers when the app receives requests. diff --git a/deployments/multi-region-deployment.mdx b/deployments/multi-region-deployment.mdx index 4de050d9..0318e1a6 100644 --- a/deployments/multi-region-deployment.mdx +++ b/deployments/multi-region-deployment.mdx @@ -12,13 +12,13 @@ Deploy apps globally across multiple continents to reduce latency through co-loc about features/functionality you would like to see -## Why Use Multi-Region Deployment +## Why use multi-region deployment - **Reduced Latency**: Deploy closer to users for faster response times and better experience with real-time applications like voice agents and LLMs - **Data Residency**: Meet data protection requirements by keeping sensitive data within specific geographic regions to comply with regulations like GDPR and CCPA - **High Availability**: Ensure fault tolerance and continuous service through geographic redundancy, disaster recovery, and load balancing across multiple regions -## Available Regions +## Available regions Cerebrium supports deployment across three major continents with the following regions: @@ -42,7 +42,7 @@ Cerebrium supports deployment across three major continents with the following r region not currently listed. -## CLI Configuration +## CLI configuration Configure the CLI to work with different regions in two ways: @@ -68,7 +68,7 @@ cerebrium download results.json -r eu-north-1 `cerebrium region set` is not modified. -## App Deployment +## App deployment Set the deployment region using the `region` parameter in the `[cerebrium.hardware]` section of `cerebrium.toml`: @@ -85,7 +85,7 @@ memory = 8.0 Pricing varies by region based on local infrastructure costs and availability: -## GPU Availability by Region +## GPU availability by region GPU availability and pricing vary across regions due to infrastructure constraints and local demand: diff --git a/endpoints/webhook.mdx b/endpoints/webhook.mdx index c0632f15..2b588cf6 100644 --- a/endpoints/webhook.mdx +++ b/endpoints/webhook.mdx @@ -14,7 +14,7 @@ curl -X POST https://api.cerebrium.ai/v4/p-xxxxxxxx//run?webhookEndpoi The proxy forwards the response body as a POST request to the specified webhook — no code changes required. Ensure the webhook endpoint accepts POST requests. Webhook forwarding works with both Cortex and Custom runtimes, but only for HTTP requests (not WebSockets). -## Retry Behavior +## Retry behavior Webhook requests are sent asynchronously and do not block the function's response. Failed deliveries are retried automatically: @@ -51,7 +51,7 @@ curl -X POST https://webhook.site/'\ --data '{"smile": "wave"}' ``` -## Webhook Signature Verification +## Webhook signature verification Verify that webhooks originate from Cerebrium using signature verification: diff --git a/endpoints/websockets.mdx b/endpoints/websockets.mdx index 0ea371d3..d49197f6 100644 --- a/endpoints/websockets.mdx +++ b/endpoints/websockets.mdx @@ -37,7 +37,7 @@ Test the WebSocket endpoint using websocat, a command-line WebSocket client: websocat wss://api.cerebrium.ai/v4/p-xxxxxxxx// ``` -## Implementing the WebSocket Endpoint +## Implementing the WebSocket endpoint Example WebSocket endpoint using FastAPI: @@ -54,7 +54,7 @@ async def websocket_endpoint(websocket: WebSocket): await websocket.close() ``` -## Additional Info +## Additional info Client-side Implementation: Handle the WebSocket connection properly on the client, including error handling and reconnection logic. diff --git a/migrations/mystic.mdx b/migrations/mystic.mdx index 66bb56ea..3c167240 100644 --- a/migrations/mystic.mdx +++ b/migrations/mystic.mdx @@ -9,7 +9,7 @@ Mystic AI is sunsetting their services. They were an early pioneer that pushed t It covers converting existing Mystic code (using a stable diffusion example) and configuration to the Cerebrium platform, including deployment optimization for performance and cost efficiency. -## Key Differences +## Key differences Cerebrium helps teams deploy and run models efficiently. The infrastructure is designed for reliable performance: @@ -19,9 +19,9 @@ Cerebrium helps teams deploy and run models efficiently. The infrastructure is d Cerebrium provides precise control over computing resources. Instead of managing entire instances, select the exact CPU, memory, and GPU power needed. Billing is per-second for actual resource usage. Use the [pricing calculator](https://cerebrium.ai/pricing) for cost estimates. -## Migration Process +## Migration process -### 1. Project Setup and Configuration +### 1. Project setup and configuration Install Cerebrium's command-line tool and create the project: @@ -92,7 +92,7 @@ xformers = "latest" ``` -### 2. Code Migration +### 2. Code migration Convert the model implementation. A typical Mystic pipeline: @@ -283,7 +283,7 @@ curl --location 'https://api.cerebrium.ai/v4/p-xxxxxxxx/stable-diffusion/predict The Cerebrium platform provides the tools and support needed for a smooth transition. -## Join the Community +## Join the community Connect with other developers and the Cerebrium team for faster response and issue resolution: diff --git a/v4/examples/aiVoiceAgents.mdx b/v4/examples/aiVoiceAgents.mdx index 32e7bdd5..ac2e10ad 100644 --- a/v4/examples/aiVoiceAgents.mdx +++ b/v4/examples/aiVoiceAgents.mdx @@ -28,7 +28,7 @@ See the [Partner Services page](/partner-services/deepgram) to deploy a Deepgram you must use their API endpoint below. -### LLM Deployment +### LLM deployment The LLM is an OpenAI-compatible Llama-3 endpoint using the vLLM framework. For low TTFT, a quantized version is used (RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8). @@ -413,7 +413,7 @@ def start_bot(room_url: str, token: str = None): The result is a fully functioning AI bot that interacts with users through speech in ~500ms. The next section creates a user interface. -## Creating Meeting Room +## Creating a meeting room Cerebrium runs any Python code, not just AI workloads. The following two functions use the Daily REST API to create a room and temporary token, both valid for 5 minutes. diff --git a/v4/examples/comfyUI.mdx b/v4/examples/comfyUI.mdx index 14e65596..c9e8d066 100644 --- a/v4/examples/comfyUI.mdx +++ b/v4/examples/comfyUI.mdx @@ -12,11 +12,11 @@ ComfyUI is a popular no-code interface for building complex stable diffusion wor Production-scale deployment guidance for ComfyUI is limited. This tutorial covers deploying ComfyUI pipelines on Cerebrium as autoscaling API endpoints with pay-as-you-go compute. Find the full example code [here](https://github.com/CerebriumAI/examples/tree/master/7-image-and-video/1-comfyui). -### Creating your Comfy UI workflow locally +### Creating your ComfyUI workflow locally Create the workflow locally or using a rented GPU from [Lambda Labs](https://lambdalabs.com/). Ensure [ComfyUI is installed](https://github.com/comfyanonymous/ComfyUI#installing) in the local environment. -This tutorial uses Stable Diffusion XL and ControlNet to create custom QR codes. Skip to "Export ComfyUI Workflow" if a workflow already exists. +This tutorial uses Stable Diffusion XL and ControlNet to create custom QR codes. Skip to "Export the ComfyUI workflow" if a workflow already exists. 1. Create the Cerebrium project: `cerebrium init 1-comfyui` 2. Clone the ComfyUI GitHub project inside the project directory: `git clone https://github.com/comfyanonymous/ComfyUI` @@ -29,7 +29,7 @@ This tutorial uses Stable Diffusion XL and ControlNet to create custom QR codes. The default ComfyUI workflow interface appears in this view. Use this locally running instance to build the image generation pipeline. -### Export ComfyUI Workflow +### Export the ComfyUI workflow The example GitHub repository contains a workflow.json file. Click the “Load” button on the right to load the workflow. It should appear populated @@ -43,7 +43,7 @@ To export the workflow in API format, click the gear icon (settings) in the top- A button appears in the right hover panel labeled “Save (API format)”. Use it to save the workflow as “workflow_api.json” -### ComfyUI Application +### ComfyUI application The main code in `main.py` uses the exported ComfyUI API workflow to create an endpoint. The code: @@ -256,7 +256,7 @@ def shutdown_event(): terminate_process() ``` -### Deploy ComfyUI Application +### Deploy the ComfyUI application Before deploying, define the deployment configuration in `cerebrium.toml`. This specifies dependencies, hardware, scaling, and any pre-run scripts: diff --git a/v4/examples/gpt-oss.mdx b/v4/examples/gpt-oss.mdx index 3d535b01..f0088e8f 100644 --- a/v4/examples/gpt-oss.mdx +++ b/v4/examples/gpt-oss.mdx @@ -3,9 +3,9 @@ title: "Serving GPT-OSS with vLLM" description: "Deploy OpenAI's Latest Open Source Model" --- -GPT recently released GPT-OSS ([gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b) and [gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b)) two state-of-the-art open-weight language models that deliver strong real-world performance at low cost. Available under the flexible Apache 2.0 license, these models outperform similarly sized open models on reasoning tasks, demonstrate strong tool use capabilities, and are optimized for efficient deployment on consumer hardware. +GPT recently released GPT-OSS ([gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b) and [gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b)): two state-of-the-art open-weight language models that deliver strong real-world performance at low cost. The models are available under the flexible Apache 2.0 license. They outperform similarly sized open models on reasoning tasks, demonstrate strong tool use capabilities, and are optimized for efficient deployment on consumer hardware. -## What Makes GPT-OSS Special? +## What makes GPT-OSS special? GPT-OSS introduces capabilities that set it apart from other open-source LLMs: @@ -15,8 +15,8 @@ GPT-OSS introduces capabilities that set it apart from other open-source LLMs: - **Harmony Response Format**: Built-in support for structured outputs like chain-of-thought reasoning and tool use. See examples from OpenAI [here](https://cookbook.openai.com/articles/openai-harmony) - Please note that in vLLM, you can only run it on NVIDIA H100, H200, B200 as - well as MI300x, MI325x, MI355x and Radeon AI PRO R9700 as of 6th August 2025 + As of 6th August 2025, vLLM only supports running this model on NVIDIA H100, + H200, and B200, as well as MI300x, MI325x, MI355x, and Radeon AI PRO R9700. This tutorial covers the simplest variation of deploying this model using `vllm serve`. For more control, see the [OpenAI compatible endpoint with vLLM guide](/v4/examples/openai-compatible-endpoint-vllm). diff --git a/v4/examples/high-throughput-embeddings.mdx b/v4/examples/high-throughput-embeddings.mdx index 6e4ffb30..288d340b 100644 --- a/v4/examples/high-throughput-embeddings.mdx +++ b/v4/examples/high-throughput-embeddings.mdx @@ -1,7 +1,7 @@ --- title: "Deploy a High Throughput Server for Embeddings and Reranking" sidebarTitle: "High-Throughput Embeddings Server" -description: "Deploy a a high-throughput, low-latency REST API for serving text-embeddings, reranking models, clip, clap and colpali" +description: "Deploy a high-throughput, low-latency REST API for serving text-embeddings, reranking models, clip, clap and colpali" --- This tutorial covers deploying a high-throughput, low-latency REST API for serving text-embeddings, reranking models, clip, clap, and colpali using the open-source framework diff --git a/v4/examples/mistral-vllm.mdx b/v4/examples/mistral-vllm.mdx index 293297c8..ce5ba4fa 100644 --- a/v4/examples/mistral-vllm.mdx +++ b/v4/examples/mistral-vllm.mdx @@ -51,9 +51,9 @@ class Item(BaseModel): Pydantic handles data validation. The `prompt` parameter is required; others are optional with default values. A missing `prompt` triggers an automatic error message. -## vLLM Implementation +## vLLM implementation -## Model Setup +### Model setup ```python import os diff --git a/v4/examples/openai-compatible-endpoint-vllm.mdx b/v4/examples/openai-compatible-endpoint-vllm.mdx index 8c0bb58a..d847ca62 100644 --- a/v4/examples/openai-compatible-endpoint-vllm.mdx +++ b/v4/examples/openai-compatible-endpoint-vllm.mdx @@ -106,7 +106,7 @@ The function: - Streams results when `stream=True` using async functionality - Returns the complete result at the end if streaming is disabled -## Deploy & Inference +## Deploy and inference To deploy the model use the following command: diff --git a/v4/examples/realtime-voice-agents.mdx b/v4/examples/realtime-voice-agents.mdx index 493de3fa..ca8d6e60 100644 --- a/v4/examples/realtime-voice-agents.mdx +++ b/v4/examples/realtime-voice-agents.mdx @@ -8,7 +8,7 @@ This tutorial creates a real-time voice agent that responds to queries via speec The app uses [PipeCat](https://www.pipecat.ai/), a framework that handles component integration, user interruptions, and audio data processing. The example joins a meeting room with a voice agent using [Daily](https://daily.co) (PipeCat's creators) and deploys on Cerebrium for scaling. -The application has 3–4 parts: +The application has three parts: - A Pipecat agent that acts as the orchestrator - A Deepgram TTS/STT service (requires a Deepgram Enterprise account) @@ -27,11 +27,11 @@ Create a Cerebrium account by signing up [here](https://dashboard.cerebrium.ai/r See the [Partner Services page](/partner-services/deepgram) to deploy a Deepgram service on Cerebrium. - You need a Deepgram Enterprise License to do deploy Deegram on Cerebrium else - you must use their API endpoint below. + You need a Deepgram Enterprise License to deploy Deepgram on Cerebrium. + Otherwise, use their API endpoint below. -### LLM Deployment +### LLM deployment The LLM is an OpenAI-compatible Llama-3 endpoint using the vLLM framework. For low TTFT, a quantized version is used (RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8). diff --git a/v4/examples/streaming-falcon-7B.mdx b/v4/examples/streaming-falcon-7B.mdx index b411a07a..d6f7af9a 100644 --- a/v4/examples/streaming-falcon-7B.mdx +++ b/v4/examples/streaming-falcon-7B.mdx @@ -52,9 +52,9 @@ class Item(BaseModel): Pydantic handles data validation. The `prompt` parameter is required; others are optional with default values. A missing `prompt` triggers an automatic error message. -## Falcon Implementation +## Falcon implementation -## Model Setup +### Model setup ```python from transformers import ( @@ -80,7 +80,7 @@ model = AutoModelForCausalLM.from_pretrained( The tokenizer and model instantiate outside the `predict` function, ensuring model weights load only once at startup. -## Streaming Implementation +### Streaming implementation The `stream` function handles streaming results from the endpoint: diff --git a/v4/examples/transcribe-whisper.mdx b/v4/examples/transcribe-whisper.mdx index 6b0a230b..bc9e8f13 100644 --- a/v4/examples/transcribe-whisper.mdx +++ b/v4/examples/transcribe-whisper.mdx @@ -70,7 +70,7 @@ Pydantic handles data validation. While `audio` and `file_url` are optional para Note: Cerebrium has a 3-minute timeout for each inference request. For long audio files (2+ hours) that take several minutes to process, use a `webhook_endpoint` — a URL where Cerebrium sends a POST request with the function's results. -## Setup Model and inference +## Set up model and inference Import the required packages and load the Whisper model. The model downloads during initial deployment and is automatically cached in persistent storage for subsequent use. Loading the model outside the `predict` function ensures this code only runs on cold start (startup). For warm containers, only the `predict` function executes for inference. diff --git a/v4/examples/twilio-voice-agent.mdx b/v4/examples/twilio-voice-agent.mdx index 3f12338b..a57d5929 100644 --- a/v4/examples/twilio-voice-agent.mdx +++ b/v4/examples/twilio-voice-agent.mdx @@ -119,7 +119,7 @@ Purchase a local number (not toll-free) from the [phone numbers page](https://co Save the changes and proceed to set up the AI agent. -### AI Agent Setup +### AI agent setup Create `bot.py` to set up the AI agent using PipeCat for component integration, interruption handling, and audio processing: @@ -248,7 +248,7 @@ A successful deployment looks like this: Test by calling the Twilio number — the agent responds automatically. -### Scaling Pipecat +### Scaling PipeCat For scaling PipeCat on Cerebrium: