diff --git a/docsrc/user_guide/alpamayo_fp8_edge_exporter.rst b/docsrc/user_guide/alpamayo_fp8_edge_exporter.rst new file mode 100644 index 00000000000..e1179ec07a1 --- /dev/null +++ b/docsrc/user_guide/alpamayo_fp8_edge_exporter.rst @@ -0,0 +1,605 @@ +.. _alpamayo_fp8_edge_exporter: + +Alpamayo 1.5 ModelOpt FP8 Edge Export +===================================== + +This guide walks through the complete workflow for: + +1. quantizing ``nvidia/Alpamayo-1.5-10B`` to FP8 with NVIDIA ModelOpt; +2. checking the quantized checkpoint with Alpamayo evaluation; +3. building the TensorRT-Edge-LLM plugin; +4. exporting Alpamayo's vision, language, and action components to TensorRT; +5. checking eager-versus-TensorRT parity and the generated engine artifacts. + +The Edge exporter produces three engines and a small ``torch.export`` program +that chains them: + +.. code-block:: text + + camera patches + │ + ▼ + vision.engine ──► visual token insertion + DeepStack packing + │ + ▼ + language.engine ──► prefix K/V cache + │ + ▼ + action.engine ──► one diffusion velocity step [batch, 64, 2] + +The export smoke test compiles and checks one action-denoising step. Alpamayo's +autoregressive reasoning loop and Python-controlled diffusion sampler remain +host-side orchestration; they are not represented as one static graph. + +.. warning:: + + Alpamayo and the PhysicalAI dataset are gated. Request access before + starting, and use ``hf auth login``. Do not put Hugging Face tokens directly + in shell commands, documentation, or logs. + +Validated component requirements +-------------------------------- + +Use Python 3.12. The quantization recipe and Edge exporter intentionally use +different environments because their dependency requirements differ. + +.. list-table:: + :header-rows: 1 + :widths: 24 36 40 + + * - Component + - Quantization environment + - Edge export environment + * - PyTorch + - ``2.8.0`` + - Version required by the installed Torch-TensorRT wheel + * - Transformers + - ``4.57.1`` + - ``>=5.4.0`` + * - ModelOpt + - ``0.43.0`` + - ``>=0.44.0`` + * - TensorRT + - Not used to create the checkpoint + - Must match Torch-TensorRT and the Edge-LLM plugin + * - CUDA + - CUDA 12 or 13, matching PyTorch + - Must match the plugin build + +The ModelOpt checkpoint format carries ``modelopt_state.pth`` so the newer +export environment can reconstruct quantized modules when +``mto.enable_huggingface_checkpointing()`` is enabled before +``from_pretrained``. + +Suggested source layout +----------------------- + +The commands below use this layout: + +.. code-block:: text + + /workspace/ + ├── alpamayo-recipes/ + │ └── recipes/alpamayo1_5_quant/ + ├── TensorRT/ + ├── TensorRT-Edge-LLM/ + └── TensorRT-Torch/ + +Set paths once: + +.. code-block:: bash + + export WORKSPACE=/workspace + export ALPAMAYO_RECIPES="$WORKSPACE/alpamayo-recipes" + export TORCH_TRT_ROOT="$WORKSPACE/TensorRT-Torch" + export EDGE_LLM_ROOT="$WORKSPACE/TensorRT-Edge-LLM" + export QUANT_DIR="$ALPAMAYO_RECIPES/recipes/alpamayo1_5_quant" + export QUANT_OUTPUT="$QUANT_DIR/outputs" + export ENGINE_DIR="$WORKSPACE/alpamayo_fp8_edge" + export HF_HOME="$WORKSPACE/.cache/huggingface" + +Use storage with enough space for the base checkpoint, calibration data, +quantized checkpoint, and TensorRT build artifacts. The base model is +approximately 22 GB in BF16; the FP8 parameter payload is approximately 11 GB. +TensorRT compilation also needs temporary host and GPU memory. + +0. Obtain the sources and gated assets +-------------------------------------- + +Clone the required repositories: + +.. code-block:: bash + + git clone https://github.com/NVlabs/alpamayo-recipes.git "$ALPAMAYO_RECIPES" + git clone https://github.com/pytorch/TensorRT.git "$TORCH_TRT_ROOT" + git clone https://github.com/NVIDIA/TensorRT-Edge-LLM.git "$EDGE_LLM_ROOT" + +Check out the Torch-TensorRT revision containing Alpamayo Edge exporter support +when it has not yet landed on your default branch. + +Request access to: + +* `Alpamayo-1.5-10B `_ +* `PhysicalAI-Autonomous-Vehicles + `_ + +Authenticate interactively: + +.. code-block:: bash + + hf auth login + hf auth whoami + +Configure the PhysicalAI dataset paths required by the Alpamayo recipe: + +.. code-block:: bash + + export ALPAMAYO_WORKSPACE="$ALPAMAYO_RECIPES" + export ALPAMAYO_MODEL_DIR="$WORKSPACE/alpamayo_model_converted_from_hf" + export ALPAMAYO_PAI_LOCAL_DIR="$WORKSPACE/PAI_mini" + export ALPAMAYO_LOG_DIR="$WORKSPACE/alpamayo_logs" + +Follow the Alpamayo repository's dataset setup instructions to populate +``ALPAMAYO_PAI_LOCAL_DIR`` and obtain the calibration parquet file referenced by +the recipe. + +1. Create the ModelOpt quantization environment +----------------------------------------------- + +Create the recipe environment with ``uv``: + +.. code-block:: bash + + export UV_CACHE_DIR="$WORKSPACE/.cache/uv" + cd "$QUANT_DIR" + + uv venv am15_quant + source am15_quant/bin/activate + + # Install torch first, then build flash-attn against it. + uv sync --active --no-install-package flash-attn + MAX_JOBS=4 uv sync --active + +Verify the important versions: + +.. code-block:: bash + + python - <<'PY' + import modelopt + import torch + import transformers + + print("torch", torch.__version__) + print("transformers", transformers.__version__) + print("modelopt", modelopt.__version__) + print("cuda available", torch.cuda.is_available()) + PY + +The recipe is defined for Python 3.12, PyTorch 2.8, Transformers 4.57.1, and +ModelOpt 0.43. + +2. Run the ModelOpt Alpamayo FP8 example +---------------------------------------- + +This workflow uses the public +`Alpamayo 1.5 ModelOpt quantization example +`_. +Its ``quantize.py``, ``eval.py``, calibration parquet, and pinned ``uv`` +environment are the source of truth for checkpoint creation and evaluation. +The quantization procedure is also documented separately in +:ref:`alpamayo_modelopt_fp8`. The steps below use the same full-model +PhysicalAI calibration and compressed FP8 output, then continue into +Torch-TensorRT Edge export. + +.. code-block:: bash + + cd "$QUANT_DIR" + source am15_quant/bin/activate + + uv run --active quantize.py \ + --quant_format=fp8 \ + --num_of_calib_clips=100 \ + --save_model_dir=./outputs + +The expected output directory is: + +.. code-block:: text + + outputs/alpamayo1.5_fp8_calib100/ + ├── config.json + ├── modelopt_state.pth + ├── model*.safetensors + └── tokenizer and processor assets + +The calibration loop exercises the full VLM rollout and diffusion path with +the selected PhysicalAI clips. By default, ``quantize.py`` calls +``mtq.compress(model)`` before saving, so this checkpoint contains real FP8 +weights. + +Confirm that ModelOpt state was saved: + +.. code-block:: bash + + export ALPAMAYO_FP8_CKPT="$QUANT_DIR/outputs/alpamayo1.5_fp8_calib100" + + test -f "$ALPAMAYO_FP8_CKPT/config.json" + test -f "$ALPAMAYO_FP8_CKPT/modelopt_state.pth" + ls -lh "$ALPAMAYO_FP8_CKPT" + +For a long-running calibration, use the upstream background command: + +.. code-block:: bash + + nohup uv run --active quantize.py \ + --quant_format=fp8 \ + --num_of_calib_clips=100 \ + --save_model_dir=./outputs \ + > quantize_fp8.log 2>&1 & + +3. Evaluate the quantized checkpoint +------------------------------------ + +Run the official evaluation command: + +.. code-block:: bash + + cd "$QUANT_DIR" + source am15_quant/bin/activate + + uv run --active eval.py \ + --ckpt ./outputs/alpamayo1.5_fp8_calib100 + +For a shorter smoke test before the complete evaluation: + +.. code-block:: bash + + uv run --active eval.py \ + --ckpt "$ALPAMAYO_FP8_CKPT" \ + --limit 10 \ + --num_traj_samples 6 \ + --seed 42 \ + --print_every 1 + +The evaluation should: + +* restore ``modelopt_state.pth`` without missing-module errors; +* print the quantization summary for a non-base checkpoint; +* report finite per-clip minADE values; +* finish with average minADE and average evaluation time. + +Increase ``--limit`` after the smoke test to obtain the customer acceptance +metric. + +4. Prepare the Edge export environment +-------------------------------------- + +Deactivate the quantization environment and activate or create an environment +containing the Torch-TensorRT build used for export: + +.. code-block:: bash + + deactivate 2>/dev/null || true + source /path/to/edge-export/bin/activate + +Install the model and exporter dependencies: + +.. code-block:: bash + + python -m pip install \ + "transformers>=5.4.0" \ + "nvidia-modelopt[hf]>=0.44.0" \ + physical-ai-av \ + einops hydra-core pillow + + python -m pip install \ + "git+https://github.com/NVlabs/alpamayo1.5.git" + +Install a Torch-TensorRT wheel built for this environment, or build/install +Torch-TensorRT from the selected source revision. Confirm that importing +``torch_tensorrt`` resolves to the installed package: + +.. code-block:: bash + + cd "$TORCH_TRT_ROOT" + + # tools/hf contains the experimental exporter. Do not add $TORCH_TRT_ROOT/py + # unless the source tree has been built and generated _version.py. + export PYTHONPATH="$TORCH_TRT_ROOT/tools/hf" + + python - <<'PY' + import modelopt + import torch + import torch_tensorrt + import transformers + + print("torch", torch.__version__) + print("torch_tensorrt", torch_tensorrt.__version__) + print("torch_tensorrt path", torch_tensorrt.__file__) + print("transformers", transformers.__version__) + print("modelopt", modelopt.__version__) + PY + +``torch_tensorrt.__file__`` should point to ``site-packages`` unless you built +the source tree in place. + +5. Build the TensorRT-Edge-LLM plugin +--------------------------------------------- + +The language engine contains TensorRT-Edge-LLM attention plugins. Build the +plugin against the same CUDA and TensorRT major versions used by +Torch-TensorRT. + +Initialize dependencies: + +.. code-block:: bash + + cd "$EDGE_LLM_ROOT" + git submodule update --init --recursive + +Configure an x86 CUDA 13 / TensorRT 11 build: + +.. code-block:: bash + + export CUDA_HOME=/usr/local/cuda-13.0 + export PATH="$CUDA_HOME/bin:$PATH" + + cmake -S . -B build-alpamayo-export \ + -DCMAKE_BUILD_TYPE=Release \ + -DTRT_PACKAGE_DIR=/usr \ + -DCUDA_CTK_VERSION=13.0 \ + -DENABLE_CUTE_DSL=OFF \ + -DBUILD_UNIT_TESTS=OFF \ + -DBUILD_PYTHON_BINDINGS=OFF + + cmake --build build-alpamayo-export \ + --target NvInfer_edgellm_plugin \ + -j"$(nproc)" + +If TensorRT headers and libraries are installed under a separate SDK root, use +that path for ``TRT_PACKAGE_DIR``. It must contain ``include/NvInfer.h`` and a +``lib`` or ``lib64`` directory containing ``libnvinfer.so``. + +Set and verify the plugin: + +.. code-block:: bash + + export EDGE_LLM_PLUGIN_SO="$EDGE_LLM_ROOT/build-alpamayo-export/libNvInfer_edgellm_plugin.so.1.0" + export LD_LIBRARY_PATH="$(dirname "$EDGE_LLM_PLUGIN_SO"):${LD_LIBRARY_PATH:-}" + + test -f "$EDGE_LLM_PLUGIN_SO" + + python - <<'PY' + import ctypes + import os + + path = os.environ["EDGE_LLM_PLUGIN_SO"] + ctypes.CDLL(path) + print("loaded", path) + PY + +Use the CUDA and TensorRT versions available on your target system rather than +copying the example values blindly. + +6. Choose an export sample +-------------------------- + +The exporter uses one PhysicalAI clip to construct realistic image, language, +trajectory-history, mRoPE, DeepStack, and action inputs. Select a clip from the +calibration parquet: + +.. code-block:: bash + + cd "$QUANT_DIR" + source am15_quant/bin/activate + + export ALPAMAYO_CLIP_ID="$( + python - <<'PY' + from alpamayo1_5_quant.utils import read_clip_ids_from_parquet + + clips = read_clip_ids_from_parquet( + "0417_5k_train_set_for_calibration_25.10.parquet" + ) + print(clips[0]) + PY + )" + + echo "$ALPAMAYO_CLIP_ID" + +Return to the Edge export environment before running the exporter. + +7. Export Alpamayo to TensorRT +------------------------------ + +Load the plugin and run the unified exporter: + +.. code-block:: bash + + cd "$TORCH_TRT_ROOT" + source /path/to/edge-export/bin/activate + + export PYTHONPATH="$TORCH_TRT_ROOT/tools/hf" + export EDGE_LLM_PLUGIN_SO="$EDGE_LLM_ROOT/build-alpamayo-export/libNvInfer_edgellm_plugin.so.1.0" + export LD_LIBRARY_PATH="$(dirname "$EDGE_LLM_PLUGIN_SO"):${LD_LIBRARY_PATH:-}" + + rm -rf "$ENGINE_DIR" + + python tools/hf/run_export.py alpamayo \ + --checkpoint "$ALPAMAYO_FP8_CKPT" \ + --clip-id "$ALPAMAYO_CLIP_ID" \ + --t0-us 5100000 \ + --engine-dir "$ENGINE_DIR" \ + --max-seq-len 4096 \ + --dtype float16 \ + --device cuda:0 + +During a successful run, the exporter: + +1. enables ModelOpt Hugging Face checkpoint restoration; +2. loads the Alpamayo checkpoint and sample clip; +3. captures unpatched eager outputs; +4. temporarily patches the original Qwen3-VL and Alpamayo ``forward`` methods; +5. compiles the vision, language, and action engines; +6. restores the original class methods; +7. compares eager and TensorRT component outputs; +8. exports and executes the outer engine-chaining graph. + +TensorRT autotuning can remain silent for several minutes. High CPU or GPU +utilization during this period usually means the build is still progressing. + +8. Validate the result +---------------------- + +The runner prints: + +* the engine mapping; +* runtime input keys; +* eager-versus-TensorRT parity for each component; +* eager and TensorRT timings; +* the final action velocity shape and mean. + +The final tensor should have shape: + +.. code-block:: text + + velocity (1, 64, 2) + +Inspect the output tree: + +.. code-block:: bash + + ls -lh "$ENGINE_DIR"/vision + ls -lh "$ENGINE_DIR"/language + ls -lh "$ENGINE_DIR"/action + +Expected layout: + +.. code-block:: text + + $ENGINE_DIR/ + ├── vision/ + │ ├── visual.engine + │ └── config.json + ├── language/ + │ ├── language.engine + │ └── config.json + └── action/ + ├── action.engine + └── config.json + +The ``config.json`` files record logical input/output names, output shapes, +dtypes, and engine filenames. The outer ``ExportedProgram`` calls these engines +through ``torch.ops.edge_llm.execute_engine``. + +Parity interpretation +^^^^^^^^^^^^^^^^^^^^^ + +Review component parity independently: + +* vision compares merged Qwen3-VL image features; +* language compares the final hidden states from multimodal prefill; +* action compares one velocity prediction with identical noisy actions, + timestep, position IDs, mask, and prefix K/V. + +Do not accept NaN/Inf values. Investigate large errors before measuring +performance. FP8 tolerances are necessarily looser than FP16, but a result that +is effectively uncorrelated with eager output indicates a packing, mRoPE, +DeepStack, mask, or checkpoint-restore mismatch. + +9. Run focused tests +-------------------- + +From the Torch-TensorRT checkout: + +.. code-block:: bash + + cd "$TORCH_TRT_ROOT" + export PYTHONPATH="$TORCH_TRT_ROOT/tools/hf" + + python -m pytest \ + -o addopts='' \ + tools/hf/exporters/tests/test_edge_exporter.py \ + -k alpamayo \ + -q + +These tests cover registration, fixed-grid and action patch wiring, visual-token +packing, and multimodal generation-position extension. They do not replace the +GPU export and parity run. + +End-to-end checklist +-------------------- + +1. [ ] Access granted for the Alpamayo model and PhysicalAI dataset. +2. [ ] Quantization environment uses Python 3.12 and the recipe-pinned packages. +3. [ ] FP8 checkpoint contains ``config.json`` and ``modelopt_state.pth``. +4. [ ] Quantized checkpoint produces finite minADE on a small evaluation subset. +5. [ ] Edge environment imports the intended Torch-TensorRT installation. +6. [ ] Edge environment uses Transformers 5.4 or newer and ModelOpt 0.44 or newer. +7. [ ] Edge-LLM plugin is built against matching CUDA/TensorRT versions. +8. [ ] ``ctypes.CDLL(EDGE_LLM_PLUGIN_SO)`` succeeds. +9. [ ] ``PYTHONPATH`` contains ``tools/hf`` but does not accidentally shadow + the installed Torch-TensorRT package with an unbuilt ``py/`` directory. +10. [ ] A valid PhysicalAI clip ID and ``t0_us`` are selected. +11. [ ] Vision, language, and action engines are written. +12. [ ] Component parity is finite and within the acceptance threshold. +13. [ ] Final velocity output has shape ``[1, 64, 2]``. + +Troubleshooting +--------------- + +.. list-table:: + :header-rows: 1 + :widths: 38 62 + + * - Symptom + - Resolution + * - ``No module named torch_tensorrt._version`` + - An unbuilt source ``py/`` directory is shadowing the installed package. + Set ``PYTHONPATH=$TORCH_TRT_ROOT/tools/hf`` and verify + ``torch_tensorrt.__file__`` points to the intended installation. + * - ``Set EDGE_LLM_PLUGIN_SO ...`` + - Build the Edge-LLM plugin, set the environment variable to the real + ``.so`` file, and add its directory to ``LD_LIBRARY_PATH``. + * - ``Plugin missing`` + - A documentation placeholder was copied literally. Use the actual build + artifact path and check it with ``test -f``. + * - ``undefined symbol`` while loading the plugin + - The plugin and Python environment use different CUDA or TensorRT + versions. Rebuild the plugin against the same TensorRT SDK. + * - ``No module named transformers.exporters`` + - Upgrade the Edge environment to Transformers 5.4 or newer. Do not use + the recipe's Transformers 4.57 environment for Edge export. + * - ModelOpt state is ignored or quantizers are absent + - Ensure ``modelopt_state.pth`` is beside the checkpoint and ModelOpt is + installed. The Alpamayo loader enables ModelOpt checkpointing before + ``from_pretrained``. + * - Real FP8 checkpoint fails to restore or export + - Retry with the ``--fake_quant`` checkpoint first. Real compressed + checkpoint restore is experimental. + * - Dataset access, 401, or missing clip + - Confirm gated access with ``hf auth whoami`` and verify + ``ALPAMAYO_PAI_LOCAL_DIR`` plus the requested clip ID. + * - Image token count does not match visual features + - Use the same processor, image set, ``image_grid_thw``, and checkpoint + tokenizer. Do not mix processor assets from another Alpamayo run. + * - Language parity is poor while vision is good + - Check multimodal position IDs, RoPE delta extension, image-token masks, + and dense DeepStack insertion. + * - Action parity is poor + - Check prefix K/V layer/head layout, non-causal expert mask, timestep + dtype, and action dimensions ``[64, 2]``. + * - CUDA out of memory during compile + - Stop unrelated GPU workloads, use a larger-memory GPU, and keep + checkpoint/cache/build directories off the root filesystem. + * - TensorRT build appears hung + - TensorRT may be autotuning without console output. Check CPU/GPU + utilization before interrupting it. + +Security and cleanup +-------------------- + +Do not export tokens directly in commands that may be captured in shell history. +If a token appears in logs, revoke it immediately and create a replacement. + +Engine files are specific to the TensorRT version, plugin build, GPU +architecture, shapes, and compilation settings used to produce them. Rebuild +the engines after changing any of those inputs. diff --git a/docsrc/user_guide/alpamayo_modelopt_fp8.rst b/docsrc/user_guide/alpamayo_modelopt_fp8.rst new file mode 100644 index 00000000000..e50ff7ff548 --- /dev/null +++ b/docsrc/user_guide/alpamayo_modelopt_fp8.rst @@ -0,0 +1,289 @@ +.. _alpamayo_modelopt_fp8: + +Alpamayo 1.5 ModelOpt FP8 Quantization +====================================== + +This guide follows the public +`Alpamayo 1.5 quantization recipe +`_ +to post-training quantize the complete Alpamayo model with NVIDIA ModelOpt and +evaluate the resulting checkpoint with minADE. + +Unlike an isolated expert smoke test, this workflow: + +* loads the full Alpamayo vision-language-action model; +* calibrates with PhysicalAI driving clips; +* exercises both VLM rollout and diffusion paths; +* compresses fake-quant weights to real FP8 by default; +* saves ``modelopt_state.pth`` with the Hugging Face checkpoint; +* reloads the checkpoint and evaluates trajectory accuracy. + +The commands below intentionally match the upstream recipe. After completing +this guide, continue with :ref:`alpamayo_fp8_edge_exporter` to build TensorRT +engines. + +Prerequisites +------------- + +The upstream recipe is tested with: + +* NVIDIA RTX 5090 with CUDA 12; +* NVIDIA B300 with CUDA 13; +* Python 3.12; +* PyTorch 2.8.0; +* torchvision 0.23.0; +* NVIDIA ModelOpt 0.43.0. + +Request access to: + +* `Alpamayo-1.5-10B `_ +* `PhysicalAI-Autonomous-Vehicles + `_ + +Clone the official recipe: + +.. code-block:: bash + + export YOUR_HOME=/path/to/your/workspace + git clone https://github.com/NVlabs/alpamayo-recipes.git \ + "$YOUR_HOME/alpamayo-recipes" + +1. Create the recipe environment +-------------------------------- + +.. code-block:: bash + + export UV_CACHE_DIR="$YOUR_HOME/.cache/uv" + + cd "$YOUR_HOME/alpamayo-recipes/recipes/alpamayo1_5_quant" + uv venv am15_quant + source am15_quant/bin/activate + + # Install all dependencies except flash-attn first so torch is available. + uv sync --active --no-install-package flash-attn + + # Build flash-attn against the installed torch. + MAX_JOBS=4 uv sync --active + +Verify the pinned environment: + +.. code-block:: bash + + python - <<'PY' + import modelopt + import torch + import torchvision + import transformers + + print("torch", torch.__version__) + print("torchvision", torchvision.__version__) + print("transformers", transformers.__version__) + print("modelopt", modelopt.__version__) + print("cuda available", torch.cuda.is_available()) + PY + +2. Configure model, dataset, and cache paths +----------------------------------------------- + +.. code-block:: bash + + export ALPAMAYO_WORKSPACE="$YOUR_HOME/alpamayo-recipes" + export ALPAMAYO_MODEL_DIR="$YOUR_HOME/alpamayo_model_converted_from_hf" + export ALPAMAYO_PAI_LOCAL_DIR="$YOUR_HOME/PAI_mini" + export ALPAMAYO_LOG_DIR="$YOUR_HOME/alpamayo_logs" + export HF_HOME="$YOUR_HOME/.cache/huggingface" + +``ALPAMAYO_PAI_LOCAL_DIR`` must contain the PhysicalAI data expected by +``load_physical_aiavdataset``. The recipe directory already contains: + +* ``0417_5k_train_set_for_calibration_25.10.parquet`` for calibration; +* ``1005_7cam_gold_eval_metadb_public.parquet`` for evaluation. + +For offline execution after the assets are cached: + +.. code-block:: bash + + export HF_HUB_OFFLINE=1 + export TRANSFORMERS_OFFLINE=1 + +3. Authenticate with Hugging Face +------------------------------------- + +Authenticate interactively so the token is not written into shell history: + +.. code-block:: bash + + hf auth login + hf auth whoami + +4. Quantize the full model to FP8 +------------------------------------- + +Run the upstream FP8 command: + +.. code-block:: bash + + cd "$YOUR_HOME/alpamayo-recipes/recipes/alpamayo1_5_quant" + source am15_quant/bin/activate + + uv run --active quantize.py \ + --quant_format=fp8 \ + --num_of_calib_clips=100 \ + --save_model_dir=./outputs + +The calibration loop loads the requested PhysicalAI clips and calls +``sample_trajectories_from_data_with_vlm_rollout``. This observes activation +ranges through the VLM, expert, action projections, and diffusion-related +paths. + +By default, ``quantize.py`` calls ``mtq.compress(model)`` before saving. The +checkpoint therefore contains real FP8 weights and is expected at: + +.. code-block:: text + + outputs/alpamayo1.5_fp8_calib100/ + ├── config.json + ├── modelopt_state.pth + ├── model*.safetensors + └── tokenizer and processor assets + +Capture logs for a long-running calibration: + +.. code-block:: bash + + nohup uv run --active quantize.py \ + --quant_format=fp8 \ + --num_of_calib_clips=100 \ + --save_model_dir=./outputs \ + > quantize_fp8.log 2>&1 & + + tail -f quantize_fp8.log + +Check the result: + +.. code-block:: bash + + export ALPAMAYO_FP8_CKPT="$PWD/outputs/alpamayo1.5_fp8_calib100" + + test -f "$ALPAMAYO_FP8_CKPT/config.json" + test -f "$ALPAMAYO_FP8_CKPT/modelopt_state.pth" + ls -lh "$ALPAMAYO_FP8_CKPT" + +Fake-quant checkpoint option +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The upstream recipe also supports ``--fake_quant`` for downstream SDK +debugging. It preserves the original weights while saving Q/DQ state: + +.. code-block:: bash + + uv run --active quantize.py \ + --quant_format=fp8 \ + --num_of_calib_clips=100 \ + --save_model_dir=./outputs \ + --fake_quant + +This writes ``alpamayo1.5_fp8_calib100_fakequant``. It is not the default +workflow and does not provide the real-weight memory reduction of the +compressed checkpoint. + +5. Evaluate the FP8 checkpoint +------------------------------ + +Run the official evaluation command: + +.. code-block:: bash + + uv run --active eval.py \ + --ckpt ./outputs/alpamayo1.5_fp8_calib100 + +For a short smoke test before the complete evaluation: + +.. code-block:: bash + + uv run --active eval.py \ + --ckpt ./outputs/alpamayo1.5_fp8_calib100 \ + --limit 10 \ + --num_traj_samples 6 \ + --seed 42 \ + --print_every 1 + +Capture a full evaluation in the background: + +.. code-block:: bash + + nohup uv run --active eval.py \ + --ckpt ./outputs/alpamayo1.5_fp8_calib100 \ + > eval_fp8.log 2>&1 & + + tail -f eval_fp8.log + +A correct run reports: + +* the number of clip IDs loaded from the evaluation parquet; +* per-clip minADE and inference time; +* failed clips, if any; +* average minADE over successful clips; +* average evaluation time per clip. + +Compare the FP8 result with a base-model run using the same settings: + +.. code-block:: bash + + uv run --active eval.py \ + --ckpt nvidia/Alpamayo-1.5-10B \ + --limit 10 \ + --num_traj_samples 6 \ + --seed 42 \ + --print_every 1 + +Keep ``--limit``, ``--num_traj_samples``, and ``--seed`` identical when +comparing checkpoints. + +6. Continue to TensorRT Edge export +----------------------------------- + +The checkpoint consumed by the Edge exporter is: + +.. code-block:: bash + + export ALPAMAYO_FP8_CKPT="$YOUR_HOME/alpamayo-recipes/recipes/alpamayo1_5_quant/outputs/alpamayo1.5_fp8_calib100" + +Continue with :ref:`alpamayo_fp8_edge_exporter`. That guide creates a separate +export environment, restores ``modelopt_state.pth``, builds the Edge-LLM +plugin, compiles the vision/language/action engines, and checks component +parity. + +Troubleshooting +--------------- + +``401`` or gated-repository error +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Confirm access and run ``hf auth login`` again. Do not paste tokens into +commands or logs. + +``No space left on device`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Move ``HF_HOME``, ``UV_CACHE_DIR``, and ``outputs`` to storage with enough +capacity for the base and quantized checkpoints. + +FlashAttention build failure +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Install all other dependencies first, confirm PyTorch imports, then rerun +``MAX_JOBS=4 uv sync --active``. + +ModelOpt state is not restored +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Confirm ``modelopt_state.pth`` is in the checkpoint directory and that the +loading process calls ``mto.enable_huggingface_checkpointing()`` before +``Alpamayo1_5.from_pretrained``. + +Evaluation produces no successful clips +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Verify ``ALPAMAYO_PAI_LOCAL_DIR``, the parquet path, dataset access, and +``t0_us``. Test one known clip directly before starting the full evaluation. diff --git a/docsrc/user_guide/edge_exporter.rst b/docsrc/user_guide/edge_exporter.rst index 74009efe52e..a63f93480ca 100644 --- a/docsrc/user_guide/edge_exporter.rst +++ b/docsrc/user_guide/edge_exporter.rst @@ -20,8 +20,8 @@ difference is what happens inside. Instead of tracing the whole policy as one graph, Edge compiles one TensorRT engine per component, then records a small outer graph that only *calls* those engines. -The code is in ``tools/hf``. The entry points are ``tools/hf/run_pi05_export.py``, -``run_groot_export.py``, and ``run_nemotron_export.py``. +The code is in ``tools/hf``. Use ``tools/hf/run_export.py`` and select a model +family such as ``pi05``, ``groot``, ``nemotron``, or ``alpamayo``. .. note:: @@ -49,6 +49,13 @@ The code is in ``tools/hf``. The entry points are ``tools/hf/run_pi05_export.py` - ``nemotron_h`` - ``language`` - ``nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16`` + * - Alpamayo 1.5 + - ``alpamayo`` + - ``vision``, ``language``, ``action`` + - ``nvidia/Alpamayo-1.5-10B`` or a ModelOpt FP8 checkpoint + +For the complete ModelOpt quantization and export workflow, see +:ref:`alpamayo_fp8_edge_exporter`. Installation ------------ diff --git a/docsrc/user_guide/index.rst b/docsrc/user_guide/index.rst index 09be207dcf6..450e471c6cb 100644 --- a/docsrc/user_guide/index.rst +++ b/docsrc/user_guide/index.rst @@ -9,6 +9,8 @@ Conceptual guides and how-tos for Torch-TensorRT. torch_tensorrt_explained compilation/index edge_exporter + alpamayo_modelopt_fp8 + alpamayo_fp8_edge_exporter shapes_precision/index runtime_performance/index performance_tuning diff --git a/docsrc/user_guide/shapes_precision/quantization.rst b/docsrc/user_guide/shapes_precision/quantization.rst index e703e6f3725..151d2a8e96e 100644 --- a/docsrc/user_guide/shapes_precision/quantization.rst +++ b/docsrc/user_guide/shapes_precision/quantization.rst @@ -26,6 +26,11 @@ Hardware requirements: * **FP8**: NVIDIA Hopper (H100) or newer. * **FP4 (NVFP4)**: NVIDIA Blackwell (B100/B200) or newer; requires TensorRT ≥ 10.8. +For full-model Alpamayo 1.5 quantization with representative PhysicalAI +calibration and minADE evaluation, see :ref:`alpamayo_modelopt_fp8`. To +continue from that checkpoint through TensorRT Edge export, see +:ref:`alpamayo_fp8_edge_exporter`. + ---- INT8 / FP8 PTQ Workflow