-
Notifications
You must be signed in to change notification settings - Fork 33
Add GStreamer AI_ML test suite #279
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| metadata: | ||
| name: gstreamer-ai-ml | ||
| format: "Lava-Test Test Definition 1.0" | ||
| description: "Validates AI/ML inference pipelines using GStreamer (gst-launch-1.0) with Qualcomm AI RT (QAIRT) runtime components for image classification and object detection use cases." | ||
| os: | ||
| - linux | ||
| scope: | ||
| - functional | ||
|
|
||
| params: | ||
| TIMEOUT: 60 # Timeout in seconds for each pipeline (default in script is 60) | ||
| GST_DEBUG_LEVEL: 2 # GStreamer debug level (default in script is 2) | ||
| OUTPUT_VIDEO_PATH: "./video_out.mp4" # Path for encoded output video | ||
|
|
||
| run: | ||
| steps: | ||
| - REPO_PATH=$PWD | ||
| - cd Runner/suites/Multimedia/GSTreamer/AI_ML/ | ||
| - ./run.sh --timeout "${TIMEOUT}" --gstdebug "${GST_DEBUG_LEVEL}" --output-video "${OUTPUT_VIDEO_PATH}" || true | ||
| - $REPO_PATH/Runner/utils/send-to-lava.sh AI_ML.res || true | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,192 @@ | ||
| # AI_ML — Runner Test | ||
|
|
||
| This directory contains the **AI_ML** validation test for Qualcomm Linux Testkit runners. | ||
|
|
||
| It validates **AI/ML inference pipelines** using **GStreamer (`gst-launch-1.0`)** with Qualcomm AI RT (QAIRT) runtime components for computer vision use cases: | ||
| - **Image Classification** | ||
| Streams a video (`video.mp4`) through **Inception‑v3** TensorFlow‑Lite quantized model. | ||
| The frames are decoded, and the top‑5 class scores are over‑laid on‑frame. The pipeline re‑encodes the video and writes the result to the path supplied via `--output-video`. | ||
|
|
||
| - **Object Detection** \ | ||
| Streams the video (`video.mp4`) through **YoloX** TensorFlow‑Lite quantized model. After decoding, each frame is sent to model and detected bounding boxes are composited onto the video and displayed full‑screen on Wayland socket. | ||
|
|
||
| The script is designed to be **CI/LAVA-friendly**: | ||
| - Writes **PASS/FAIL** into `AI_ML.res` | ||
| - Always **exits 0** (even on FAIL) to avoid terminating LAVA jobs early | ||
| - Logs detailed output for each pipeline test | ||
| - Downloads required AI models automatically when connected to WiFi | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. says downloads happen “when connected to WiFi”, but the implementation is generic network access |
||
|
|
||
| --- | ||
|
|
||
| ## Location in repo | ||
|
|
||
| Expected path: | ||
|
|
||
| ``` | ||
| Runner/suites/Multimedia/GSTreamer/AI_ML/run.sh | ||
| ``` | ||
|
|
||
| Required shared utils (sourced from `Runner/utils` via `init_env`): | ||
| - `functestlib.sh` | ||
| - `lib_gstreamer.sh` | ||
|
|
||
| --- | ||
|
|
||
| ## What this test does | ||
|
|
||
| At a high level, the test: | ||
|
|
||
| 1. Finds and sources `init_env` | ||
| 2. Sources `$TOOLS/functestlib.sh` and `$TOOLS/lib_gstreamer.sh` | ||
| 3. Sets Wayland environment variables for proper display | ||
| 4. Connects to Ethernet network to download required AI models | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. “Connects to Ethernet network”, which the script does not do explicitly |
||
| 5. Defines and validates **2 different AI pipeline tests** covering: | ||
| - Image Classification (Inception-v3) | ||
| - Object Detection (YoloX) | ||
| 6. For each pipeline: | ||
| - Checks if all required GStreamer elements are available | ||
| - Runs the pipeline with a timeout | ||
| - Validates successful execution based on GStreamer state changes | ||
| - Logs detailed output for diagnostics | ||
| 7. Generates a comprehensive report with pass/fail status for each pipeline | ||
| 8. Produces a final PASS/FAIL result for the entire suite | ||
|
|
||
| --- | ||
|
|
||
| ## PASS / FAIL criteria | ||
|
|
||
| ### PASS | ||
| - **All pipelines** successfully run to PLAYING state without critical errors | ||
| - Output file contains `AI_ML PASS` | ||
| - Exit code 0 | ||
|
|
||
| ### FAIL | ||
| - **Any pipeline** fails to reach PLAYING state or encounters critical errors | ||
| - Output file contains `AI_ML FAIL` | ||
| - Exit code 1 | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. FAIL uses exit code 1, but run.sh always exits 0 and uses the .res file as the source of truth |
||
|
|
||
| **Note:** The test always exits `0` even for FAIL. The `.res` file is the source of truth. | ||
|
|
||
| --- | ||
|
|
||
| ## Logs and artifacts | ||
|
|
||
| By default, logs are written relative to the script working directory: | ||
|
|
||
| ``` | ||
| ./AI_ML.res | ||
| ./logs/ | ||
| object_detection_console.log | ||
| object_detection_gst_debug.log | ||
| image_classification_console.log | ||
| image_classification_gst_debug.log | ||
| ``` | ||
|
|
||
| Each pipeline log contains detailed GStreamer debug output that can be examined for failures. | ||
|
|
||
| --- | ||
|
|
||
| ## Dependencies | ||
|
|
||
| ### Required | ||
| - `gst-launch-1.0` | ||
| - `gst-inspect-1.0` | ||
| - `curl`, `unzip` (for model downloads) | ||
| - Internet connectivity (to download models) | ||
| - Proper QAIRT runtime installation (QNN) | ||
|
|
||
| ### Required AI Models (downloaded automatically) | ||
| - `inception_v3_quantized.tflite` | ||
| - `yolox_quantized.tflite` | ||
| - `video.mp4` | ||
| - Labels from `labels.zip` (containing classification and detection labels) | ||
|
|
||
| --- | ||
|
|
||
| ## Pipeline validation behavior | ||
|
|
||
| The test validates pipelines by: | ||
| 1. First checking if all required GStreamer elements are available using `gst-inspect-1.0` | ||
| 2. Running each pipeline with a timeout (default 60 seconds) | ||
| 3. Checking for the "Setting pipeline to PLAYING" message in logs | ||
| 4. Ensuring no critical errors occur after reaching PLAYING state | ||
| 5. Special handling for timeout: if pipeline reached PLAYING before timeout, it's still considered a PASS | ||
|
|
||
| --- | ||
|
|
||
| ## Usage | ||
|
|
||
| Run: | ||
|
|
||
| ``` | ||
| ./run.sh [options] | ||
| ``` | ||
|
|
||
| ### Options | ||
|
|
||
| - `--timeout <seconds>` | ||
| - Sets pipeline execution timeout (default: 60 seconds) | ||
| - `--gstdebug <level>` | ||
| - Sets GStreamer debug level (default: 2) | ||
| - `--output-video <path>` | ||
| - Path for the encoded video generated by the classification pipeline (default: ./video_out.mp4) | ||
|
|
||
| --- | ||
|
|
||
| ## Examples | ||
|
|
||
| ### 1) Basic execution with default timeout | ||
|
|
||
| ``` | ||
| ./run.sh | ||
| ``` | ||
|
|
||
| ### 2) Increase timeout to 120 seconds and set debug level | ||
|
|
||
| ``` | ||
| ./run.sh --timeout 120 --gstdebug 3 | ||
| ``` | ||
|
|
||
| ### 3) Specify custom output video path | ||
|
|
||
| ``` | ||
| ./run.sh --output-video /tmp/custom_video.mp4 | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
| ### A) "FAIL: Test directory not found" | ||
| - Ensure the test is located in the expected directory structure within Runner | ||
|
|
||
| ### B) "Could not find init_env" | ||
| - Verify Runner environment is properly set up | ||
| - Check if `init_env` exists in parent directories | ||
|
|
||
| ### C) Internet connection failures | ||
| - Confirm internet interface is available and functional | ||
| - The script uses `ensure_network_online` which automatically connects to available networks | ||
| - Verify internet connectivity with ping test | ||
|
|
||
| ### D) Pipeline element missing | ||
| - Verify all required GStreamer plugins are installed | ||
| - Check for proper QAIRT GStreamer plugin integration | ||
|
|
||
| ### E) Model download failures | ||
| - Confirm internet connectivity | ||
| - Verify sufficient storage space for model downloads | ||
|
|
||
| --- | ||
|
|
||
| ## Notes for CI / LAVA | ||
|
|
||
| - The test always exits `0` even for FAIL conditions | ||
| - Use the `.res` file for result determination: | ||
| - `AI_ML PASS` | ||
| - `AI_ML FAIL` | ||
| - Requires Internet connectivity to download models | ||
| - Test duration depends on platform performance and number of successful pipelines | ||
| - Each pipeline test has its own timeout to prevent hangs | ||
|
|
||
| --- | ||
Uh oh!
There was an error while loading. Please reload this page.