Add video frames input support via video_url in chat/completions - #4583
Open
chenrui7019 wants to merge 1 commit into
Open
chenrui7019 wants to merge 1 commit into
chenrui7019 wants to merge 1 commit into
Conversation
Extract frames client-side and send as video_url content part with an
array of frame references. Server decodes frames via existing stb_image
(no new dependency), stacks them into an {N,H,W,C} tensor and forwards
to GenAI VLM pipelines (continuous batching and legacy).
- video_utils: loadVideoFrames() reuses loadImage(), validates uniform
frame dimensions, MAX_VIDEO_FRAMES cap
- VideoFramesProcessor: injection guard, decodes video_url frame array,
rewrites to <ov_genai_video_N> tag before chat template
- openai_completions: validate video_url.url as non-empty string array
- wire VideoFramesProcessor into isVLM input processor chain
- pass inputVideos to CB add_request and legacy generate overloads
- unit + integration tests, docs for video_url
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The current frame cap permits excessive memory use and potential server exhaustion; the decoded-byte or megapixel budget must be addressed.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Adds video_url frame-array support to VLM chat completions, including decoding, validation, tensor forwarding, tests, and documentation.
Changes:
- Added video frame loading, validation, tagging, and security checks.
- Wired video tensors through legacy and continuous-batching pipelines.
- Added unit/integration tests and REST API documentation.
File summaries
| File | Summary |
|---|---|
src/test/llm/input_processing/video_frames_processor_test.cpp |
Video processor unit tests |
src/test/llm/input_processing/input_processing_integration_test.cpp |
Input-processing integration tests |
src/llm/visual_language_model/legacy/legacy_executor.cpp |
Legacy video forwarding |
src/llm/visual_language_model/continuous_batching/servable.cpp |
Continuous-batching video forwarding |
src/llm/io_processing/video_utils.hpp |
Video frame loading API |
src/llm/io_processing/video_utils.cpp |
Frame decoding and tensor stacking |
src/llm/io_processing/input_processors/video_frames_processor.hpp |
Video processor interface |
src/llm/io_processing/input_processors/video_frames_processor.cpp |
Video decoding and prompt tagging |
src/llm/io_processing/input_processor.cpp |
Processor-chain integration |
src/llm/BUILD |
Build target updates |
src/llm/apis/openai_completions.cpp |
video_url validation |
docs/model_server_rest_api_chat.md |
API documentation |
Review details
Suppressed comments (1)
src/test/llm/input_processing/video_frames_processor_test.cpp:16
- This new test uses
std::moveinmakeChatRequest, but the file does not include<utility>. It currently depends on a transitive include from the GenAI headers, which is not guaranteed and can break when those headers change; include the standard header directly.
#include <string>
- Files reviewed: 12/12 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+34
to
+39
| if (static_cast<int64_t>(frameSources.size()) > MAX_VIDEO_FRAMES) { | ||
| return absl::InvalidArgumentError("Number of video frames exceeds the allowed maximum of " + std::to_string(MAX_VIDEO_FRAMES)); | ||
| } | ||
|
|
||
| std::vector<ov::Tensor> frames; | ||
| frames.reserve(frameSources.size()); |
Comment on lines
+26
to
+29
| // Decodes video_url content entries from ChatHistory messages into video tensors | ||
| // and rewrites each video_url part into a text part carrying an | ||
| // <ov_genai_video_N> tag. Runs before ImageDecodingProcessor, which then flattens | ||
| // the message content (text + image tags + video tags) into a single string. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extract frames client-side and send them as a
video_urlcontent partcontaining an array of frame references. The server decodes each frame via the
existing stb_image path (no new dependency), stacks them into an
{N, H, W, C}tensor, and forwards it to the GenAI VLM pipelines (continuous batching and legacy).
video_utils:loadVideoFrames()reusesloadImage(), validates uniformframe dimensions, enforces a
MAX_VIDEO_FRAMEScapVideoFramesProcessor: injection guard, decodes thevideo_urlframe array,rewrites each entry to an
<ov_genai_video_N>tag before chat template renderingopenai_completions: validatesvideo_url.urlas a non-empty array of stringsVideoFramesProcessorinto theisVLMinput processor chaininputVideosto the CBadd_requestand legacygenerateoverloadsvideo_urldocumentation🛠 Summary
Adds video input support to the
chat/completionsendpoint via a newvideo_urlcontent type. Frames are extracted on the client (mirroring the GenAI
video_to_text_chat.pyapproach) and passed as an array; the server reuses theexisting image decoding path, so no video codec / FFmpeg dependency is introduced.
Reuses the existing
allowedLocalMediaPath/allowedMediaDomainsmediawhitelists for per-frame validation.
🧪 Checklist
``