Skip to content

OVMS requiring manual WEIGHTS_PATH #4580

Description

@WizardlyBump17

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

  1. podman run --interactive --tty --publish=1235:8080 --volume=/home/davi/AI/models/:/models/ --device=/dev/dri/renderD128 --user=root docker.io/openvino/model_server:2026.4.0-gpu --rest_port 8080 --model_repository_path /models/ --source_model OpenVINO/Qwen3.6-35B-A3B-int4-ov --target_device GPU --task text_generation --kv_cache_precision u8 --plugin_config '{"OFFLOAD_RATIO": 1}' --max_num_batched_tokens 4000
  2. See error

Expected behavior
WEIGHTS_PATH should not be required to be manually added

Logs

davi@davi:~$ podman run --interactive --tty --publish=1235:8080 --volume=/home/davi/AI/models/:/models/ --device=/dev/dri/renderD128 --user=root docker.io/openvino/model_server:2026.4.0-gpu --rest_port 8080 --model_repository_path /models/ --source_model OpenVINO/Qwen3.6-35B-A3B-int4-ov --target_device GPU --task text_generation --kv_cache_precision u8 --plugin_config '{"OFFLOAD_RATIO": 1}' --max_num_batched_tokens 4000
[2026-09-18 02:41:48.761][1][serving][info][server.cpp:115] OpenVINO Model Server 2026.4.0.869b2186a
[2026-09-18 02:41:48.761][1][serving][info][server.cpp:116] OpenVINO backend 2026.4.0-22959-99c81491cc3-releases/2026/4
[2026-09-18 02:41:48.761][1][serving][info][server.cpp:119] OpenVINO GenAI backend 2026.4.0.0-3407-7ea2546852a
[2026-09-18 02:41:48.761][1][serving][info][pythoninterpretermodule.cpp:37] PythonInterpreterModule starting
[2026-09-18 02:41:48.761][1][serving][info][pythoninterpretermodule.cpp:40] Initializing python interpreter
Python version:
3.12.3 (main, Aug 31 2026, 10:18:26) [GCC 13.3.0]
Python sys.path output:
['', '/ovms/lib/python', '/ovms/python_deps', '/lib/python312.zip', '/lib/python3.12', '/lib/python3.12/lib-dynload']
[2026-09-18 02:41:48.843][1][serving][info][pythoninterpretermodule.cpp:59] PythonInterpreterModule started
[2026-09-18 02:41:48.844][8][modelmanager][info][servable_loading_queue.cpp:89] Started servable loading queue thread
[2026-09-18 02:41:48.947][1][modelmanager][info][modelmanager.cpp:252] Available devices for Open VINO: CPU, GPU
[2026-09-18 02:41:48.947][1][serving][info][modelmanager.cpp:243] Running inside Docker container
[2026-09-18 02:41:48.947][1][serving][info][modelmanager.cpp:244] cpu quota: 0, cpu affinity: 16, max_open_files: 524288
[2026-09-18 02:41:48.947][1][serving][info][capimodule.cpp:40] C-APIModule starting
[2026-09-18 02:41:48.947][1][serving][info][capimodule.cpp:42] C-APIModule started
[2026-09-18 02:41:48.947][1][serving][info][grpcservermodule.cpp:106] GRPCServerModule starting
[2026-09-18 02:41:48.947][1][serving][info][grpcservermodule.cpp:110] GRPCServerModule started
[2026-09-18 02:41:48.947][1][serving][info][grpcservermodule.cpp:111] Port was not set. GRPC server will not be started.
[2026-09-18 02:41:48.947][1][serving][info][httpservermodule.cpp:35] HTTPServerModule starting
[2026-09-18 02:41:48.947][1][serving][info][httpservermodule.cpp:39] Will start 16 REST workers
[2026-09-18 02:41:48.948][10][serving][info][drogon_http_server.cpp:157] Binding REST server to address: 0.0.0.0:8080
[2026-09-18 02:41:48.998][1][serving][info][drogon_http_server.cpp:184] REST server listening on port 8080 with 16 unary threads and 16 streaming threads
[2026-09-18 02:41:48.998][1][serving][info][http_server.cpp:242] API key not provided via --api_key_file or API_KEY environment variable. Authentication will be disabled.
[2026-09-18 02:41:48.998][1][serving][info][httpservermodule.cpp:52] HTTPServerModule started
[2026-09-18 02:41:48.998][1][serving][info][httpservermodule.cpp:53] Started REST server at 0.0.0.0:8080
[INFO] LFS resume: attempts=5 interval=10 s
[2026-09-18 02:41:49.049][1][serving][info][libgit2.cpp:1342] Path "/models/OpenVINO/Qwen3.6-35B-A3B-int4-ov" exists but is not a git repository. Skipping download and using existing files.
Model: OpenVINO/Qwen3.6-35B-A3B-int4-ov downloaded to: /models/OpenVINO/Qwen3.6-35B-A3B-int4-ov
Graph: graph.pbtxt created in: /models/OpenVINO/Qwen3.6-35B-A3B-int4-ov
[2026-09-18 02:41:49.051][1][serving][info][servablemanagermodule.cpp:51] ServableManagerModule starting
[2026-09-18 02:41:49.052][8][serving][info][mediapipegraphdefinition.cpp:643] MediapipeGraphDefinition initializing graph nodes
[2026-09-18 02:41:49.052][8][modelmanager][info][servable_initializer.cpp:688] Initializing Visual Language Model Continuous Batching servable
[2026-09-18 02:41:56.687][8][serving][error][servable_initializer.cpp:284] Error during llm node initialization for models_path: /models/OpenVINO/Qwen3.6-35B-A3B-int4-ov/./ exception: Exception from src/inference/src/cpp/core.cpp:117:
Exception from src/inference/src/dev/plugin.cpp:54:
Check '!weights_path.empty()' failed at src/plugins/intel_gpu/src/plugin/ops/moe.cpp:55:
ov::weights_path property is not set. OTD requires a valid path to the model .bin file. Please set ov::weights_path when compiling the model.



[2026-09-18 02:41:56.687][8][modelmanager][error][servable_initializer.cpp:693] Error during LLM node resources initialization: The LLM Node resource initialization failed
[2026-09-18 02:41:56.687][8][serving][error][llm_node_initializer.cpp:60] Failed to process LLM node graph OpenVINO/Qwen3.6-35B-A3B-int4-ov
[2026-09-18 02:41:56.687][8][modelmanager][info][pipelinedefinitionstatus.hpp:60] Mediapipe: OpenVINO/Qwen3.6-35B-A3B-int4-ov state changed to: LOADING_PRECONDITION_FAILED after handling: ValidationFailedEvent: 
[2026-09-18 02:41:56.687][1][modelmanager][error][modelmanager.cpp:286] Couldn't start model manager
[2026-09-18 02:41:56.687][1][serving][error][servablemanagermodule.cpp:58] ovms::ModelManager::Start() Error: The LLM Node resource initialization failed
[2026-09-18 02:41:56.687][1][serving][info][grpcservermodule.cpp:195] GRPCServerModule shutting down
[2026-09-18 02:41:56.687][1][serving][info][grpcservermodule.cpp:205] GRPCServerModule shutdown
[2026-09-18 02:41:56.687][1][serving][info][httpservermodule.cpp:59] HTTPServerModule shutting down
[2026-09-18 02:41:56.688][1][serving][info][httpservermodule.cpp:64] Shutdown HTTP server
[2026-09-18 02:41:56.688][1][serving][info][servablemanagermodule.cpp:65] ServableManagerModule shutting down
[2026-09-18 02:41:56.688][8][modelmanager][info][servable_loading_queue.cpp:113] Stopped servable loading queue thread
[2026-09-18 02:41:56.693][1][serving][info][servablemanagermodule.cpp:71] ServableManagerModule shutdown
[2026-09-18 02:41:56.693][1][serving][info][pythoninterpretermodule.cpp:73] PythonInterpreterModule shutting down
[2026-09-18 02:41:56.693][1][serving][info][pythoninterpretermodule.cpp:77] PythonInterpreterModule shutdown
[2026-09-18 02:41:56.701][1][serving][info][capimodule.cpp:50] C-APIModule shutting down
[2026-09-18 02:41:56.701][1][serving][info][capimodule.cpp:52] C-APIModule shutdown

Configuration
Ryzen 7 5700X3D, B580

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions