NPU support

[iofirag]

is Docker Model-Runner support using Intel NPU AI Boost hardware?

[ericcurtin]

The team doesn't have this hardware to test and enable, but if you want to help out, first thing to do is make the relevant contributions to the following projects:

https://github.com/ericcurtin/inferrs
https://github.com/ggml-org/llama.cpp

[ericcurtin]

What a machine btw, an NPU and 2 GPUs, that's pretty rare :)

[iofirag]

people over the internet talks about OpenVINO for making theirs NPU works.
it looks like it already supported various npu's.

is it can be a solution?

[iofirag]

i have been tried OpenVino in my machine with some lines of python as gpt says:

requirements.txt:

openvino>=2024.0.0
openvino-genai
huggingface_hub

run_npu.py

from huggingface_hub import snapshot_download
from openvino_genai import LLMPipeline

model_path = snapshot_download(
    repo_id="OpenVINO/TinyLlama-1.1B-Chat-v1.0-int4-ov"
)
pipe = LLMPipeline(model_path, device="NPU")
print(pipe("What is an NPU in simple words?"))

output was:

$ py run_npu.py 
Fetching 15 files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 15/15 [00:00<00:00, 47590.44it/s]
Download complete: : 0.00B [00:00, ?B/s]              An NPU (Neuromorphic Processing Unit) is a type of artificial neural network that uses the principles of biological neural networks to simulate the behavior of biological neural networks. It is a type of artificial intelligence that can learn and make decisions based on data, and it can be used in a wide range of applications, including healthcare, robotics, and autonomous vehicles. An NPU is a complex computer system that combines the principles of biological neural networks with the capabilities of modern computing. It is designed to mimic the behavior of biological neural networks, which are highly complex and adaptive systems that can learn and make decisions based on data.
Download complete: : 0.00B [00:09, ?B/s]

and i have seen it really works in win 11 Task Manager resources usage.

[ericcurtin]

I would say a good first step would be to work with @wine99 @ibelem @ravi9 in:

https://github.com/ggml-org/llama.cpp

until we get to the point that there are released llama.cpp openvino binaries for Windows (I'm assuming you would like to use llama.cpp backend with native Windows, not Linux)