[2026.4] Gemma4 native tool calling: parser, guided generation and template hardening

docs/gemma4/.gitattributes

@@ -0,0 +1,4 @@
+# Copyright (c) 2026 Intel Corporation
+# Preserve upstream patch context and raw SSE framing without whitespace rewriting.
+dependencies/*.patch -text whitespace=-blank-at-eol,-blank-at-eof
+evidence/tools/*-response.txt -text whitespace=-blank-at-eol,-blank-at-eof

docs/gemma4/UPSTREAM-HANDOFF.md

@@ -0,0 +1,57 @@
+# Gemma4 2026.4 stack transfer — draft
+
+Source: downstream RC `170644006a5334cb971b05824e4a8c95b495c4e2`. Target: upstream releases/2026/4 at `869b2186a004c6d7eba654db1b03b701bd80757f`.
+
+## Scope
+
+This transfers the custom registry-aware native/JSON tool parser, quoted call boundaries and nested arguments, independent reasoning parser, reasoning-to-tool routing, Google-template/history and rendered-prompt adaptation, auto/required/named guided grammars, parallel_tool_calls validation, opt-in session journal/seed state, and actual terminal streamer finish reasons/incomplete-frame diagnostics. Tool schemas carry max_whitespace_cnt=2 through the typed GenAI API.
+
+Session persistence activates through OVMS_SESSION_STORE_DIR plus X-OVMS-Session-ID. Bodies are journaled on disk with bounded size/cache limits. This storage/API addition and endpoint-wide hard-choice validation need separate upstream design/security review and may need separate PRs.
+
+The upstream logprob fix and all existing upstream Gemma4 parser tests are retained. Windows build-policy changes, other modalities, fork branding and later cache diagnostics/cache-off experiments are excluded. Generic utility helpers remain available for upstream users.
+
+## Dependency blocker
+
+The target GenAI dependency lacks JSONSchema(schema, optional whitespace_bound). Companion draft [GenAI #4477](https://github.com/openvinotoolkit/openvino.genai/pull/4477) carries the API and frozen XGrammar revision, with four native tests. The attached dependency patch has only a license-comment preamble added; it is NOT applied by the upstream build. Review/merge and a coordinated dependency update are required before this OVMS draft can compile. An unbounded fallback would invalidate the repair.
+
+Downstream runtime tuple: OpenVINO `227c33757d1ef95d4da506d00686f923fdd2a535`, GenAI base `7ea2546852a382cd16bd22dea0cfad2db70ed744` plus attached patch, Tokenizers `a04accf6282d9b304214b492694b18c3979f667a`, XGrammar `9aa840b6d16abf094f3e8e2ac9c10465b77656c9`.
+
+[Frozen package and provenance](https://github.com/DassaultFalconKing/gemmamonster_model_server_OVMS/releases/tag/gemmamonster-2026.4-rc-whitespace-17064400).
+
+## Downstream live evidence
+
+2026-09-14 22:44–22:49 UTC; already-running Windows RC, Intel Arc 140V 16GB, driver 32.0.101.8991, GPU/VLM_CB, Gemma4 26B A4B heretic INT4, prefix caching and DEBUG enabled. Concurrent client traffic was not controlled.
+
+| Check on frozen RC | Result |
+|---|---|
+| Non-stream benchmark | 14/14 HTTP 200 |
+| Long requests, curl | 1797 actual tokens / 72.979 s = 24.624 tok/s |
+| Long requests, Invoke-WebRequest | 2186 actual tokens / 86.639 s = 25.231 tok/s |
+| Named single echo | PASS, exact arguments, 1 call, tool_calls finish |
+| Two same-name parallel echo calls | PASS, exact arguments, 2 calls, tool_calls finish |
+| Named SSE echo | PASS, reconstructed arguments, 1 call, tool_calls finish |
+
+Raw synthetic requests/responses and summaries are under evidence. All original traces, including CSV/NDJSON and SSE, are preserved byte-for-byte in evidence/raw-traces.tar.gz; JSON requests/responses remain browsable. Tool cases: temperature=0, seed=170644, max_tokens=256. Benchmark: temperature=1, top_k=64, top_p=0.95, preserved seeds/prompts; timing includes prefill/HTTP/possible queueing. Long outputs stopped before max_tokens and metrics use actual usage. Cold startup, TTFT, concurrency, factual accuracy of free benchmark texts, multi-turn/real-tool execution and session persistence were NOT RUN in this campaign.
+
+These results belong to the frozen downstream RC, NOT the assembled upstream head. The original candidate manifest's historical live acceptance NOT_RUN is not overwritten.
+
+## Merge gates
+
+- Review/land the companion GenAI API and update dependencies coherently.
+- Build product and run the six targets below on this exact PR head.
+- Run every retained upstream parser regression and generic parser/streamer coverage plus Linux/Windows CI; no unsupported exclusions.
+- Review journal filesystem/seed/API behavior and hard-choice policy; split scope if maintainers prefer.
+- Run packaged repeated, streaming, multi-turn/tool-result semantic acceptance with raw evidence on this head.
+
+```text
+//src/test/llm/gemma4_fast:gemma4_parser_contract_test
+//src/test/llm/generation_config:gemma4_generation_contract_test
+//src/test/llm/generation_config:gemma4_prompt_state_generation_contract_test
+//src/test/llm/generation_config:openai_parallel_tool_calls_contract_test
+//src/test/llm/gemma4_overlay:gemma4_chat_template_overlay_contract_test
+//src/test/llm/gemma4_overlay:gemma4_google_jinja_contract_test
+```
+
+Local product build/executable tests on the assembled OVMS head: NOT RUN. Jenkins job 1 reported ERROR on the initial head dd7ac8de89fe76cfe00d04d4f69620e00e5aa37e; a successful complete pipeline on the updated head remains required. License scanner preflight with Linux paths/UTF-8 passed; the three new BUILD files have Apache headers. ownsToolCallBoundaries is present in OutputParsingConfig, avoiding the earlier #4525 missing-field wiring error. These checks do not constitute a product build.
+
+Companion GenAI real-header standalone contracts: original API RED at compilation; transferred API GREEN for legacy/bounded/zero/equality assertions. Full GenAI/native tests remain NOT RUN. This remains draft while gates are open.

docs/gemma4/dependencies/genai-gemma4-bounded-whitespace.patch

@@ -0,0 +1,54 @@
+# Copyright (c) 2026 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+# Frozen dependency diff; comment preamble added for the upstream license scanner.
+diff --git a/src/cpp/CMakeLists.txt b/src/cpp/CMakeLists.txt
+index 463542c9..42bbe6a3 100644
+--- a/src/cpp/CMakeLists.txt
++++ b/src/cpp/CMakeLists.txt
+@@ -152,7 +152,7 @@ if(ANDROID)
+ endif()
+
+ if(ENABLE_XGRAMMAR)
+-    set(XGRAMMAR_VERSION v0.1.31)
++    set(XGRAMMAR_VERSION 9aa840b6d16abf094f3e8e2ac9c10465b77656c9)
+     set(XGRAMMAR_DIR ${CMAKE_BINARY_DIR}/xgrammar)
+
+     FetchContent_Declare(
+diff --git a/src/cpp/include/openvino/genai/generation_config.hpp b/src/cpp/include/openvino/genai/generation_config.hpp
+index 26b230d3..9724c1a7 100644
+--- a/src/cpp/include/openvino/genai/generation_config.hpp
++++ b/src/cpp/include/openvino/genai/generation_config.hpp
+@@ -5,6 +5,7 @@
+
+ #include <filesystem>
+ #include <limits>
++#include <optional>
+ #include <variant>
+ #include <string>
+ #include <sstream>
+@@ -157,17 +158,21 @@ public:
+      */
+     struct JSONSchema {
+         std::string value;
++        std::optional<int> max_whitespace_cnt;
+
+         JSONSchema() = default;
+-        JSONSchema(const std::string& schema) : value(schema) {}
++        JSONSchema(const std::string& schema, std::optional<int> whitespace_bound = std::nullopt)
++            : value(schema), max_whitespace_cnt(whitespace_bound) {}
+         std::string to_string() const {
+-            return "JSONSchema(\"" + value + "\")";
++            return "JSONSchema(\"" + value + "\"" +
++                (max_whitespace_cnt ? ", max_whitespace_cnt=" + std::to_string(*max_whitespace_cnt) : "") + ")";
+         }
+         std::string to_json() const {
+-            return std::string("{\"type\": \"json_schema\", \"json_schema\": ") + value + "}";
++            return std::string("{\"type\": \"json_schema\", \"json_schema\": ") + value +
++                (max_whitespace_cnt ? ", \"max_whitespace_cnt\": " + std::to_string(*max_whitespace_cnt) : "") + "}";
+         }
+         bool operator==(const JSONSchema& other) const {
+-            return value == other.value;
++            return value == other.value && max_whitespace_cnt == other.max_whitespace_cnt;
+         }
+     };
+

docs/gemma4/evidence/benchmark/aggregate.json

@@ -0,0 +1,16 @@
+[
+  {
+    "transport": "curl",
+    "cases": 2,
+    "completion_tokens": 1797,
+    "elapsed_s": 72.979,
+    "weighted_tokens_per_s": 24.624
+  },
+  {
+    "transport": "InvokeWebRequest",
+    "cases": 2,
+    "completion_tokens": 2186,
+    "elapsed_s": 86.639,
+    "weighted_tokens_per_s": 25.231
+  }
+]

docs/gemma4/evidence/benchmark/gpu-512-InvokeWebRequest-request.json

@@ -0,0 +1 @@
+{"model":"gemma4-26-heretic","messages":[{"role":"user","content":"Write a continuous detailed technical explanation of GPU transformer inference optimization. Do not use tools, headings, lists, or a conclusion. Keep expanding the explanation until the token limit."}],"max_tokens":512,"temperature":1.0,"top_k":64,"top_p":0.95,"seed":777}

docs/gemma4/evidence/benchmark/gpu-512-InvokeWebRequest-response.json

@@ -0,0 +1 @@
+{"choices":[{"finish_reason":"length","index":0,"logprobs":null,"message":{"content":"The optimization of transformer inference on a GPU is a multifaceted engineering challenge that necessitates a deep understanding of the interplay between memory bandwidth, compute throughput, and the fundamental mathematical structures of the attention mechanism. At the core of the transformer architecture lies the self-attention mechanism, which scales quadratically with sequence length, creating a significant computational bottleneck. To optimize this, practitioners employ techniques such as FlashAttention, which addresses the memory-bound nature of the attention computation by utilizing tiling and recomputation to minimize the movement of data between the high-bandwidth memory (HBM) and the on-chip SRAM. By restructuring the computation into blocks that fit within the fast SRAM, FlashAttention reduces the number of read/write operations to the slower HBM, effectively transforming a memory-bound operation into a compute-bound one through better utilization of the GPU's streaming multiprocessors. Furthermore, the inference process is divided into two distinct phases: the prefill phase and the decoding phase. The prefill phase involves processing the initial input prompt, which is highly parallelizable and compute-intensive, while the decoding phase involves generating tokens one by one, which is inherently sequential and memory-bound due to the low arithmetic intensity of loading the entire Key-Value (KV) cache for each new token. To mitigate the overhead of the decoding phase, techniques like PagedAttention are employed, which manage the KV cache using a memory management scheme inspired by operating systems, utilizing non-contiguous memory blocks to reduce fragmentation and allow for higher batch sizes. This prevents the \"out of memory\" errors often encountered when using static memory allocation for long sequences. As the model moves from prefill to decoding, the bottleneck shifts from compute-bound to memory-bandwidth-bound, necessitating optimizations like continuous batching, where new requests are interleaved with ongoing generation to maximize GPU utilization without waiting for a full batch to complete. Quantization also plays a critical role in increasing throughput by reducing the precision of the weights and activations. By converting 16-bit floating-point (FP16 or BF16) weights to 8-bit (INT8) or even 4-bit (INT4) integers, one can significantly reduce the memory footprint and the amount of data transferred from HBM to the registers. This reduction in data movement is crucial because, in many modern LLM inference scenarios, the GPU's performance is limited by how fast it can pull data from memory rather than how many floating-point operations it can perform per second. However, low-precision arithmetic requires careful","role":"assistant","tool_calls":[]}}],"created":1789426100,"model":"gemma4-26-heretic","object":"chat.completion","usage":{"prompt_tokens":47,"completion_tokens":512,"total_tokens":559}}

docs/gemma4/evidence/benchmark/gpu-512-curl-request.json

@@ -0,0 +1 @@
+{"model":"gemma4-26-heretic","messages":[{"role":"user","content":"Write a continuous detailed technical explanation of GPU transformer inference optimization. Do not use tools, headings, lists, or a conclusion. Keep expanding the explanation until the token limit."}],"max_tokens":512,"temperature":1.0,"top_k":64,"top_p":0.95,"seed":777}

docs/gemma4/evidence/benchmark/gpu-512-curl-response.json

@@ -0,0 +1 @@
+{"choices":[{"finish_reason":"length","index":0,"logprobs":null,"message":{"content":"The optimization of transformer inference on modern graphics processing units (GPUs) necessitates a multi-faceted approach that addresses the fundamental bottlenecks of memory bandwidth and compute throughput, specifically focusing on the distinct phases of the auto-regressive generation process. During the prefill phase, where the entire input prompt is processed in parallel to compute the initial key and value states, the operation is compute-bound, meaning the primary objective is maximizing the utilization of tensor cores through high-performance matrix multiplications. As the model transitions into the decoding phase, where tokens are generated one by one, the computational profile shifts dramatically toward being memory-bandwidth bound due to the repetitive loading of the existing key-value cache from high-bandwidth memory (HBM) to the much faster but smaller on-chip SRAM. To mitigate this bottleneck, techniques like PagedAttention revolutionize memory management by implementing a virtual memory paging system for the KV cache, allowing for non-contiguous physical memory allocation which eliminates the fragmentation problems inherent in traditional static allocation methods. By utilizing a block-based memory management strategy similar to operating system paging, the system can allocate memory dynamically for the key-value pairs of each sequence, significantly increasing the effective batch size and reducing the waste of VRAM. Parallel to memory management, the optimization of kernel execution involves fusing multiple operations into a single GPU kernel to reduce the overhead of memory round-trips between the global memory and the register files. Kernel fusion is critical when dealing with the activation functions and normalization layers following a projection layer; instead of writing the intermediate result of a linear transformation back to HBM and then reading it back for a subsequent Softmax or LayerNorm, the operations are fused into a single computational unit that keeps the data in the L1 cache or shared memory. Another critical component is the implementation of quantization techniques such as FP8, INT8, or even INT4 precision for the weights and activations. Quantization reduces the memory footprint of the model, allowing larger models to fit within the constraints of a single GPU or enabling larger batch sizes, while simultaneously increasing throughput by reducing the total amount of data that must be moved from HBM to the streaming multiprocessors. However, quantization introduces the risk of precision loss, which is managed through sophisticated scaling factors and techniques like SmoothQuant, which balances the dynamic range between weights and activations to ensure that quantization errors do not accumulate across the deep layers of the transformer architecture. Furthermore, the deployment of speculative decoding introduces a computational strategy where a smaller, faster \"draft\" model predicts a sequence of tokens that are subsequently verified by the","role":"assistant","tool_calls":[]}}],"created":1789426080,"model":"gemma4-26-heretic","object":"chat.completion","usage":{"prompt_tokens":47,"completion_tokens":512,"total_tokens":559}}

docs/gemma4/evidence/benchmark/long-1024-InvokeWebRequest-request.json

@@ -0,0 +1 @@
+{"model":"gemma4-26-heretic","messages":[{"role":"user","content":"Write a detailed continuous technical discussion of local large-language-model inference, memory bandwidth, KV caching, batching, and latency. Do not use tools, headings, bullet lists, or conclude early. Keep expanding naturally until the 1024 token output limit."}],"max_tokens":1024,"temperature":1.0,"top_k":64,"top_p":0.95,"seed":8024}

docs/gemma4/evidence/benchmark/long-1024-InvokeWebRequest-response.json

@@ -0,0 +1 @@
+{"choices":[{"finish_reason":"stop","index":0,"logprobs":null,"message":{"content":"The process of local large language model inference is fundamentally a struggle against the physical limitations of hardware, specifically the dichotomy between computational throughput and memory bandwidth. When running a transformer-based architecture locally, the primary bottleneck is rarely the raw arithmetic capability of the GPU cores—measured in TFLOPS—but rather the speed at which data can be moved from the VRAM to the registers, a metric known as memory bandwidth. During the generation phase of inference, the model operates in an autoregressive fashion, meaning it predicts one token at a time. For each new token produced, the entire set of model weights must be read from the memory and loaded into the processor to perform the matrix-vector multiplications required for the forward pass. Because the arithmetic intensity—the ratio of computations to memory accesses—is relatively low during the generation phase, the GPU spends most of its time waiting for the weights to arrive from the memory bus, making the inference process \"memory-bandwidth bound.\"\n\nAs the sequence length grows, a new phenomenon emerges that complicates this landscape: the KV cache. To avoid the redundant computation of recalculating the attention mechanism for all previous tokens every time a new token is generated, the system stores the Key and Value matrices of all preceding tokens in a dedicated memory buffer known as the KV cache. While this drastically reduces the computational load by turning an $O(n^2)$ problem into something more manageable, it introduces a severe memory footprint issue. The KV cache scales linearly with the sequence length, the batch size, and the number of attention heads. In a local environment with limited VRAM, the KV cache can quickly consume the majority of the available memory, leaving little room for the actual model weights. This creates a direct trade-off between the maximum context window a user can support and the batch size of the inference.\n\nTo increase throughput in a server-side or multi-user context, batching is employed, where multiple independent queries are processed simultaneously in a single forward pass. This increases the arithmetic intensity because a single weight loading operation can be reused for multiple queries, effectively amortizing the cost of memory access across many tokens. However, batching is the enemy of latency. As the batch size increases, the time required to process a single batch increases, leading to higher per-token latency. For a local user, high latency is the primary metric of quality, as it dictates the \"perceived\" speed of the model; if the time between one token being generated and the next exceeds the human reading speed, the experience degrades. This creates a complex optimization problem where one must balance the throughput gains of batching against the latency requirements of a single user.\n\nThe interaction between memory bandwidth, KV cache, and batching reaches a breaking point at high context lengths. When the KV cache grows so large that it spills out of high-speed VRAM into slower system RAM, the performance collapses due to the massive latency penalty of moving data across the PCIe bus. Techniques like PagedAttention, which manages the KV cache in non-contiguous memory blocks similar to virtual memory in an operating system, have emerged to mitigate this by reducing fragmentation and allowing more efficient use of available memory. Furthermore, quantization techniques like 4-bit or 8-bit integer arithmetic serve as a proxy for bandwidth relief. By reducing the precision of the weights and the KV cache, we effectively increase the \"virtual\" bandwidth, allowing more data to be moved per clock cycle and enabling larger batch sizes or longer context windows within the same hardware constraints. The ultimate goal in local inference is finding the \"sweet spot\" where the quantization level is low enough to fit the model and cache into memory, the batch size is high enough to saturate the compute units, but the latency remains low enough to maintain a fluid, real-time interaction. This delicate balance of memory management, precision, and parallelization defines the current state of local LLM deployment.","role":"assistant","tool_calls":[]}}],"created":1789425921,"model":"gemma4-26-heretic","object":"chat.completion","usage":{"prompt_tokens":68,"completion_tokens":789,"total_tokens":857}}