gemini_context_cache_manager.create gate uses request total tokens, not cacheable prefix tokens

[rannn505]

Summary

GeminiContextCacheManager._create_new_cache_with_contents decides whether to call caches.create() using a token count that represents the full request (system_instruction + tools + all chat history + user turn), but the actual caches.create() call only sends the cacheable prefix (system_instruction + tools + contents[:cache_contents_count]).

This mismatch causes caches.create() to fail with 400 INVALID_ARGUMENT ("The cached content is of tokens. The minimum token count to start caching is 4096.") on every cold-prefix request whose chat history happens to push the full token count over 4096 while the prefix itself stays below.

Worse, on failure the manager returns CacheMetadata(fingerprint=..., contents_count=..., cache_name=None) (no cache_name), so the next request with the same fingerprint hits the same failure path and tries to create again. Infinite create-fail loop, zero cache hits, flood of Failed to create cache: 400 INVALID_ARGUMENT warnings.

Affected version

• google-adk == 1.34.1

Environment

• Model: gemini-3.5-flash on Vertex AI
• ContextCacheConfig(cache_intervals=10, ttl_seconds=1800, min_tokens=4096)
• App-level config (App.context_cache_config)
• ~24h prod observation: hundreds of failures/min, cache_hit_pct ~= 0% across all routed models

Reproduction (code path walkthrough)

1. google/adk/flows/llm_flows/context_cache_processor.py:81

   llm_request.cacheable_contents_token_count = previous_token_count

   previous_token_count is the prior response's usage_metadata.prompt_token_count — i.e. the full prompt the model just saw (incl. chat history + tools + system_instruction + the latest user turn).

2. google/adk/models/gemini_context_cache_manager.py:312-324 (gate inside _create_new_cache_with_contents):

   if cacheable_contents_token_count < ctx_cache_config.min_tokens:
       logger.info("Previous request too small to cache: ...")
       return None
   if cacheable_contents_token_count < _GEMINI_MIN_CACHE_TOKENS:
       logger.info("Token count below Gemini minimum: ...")
       return None

   This uses the full prompt count from step 1 as the gate.

3. google/adk/models/gemini_context_cache_manager.py:372-441 (_create_gemini_cache):

   cache_contents = llm_request.contents[:cache_contents_count]   # line 388
   ...
   cache_config = CreateCachedContentConfig(
       system_instruction=llm_request.config.system_instruction,
       tools=llm_request.config.tools,
       contents=cache_contents,
       ...
   )
   await self.genai_client.aio.caches.create(model=..., config=cache_config)   # line 423

   The actual caches.create() call only sends the prefix slice. The server measures that prefix, finds it < 4096, and rejects with 400.

4. On failure (gemini_context_cache_manager.py:~133), the manager returns:

   CacheMetadata(fingerprint=..., contents_count=cache_contents_count)

   with no cache_name. The next request matches the same fingerprint, sees no cache_name, and goes back through _create_new_cache_with_contents — same gate, same failure.

Observed log spam (production)

WARNING Failed to create cache: 400 INVALID_ARGUMENT.
{'error': {'code': 400, 'message': 'The cached content is of 1820 tokens. The minimum token count to start caching is 4096.', 'status': 'INVALID_ARGUMENT'}}

Token counts in real failures: 1820, 1857, 1991, 1829 — all well below 4096 because the prefix (system_instruction + tools + few-content sub-agents) is small even when the full chat history pushes the gate's measurement above 4096.

Suggested fix

The gate must be measured on the same payload caches.create() will receive. Two options:

1. Estimate prefix tokens at gate time. Compute system_instruction + tools + contents[:cache_contents_count] token count locally and compare against min_tokens / _GEMINI_MIN_CACHE_TOKENS. This avoids the extra count_tokens API round trip if you can use the on-device tokenizer; otherwise one client.models.count_tokens call is cheap relative to letting the request fail at the server.
2. Cache the negative result by fingerprint. Even with the gate fix, transient API rejections still happen; refusing to retry the same fingerprint for ttl_seconds would prevent the infinite-loop log flood.

Either fix on its own breaks the loop; both together would be ideal.

Workaround for users

Set App.context_cache_config = None to disable caching entirely until a fix lands. There is no per-agent disable in 1.34.1 — App.context_cache_config is global (llm_agent.py:266), and the only safe workaround is full disable.

[rannn505]

For anyone bisecting: this is not a 1.34.1 regression. ADK 1.34.0 has the identical bug.

google/adk/models/gemini_context_cache_manager.py and google/adk/flows/llm_flows/context_cache_processor.py are byte-identical between 1.34.0 and 1.34.1:

$ diff 1.34.0/google/adk/models/gemini_context_cache_manager.py \
       1.34.1/google/adk/models/gemini_context_cache_manager.py
(empty)

$ diff 1.34.0/google/adk/flows/llm_flows/context_cache_processor.py \
       1.34.1/google/adk/flows/llm_flows/context_cache_processor.py
(empty)

Both versions ship the _GEMINI_MIN_CACHE_TOKENS = 4096 constant, the cacheable_contents_token_count-based gate, and the contents[:cache_contents_count] prefix slice at caches.create(). The mismatch existed at least since 1.34.0.

Skip the bisect — it's load-bearing in both, fix needed on main.

[surajksharma07]

@rannn505 the gate in _create_new_cache_with_contents uses cacheable_contents_token_count (full prompt tokens) but caches.create() only receives the prefix slice, so the two counts never match on long conversations.

One approach worth trying: subclass GeminiContextCacheManager and override _create_new_cache_with_contents to estimate tokens on contents[:cache_contents_count] + system_instruction + tools (mirroring the existing _estimate_request_tokens pattern at line 339) and gate on that prefix count instead.

Could you give this a try and confirm whether it stops the 400s on your end?

In the meantime, setting App.context_cache_config = None disables caching globally as a quick interim workaround while you test!

[rannn505]

@surajksharma07 thanks for the suggested patch. We tried it in production and have feedback.

What we did

Implemented a FixedGeminiContextCacheManager subclass that overrides _create_new_cache_with_contents to estimate tokens on the prefix-only request (mirroring _estimate_request_tokens at line 339). Conservative gate: only attempt cache when estimated prefix tokens >= max(_GEMINI_MIN_CACHE_TOKENS * 1.5, cache_config.min_tokens) to give headroom against the char/4 heuristic vs Vertex's real tokenization.

Result on the 400 errors

Patch successfully eliminated the 400 INVALID_ARGUMENT spam. The pre-check correctly skips requests that wouldn't make valid caches. ✅

Result on production latency

Had to revert within hours due to latency regression:

• Request avg latency: ~2x slowdown
• p95 hit our 600s task-queue timeout
• Queue stalled (30 processing slots full, growing backlog of DEAD tasks)

Suspected root causes (latency, not the original 400 bug)

1. Every cacheable request makes a caches.create() API roundtrip when the prefix doesn't match an existing cache — adds ~150ms+ per request even when the gate decides "attempt"
2. Cache cleanup race: many 404 NOT_FOUND warnings when ADK tries to delete caches that already expired. Suggests cleanup runs synchronously in the request path
3. OTel side effect: Calling end() on an ended span warnings spiked while caching was active, dropped to 0 after revert. Likely double-span-end from sync cleanup hitting cancelled async context (related: OpenTelemetry contextvars leak across asynchronous generator boundaries in BaseAgent and Runner #5722)

Suggested fix directions

The fundamental issue isn't the gate — it's that cache create + cleanup happen synchronously in the request path. Three directions:

• Memoize cache creation per prefix fingerprint. Reuse cached prefixes within TTL instead of calling caches.create() again for near-identical prefixes seconds apart
• Make cache creation non-blocking. First request returns without waiting for caches.create(); subsequent requests benefit once the cache is ready
• Add a config flag ContextCacheConfig.create_cache: bool = True. Lets operators disable cache creation while keeping the rest of the caching infrastructure (e.g., to use pre-created caches without auto-create overhead)

Happy to test any candidate patch — we have full observability (Cloud Trace + OTel) ready for fast feedback.

— @rannn505 (production deployment of ADK 1.34.1 across multiple internal services)

[rannn505]

Also filed a related feature request for the trace pollution side: #5889

[surajksharma07]

@rannn505 The gate fix confirming the 400s are gone validates the root cause, and the latency regression is a separate, deeper problem worth addressing properly.

The latency is coming from two places that both sit on the synchronous request path in handle_context_caching:

Cleanup round-trip (line 103 in gemini_context_cache_manager.py):

await self.cleanup_cache(old_cache_metadata.cache_name)

Every time a cache is found to be invalid (expired), this fires await self.genai_client.aio.caches.delete(...) synchronously before anything else proceeds.
When Vertex has already expired the cache server-side (TTL elapsed), the delete comes back 404 — but you've still paid the full API round-trip. Changing this line to asyncio.create_task(self.cleanup_cache(...)) fires the delete in the background and removes it from the hot path entirely.
The cleanup_cache method at line 455 already swallows all exceptions, so this is safe to background.

caches.create() latency (~150ms, line 423 in _create_gemini_cache):

cached_content = await self.genai_client.aio.caches.create(model=..., config=cache_config)

This is awaited inside _create_gemini_cache, which is itself awaited from handle_context_caching at line 116.
Every cache-creation attempt blocks the full request path for the duration of the Vertex API call.

For Direction 2 (non-blocking create): maintain a class-level _pending_creates: dict[str, asyncio.Task[Optional[CacheMetadata]]] keyed by fingerprint.
When a cold fingerprint is seen, fire asyncio.create_task(_create_gemini_cache(...)), store the task, and return a fingerprint-only CacheMetadata for this request (cache miss, no blocking).
On the next request with the same fingerprint, check task.done() — if done, retrieve the result; if still in flight, proceed as a cache miss again.
This removes caches.create() from the critical path completely.
The asyncio.create_task() also gives the task a clean copy of the current context, which reduces the surface for the OTel double-end() you're seeing from the #5722 contextvars leak.

For Direction 1 (dedup concurrent creates): if multiple requests hit the same cold fingerprint simultaneously (likely under your 30-slot load), each currently fires its own caches.create() and creates redundant caches.
An asyncio.Event per fingerprint would let the first caller create while others await the event result — preventing the N-parallel-create storm.
This complements Direction 2 rather than replacing it.

For Direction 3 (create_cache: bool = True flag): simplest to land. A one-field addition to ContextCacheConfig and a two-line guard at the top of _create_new_cache_with_contents (line 292):

if not llm_request.cache_config.create_cache:
    return None

This lets you validate the cache-lookup path with pre-created caches before re-enabling auto-create, and gives operators a safe escape hatch independent of the architectural fixes above.

Suggested path: Direction 3 + the create_task cleanup change are the safest, smallest PRs and would give you immediate relief without touching the create flow.
Direction 2 is the right fix for the 150ms issue — worth a PR once the fingerprint lifecycle and task cleanup (expired task entries, TTL-aligned eviction) is worked out.

Happy to look at a draft for any of these; your Cloud Trace setup should make the before/after comparison straightforward!