fix: [#1009] activate Snowflake Cortex prompt caching for Claude models

CHANGELOG.md

@@ -5,6 +5,20 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+### Added
+
+- **Snowflake Cortex: `claude-sonnet-5`, `claude-opus-4-8`, and OpenAI GPT-5.4 (`openai-gpt-5.4`, `-mini`, `-nano`) in the model picker.** All verified live against Cortex, including prompt caching and tool calling on the new Claude models.
+
+### Changed
+
+- **Snowflake Cortex model catalog refreshed against the live service (2026-07-20).** Removed models Snowflake has deprecated (July 8, 2026: `deepseek-r1`, `mistral-large`, `llama3.1-405b`, `snowflake-llama-3.3-70b`) or delisted (`claude-3-7-sonnet`, `claude-3-5-sonnet`, `openai-gpt-5-chat`, `llama4-scout`, `mixtral-8x7b`, `snowflake-llama-3.1-405b`, `gemini-3.1-pro`) — requests to these now hard-fail on Cortex. Locally registered models via `altimate-code.json` are unaffected.
+
+### Fixed
+
+- **Snowflake Cortex prompt caching now activates for Claude models.** Cortex only honors caching markers placed inside content blocks (`messages[].content[].cache_control`), but requests carried them as message-level fields — so every request billed the full input rate (`cache_read_input`/`cache_write_input` stayed NULL in `TOKENS_GRANULAR`). The provider now relocates the markers into content blocks (system prompt + trailing messages, max 4 breakpoints), cutting repeated-prefix input cost by up to 90% on long agent sessions. If a Cortex account rejects the marked shape, the request is retried once without markers and marker injection pauses for a 5-minute cooldown. (#1009)
+
 ## [0.9.1] - 2026-07-08
 
 This release rebases altimate-code onto **upstream OpenCode v1.17.9** (bridged up from v1.4.0 — ~165 upstream commits) behind the fork's own fixes and hardening. It is a larger-than-usual jump from 0.8.10; the upgrade is automatic and in-place (see **Upgrading from 0.8.10** below).

docs/docs/configure/providers.md

@@ -276,15 +276,18 @@ Create a PAT in Snowsight: **Admin > Security > Programmatic Access Tokens**.
 
 Billing flows through your Snowflake credits — no per-token costs.
 
-**Available models:**
+Prompt caching is applied automatically for Claude models: cache markers are placed on the system prompt and trailing messages, so repeated context in long sessions is billed at Snowflake's cached-input rate (a 90% discount on cache reads, 5-minute TTL). Cache activity appears as `cache_read_input`/`cache_write_input` in Snowflake's `CORTEX_FUNCTIONS_QUERY_USAGE_HISTORY`/`TOKENS_GRANULAR` telemetry. OpenAI models are cached automatically by Cortex itself; other model families don't support caching.
+
+**Available models** (catalog verified live against Cortex on 2026-07-20):
 
 | Model | Tool Calling |
 |-------|-------------|
-| `claude-opus-4-7`, `claude-sonnet-4-6`, `claude-opus-4-6`, `claude-sonnet-4-5`, `claude-opus-4-5`, `claude-haiku-4-5`, `claude-4-sonnet`, `claude-3-7-sonnet`, `claude-3-5-sonnet` | Yes |
-| `openai-gpt-4.1`, `openai-gpt-5`, `openai-gpt-5.1`, `openai-gpt-5.2`, `openai-gpt-5-mini`, `openai-gpt-5-nano`, `openai-gpt-5-chat` | Yes |
-| `llama4-maverick`, `llama4-scout`, `llama3.3-70b`, `snowflake-llama-3.3-70b`, `snowflake-llama-3.1-405b`, `llama3.1-70b`, `llama3.1-405b`, `llama3.1-8b` | No |
-| `mistral-large`, `mistral-large2`, `mistral-7b`, `mixtral-8x7b` | No |
-| `deepseek-r1`, `gemini-3.1-pro` | No |
+| `claude-sonnet-5`, `claude-opus-4-8`, `claude-opus-4-7`, `claude-sonnet-4-6`, `claude-opus-4-6`, `claude-sonnet-4-5`, `claude-opus-4-5`, `claude-haiku-4-5`, `claude-4-sonnet` | Yes |
+| `openai-gpt-4.1`, `openai-gpt-5`, `openai-gpt-5.1`, `openai-gpt-5.2`, `openai-gpt-5.4`, `openai-gpt-5.4-mini`, `openai-gpt-5.4-nano`, `openai-gpt-5-mini`, `openai-gpt-5-nano` | Yes |
+| `llama4-maverick`, `llama3.3-70b`, `llama3.1-70b`, `llama3.1-8b` | No |
+| `mistral-large2`, `mistral-7b` | No |
+
+Snowflake deprecated `deepseek-r1`, `mistral-large`, `llama3.1-405b`, and `snowflake-llama-3.3-70b` on July 8, 2026, and has delisted `claude-3-7-sonnet`, `claude-3-5-sonnet`, `openai-gpt-5-chat`, `llama4-scout`, `mixtral-8x7b`, and `gemini-3.1-pro` — requests to these now fail.
 
 !!! note
     Model availability depends on your Snowflake region. Enable cross-region inference with `ALTER ACCOUNT SET CORTEX_ENABLED_CROSS_REGION = 'ANY_REGION'` for full model access.
@@ -309,7 +312,7 @@ Snowflake Cortex adds models faster than this list can be updated. If a model is
 }
 ```
 
-The entry merges with the built-in list, so the model appears in the picker and can be selected as `snowflake-cortex/your-new-model-id`. Set `"tool_call": false` for models that don't support tools on Cortex (Llama, Mistral, DeepSeek, Gemini today) — otherwise requests with tools will fail.
+The entry merges with the built-in list, so the model appears in the picker and can be selected as `snowflake-cortex/your-new-model-id`. Set `"tool_call": false` for models that don't support tools on Cortex (Llama and Mistral today) — otherwise requests with tools will fail.
 
 The `tool_call` field uses snake_case (matching the rest of the `altimate-code.json` schema) and maps to the picker's `capabilities.toolcall`. The request transform reads the same value, so a user-added model marked `tool_call: true` keeps `tools` and `tool_choice` in outgoing requests — and one marked `tool_call: false` has them stripped, the same as the built-in non-tool entries.
 

packages/opencode/src/altimate/plugin/snowflake.ts

@@ -1,5 +1,8 @@
 import type { Hooks, PluginInput } from "@opencode-ai/plugin"
 import { Auth, OAUTH_DUMMY_KEY } from "@/auth"
+import { Log } from "@/altimate/util/log"
+
+const log = Log.create({ service: "snowflake-cortex" })
 
 /**
  * Build the set of Snowflake Cortex model IDs that support tool calling.
@@ -40,6 +43,137 @@ export function buildToolCapableSet(
 /** Snowflake account identifiers contain only alphanumeric, hyphen, underscore, and dot characters. */
 export const VALID_ACCOUNT_RE = /^[a-zA-Z0-9._-]+$/
 
+/** Snowflake Cortex accepts at most 4 cache breakpoints per request (Anthropic limit). */
+const CACHE_BREAKPOINT_LIMIT = 4
+
+/**
+ * How long to stop injecting cache markers after Cortex rejects a cache-marked
+ * request. Matches Cortex's 5-minute ephemeral cache TTL: by the time the
+ * cooldown expires the cache would be cold anyway, so re-trying markers costs
+ * almost nothing — while a false trip (transient 400 whose stripped retry
+ * happened to succeed) self-heals instead of silently billing the whole
+ * session at the uncached rate.
+ */
+const CACHE_DISABLE_COOLDOWN_MS = 5 * 60 * 1000
+
+// `any` is deliberate: these guards navigate arbitrary request-body JSON whose
+// message/block properties are read and mutated without a fixed schema.
+function isRecord(value: unknown): value is Record<string, any> {
+  return typeof value === "object" && value !== null && !Array.isArray(value)
+}
+
+/** Roles whose content-block cache markers Cortex honors (verified live 2026-07-20). */
+const CACHEABLE_ROLES = new Set(["system", "user", "assistant"])
+
+/**
+ * Attach a cache marker to the last non-empty content block of a message.
+ * String content is wrapped into a text block. Returns false when the message
+ * has no block that can carry a marker (e.g. empty content).
+ */
+function attachMarker(msg: Record<string, any>, marker: Record<string, any>): boolean {
+  if (typeof msg.content === "string") {
+    if (msg.content.length === 0) return false
+    msg.content = [{ type: "text", text: msg.content, cache_control: marker }]
+    return true
+  }
+  if (!Array.isArray(msg.content)) return false
+  for (let i = msg.content.length - 1; i >= 0; i--) {
+    const block = msg.content[i]
+    if (!isRecord(block)) continue
+    if (block.type === "text" && (typeof block.text !== "string" || block.text.length === 0)) continue
+    if (!isRecord(block.cache_control)) block.cache_control = marker
+    return true
+  }
+  return false
+}
+
+/**
+ * Relocate prompt-cache markers into content blocks for Claude models.
+ *
+ * Cortex only honors caching via `messages[].content[].cache_control`
+ * (ephemeral, max 4 breakpoints) — and only on system/user/assistant messages;
+ * markers on `role:"tool"` messages are accepted but silently ignored
+ * (verified against a live Cortex account). The OpenAI-compatible AI SDK
+ * serializes our cache providerOptions as message-level fields — on system
+ * messages, on single-text-part user messages (collapsed to string content),
+ * on tool messages, and into assistant `tool_calls` entries — all of which
+ * Cortex ignores, so every request bills the full input rate.
+ *
+ * Message-level markers move into that message's content blocks; a marker on
+ * a tool message (or a message with no attachable block) walks back to the
+ * nearest earlier cacheable message, keeping the breakpoint as close to the
+ * end of the conversation as Cortex allows.
+ *
+ * Returns true when the request carries block-level markers after relocation.
+ */
+export function relocateCacheControl(parsed: Record<string, any>): boolean {
+  // Every Claude model Cortex serves uses a `claude-` prefixed ID; a prefix
+  // match avoids false positives on user-registered aliases that merely
+  // contain the substring.
+  if (typeof parsed.model !== "string" || !parsed.model.startsWith("claude-")) return false
+  if (!Array.isArray(parsed.messages)) return false
+  const messages = parsed.messages
+
+  for (let idx = 0; idx < messages.length; idx++) {
+    const msg = messages[idx]
+    if (!isRecord(msg)) continue
+
+    // The SDK spreads part metadata into tool_calls entries — invalid location.
+    if (Array.isArray(msg.tool_calls)) {
+      for (const call of msg.tool_calls) {
+        if (isRecord(call)) delete call.cache_control
+      }
+    }
+
+    const marker = isRecord(msg.cache_control) ? msg.cache_control : undefined
+    delete msg.cache_control
+    if (!marker) continue
+
+    for (let t = idx; t >= 0; t--) {
+      const target = messages[t]
+      if (!isRecord(target) || !CACHEABLE_ROLES.has(target.role)) continue
+      if (attachMarker(target, marker)) break
+    }
+  }
+
+  // Enforce the breakpoint limit, keeping the last markers (longest prefixes).
+  // Markers on non-cacheable roles are dead weight against the limit — drop them.
+  const marked: Record<string, any>[] = []
+  for (const msg of messages) {
+    if (!isRecord(msg) || !Array.isArray(msg.content)) continue
+    for (const block of msg.content) {
+      if (!isRecord(block) || !block.cache_control) continue
+      if (!CACHEABLE_ROLES.has(msg.role)) {
+        delete block.cache_control
+        continue
+      }
+      marked.push(block)
+    }
+  }
+  for (const block of marked.slice(0, Math.max(0, marked.length - CACHE_BREAKPOINT_LIMIT))) {
+    delete block.cache_control
+  }
+
+  return marked.length > 0
+}
+
+function stripParsedCacheControl(parsed: Record<string, any>) {
+  if (!Array.isArray(parsed.messages)) return
+  for (const msg of parsed.messages) {
+    if (!isRecord(msg)) continue
+    delete msg.cache_control
+    if (Array.isArray(msg.tool_calls)) {
+      for (const call of msg.tool_calls) {
+        if (isRecord(call)) delete call.cache_control
+      }
+    }
+    if (!Array.isArray(msg.content)) continue
+    for (const block of msg.content) {
+      if (isRecord(block)) delete block.cache_control
+    }
+  }
+}
+
 /** Parse a `account::token` PAT credential string. */
 export function parseSnowflakePAT(code: string): { account: string; token: string } | null {
   const sep = code.indexOf("::")
@@ -53,16 +187,23 @@ export function parseSnowflakePAT(code: string): { account: string; token: strin
 
 /**
  * Transform a Snowflake Cortex request body string.
- * Returns a Response to short-circuit the fetch (synthetic stop), or undefined to continue normally.
  *
  * @param toolCapable Model IDs that should retain `tools` / `tool_choice` / tool messages.
  *                    Build via `buildToolCapableSet(provider.models)` at loader time so
  *                    user-added models with `tool_call: true` in `altimate-code.json` are honored.
+ * @param cacheControl When false, strip every cache marker instead of relocating
+ *                     them into content blocks (set after Cortex rejected a
+ *                     cache-marked request).
+ * @returns `body` — the rewritten request body; `syntheticStop` — a Response that
+ *          short-circuits the fetch entirely (trailing-assistant continuation);
+ *          `cacheApplied` — true when the body carries block-level cache markers.
+ * @throws SyntaxError when `bodyText` is not valid JSON.
  */
 export function transformSnowflakeBody(
   bodyText: string,
   toolCapable: ReadonlySet<string>,
-): { body: string; syntheticStop?: Response } {
+  cacheControl = true,
+): { body: string; syntheticStop?: Response; cacheApplied?: boolean } {
   const parsed = JSON.parse(bodyText)
 
   // Snowflake uses max_completion_tokens instead of max_tokens
@@ -71,6 +212,15 @@ export function transformSnowflakeBody(
     delete parsed.max_tokens
   }
 
+  let cacheApplied = false
+  if (cacheControl) {
+    cacheApplied = relocateCacheControl(parsed)
+  } else if (typeof parsed.model === "string" && parsed.model.startsWith("claude-")) {
+    // Disabled mode only unwinds what relocation would have produced — keep
+    // non-Claude payloads untouched (Cortex ignores their markers anyway).
+    stripParsedCacheControl(parsed)
+  }
+
   // Strip tools for models that don't support tool calling on Snowflake Cortex.
   // Also remove orphaned tool_calls from messages to avoid Snowflake API errors.
   if (!toolCapable.has(parsed.model)) {
@@ -105,6 +255,7 @@ export function transformSnowflakeBody(
       })
       return {
         body: JSON.stringify(parsed),
+        cacheApplied,
         syntheticStop: new Response(stream, {
           status: 200,
           headers: { "content-type": "text/event-stream", "cache-control": "no-cache" },
@@ -113,7 +264,7 @@ export function transformSnowflakeBody(
     }
   }
 
-  return { body: JSON.stringify(parsed) }
+  return { body: JSON.stringify(parsed), cacheApplied }
 }
 
 export async function SnowflakeCortexAuthPlugin(_input: PluginInput): Promise<Hooks> {
@@ -137,6 +288,11 @@ export async function SnowflakeCortexAuthPlugin(_input: PluginInput): Promise<Ho
         // request time because a static hardcoded set never sees user additions.
         const toolCapable = buildToolCapableSet(provider.models)
 
+        // Cortex documents block-level cache_control for Claude models, but not
+        // for every message role. If an account rejects a cache-marked request,
+        // fall back once and pause marker injection for a cooldown period.
+        let cacheDisabledUntil = 0
+
         return {
           apiKey: OAUTH_DUMMY_KEY,
           async fetch(requestInput: RequestInfo | URL, init?: RequestInit) {
@@ -162,29 +318,74 @@ export async function SnowflakeCortexAuthPlugin(_input: PluginInput): Promise<Ho
             headers.set("X-Snowflake-Authorization-Token-Type", "PROGRAMMATIC_ACCESS_TOKEN")
 
             let body = init?.body
+            let text = ""
+            let cacheApplied = false
             if (body) {
-              try {
-                let text: string
-                if (typeof body === "string") {
-                  text = body
-                } else if (body instanceof Uint8Array || body instanceof ArrayBuffer) {
-                  text = new TextDecoder().decode(body)
-                } else {
-                  // ReadableStream, Blob, FormData — pass through untransformed
-                  text = ""
+              if (typeof body === "string") {
+                text = body
+              } else if (body instanceof Uint8Array || body instanceof ArrayBuffer) {
+                text = new TextDecoder().decode(body)
+              }
+              // ReadableStream, Blob, FormData — pass through untransformed
+              if (text) {
+                let result
+                try {
+                  result = transformSnowflakeBody(text, toolCapable, Date.now() >= cacheDisabledUntil)
+                } catch (error) {
+                  // Non-JSON body — pass through untransformed. Anything else
+                  // is a transform bug and must surface, not degrade silently.
+                  if (!(error instanceof SyntaxError)) throw error
                 }
-                if (text) {
-                  const result = transformSnowflakeBody(text, toolCapable)
+                if (result) {
                   if (result.syntheticStop) return result.syntheticStop
                   body = result.body
+                  cacheApplied = result.cacheApplied === true
                   headers.delete("content-length")
                 }
-              } catch {
-                // JSON parse error — pass original body through untransformed
               }
             }
 
-            return fetch(requestInput, { ...init, headers, body })
+            const response = await fetch(requestInput, { ...init, headers, body })
+
+            // If Cortex rejects a cache-marked request, retry once without the
+            // markers; when the stripped retry succeeds, the markers were the
+            // problem — pause marker injection for a cooldown period. The retry
+            // fires on any 400 (Cortex's marker-rejection error text is
+            // undocumented, so there is no reliable signal to scope on); a
+            // deterministic non-marker 400 recurs identically on the retry and
+            // never trips the cooldown.
+            if (response.status === 400 && cacheApplied) {
+              const stripped = transformSnowflakeBody(text, toolCapable, false).body
+              if (stripped !== body) {
+                // Pause markers eagerly so concurrent in-flight requests stop
+                // sending the rejected shape while this probe retry runs;
+                // restored below if the retry shows markers weren't the cause.
+                const previousDisabledUntil = cacheDisabledUntil
+                cacheDisabledUntil = Date.now() + CACHE_DISABLE_COOLDOWN_MS
+                let retry: Response
+                try {
+                  retry = await fetch(requestInput, { ...init, headers, body: stripped })
+                } catch {
+                  // Transport failure on the probe — surface the original 400.
+                  cacheDisabledUntil = previousDisabledUntil
+                  return response
+                }
+                if (retry.ok) {
+                  log.warn("Cortex rejected cache-marked request; pausing prompt caching", {
+                    status: response.status,
+                    cooldownMs: CACHE_DISABLE_COOLDOWN_MS,
+                  })
+                  void response.body?.cancel().catch(() => {})
+                  return retry
+                }
+                // Retry failed too — the 400 wasn't the markers; resume injection.
+                cacheDisabledUntil = previousDisabledUntil
+                void retry.body?.cancel().catch(() => {})
+                return response
+              }
+            }
+
+            return response
           },
         }
       },