build: guards for SKEEP-003's acceptance criteria (#1501)

.github/workflows/build.yml

@@ -76,10 +76,11 @@ jobs:
           # associates with dumps (#1171 merged a stale dump exactly that way; #1174 repaired it).
           - name: api-compatibility
             tasks: apiCheck
-          # Every NotImplementedError / TODO( in production code is either @NotImplemented
-          # (so it shows in the ops coverage matrix) or listed in config/stub-allowlist.txt.
+          # Both guards scan sources only, so they share one leg: stubs are either @NotImplemented
+          # or in config/stub-allowlist.txt, and downcasts to concrete TensorData classes outside
+          # the data package (SKEEP-003) are in config/tensor-data-downcast-allowlist.txt.
           - name: stub-markers
-            tasks: checkUnmarkedStubs
+            tasks: checkUnmarkedStubs checkTensorDataDowncasts
 
     permissions:
       contents: read

CHANGELOG.md

@@ -2,6 +2,14 @@
 
 ## [Unreleased]
 
+### Added
+
+- **Guards for SKEEP-003's acceptance criteria** (build, `skainet-backend-cpu`, docs, #1501): `./gradlew checkTensorDataDowncasts` counts the downcasts to concrete `TensorData` classes outside the data package (162 in 21 files today) against `config/tensor-data-downcast-allowlist.txt`, so a new one fails the build and the list can only shrink; it runs in the `stub-markers` CI leg. `ForwardScopeFlatMemoryTest` runs 300 forward passes under a forward scope on the JVM and asserts that direct memory, the live heap and the slab stay flat. `reference/benchmark-baseline.adoc` records the JMH numbers for matmul, elementwise and reductions that a dispatch or storage change is compared against, replacing the baseline deleted in #1106.
+
+### Changed
+
+- **SKEEP-003 is back at `Accepted`, with an implementation-status table** (docs, #1500): an audit at 0.64.0 found the new storage model in use and the old layer (`TensorStorage`, `BufferHandle`, `TensorData` downcasts in the ops) still in place, so three of the five acceptance criteria are open. The SKEEP now says which, and what tracks each (#932). SKEEP-003a no longer lists #1148 as open, SKEEP-002's status records the Q4_K / Q6_K mapped staging of #1190 and its device measurement, `memory-model.adoc` drops a limit that #1190 removed, and the `TensorStorage` KDoc stops telling new code to target it.
+
 ## [0.64.0] - 2026-11-11
 
 Headline: **every differentiable op has a backward rule, and mixed precision is finished: it trains, exports without

build-logic/convention/src/main/kotlin/CheckTensorDataDowncastsTask.kt

@@ -0,0 +1,140 @@
+import org.gradle.api.DefaultTask
+import org.gradle.api.GradleException
+import org.gradle.api.file.ConfigurableFileCollection
+import org.gradle.api.file.DirectoryProperty
+import org.gradle.api.file.RegularFileProperty
+import org.gradle.api.tasks.*
+import java.io.File
+
+/**
+ * Guards a SKEEP-003 acceptance criterion: no dispatch site downcasts to a concrete `TensorData`
+ * class outside the data package itself. Kernels are meant to be chosen from a view's
+ * `Format(dtype, encoding)`, not from the Kotlin class that happens to hold the bytes.
+ *
+ * A downcast is an `is` / `!is` / `as` / `as?` whose target is a class named `…TensorData`
+ * (the generic base `TensorData` itself does not count) or one of the storage markers in
+ * [markerTypes]. Production code only; the packages that define the classes are exempt.
+ *
+ * The sites that exist today are listed in [allowlistFile] as `path|count|reason`, where `count`
+ * is the exact number of downcasts in the file. A new downcast fails the build until the entry is
+ * raised, which a reviewer sees; removing one fails until the entry is lowered, so the list can
+ * only describe the code as it is. The goal is an empty list.
+ */
+@CacheableTask
+abstract class CheckTensorDataDowncastsTask : DefaultTask() {
+
+    @get:InputFiles
+    @get:PathSensitive(PathSensitivity.RELATIVE)
+    abstract val sources: ConfigurableFileCollection
+
+    /** Repository root that report and allowlist paths are relative to. */
+    @get:Internal
+    abstract val rootDirectory: DirectoryProperty
+
+    @get:InputFile
+    @get:PathSensitive(PathSensitivity.RELATIVE)
+    abstract val allowlistFile: RegularFileProperty
+
+    @get:OutputFile
+    abstract val reportFile: RegularFileProperty
+
+    private val cast = Regex("""(?:!is|\bis|\bas\?|\bas)\s+(?:[A-Za-z_][\w]*\.)*([A-Z]\w*)""")
+    private val mainSourceDir = Regex("""/src/[A-Za-z0-9]*[Mm]ain/""")
+
+    /** Marker interfaces that stand for a concrete storage class in a dispatch `when`. */
+    private val markerTypes = setOf(
+        "PackedBlockStorage",
+        "RowDequantSource",
+        "MemorySegmentBackedData",
+        "Q4MemorySegmentMarker",
+        "Q8MemorySegmentMarker",
+    )
+
+    /** Packages that define the classes and may inspect them. */
+    private val exemptPackages = listOf(
+        "/sk/ainet/lang/tensor/data/",
+        "/sk/ainet/lang/tensor/storage/",
+        "/sk/ainet/lang/memory/",
+    )
+
+    @TaskAction
+    fun check() {
+        val root = rootDirectory.get().asFile
+        val allowlist = parseAllowlist(allowlistFile.get().asFile)
+        val found = mutableMapOf<String, MutableList<Int>>()
+
+        sources.files
+            .filter { it.extension == "kt" }
+            .forEach { file ->
+                val rel = file.relativeTo(root).invariantSeparatorsPath
+                if (!mainSourceDir.containsMatchIn("/$rel")) return@forEach
+                if (exemptPackages.any { it in "/$rel" }) return@forEach
+                file.readLines().forEachIndexed { index, line ->
+                    val trimmed = line.trimStart()
+                    if (trimmed.startsWith("//") || trimmed.startsWith("*") || trimmed.startsWith("/*")) return@forEachIndexed
+                    val code = stripStrings(line).substringBefore("//")
+                    val hits = cast.findAll(code).count { isConcreteStorageType(it.groupValues[1]) }
+                    repeat(hits) { found.getOrPut(rel) { mutableListOf() } += index + 1 }
+                }
+            }
+
+        val problems = mutableListOf<String>()
+        found.toSortedMap().forEach { (path, hits) ->
+            val expected = allowlist[path]
+            when {
+                expected == null -> hits.forEach { problems += "$path:$it — downcast to a concrete TensorData class" }
+                expected != hits.size -> problems += "$path — ${hits.size} downcasts, allowlist expects $expected (lines ${hits.joinToString()})"
+            }
+        }
+        allowlist.keys.filter { it !in found }.sorted().forEach { path ->
+            problems += "$path — allowlisted but has no downcasts any more; remove the entry"
+        }
+
+        reportFile.get().asFile.apply { parentFile.mkdirs() }.writeText(
+            found.toSortedMap().entries.joinToString("\n", postfix = "\n") { (path, hits) -> "$path|${hits.size}|${hits.joinToString(",")}" },
+        )
+
+        if (problems.isNotEmpty()) {
+            throw GradleException(
+                buildString {
+                    appendLine("Downcasts to concrete TensorData classes outside the data package (SKEEP-003):")
+                    problems.forEach { appendLine("  $it") }
+                    appendLine()
+                    appendLine("Choose the kernel from the view's Format(dtype, encoding) instead of the class. If the")
+                    appendLine("count went down, lower it in ${allowlistFile.get().asFile.relativeTo(root).invariantSeparatorsPath} (path|count|reason).")
+                },
+            )
+        }
+        logger.lifecycle("✅ TensorData downcasts match the allowlist (${found.values.sumOf { it.size }} in ${found.size} files)")
+    }
+
+    private fun isConcreteStorageType(name: String): Boolean =
+        name in markerTypes || (name.endsWith("TensorData") && name != "TensorData")
+
+    /** Blank out the contents of `"…"` literals on one line (escape-aware). */
+    private fun stripStrings(line: String): String = buildString {
+        var inString = false
+        var i = 0
+        while (i < line.length) {
+            val ch = line[i]
+            when {
+                inString && ch == '\\' -> i++
+                ch == '"' -> { inString = !inString; append(ch) }
+                !inString -> append(ch)
+            }
+            i++
+        }
+    }
+
+    private fun parseAllowlist(file: File): Map<String, Int> =
+        file.readLines()
+            .map { it.trim() }
+            .filter { it.isNotEmpty() && !it.startsWith("#") }
+            .associate { line ->
+                val parts = line.split("|").map { it.trim() }
+                if (parts.size < 3 || parts[2].isEmpty()) {
+                    throw GradleException("Malformed allowlist line (want path|count|reason): $line")
+                }
+                parts[0] to (parts[1].toIntOrNull() ?: throw GradleException("Bad count in allowlist line: $line"))
+            }
+}

build-logic/convention/src/main/kotlin/DocumentationPlugin.kt

@@ -57,6 +57,22 @@ class DocumentationPlugin : Plugin<Project> {
             }
         })
 
+        // Guard (SKEEP-003): ops choose kernels from a view's Format, not by downcasting to a
+        // concrete TensorData class. Today's sites are allowlisted; the list may only shrink.
+        project.tasks.register("checkTensorDataDowncasts", CheckTensorDataDowncastsTask::class.java, object : Action<CheckTensorDataDowncastsTask> {
+            override fun execute(task: CheckTensorDataDowncastsTask) {
+                task.group = "verification"
+                task.description = "Fail on downcasts to concrete TensorData classes that are not in the allowlist"
+                val root = project.rootProject.layout.projectDirectory
+                project.rootProject.allprojects.forEach { p ->
+                    task.sources.from(p.fileTree("src") { include("**/*.kt") })
+                }
+                task.rootDirectory.set(root)
+                task.allowlistFile.set(root.file("config/tensor-data-downcast-allowlist.txt"))
+                task.reportFile.set(project.layout.buildDirectory.file("reports/downcasts/tensor-data-downcasts.txt"))
+            }
+        })
+
         // Register schema validation task in plugin (migrated from skainet-lang-export-ops)
         val validateTaskProvider = project.tasks.register("validateOperatorSchema", SchemaValidationTask::class.java, object : Action<SchemaValidationTask> {
             override fun execute(task: SchemaValidationTask) {

config/tensor-data-downcast-allowlist.txt

@@ -0,0 +1,34 @@
+# Downcasts to concrete TensorData classes outside the data package. Checked by
+# `./gradlew checkTensorDataDowncasts`.
+#
+# SKEEP-003 acceptance criterion: no dispatch site downcasts to a concrete `TensorData` class
+# outside the storage/data package itself; a kernel is chosen from the view's
+# Format(dtype, encoding). These are the sites that still do. The goal is an empty file.
+#
+# Format: path|count|issue or reason
+#   count = exact number of `is` / `!is` / `as` / `as?` whose target is a `…TensorData` class
+#   or a storage marker (PackedBlockStorage, RowDequantSource, MemorySegmentBackedData,
+#   Q4/Q8MemorySegmentMarker). Removing a downcast means lowering the count (or deleting the
+#   line). Do not raise a count: dispatch on the format instead.
+
+skainet-apps/skainet-grayscale-cli/src/main/kotlin/sk/ainet/apps/grayscale/HloCompiler.kt|1|#1507
+skainet-apps/skainet-grayscale-cli/src/main/kotlin/sk/ainet/apps/grayscale/TensorConversionPipeline.kt|1|#1507
+skainet-apps/skainet-tensor-tools/src/main/kotlin/sk/ainet/apps/tool/quantize/QuantizeCli.kt|3|#1507
+skainet-backends/skainet-backend-cpu/src/appleMain/kotlin/sk/ainet/exec/tensor/ops/AccelerateCpuOps.kt|18|#1507
+skainet-backends/skainet-backend-cpu/src/commonMain/kotlin/sk/ainet/exec/tensor/ops/DefaultCpuOps.kt|51|#1507
+skainet-backends/skainet-backend-cpu/src/jvmMain/kotlin/sk/ainet/exec/tensor/ops/DefaultCpuOpsJvm.kt|48|#1507
+skainet-backends/skainet-backend-wasm-simd/src/wasmJsMain/kotlin/sk/ainet/exec/wasm/WasmCpuOps.kt|10|#1507
+skainet-compile/skainet-compile-dag/src/commonMain/kotlin/sk/ainet/lang/trace/TraceToGraphBuilder.kt|3|#1507
+skainet-compile/skainet-compile-hlo/src/commonMain/kotlin/sk/ainet/compile/hlo/converters/BlockQuantMatmulConverter.kt|1|#1507
+skainet-compile/skainet-compile-hlo/src/commonMain/kotlin/sk/ainet/compile/hlo/converters/ConstantOperationsConverter.kt|1|#1507
+skainet-compile/skainet-compile-opt/src/commonMain/kotlin/sk/ainet/compile/opt/passes/BlockQuantMatmulFusionPass.kt|1|#1507
+skainet-io/skainet-io-gguf/src/commonMain/kotlin/sk/ainet/io/gguf/StreamingGgufParametersLoader.kt|2|#1507
+skainet-io/skainet-io-gguf/src/commonMain/kotlin/sk/ainet/io/gguf/export/GGUFWriter.kt|2|#1507
+skainet-io/skainet-io-safetensors/src/commonMain/kotlin/sk/ainet/io/safetensors/SafeTensorsMaterializer.kt|1|#1507
+skainet-lang/skainet-lang-core/src/commonMain/kotlin/sk/ainet/lang/tensor/IndexTensor.kt|1|#1507
+skainet-lang/skainet-lang-core/src/commonMain/kotlin/sk/ainet/lang/tensor/ops/QuantizedMatmul.kt|9|#1507
+skainet-lang/skainet-lang-core/src/commonMain/kotlin/sk/ainet/lang/tensor/ops/TensorSpecEncoding.kt|1|#1507
+skainet-lang/skainet-lang-core/src/commonMain/kotlin/sk/ainet/lang/tensor/ops/TernaryMatmul.kt|3|#1507
+skainet-lang/skainet-lang-core/src/commonMain/kotlin/sk/ainet/lang/trace/TraceSession.kt|1|#1507
+skainet-lang/skainet-lang-core/src/jvmMain/kotlin/sk/ainet/lang/tensor/StorageBenchmarks.kt|3|#1507
+skainet-test/skainet-test-groundtruth/src/commonMain/kotlin/sk/ainet/test/groundtruth/TensorAssertions.kt|1|#1507

docs/modules/ROOT/nav.adoc

@@ -30,6 +30,7 @@
 ** xref:reference/operators/generated/index.adoc[Operator reference]
 ** xref:reference/ops-status-matrix.adoc[Operator coverage matrix]
 ** xref:reference/kernel-support-matrix.adoc[Kernel × platform support]
+** xref:reference/benchmark-baseline.adoc[Benchmark baseline]
 ** xref:reference/api.adoc[API reference (Dokka)]
 * Explanation
 ** xref:explanation/skainet-for-ai.adoc[SKaiNET for AI/ML]

docs/modules/ROOT/pages/explanation/memory-model.adoc

@@ -204,7 +204,7 @@ ARMv8.2 Cortex-A55 reference board.
 [cols="2,3",options="header"]
 |===
 | Delivered | Notes
-| `Storage` / `Scope` / `TensorView` / `Format` / `Layout` | All targets; `TensorData` implementations are façades over views
+| `Storage` / `Scope` / `TensorView` / `Format` / `Layout` | All targets; every `TensorData` family used by the kernels exposes a view. Scoped activations (`StorageFloatTensorData`) sit on a `Storage`; the other classes still own their bytes and wrap the view around them (xref:skeep:003-unified-tensor-storage.adoc[SKEEP-003], implementation status)
 | `KernelKey` dispatch for matmul | Reference kernel plus packed and ternary kernel packs
 | Ternary encodings and `bitnet_gemv` | `TQ1_0`, `TQ2_0`, BitNet b1.58; reference kernel plus a NEON pack on Android
 | Memory plan, device profiles, fit check | `skainet-plan`, mobile/desktop/native profiles, two-pool device check
@@ -247,9 +247,10 @@ anything is loaded rather than after an OOM.
 
 Known limits, stated rather than implied:
 
-* *Packed weights still reach the managed heap under mapped staging.* The packed matmul SPI takes
-`ByteArray`s, so a buffer-aware kernel is future work. Mapping therefore lifts the Android heap
-ceiling for dense checkpoints, not yet for a Q4_K_M one.
+* *Some packed weights still reach the managed heap under mapped staging.* Q4_K and Q6_K weights
+stay in the file mapping (`BufferPackedTensorData`, read by row-major kernels through a direct
+buffer, #1190), so a Q4_K_M checkpoint larger than the Android heap loads. The other packed
+formats are still handed to their kernels as `ByteArray`s.
 * *Whole-file mapping only.* Files larger than 2 GB are refused rather than mapped in windows.
 * *Graph-level planning* (allocating a whole forward pass at once) stays in the downstream
 compiler by decision (#1134). Device placement itself was resolved in 0.49.0: `AllocationResolver`