Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@

## [Unreleased]

### Changed

- **SKEEP-003 is back at `Accepted`, with an implementation-status table** (docs, #1500): an audit at 0.64.0 found the new storage model in use and the old layer (`TensorStorage`, `BufferHandle`, `TensorData` downcasts in the ops) still in place, so three of the five acceptance criteria are open. The SKEEP now says which, and what tracks each (#932). SKEEP-003a no longer lists #1148 as open, SKEEP-002's status records the Q4_K / Q6_K mapped staging of #1190 and its device measurement, `memory-model.adoc` drops a limit that #1190 removed, and the `TensorStorage` KDoc stops telling new code to target it.

## [0.64.0] - 2026-11-11

Headline: **every differentiable op has a backward rule, and mixed precision is finished: it trains, exports without
Expand Down
9 changes: 5 additions & 4 deletions docs/modules/ROOT/pages/explanation/memory-model.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -204,7 +204,7 @@ ARMv8.2 Cortex-A55 reference board.
[cols="2,3",options="header"]
|===
| Delivered | Notes
| `Storage` / `Scope` / `TensorView` / `Format` / `Layout` | All targets; `TensorData` implementations are façades over views
| `Storage` / `Scope` / `TensorView` / `Format` / `Layout` | All targets; every `TensorData` family used by the kernels exposes a view. Scoped activations (`StorageFloatTensorData`) sit on a `Storage`; the other classes still own their bytes and wrap the view around them (xref:skeep:003-unified-tensor-storage.adoc[SKEEP-003], implementation status)
| `KernelKey` dispatch for matmul | Reference kernel plus packed and ternary kernel packs
| Ternary encodings and `bitnet_gemv` | `TQ1_0`, `TQ2_0`, BitNet b1.58; reference kernel plus a NEON pack on Android
| Memory plan, device profiles, fit check | `skainet-plan`, mobile/desktop/native profiles, two-pool device check
Expand Down Expand Up @@ -247,9 +247,10 @@ anything is loaded rather than after an OOM.

Known limits, stated rather than implied:

* *Packed weights still reach the managed heap under mapped staging.* The packed matmul SPI takes
`ByteArray`s, so a buffer-aware kernel is future work. Mapping therefore lifts the Android heap
ceiling for dense checkpoints, not yet for a Q4_K_M one.
* *Some packed weights still reach the managed heap under mapped staging.* Q4_K and Q6_K weights
stay in the file mapping (`BufferPackedTensorData`, read by row-major kernels through a direct
buffer, #1190), so a Q4_K_M checkpoint larger than the Android heap loads. The other packed
formats are still handed to their kernels as `ByteArray`s.
* *Whole-file mapping only.* Files larger than 2 GB are refused rather than mapped in windows.
* *Graph-level planning* (allocating a whole forward pass at once) stays in the downstream
compiler by decision (#1134). Device placement itself was resolved in 0.49.0: `AllocationResolver`
Expand Down
33 changes: 17 additions & 16 deletions docs/modules/skeep/pages/002-android-offheap-tensor-storage.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -232,14 +232,13 @@ keeps working unchanged and the overload is purely opt-in. Expected consumers:
(packed quant blocks) and a synthesized SafeTensors file (dense
F32/F16/BF16) produce bit-identical tensors under either placement.

== Implementation status (updated 2026-08-26, for 0.49.0)
== Implementation status (updated 2026-11-11, at 0.64.0)

Tracked by SKEEP-003's M2 slice
https://github.com/SKaiNET-developers/SKaiNET/issues/1038[#1038]. This SKEEP stays *Draft*: the
mechanism is in `develop` (and since 0.49.0 the loader speaks one `WeightForm` — the legacy
`StagingPolicy`/`QuantPolicy`/`WeightOrientation` axes were removed in #1159), the acceptance
criteria that need a physical device are not met yet, and one of them cannot be met until an
unrelated contract is fixed.
`StagingPolicy`/`QuantPolicy`/`WeightOrientation` axes were removed in #1159), one acceptance
criterion has a device measurement behind it, and the others are not recorded.

Landed:

Expand All @@ -257,20 +256,22 @@ Landed:
`AndroidGguf.fits(context, path, ctx)` answers from the GGUF header before a byte of payload is
read, naming the pool that runs out and what to do about it.

* *Phase 4 for Q4_K and Q6_K* — `BufferPackedTensorData` keeps a packed weight in the file
mapping, and row-major kernels read it through a direct `ByteBuffer`
(https://github.com/SKaiNET-developers/SKaiNET/issues/1189[#1189], PR #1190). Measured on a Pixel 8a with the 256 MB heap cap
(https://github.com/SKaiNET-developers/SKaiNET/issues/1130[#1130]): Qwen2.5-1.5B Q4_K_M, 1.0 GB on disk, loads with 566 KB of
weights on the managed heap and decodes at 153 ms per step; before, it ran out of memory at
load. That is the acceptance criterion "a Q4_K model larger than the heap loads and generates".

Not landed, and why:

* *Packed weights still reach the managed heap.* Mapped staging serves dense F32 tensors as
zero-heap views, but every quantized tensor is still handed to its kernel as a `ByteArray`,
because the packed matmul SPI takes arrays. That is Phase 4, and it is blocked on
https://github.com/SKaiNET-developers/SKaiNET/issues/973[#973] — the packed-quant byte-order
contract — since a buffer-aware kernel first needs an unambiguous answer to *which* byte order it
is reading. Until then the heap ceiling is lifted for dense checkpoints, not for a Q4_K_M one,
which is exactly the acceptance criterion below that remains open.
* *The device numbers.* "≤ 40 MB managed heap for SmolLM2-135M Q8_0" and "a ~600 MB Q4_K model
loads on a 256 MB heap" are measurements on a physical device; the repository's CI has none. They
belong to the `skainet-decode` sample in SKaiNET-transformers, which owns a model and a device
lane. `androidHostTest` covers what can be proven without hardware: the Android compilation
builds, maps, loads, and produces bit-identical tensors under either staging.
* *Other packed formats still reach the managed heap.* Q8_0, Q4_0, Q5_0, Q5_1, Q5_K and the
ternary encodings are handed to their kernels as `ByteArray`s under mapped staging; only Q4_K
and Q6_K have the buffer-reading kernels.
* *The SmolLM2-135M Q8_0 heap figure and the mapped-versus-heap throughput comparison* are not
recorded. Q8_0 is one of the formats above, so the "less than 40 MB" criterion cannot hold yet.
* *Whole-file mapping only.* A file larger than 2 GB is refused instead of being mapped in
windows (https://github.com/SKaiNET-developers/SKaiNET/issues/1422[#1422] asks for the same in the IRPA writer).

== Risks

Expand Down
57 changes: 55 additions & 2 deletions docs/modules/skeep/pages/003-unified-tensor-storage.adoc
Original file line number Diff line number Diff line change
@@ -1,11 +1,64 @@
= SKEEP-003: Unifying the tensor storage model — one byte-owner, enforced ownership, coherent dtype/encoding
:description: SKaiNET proposal to converge the TensorData and TensorStorage layers into one storage model with real ownership, a single view mechanism, and dtype/encoding coherence.

Status: Implemented (accepted 2026-08-22; M0/M1/M2 + the P7/P8 resolution shipped in 0.49.0) +
Status: Accepted (2026-08-22; M0/M1/M2 and the placement resolution shipped in 0.49.0; the removal of the old layer is open, see <<implementation-status,Implementation status>>) +
Audience: SKaiNET maintainers and contributors +
Created: 2026-08-10 +
Tracking issue: https://github.com/SKaiNET-developers/SKaiNET/issues/932[#932] (umbrella) → https://github.com/SKaiNET-developers/SKaiNET/issues/1001[#1001 M0] · https://github.com/SKaiNET-developers/SKaiNET/issues/1002[#1002 M1] · https://github.com/SKaiNET-developers/SKaiNET/issues/1003[#1003 M2] +
Implemented: the model as built is documented in xref:ROOT:explanation/memory-model.adoc[The memory model] and xref:ROOT:explanation/packed-weight-layout.adoc[Packed weight layout]; the proposal and milestone PRD that led there are in the pull requests and issues linked below (#932, #1001, #1002, #1003).
As built: the model is documented in xref:ROOT:explanation/memory-model.adoc[The memory model] and xref:ROOT:explanation/packed-weight-layout.adoc[Packed weight layout]; the proposal and milestone PRD that led there are in the pull requests and issues linked below (#932, #1001, #1002, #1003).

[#implementation-status]
== Implementation status (audited 2026-11-11, at 0.64.0)

The model is built and in use on every target: `Storage`, `TensorView`, `Format`, `Scope`, the
memory plan, the tracer, mapped staging. The old layer it was meant to replace is still in the
code, so this proposal is back at `Accepted` until the open rows below are closed. The umbrella
https://github.com/SKaiNET-developers/SKaiNET/issues/932[#932] lists the work.

[cols="3,1,3",options="header"]
|===
| Acceptance criterion | State | Where it stands

| One abstraction owns bytes, ownership enforced
| Partly
| Enforcement is tested (`StorageTest`, `ScopeTest`, `TensorViewTest`, `JvmStorageTest`). `TensorStorage` and `BufferHandle` still exist beside `Storage`, and the streaming readers' `loadTensorStorage*` return the old type (https://github.com/SKaiNET-developers/SKaiNET/issues/1502[#1502], https://github.com/SKaiNET-developers/SKaiNET/issues/1507[#1507]).

| One view mechanism
| Partly
| `OneViewMechanismTest` covers slice, transpose and unsqueeze on the new view. The public `slice()` family still returns the deprecated `SlicedTensorView` (https://github.com/SKaiNET-developers/SKaiNET/issues/1502[#1502]).

| A packed tensor reports FP32 and its encoding; no dispatch site downcasts to a concrete `TensorData` class outside the data package
| Not met
| The first half holds (`FormatTest`). About 130 downcasts remain in the CPU, Accelerate and wasm ops, and matmul reaches `KernelKey` dispatch only as its last fallback (https://github.com/SKaiNET-developers/SKaiNET/issues/1501[#1501] for the guard, https://github.com/SKaiNET-developers/SKaiNET/issues/1507[#1507] for the migration).

| No kernel regression against a baseline
| Partly
| The benchmarks exist; the recorded baseline was deleted with `docs/design/` in #1106 (https://github.com/SKaiNET-developers/SKaiNET/issues/1501[#1501]).

| Flat direct memory under a forward scope on the JVM
| No test
| Slab reuse is tested (`ScopedOpOutputsTest`), direct memory only under the GC arena (`MemSegArenaLeakTest`) (https://github.com/SKaiNET-developers/SKaiNET/issues/1501[#1501]).
|===

Decisions 1, 2 (naming and package), 6, 9, 11, 12 and 13 are delivered; `LogicalDType` is
deprecated and its removal is https://github.com/SKaiNET-developers/SKaiNET/issues/1502[#1502]. The rest:

[cols="1,4,3",options="header"]
|===
| # | Open part | Tracked by

| 1 | Façades: only `StorageFloatTensorData` sits on a `Storage`; the other `TensorData` classes own their bytes and wrap a view around them on each read | https://github.com/SKaiNET-developers/SKaiNET/issues/1507[#1507]
| 2 | `sk.ainet.lang.graph.Layout` is not deprecated; `BlockOrderLayout` implements it now | SKEEP-004, which owns the compile-side layout
| 3 | `get()` on a packed view decodes; the debug-mode warning on hot paths does not exist | https://github.com/SKaiNET-developers/SKaiNET/issues/1503[#1503]
| 4 | The `(head, tail)` pair and its gather fallback exist (`WindowedKV`); only the reference attention consumes them | https://github.com/SKaiNET-developers/SKaiNET/issues/1504[#1504], outside this SKEEP
| 5 | `BlockSpec.activation` drives the eager dispatcher; the StableHLO export does not read it | https://github.com/SKaiNET-developers/SKaiNET/issues/1505[#1505], outside this SKEEP
| 7 | `SuspendingRandomAccessSource` exists in `skainet-io-core`; there is no remote module | https://github.com/SKaiNET-developers/SKaiNET/issues/1506[#1506], outside this SKEEP
| 8 | Axis labels on `Shape` | https://github.com/SKaiNET-developers/SKaiNET/issues/1503[#1503]
| 10 | Activation ids exist when tracing, not in eager debug mode | https://github.com/SKaiNET-developers/SKaiNET/issues/1503[#1503]
|===

Compiled parity (P7 below) was closed as superseded: IREE validation lives in the IREE-tools and
conformity repositories (#1148), and this repository has no eager-versus-compiled parity test.

== Decision (2026-08-22)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -98,9 +98,9 @@ The carry work landed as the #1178/#1179/#1180 arc (0.49.0): `TensorRef` carries
encoding and block order; `TensorSpec` metadata accessors follow the untyped precedent; the
structural `skainet.tensor_layouts` module attribute and `ExternalParameterRef.blockOrder` declare
the facts to the consumer; and `LayoutAssignmentPass` runs on the production path via
`HloGenerator.generate(target = …)`, populating the pre-built `ResolvedComputeGraph` seams. Until
the conformity pipeline exists (#1148, the future IREE-validation SKEEP), the carriage tests are
the metadata's reader.
`HloGenerator.generate(target = …)`, populating the pre-built `ResolvedComputeGraph` seams. The
conformity pipeline that reads the metadata lives outside this repository (#1148 was closed as
superseded); here, the carriage tests are the metadata's reader.

== Where everything landed

Expand All @@ -113,7 +113,7 @@ the metadata's reader.
| `memoryScope` wired into creation | #1145 | PR #1156
| Op outputs through the scope + offset-aware SIMD | #1146, #1173 | landed (PRs #1168/#1175)
| Layout/placement carriage to StableHLO | #1147 (#1178/#1179/#1180) | landed (PRs #1182/#1183/#1184)
| `.vmfb` parity acceptance run | #1148 | open (follow-up)
| `.vmfb` parity acceptance run | #1148 | closed as superseded 2026-09-02: IREE validation lives in the IREE-tools and conformity repositories; no eager-versus-compiled parity test in this repository
|===

The SKEEP-003 items "`StorageSpec`, `MemoryPlanner` and the annotations are deleted unless wired"
Expand Down
6 changes: 3 additions & 3 deletions docs/modules/skeep/pages/index.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -116,12 +116,12 @@ Every proposal should include:
| Off-heap tensor storage on Android

| xref:skeep:003-unified-tensor-storage.adoc[SKEEP-003]
| Implemented
| Accepted
| Unifying the tensor storage model

| xref:skeep:003a-placement-and-planning-resolution.adoc[SKEEP-003a]
| Resolved
| Placement, eager lifetime and graph-level planning — the P7/P8 answers
| Implemented
| Addendum to SKEEP-003: placement, eager lifetime and graph-level planning

| xref:skeep:004-virtual-tensor-layout.adoc[SKEEP-004]
| Draft
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,19 +8,20 @@ import sk.ainet.lang.types.DType
/**
* Runtime descriptor for a tensor's backing memory.
*
* [TensorStorage] is the main architectural type that replaces ad-hoc
* array passing between loaders, planners, and backends. It carries enough
* information to handle a tensor without inspecting its bytes:
* [TensorStorage] describes a tensor read from a file before it becomes a
* tensor: the streaming GGUF and SafeTensors readers return it. It carries
* enough information to handle a tensor without inspecting its bytes:
*
* - **What** the values mean: [logicalType]
* - **How** they are stored: [encoding]
* - **Where** the bytes live: [buffer] + [placement]
* - **Layout**: [shape], [byteOffset], [strides], [isContiguous]
* - **Ownership**: via [buffer]'s [BufferHandle] subtype
*
* Existing [sk.ainet.lang.tensor.data.TensorData] remains as a
* compatibility façade. New loaders, planners, and backends should target
* [TensorStorage] directly.
* This is the older of two storage descriptions. The model new code targets is
* [sk.ainet.lang.memory.Storage] (owns bytes) with
* [sk.ainet.lang.memory.TensorView] (interprets them); SKEEP-003 replaces this
* type with a view once the readers return one (#1507).
*/
public data class TensorStorage(
val shape: Shape,
Expand Down
Loading