Skip to content

support_get_rows predates K-quant get_rows support, silently 4x-ing embedding RAM #1930

Description

@opumep

Embedding::init_params falls back to F32 whenever support_get_rows() rejects the stored type, and the allowlist only contains F16/Q8_0/Q5_1/Q5_0/Q4_1/Q4_0 (src/core/ggml_extend.hpp). ggml's CPU get_rows has dispatched all the K-quants for a while now (ggml-cpu/ops.cpp, ggml_compute_forward_get_rows_q).

Practical impact: city96's umt5-xxl Q3_K_S stores token_embd.weight as q6_K (862MB). It loads as F32 = 4.2GB, which is the difference between running and OOM on an 8GB Android device. Nothing in the logs points at the embedding; the file size suggests it should fit.

Locally we extended the allowlist with Q2_K..Q6_K and verified q6_K end-to-end on-device (wan2.1 video pipeline, CPU backend). Two things we did NOT verify, which is why this is an issue rather than a PR: coverage of the other K-quants beyond q6_K, and whether every non-CPU backend's get_rows handles K-quants (if not, a ggml_backend_supports_op-style check may be better than a static list).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions