diff --git a/README.md b/README.md index 60bcc51..0954c99 100644 --- a/README.md +++ b/README.md @@ -113,6 +113,8 @@ The goal is not to flood the space with near-duplicates offering negligible KLD MagicQuant is built around transparency, honesty, maintainability, and most importantly trust. As it evaluates new architectures and quant families, it doesn’t invent quantization schemes in isolation. Instead, it learns from proven tensor assignments provided by trusted sources like llama.cpp and Unsloth. If those baselines are stable, MagicQuant operates within that same safe space, extending rather than reinventing. +Historical sources expand that tensor vocabulary; they do not vote on the current winner. MagicQuant pins the source revision, rebuilds the available recipes under current controlled conditions, and relearns their effects rather than replaying an old final mixture. + That said, the system is designed to adapt. Edge cases can exist, but the architecture is intentionally flexible to handle them. ### How MagicQuant Works @@ -181,6 +183,7 @@ That said, the system is designed to adapt. Edge cases can exist, but the archit │ ───────────────────────── │ │ - Group-level testing │ │ - Rank-safe prediction │ + │ - Controlled context tests │ └────────────┬──────────────┘ │ │ Build real GGUF @@ -190,6 +193,7 @@ That said, the system is designed to adapt. Edge cases can exist, but the archit │ ───────────────────────── │ │ - KLD (primary) │ │ - PPL (secondary) │ + │ - Measured GPU scheduling │ └────────────┬──────────────┘ │ │ Final decision @@ -201,4 +205,18 @@ That said, the system is designed to adapt. Edge cases can exist, but the archit │ - Nonlinear winners │ │ - Spacing collapse │ └────────────────────────────┘ -``` \ No newline at end of file +``` + +The controlled context tests check whether a promising group choice still behaves the same way when the surrounding model moves from a Q4-or-better regime into more aggressive compression. They are bounded and evidence-driven because exhaustive context testing would recreate the full combinatorial problem. + +GPU scheduling is also measured rather than assumed. A large benchmark can use multiple GPUs in one shared process, while batches of smaller candidates can run concurrently on independent GPUs when that produces higher aggregate throughput. + +The final release is a curated survivor menu. Research campaigns and cross-run audits should preserve the full nondominated evidence frontier before applying spacing, so that a presentation decision does not erase valid results. + +## Deep Dive Documentation + +- [Wiki index](./wiki/index.md) +- [Prediction Engine](./wiki/docs/Prediction-Engine.md) +- [Regime-Aware Tensor Search](./wiki/docs/Regime-Aware-Search.md) +- [GPU Benchmark Scheduling](./wiki/docs/GPU-Benchmark-Scheduling.md) +- [Pareto Archives and Reproducibility](./wiki/docs/Pareto-Archives-And-Reproducibility.md) diff --git a/wiki/docs/GPU-Benchmark-Scheduling.md b/wiki/docs/GPU-Benchmark-Scheduling.md new file mode 100644 index 0000000..0cf4d32 --- /dev/null +++ b/wiki/docs/GPU-Benchmark-Scheduling.md @@ -0,0 +1,318 @@ +# GPU Benchmark Scheduling and Measured Topology Planning + +MagicQuant performs many real `llama.cpp` benchmarks. + +On a machine with multiple GPUs, the obvious question sounds simple: + +> Should every benchmark use every GPU, or should MagicQuant run one benchmark per GPU? + +The honest answer is: + +> **It depends on the model size, the GPUs, the stable offload depth, and whether there is actually a batch of independent jobs waiting.** + +MagicQuant therefore treats GPU use as a measured scheduling problem rather than a fixed command-line preference. + +--- + +## The Important Correction + +`llama.cpp` benchmarking is not inherently limited to one GPU. + +A single benchmark process can split a model across multiple visible GPUs. That can be the only practical way to keep a larger GGUF fully or mostly offloaded. + +But using every GPU for every job is not automatically the fastest way to finish a batch. + +Consider two strategies on a two-GPU system: + +```text +shared topology: + benchmark A uses GPU 0 + GPU 1 + benchmark B waits + +independent topology: + benchmark A uses GPU 0 + benchmark B uses GPU 1 +``` + +The shared process may finish one large job faster. The independent topology may finish two smaller jobs faster in aggregate. + +Those are different optimization targets: + +```text +single-job latency != total batch throughput +``` + +MagicQuant measures both when the hardware and workload allow it. + +--- + +## The Two Topologies + +### Shared multi-GPU topology + +A shared slot exposes multiple GPUs to one benchmark process. + +Conceptually: + +```text +candidate.gguf + | + v +llama.cpp benchmark process + | + +---- GPU 0 + | + +---- GPU 1 +``` + +This topology is useful when: + +- the candidate is too large to fit well on one GPU +- one process needs the combined VRAM pool +- only one benchmark is ready +- the caller is performing an adaptive or sequential benchmark +- measured shared performance is better for that part of the size range + +### Independent per-GPU topology + +An independent topology creates one benchmark slot per stable GPU. + +Conceptually: + +```text +candidate-A.gguf ---- benchmark process A ---- GPU 0 +candidate-B.gguf ---- benchmark process B ---- GPU 1 +``` + +This topology is useful when: + +- multiple independent candidates are ready at the same time +- each candidate fits within an individual GPU's measured capacity +- concurrent workers produce higher aggregate throughput +- neither process needs to borrow the other GPU's VRAM + +Independent mode is not selected merely because two GPUs exist. There must be real batch concurrency to exploit. + +--- + +## Why Static Rules Are Not Enough + +A rule such as: + +```text +two GPUs => always run two workers +``` + +fails when the model is too large for either GPU. + +A rule such as: + +```text +two GPUs => always split every benchmark +``` + +wastes throughput when two smaller jobs could run concurrently. + +A file-size-only estimate is also incomplete. Two GPUs with the same advertised VRAM can have different usable headroom because of display usage, other processes, driver behavior, or configured safety limits. + +MagicQuant instead discovers: + +- the highest stable GPU-layer offload for the shared slot +- the highest stable offload for each independent GPU slot +- the throughput of the shared topology +- the aggregate throughput of the independent topology +- the candidate-size crossover below which independent workers are expected to win + +The result is a hardware execution plan measured on the machine that will perform the work. + +--- + +## NGL Is Candidate-Aware + +`NGL` is the number of model layers requested for GPU offload. + +The correct value is not assumed to be identical for every candidate or every GPU. + +MagicQuant starts from measured slot capability, then plans the offload depth for the actual candidate size. If a candidate cannot run stably at the planned depth, the benchmark path can reduce offload rather than treating the whole campaign as impossible. + +This matters on asymmetric systems. + +For example: + +```text +GPU 0 stable Q8 offload: 44 / 66 layers +GPU 1 stable Q8 offload: 57 / 66 layers +shared stable offload: 66 / 66 layers +``` + +The scheduler should not pretend those independent slots are interchangeable. + +When several candidates are ready, candidate-aware best-fit placement prefers: + +- a near-full-fit candidate on the weakest adequate GPU +- a candidate needing more offload help on the stronger GPU + +That preserves scarce capacity instead of assigning jobs by GPU index alone. + +--- + +## The Measured Crossover + +The crossover is a candidate-size boundary, not a universal constant. + +Below it, measured independent-worker throughput is expected to be better: + +```text +candidate size <= measured crossover +and concurrent batch intent exists + => independent GPU slots may be used +``` + +Above it, MagicQuant uses the shared multi-GPU slot: + +```text +candidate size > measured crossover + => shared multi-GPU slot +``` + +The shared topology also remains the safe choice for singleton and adaptive calls, because there is no second ready job from which to obtain concurrency. + +This distinction is important. A crossover derived from two concurrent workers should not be used to force a single benchmark onto one GPU. + +--- + +## Measured Example: Two RTX 3090 GPUs + +During the Qwen3.8 27B development campaign, a two-RTX-3090 machine produced this plan: + +| Measurement | Result | +| --- | ---: | +| Candidate maximum NGL | 66 | +| Shared stable NGL | 66 / 66 | +| GPU 0 independent Q8 NGL | 44 / 66 | +| GPU 1 independent Q8 NGL | 57 / 66 | +| Measured independent crossover | roughly 23.35–23.72 GiB | +| Configured shared tensor split | `19,23` | + +For sub-crossover Q5-sized work, the measured aggregate rates were approximately: + +| Topology | Aggregate throughput | +| --- | ---: | +| Shared multi-GPU workers | 0.529 jobs/second | +| Independent per-GPU workers | 0.762 jobs/second | + +That is roughly a 44% aggregate throughput improvement for that workload. + +This is a case study, not a promised multiplier and not a portable 23 GiB rule. Another model, CUDA build, driver, GPU pair, background load, or benchmark corpus can produce a different crossover. + +The transferable lesson is the method: + +```text +measure the machine +measure both viable topologies +schedule according to candidate size and batch shape +``` + +--- + +## Configuring Usable GPU Memory + +MagicQuant can accept per-GPU usable memory limits: + +```yaml +hardware: + gpu_memory_limits_gb: + 0: 19 + 1: 23 +``` + +These values are intentionally usable limits, not a claim about the physical capacity printed on the GPU box. + +They can reserve headroom for: + +- the display server +- another process that must remain active +- driver overhead +- a known stability margin + +When the shared slot uses multiple GPUs, MagicQuant passes the configured split in visible-GPU order. + +One small but important implementation detail is that `llama.cpp` tools do not all accept the same separator: + +```text +common llama.cpp CLI tools: --tensor-split 19,23 +llama-bench: --tensor-split 19/23 +``` + +MagicQuant builds the correct argument for the target executable. + +--- + +## Hardware Plan Caching + +Probing a Q8 or native model deeply enough to discover stable GPU behavior is useful, but it should not be paid on every run. + +MagicQuant caches the execution plan in SQLite. The cache is scoped to the things that can materially change the answer, including: + +- model identity +- imatrix identity where relevant to the generated probe artifact +- detected GPU topology +- configured memory limits +- `llama.cpp` executable fingerprint +- execution-plan schema version + +When that scope still matches, later runs can reuse the measured plan. + +When the hardware, model, binary, or configuration changes, a stale plan should not silently remain authoritative. + +To force a fresh hardware probe: + +```text +--recheck-hardware-probe +``` + +The alias `--force-refresh-hardware-probe` is also accepted. + +--- + +## Failure Behavior + +GPU planning is an optimization layer. It should not turn an otherwise valid benchmark into a brittle all-or-nothing operation. + +The intended fallback order is: + +```text +use cached measured plan when valid + | + v +probe stable shared and independent slots when needed + | + v +choose topology for candidate size and caller batch intent + | + v +reduce candidate offload if the planned NGL is unstable + | + v +fall back safely when a GPU topology cannot be validated +``` + +An unstable independent slot disables that optimization path. It does not justify fabricating a throughput estimate. + +--- + +## What This Planner Does Not Claim + +The planner does not claim: + +- that multi-GPU is always faster +- that one process per GPU is always faster +- that advertised VRAM equals usable VRAM +- that a crossover measured for one model is valid for another +- that KLD or PPL changes because of the scheduling topology + +The benchmark result remains the evidence. The planner changes how quickly MagicQuant can collect that evidence. + +Its job is simple: + +> **Spend the available GPU capacity in the highest-throughput stable shape that the current candidate batch can actually use.** diff --git a/wiki/docs/Learning-From-Existing-Quantizations.md b/wiki/docs/Learning-From-Existing-Quantizations.md index e7e45c6..e127bd7 100644 --- a/wiki/docs/Learning-From-Existing-Quantizations.md +++ b/wiki/docs/Learning-From-Existing-Quantizations.md @@ -285,6 +285,63 @@ without brute-forcing the entire combinatorial space. --- +## Historical Sources: Learn the Vocabulary, Not the Old Winner + +An older source revision can contain tensor recipes that a newer revision no longer publishes. + +That makes historical sources useful, but only at the correct level of abstraction. + +MagicQuant should learn: + +```text +which tensor assignments existed +which group recipes are independently testable +which quant families are available to the current search +``` + +It should not learn: + +```text +this old final mixture won before +therefore replay the same mixture now +``` + +Replaying historical winners would bias the current frontier toward a previous model, imatrix, benchmark corpus, and search campaign. Instead, MagicQuant digests the tensor vocabulary, rebuilds those choices under the current controlled conditions, and makes every group behavior earn support again. + +The principle is: + +> **Learn what configurations exist. Relearn what they do.** + +## Pin Every External Revision + +A provider label such as “Unsloth Dynamic v2” or “v3” is not a reproducible source identity. + +Repositories change. Files can be replaced while retaining familiar names. MagicQuant should therefore record: + +- repository identity +- exact revision or commit hash +- artifact filename +- artifact checksum where practical + +The repository configuration supports an immutable revision directly: + +```yaml +baselines: + custom_repositories: + - repo_id: provider/model-gguf + revision: exact-commit-or-repository-revision + enabled: true + allow_as_learning_baseline: true +``` + +Historical vocabulary sources can remain learning-only by disabling their use as global carriers and explicit group candidates. Their learned group recipes can then be considered through the controlled search without multiplying the carrier space or replaying the source artifact as a winner. + +This matters both for future reruns and for cross-run comparisons. Numeric SQLite IDs are local bookkeeping values, not globally stable identities. Two runs can assign the same number to different source recipes. + +For the wider evidence and identity rules, see [Pareto Archives, Release Curation, and Reproducibility](./Pareto-Archives-And-Reproducibility.md). + +--- + ## Why This Approach Works MagicQuant avoids one of the hardest problems in quantization: @@ -334,4 +391,4 @@ It does not replace quantizers. It builds on them. > **MagicQuant does not decide how to quantize a tensor. -> It decides which quantized configurations are worth keeping.** \ No newline at end of file +> It decides which quantized configurations are worth keeping.** diff --git a/wiki/docs/Nonlinear-Winners-And-Survivors.md b/wiki/docs/Nonlinear-Winners-And-Survivors.md index a6e3e33..a219961 100644 --- a/wiki/docs/Nonlinear-Winners-And-Survivors.md +++ b/wiki/docs/Nonlinear-Winners-And-Survivors.md @@ -211,6 +211,30 @@ MagicQuant avoids keeping candidates that only provide negligible improvement, r --- +## The Evidence Archive Is Not the Release Menu + +Meaningful spacing is a presentation decision. Strict dominance is a measured fact. + +Those should be stored separately: + +```text +full measured nondominated archive + => every real point that survives strict dominance + +curated release menu + => a smaller set chosen for useful user-facing spacing +``` + +A candidate removed only by spacing can still be scientifically important. It may become relevant when compared with a future run, when a user wants finer high-fidelity choices, or when the curation policy changes. + +The current spacing rule uses a fraction of the global survivor size span. This keeps a release clean, but a very wide frontier can cause useful points in a dense high-quality region to be hidden. + +MagicQuant should therefore preserve the unspaced nondominated evidence before creating the release view. + +For a complete source-aware comparison example and the reproducibility rules, see [Pareto Archives, Release Curation, and Reproducibility](./Pareto-Archives-And-Reproducibility.md). + +--- + ## Baselines Still Often Win MagicQuant hybrids do not always beat pure baselines. @@ -245,4 +269,4 @@ It is a candidate that improves the size/fidelity frontier. That is the core idea: > MagicQuant does not keep hybrids because they are different. -> It keeps them because the trade was better than expected. \ No newline at end of file +> It keeps them because the trade was better than expected. diff --git a/wiki/docs/Pareto-Archives-And-Reproducibility.md b/wiki/docs/Pareto-Archives-And-Reproducibility.md new file mode 100644 index 0000000..00f7ccd --- /dev/null +++ b/wiki/docs/Pareto-Archives-And-Reproducibility.md @@ -0,0 +1,305 @@ +# Pareto Archives, Release Curation, and Reproducibility + +MagicQuant produces two kinds of truth that should not be confused: + +```text +measured evidence archive +and +curated release menu +``` + +The archive answers: + +> What measured candidates are nondominated across everything compared? + +The release menu answers: + +> Which of those candidates are useful enough and sufficiently separated to present to users? + +Both are valuable. They serve different jobs. + +--- + +## Strict Pareto Dominance + +Candidate A strictly dominates candidate B when: + +```text +A.size <= B.size +A.KLD < B.KLD +``` + +with the expected measurement epsilon applied by the implementation. + +If A is smaller or equal in size and has lower KLD, B no longer represents a rational size/fidelity choice in that comparison space. + +The full nondominated union is the scientific frontier. + +It should be preserved even when several points are very close together. + +--- + +## Why the Release Can Be Smaller Than the Archive + +A release containing dozens of models separated by tiny size differences can be technically complete and practically unhelpful. + +MagicQuant therefore applies meaningful spacing to produce a cleaner survivor menu. + +That curation is useful, but it is not the same as dominance. + +```text +dominance: + this point is objectively unnecessary under measured size/KLD + +spacing: + this point may be valid, but it is too close to another point for the release menu +``` + +A spacing-removed candidate should not disappear from the evidence record. + +The recommended model is: + +```text +all measured candidates + | + v +strict nondominated archive + | + v +curated, meaningfully spaced release view +``` + +--- + +## The Limitation of Global-Span Spacing + +The current spacing rule derives a minimum neighbor gap from a fraction of the global size span: + +```text +minGap = (largest survivor size - smallest survivor size) + * minimumNeighborGapFraction +``` + +This is simple and predictable. + +It also has a limitation. + +When the full frontier spans from very small low-bit models to very large high-fidelity models, a global percentage can become large enough to hide useful distinctions in a dense high-quality region. + +For example: + +```text +full frontier span: very large +high-quality neighborhood: several meaningful choices within a narrow size band +global 3% spacing: collapses most of that neighborhood +``` + +That does not mean the removed points were dominated. It means the release policy chose a sparse menu. + +Future curation can improve this with quality-aware or local-density spacing. Until then, the full nondominated archive is essential for honest comparison and later recuration. + +--- + +## Cross-Run Comparison Requires Source Identity + +SQLite numeric IDs are local database identifiers. + +They are not stable artifact identities across runs. + +This is dangerous when comparing an existing release to a new campaign. Two different source generations can reuse the same local tensor-combination or baseline ID. + +The wrong merge rule is: + +```text +same numeric ID => same measured artifact +``` + +The safer identity includes: + +```text +source repository and revision +artifact filename or recipe +tensor-group assignment +model and imatrix scope +measured size and KLD +``` + +Rows should be collapsed as duplicates only when their source identity or reconstructed recipe and measurements agree. + +--- + +## Qwen3.8 27B Frontier Audit + +The Qwen3.8 27B campaign compared the existing published frontier with a new regime-aware run. + +The corrected full-Pareto run reused 43 of 43 valid startup artifacts, retained 33 measured survivors, and contained 20 local GGUFs plus 13 upstream references. Its search evidence included 212 isolation samples and 95 recorded bad trades. Reuse shortened the campaign, but every reused object still had to match its recorded identity and scope. + +The source-aware union contained: + +| Evidence | Count | +| --- | ---: | +| Published candidates | 23 | +| Challenger candidates | 33 | +| Numeric-key overlaps | 12 | +| Exact measured overlaps | 11 | +| Distinct measured artifacts | 45 | +| Nondominated measured artifacts | 41 | + +The difference between 12 numeric-key overlaps and 11 exact overlaps exposed a real identity collision: local key `100` referred to different `IQ2_XXS` measurements across source generations. + +Treating numeric IDs as global identities would have silently erased evidence. + +Of the 41 nondominated artifacts: + +- 20 came from the published frontier +- 21 came from the challenger run +- 3 published points were strictly dominated by challenger points +- 1 challenger point was strictly dominated by a published point + +The new run produced three strict measured replacements: + +| Previous size / KLD | New size / KLD | +| --- | --- | +| 22.564615 GiB / 0.001439 | 22.537283 GiB / 0.001364 | +| 15.365976 GiB / 0.011351 | 15.362543 GiB / 0.011154 | +| 14.335722 GiB / 0.014502 | 14.332670 GiB / 0.014381 | + +This proves improvement without pretending the entire old frontier became obsolete. + +When the existing 3% global-span spacing policy was applied, only 17 of the 41 nondominated measurements remained in the curated view. + +That is why both layers should be reported: + +```text +41 = complete nondominated evidence +17 = one useful release curation under the current spacing policy +``` + +--- + +## What a Reproducible Campaign Must Record + +A final README table is not enough to reproduce a search campaign. + +The campaign record should preserve: + +### Model scope + +- exact base model repository +- exact model revision +- source file checksums where practical +- architecture and tensor-group mapping used + +### Tensor configuration sources + +- repository identity for every external source +- exact revision for every source generation +- artifact filenames and checksums where practical +- learned tensor recipes, not only provider labels + +### Measurement scope + +- imatrix file identity and checksum +- KLD/PPL corpus identity +- bucket or benchmark configuration +- `llama.cpp` binary fingerprint +- hardware topology and usable memory limits + +### Search scope + +- complete YAML configuration +- prediction and pruning thresholds +- context strata and probe budgets +- random seeds where applicable +- code commit or PR revision +- explicit runtime root used by the process + +### Evidence + +- SQLite campaign database or export +- manifests +- logs +- isolation measurements +- harmful, beneficial, and suppression context evidence +- every final artifact's real size and real KLD +- the unspaced nondominated union +- the curated release decision + +This allows a future reader to separate: + +```text +what was measured +what was inferred +what was removed by dominance +what was removed only by presentation policy +``` + +An isolated campaign root is especially useful when several runs share the same machine: + +```text +--magic-quant-root /path/to/isolated/runtime +``` + +That root determines where campaign state such as SQLite data is resolved. Record it and verify the live process is using it before a long run; opening a different historical database can make cache reuse look valid when it belongs to another campaign. + +--- + +## Cached Work and Reproducibility + +Reusing previously downloaded baselines, learned tensor digestion, imatrix artifacts, or hardware plans can save enormous time. + +Reuse is valid when its scope is verified. + +The cache key must prove that the reused object belongs to the same relevant inputs. A convenient directory name is not enough. + +For example: + +```text +hardware plan reuse requires matching hardware and llama binary +tensor vocabulary reuse requires matching source revision +benchmark truth reuse requires matching model, artifact, corpus, and measurement setup +``` + +If the scope cannot be proven, the correct action is to rebuild or remeasure. + +--- + +## A Practical Comparison Checklist + +Before claiming that a new run beat an old run: + +```text +[ ] Compare real size and real KLD, not prediction rows +[ ] Build the source-aware union of both runs +[ ] Do not deduplicate by local numeric ID alone +[ ] Collapse only exact artifact/recipe and measurement matches +[ ] Recompute strict dominance across the full union +[ ] Report which source contributed each nondominated point +[ ] List every strict replacement explicitly +[ ] Preserve the full nondominated archive +[ ] Apply release spacing only after the archive is known +[ ] Record the configuration, revisions, manifests, and logs +``` + +This is more work than comparing two final README tables. + +It is also the difference between a persuasive result and a reproducible one. + +--- + +## Core Principle + +MagicQuant should be aggressive in search and conservative in claims. + +That means: + +```text +preserve measured evidence +identify artifacts by provenance, not coincidence +separate Pareto truth from release presentation +claim only the improvements the source-aware union proves +``` + +The final survivor menu is for users. + +The complete evidence archive is what keeps that menu trustworthy. diff --git a/wiki/docs/Prediction-Engine.md b/wiki/docs/Prediction-Engine.md index 9ab1ec6..6955414 100644 --- a/wiki/docs/Prediction-Engine.md +++ b/wiki/docs/Prediction-Engine.md @@ -1,11 +1,13 @@ # MagicQuant Prediction Engine -## Rank-Safe Isolation Prediction, Practical Gravity, and Contextual Anomaly Learning +## Rank-Safe Isolation Prediction, Practical Gravity, and Controlled Context Learning MagicQuant is a **prediction-guided validation system** for GGUF hybrid quantization. It measures a small number of physically real tensor-group samples, uses those samples to build a rank-safe prediction space, searches that compressed space for candidates that could meaningfully improve the final frontier, then builds and benchmarks only the candidates that deserve a real test. The main path is the DuckDB/rank-safe prediction engine. A newer secondary path, the **smart baseline-tuning fallback**, runs only after a normal dominance, premium, or interior discovery attempt fails to validate a winner. That fallback does not pretend to out-predict the full engine. It uses the same isolated measurements more conservatively, looking for small MDA-backed “free lunch” swaps and tightly budgeted protection trades that may have been intentionally pruned away from the main prediction space. +MagicQuant can also repeat a bounded number of useful comparisons inside controlled high-, mid-, and low-fidelity blankets. These probes do not replace normal isolation. They test whether an isolation ranking transfers when the surrounding model enters a different compression regime. The complete design is documented in [Regime-Aware Tensor Search](./Regime-Aware-Search.md). + The prediction engine is allowed to be approximate because it is not the final judge. Its job is to decide where compute should be spent. The real benchmark remains the final authority. The system is built around a practical belief: @@ -2106,34 +2108,45 @@ Anomaly probing asks: Does this lower-fidelity movement behave differently in a real quantized context? ``` -## Known blind spot: high-bit truth does not always transfer downward +## Controlled fidelity regimes: testing whether truth transfers downward Anomaly detection is powerful, but it is not omniscient. -Normal isolation and the current anomaly probes intentionally sample a limited number of high-signal contexts. That keeps MagicQuant practical. Testing every possible low-bit blanket, every pair, every triple, and every surrounding quantization condition would explode the sample count. +Normal isolation and anomaly probes intentionally sample a limited number of high-signal contexts. Testing every low-bit blanket, pair, triple, and surrounding quantization condition would explode the sample count. + +A useful example is a group where `IQ4_NL` looks unusually strong in isolation or in a 4-bit-and-above context. It may beat `Q4_K_M` by a meaningful amount when the surrounding groups are Q8/native exact or otherwise high fidelity. That does not guarantee the relationship survives in 3-bit-or-lower territory. -A useful example shape is a group where `IQ4_NL` looks unusually strong in isolation or in a 4-bit-and-above context. It may beat `Q4_K` by a meaningful amount when the surrounding groups are Q8/native exact or otherwise high fidelity. +In a much lower-fidelity context, the surrounding damage field changes. The local ranking can reverse. -But that does not guarantee the relationship survives in 3-bit-or-lower territory. +MagicQuant now tests this risk with bounded controlled context strata: -In very low-bit contexts, the surrounding damage field changes. A quant that hit above its weight class near Q8/BF16 may stop doing so once other groups are also heavily compressed. In that territory, a `Q4_K` choice that looked worse in high-bit isolation can become the better practical companion. +```text +high fidelity: Q6/Q5 reference blankets +mid fidelity: Q4 reference blanket +low fidelity: IQ3 reference blanket, opt-in +``` -This is an accepted blind spot of the current practical engine: +For exploratory rank checks, it compares: ```text -high-bit isolation truth is highly useful -but it is not a complete map of low-bit emergent behavior +isolation winner +versus +closest-size same-bit non-equivalent alternative ``` -MagicQuant could add more low-bit context isolation in the future, such as 3-bit blanket probes or targeted low-bit transfer checks. The tradeoff is sample growth. At present, the system chooses the practical path: respect the limited truth it has, acknowledge where that truth can fail, and let the other mechanisms — bit-stress fitting, anomaly probes, synergy second chance, fallback attempts, and real validation — power through most cases without brute-forcing the entire universe. +Both candidates are measured inside the same blanket. This provides evidence about transfer without replaying an old final mixture or brute-forcing every pair. -The important contract remains: +The resulting rule is matched against the effective non-selected surrounding groups, not merely the candidate's carrier label. Evidence can be beneficial, harmful, or suppression-only. Weak or noisy changes can therefore block unsafe promotion without creating an optimistic boost. + +This closes part of the earlier blind spot, but it does not turn a handful of blankets into a complete map: ```text -prediction may miss some low-bit context flips -real benchmark truth still decides what survives +controlled regime probes improve transfer knowledge +they do not eliminate combinatorics ``` +Low-fidelity probing remains opt-in because the extra real builds are most valuable when Q3-or-lower artifacts are an important release target. + ## Movement classification MagicQuant compares a candidate to its reference twin and classifies the movement. @@ -2839,7 +2852,9 @@ Reject invalid sparse/native-exact anomaly configs. Require quantized contextual twins. -Remember the scope limit: anomaly smoke is not an exhaustive low-bit context map. +Also identify justified context rank pairs: the isolation winner and the closest-size same-bit non-equivalent alternative for the selected group. + +Remember the scope limit: anomaly smoke and controlled strata are not an exhaustive low-bit context map. ## Step 18: Probe anomaly candidates @@ -2854,6 +2869,8 @@ full changed set Benchmark probe and reference twin. +For transfer and exploratory rank probes, rebuild both sides inside the same configured high-, mid-, or opt-in low-fidelity blanket. + Classify as beneficial, harmful, normal gravity, or suppression-only. ## Step 19: Persist anomaly rules @@ -2866,6 +2883,8 @@ Harmful rules demote matching rows. Suppression-only rules prevent repeated wasted probing. +Scope transfer rules to the measured effective surrounding-group context, excluding the groups changed by the rule itself. Do not use the carrier label as a substitute for the actual context. + ## Step 20: Query frontier candidates Use DuckDB to query candidates for: @@ -3284,6 +3303,16 @@ Reject BF16/native-exact anomaly configs. Probe subsets of changed groups. +Add bounded controlled context strata when transfer behavior matters: + +```text +high-fidelity Q6/Q5 blankets +mid-fidelity Q4 blanket +opt-in low-fidelity IQ3 blanket +``` + +For exploratory rank pairs, compare the isolation winner with the closest-size same-bit non-equivalent alternative inside the same blanket. + Classify: ```text @@ -3293,6 +3322,8 @@ normal gravity suppression-only ``` +Match learned rules against effective non-selected surrounding groups. Do not replay historical final winners and do not treat a search-row carrier label as sufficient context identity. + ## Anomaly prediction adjustment Keep normal prediction and anomaly exceptions separate: @@ -3332,7 +3363,7 @@ It does not brute-force the universe. It also does not blindly trust a simplistic predictor. -Instead, it builds a practical prediction space from isolated tensor-group measurements, stabilizes that space with monotonic rank projection, uses fitted interaction correction where enough truth exists, keeps subtle bad-trade-adjacent ideas out of the main ranking space, recovers a few of them through guarded smart fallback when prediction fails, detects contextual violations of gravity through Q8-style quantized twins, and validates every serious candidate with real benchmarks. +Instead, it builds a practical prediction space from isolated tensor-group measurements, stabilizes that space with monotonic rank projection, uses fitted interaction correction where enough truth exists, keeps subtle bad-trade-adjacent ideas out of the main ranking space, recovers a few of them through guarded smart fallback when prediction fails, detects contextual violations of gravity through quantized twins and controlled fidelity strata, and validates every serious candidate with real benchmarks. The system’s deepest idea is not merely: diff --git a/wiki/docs/Regime-Aware-Search.md b/wiki/docs/Regime-Aware-Search.md new file mode 100644 index 0000000..928a9db --- /dev/null +++ b/wiki/docs/Regime-Aware-Search.md @@ -0,0 +1,415 @@ +# Regime-Aware Tensor Search + +Isolated tensor-group testing is one of MagicQuant's most useful tools. + +It answers questions such as: + +```text +If every other group is held constant, +what happens when attn_q changes from Q4_K_M to IQ4_NL? +``` + +But there is a subtle problem: + +> **A group choice that wins inside a high-fidelity surrounding model may not remain the winner after the surrounding model becomes much more compressed.** + +MagicQuant therefore distinguishes isolation truth from regime transfer truth. + +--- + +## The Rank-Flip Problem + +Imagine an embeddings experiment inside a Q4-or-better blanket: + +```text +surrounding groups: Q4 and above + +embeddings IQ4_NL => excellent +embeddings Q4_K_M => worse +``` + +It is tempting to promote `IQ4_NL` everywhere. + +Now repeat the same head-to-head comparison while the surrounding groups are Q3 or lower: + +```text +surrounding groups: Q3 and below + +embeddings IQ4_NL => worse +embeddings Q4_K_M => better +``` + +The tensor group did not change. Its context did. + +That does not mean either measurement is wrong. It means the local ranking is conditional: + +```text +best choice for group g + may depend on +effective surrounding fidelity +``` + +This is especially important near aggressive compression regimes, where error from other groups is no longer small relative to the isolated change being measured. + +--- + +## Why Normal Isolation Cannot See It + +A standard isolation probe changes one group while keeping a single reference blanket fixed. + +Conceptually: + +```text +reference blanket = Q8 + +Q8 model + candidate embeddings +Q8 model + candidate attn_q +Q8 model + candidate ffn_down +``` + +This is excellent for learning a clean marginal signal. It reduces interference from unrelated compressed groups. + +But it cannot answer: + +```text +does the same marginal ranking survive in a Q4 blanket? +does it survive in an IQ3 blanket? +``` + +The controlled answer is not to abandon isolation. It is to repeat a small number of justified head-to-head probes inside deliberately chosen surrounding-fidelity strata. + +--- + +## Controlled Context Strata + +MagicQuant can define reference blankets at several fidelity levels: + +```yaml +synergy_detection: + transfer_probe_context_strata: + high_fidelity_reference_quants: [Q6_K, Q5_K] + mid_fidelity_reference_quants: [Q4_K_M] + low_fidelity_reference_quants: [IQ3_S] + low_fidelity_enabled: false +``` + +The default categories are conceptual rather than universal laws: + +| Stratum | Purpose | +| --- | --- | +| High fidelity | Check behavior near relatively clean surrounding groups | +| Mid fidelity | Check the common Q4 neighborhood where many release candidates live | +| Low fidelity | Check whether behavior transfers or flips under aggressive compression | + +Low-fidelity probing is opt-in because it is not free. + +It adds real builds and benchmarks. It can also generate misleading excitement if a noise-scale difference is treated as a universal winner. + +The purpose of a low-fidelity probe is not: + +```text +make every Q3-or-lower candidate more important +``` + +It is: + +```text +detect when evidence learned in a cleaner regime stops transferring +``` + +--- + +## Context Rank Pairs + +MagicQuant does not need to retest every quant type against every other quant type in every blanket. + +That would recreate the combinatorial explosion the prediction system exists to avoid. + +Instead, exploratory context probing chooses a disciplined pair: + +```text +candidate A = isolation winner +candidate B = closest-size, same-bit, non-equivalent alternative +``` + +Then both are remeasured inside the same controlled blanket. + +Why the closest-size alternative? + +Because a head-to-head comparison is most informative when it is not secretly comparing a large size jump. + +Why same-bit? + +Because the question is whether tensor behavior or quant-family behavior changes with context, not whether four bits usually beat three bits. + +Why non-equivalent? + +Because spending two real benchmarks on recipes that produce effectively identical tensor truth teaches nothing. + +The relevant controls are: + +```yaml +synergy_detection: + exploratory_context_pair_enabled: true + max_exploratory_context_pairs_per_run: 14 + exploratory_pair_bit_ranges: [4] + exploratory_pair_context_strata: [mid-fidelity, low-fidelity] +``` + +The budget keeps this as targeted scientific measurement rather than an accidental exhaustive sweep. + +--- + +## Historical Sources: Vocabulary, Not Winner Replay + +Historical quant repositories can be valuable because they expose tensor configurations that do not exist in the newest source. + +Suppose an earlier external revision used: + +```text +quant family A for embeddings +quant family B for ffn_down +an unusual protection pattern for attn_output +``` + +A later revision may no longer publish all of those choices. + +MagicQuant can digest both revisions to preserve the available tensor vocabulary. + +It must not conclude: + +```text +this old mixed model was a winner before +therefore replay the old mixture now +``` + +That would import historical frontier bias into a new model and a new measurement campaign. + +The safer abstraction is: + +```text +historical source + => additional independently testable tensor recipes + != historical final winner replay +``` + +Current isolation, context probing, prediction, and real validation still decide whether any recipe belongs in a current hybrid. + +This distinction is the heart of historical learning in MagicQuant: + +> **Learn what configurations exist. Relearn what they do.** + +--- + +## Pin the Source Revision + +Labels such as “Dynamic v2” or “Dynamic v3” are not precise enough for reproducible tensor digestion. + +A repository can change while keeping the same human-facing name. Files can be replaced. Manifests can change. A rerun months later can silently learn a different vocabulary. + +For every external tensor source, record the exact revision: + +```text +repository identity +revision or commit hash +artifact filename +artifact checksum where available +``` + +During the Qwen3.8 27B campaign, the historical sources were pinned to exact revisions rather than read from a moving branch: + +| Source generation | Pinned revision | +| --- | --- | +| Dynamic v2 source | `313447f257f7ebde0b968e4778feef774546ed81` | +| Dynamic v3 source | `4ca720788d1e01f1bff70c033e0d0028fd02e502` | + +Those hashes document what was digested. They do not endorse one generation as globally better than the other. + +--- + +## Rules Must Match Effective Context + +A context result is only useful if it is applied to compatible contexts. + +The wrong shortcut is: + +```text +the search row began from a Q4 carrier +therefore call its context Q4 +``` + +A hybrid may replace several groups. The carrier label can stop describing the effective surrounding model. + +MagicQuant therefore evaluates context using the groups that remain around the rule-selected groups: + +```text +effective context = non-selected surrounding groups +``` + +Rule-selected groups are excluded from the match because they are the intervention, not the environment. + +The application controls include: + +```yaml +synergy_detection: + context_scoped_rule_application_enabled: true + max_non_rule_group_context_mismatches: 1 +``` + +A small mismatch allowance can tolerate a nearly equivalent surrounding recipe. A broad carrier-based match would transfer evidence far beyond what was actually measured. + +--- + +## Beneficial, Harmful, and Suppression Evidence + +Context evidence is not only a source of bonuses. + +MagicQuant records three important outcomes: + +### Beneficial evidence + +The candidate improved more than expected in the measured context. + +This can justify a conservative, context-scoped prediction adjustment or a transfer probe. + +### Harmful evidence + +The candidate became worse in the measured context. + +This should block optimistic transfer into that regime. + +### Suppression-only evidence + +The apparent difference exists, but it is too small, noisy, or inconsistent to support promotion. + +This is still useful. It prevents an attractive high-fidelity result from being overgeneralized. + +The asymmetry is intentional: + +```text +weak positive evidence should not create a broad boost +harmful or uncertain evidence can still prevent unsafe transfer +``` + +This is how MagicQuant uses historical and contextual knowledge without allowing it to bias the whole search. + +--- + +## Synergy Is Not Assumed + +Two individually interesting context rules do not automatically compose into a stronger multi-group rule. + +MagicQuant can run bounded composition probes, but the combined result must earn support from measurement. + +During the Qwen3.8 27B campaign, composition probes did not justify a general synergy boost. + +That is a valid result. + +The system learned: + +```text +some effects are context-specific +some are suppressive +the measured evidence does not support broad composition +``` + +Not finding synergy is better than inventing it. + +Across the active campaign history, the persisted rule evidence was: + +| Rule evidence | Count | +| --- | ---: | +| Confirmed beneficial | 12 | +| Confirmed harmful | 95 | +| Exact-context suppression | 117 | + +The imbalance is informative. Most contextual knowledge was useful as a boundary on transfer, not as permission to improve predicted KLD broadly. + +--- + +## Qwen3.8 27B: What the Probes Actually Found + +The controlled campaign demonstrated why both promotion and restraint matter. + +### Embeddings + +`IQ4_XS` and `UD-IQ4_XS` changed rank in the IQ3 context, but the KLD difference was only `0.000031` while the size difference was about `0.157 GiB`. + +That was treated as noise-scale suppression evidence, not proof of a universal low-fidelity winner. + +### Attention query + +For `attn_q`, `IQ4_NL` remained ahead of `Q4_K_M` in both the Q4 and IQ3 controlled blankets on this model. + +The general rank-flip concern was valid, but the exact example did not reproduce for this group and model. + +### Attention output + +`Q4_K_M` and `UD-Q4_K_XL` improved the Q4 blanket but worsened the IQ3 blanket. + +This is direct evidence that a useful mid-fidelity effect should not be transferred downward blindly. + +### Feed-forward down projection + +`UD-Q4_K_XL` was smaller but worse in the Q4 context, then became a strict win in the IQ3 context. + +This is the kind of reversal that a single Q8 isolation blanket cannot reveal. + +### Frontier outcome + +One context-derived candidate reached: + +```text +size: 16.241945 GiB +KLD: 0.011087 +``` + +It was nondominated, but it did not dominate the already published point at: + +```text +size: 16.414560 GiB +KLD: 0.007412 +``` + +That distinction matters. The contextual system found valid evidence and a real frontier point. It did not justify claiming that every new probe improves the best release choice. + +--- + +## Deciding Whether Low-Fidelity Probing Is Worth It + +Enable low-fidelity context work when: + +- Q3-or-lower artifacts are an important release target +- high-fidelity isolation winners are being transferred into much lower regimes +- a model architecture shows unusually strong cross-group interactions +- external sources expose materially different same-bit tensor recipes +- the run budget can afford real controlled comparisons + +Leave it disabled when: + +- the release is focused on Q4 and above +- there is no credible alternative pair to compare +- the expected effect is below benchmark noise +- the extra builds would displace more valuable frontier validation + +The goal is not maximum probing. + +The goal is maximum useful information per real build. + +--- + +## What Regime-Aware Search Does Not Claim + +It does not claim: + +- that Q3 and lower always reverse Q4 behavior +- that historical revisions should vote on current winners +- that one controlled blanket represents every hybrid at that bit range +- that every measured rank flip deserves a prediction boost +- that context rules remove the need for final real benchmarking + +The controlled probes improve the prediction system's understanding of where evidence transfers. + +The final contract remains unchanged: + +> **Prediction spends the benchmark budget. Real size and real KLD decide the survivor.** diff --git a/wiki/index.md b/wiki/index.md index 5cf6d4d..04fb1a6 100644 --- a/wiki/index.md +++ b/wiki/index.md @@ -1,6 +1,20 @@ -**Deep Dive Documentation**: -* [Bad Trades, Early Pruning, and Search-Space Collapse](./docs/Bad-Trades-And-Early-Pruning.md) -* [Learning From Existing Quantizations (Tensor Configuration Sources)](./docs/Learning-From-Existing-Quantizations.md) -* [Nonlinear Winners and Survivor Selection](./docs/Nonlinear-Winners-And-Survivors.md) -* [Prediction Engine](./docs/Prediction-Engine.md) -* [Imatrix Dataset](./docs/Imatrix-Dataset.md) \ No newline at end of file +# MagicQuant v2 Documentation + +MagicQuant is a benchmark-driven GGUF evaluation and hybrid-discovery system. These pages explain not only what the pipeline does, but why its search, measurement, and survivor rules exist. + +## Start Here + +- [Learning From Existing Quantizations (Tensor Configuration Sources)](./docs/Learning-From-Existing-Quantizations.md) +- [Prediction Engine](./docs/Prediction-Engine.md) +- [Nonlinear Winners and Survivor Selection](./docs/Nonlinear-Winners-And-Survivors.md) + +## Search Behavior + +- [Bad Trades, Early Pruning, and Search-Space Collapse](./docs/Bad-Trades-And-Early-Pruning.md) +- [Regime-Aware Tensor Search](./docs/Regime-Aware-Search.md) + +## Operations and Evidence + +- [GPU Benchmark Scheduling and Measured Topology Planning](./docs/GPU-Benchmark-Scheduling.md) +- [Pareto Archives, Release Curation, and Reproducibility](./docs/Pareto-Archives-And-Reproducibility.md) +- [Imatrix Dataset](./docs/Imatrix-Dataset.md)