That trick is quantization, and it's quietly become the reason large models run anywhere but a datacenter. It's also older than deep learning, dumber than you'd expect, and currently going through its most interesting decade. Here's the arc.
A history in four squeezes
The signal-processing century. Quantization predates neural networks by a good margin — it's how analog signal became digital audio (pulse-code modulation, 1930s–40s). The core question hasn't changed since: how few bits can represent this signal before the error becomes audible? Neural networks inherited the question and, for a long time, punted on it — FP32 was cheap enough that nobody bothered.
The binary era (2015–2017). The first serious squeeze was aggressive to the point of comedy: Binarized Neural Networks and XNOR-Net showed you could train networks with weights of ±1 and still get non-embarrassing accuracy on small benchmarks. They didn't survive contact with large models, but they proved the central dogma — networks are wildly over-parameterized and will tolerate astonishing numeric abuse. Meanwhile NVIDIA shipped tensor cores (Volta, 2017) and mixed-precision training made FP16/BF16 the default: same model, half the memory, free speed.
The LLM crisis (2022–2023). Large language models broke the naive playbook twice. First, LLM.int8() showed that models past a few billion parameters develop outlier features — a handful of activation channels with magnitudes 100× the norm — that wreck plain INT8 quantization. Second, nobody could afford quantization-aware retraining on a 70B model, so post-training methods had to carry the load. The community's answers came fast: GPTQ (one-shot, layerwise, second-order error correction), SmoothQuant (mathematically shifting the outlier problem from activations onto weights, enabling W8A8), AWQ (protecting the ~1% of weights that activations prove matter), and QLoRA/NF4 (a 4-bit format literally designed around the normal distribution of trained weights, plus double-quantizing the quantizer's own metadata). And outside the datacenter, llama.cpp's GGUF k-quants put 4-bit models on laptops — consumer quantization as a first-class citizen, with importance-matrix calibration to spend precision where it counts.
The floating-point floor (2024–now). The newest squeeze isn't integers at all. Hopper made FP8 (E4M3/E5M2) mainstream; Blackwell arrived with NVFP4 — 4-bit floating point with two-level micro-block scaling, a format engineered so 4-bit values keep enough dynamic range to track real weight distributions. Open-weight releases now ship MXFP4/NVFP4 checkpoints directly. And the frontier moved up the stack: KV-cache quantization (FP8 and below) attacks the other memory wall — the attention cache that grows with context length — which matters more every time someone doubles the window.
The techniques, organized by what you're trading
Strip away the alphabet soup and every quantization scheme is a choice on four axes:
- When: quantization-aware training (QAT — bake it in, best quality, expensive) or post-training (PTQ — apply after, cheap, slightly lossier). Almost everyone ships PTQ.
- What: weights only (W4A16 — activations stay wide; great for memory-bound decode) or weights and activations (W8A8 and below — unlocks the fast integer/FP8 tensor-core paths).
- Granularity: one scale per tensor, per channel, per group, or per micro-block. Finer granularity is the whole trick in modern 4-bit formats — you're paying a little metadata to keep the error local.
- Format: integers (INT8/INT4), lookup-ish floats (NF4), or real microfloats (FP8, NVFP4). Formats follow hardware; hardware follows formats.
The trade is always the same three-way: memory footprint, decode bandwidth, quality. On memory-bandwidth-bound hardware — which is every unified-memory box, and most GPUs at small batch — a 4-bit model isn't 4× smaller, it's close to 4× faster at decode, because generating tokens is mostly reading weights. Quality is the axis you defend: a good 4-bit quant costs you a point of perplexity you can often win back with a light LoRA pass; a bad one silently moves your model's behavior in ways your spot-checks won't catch.
Which is the part nobody talks about: a quantized checkpoint is a new model artifact and deserves the same suspicion as one. It needs evaluation, benchmarking, and provenance — not just a smaller file on disk.
A quantized checkpoint is a new model artifact and deserves the same suspicion as one.
The cache is the model's shadow
Everything above compresses the weights. But a running model carries a second, invisible allocation that quantization forgot about for years: the KV cache — every layer's keys and values for every token in the context, kept so the model never recomputes attention over what it already read. Weights are a fixed cost you pay once; the cache grows linearly with context, and it is where long-context memory actually lives.
The math is not subtle. Cache size is 2 × layers × KV heads × head dim × bytes per value, per token. Take a current 27B-class model — 65 layers, 4 KV heads, a 256-wide head dim — that's ~260 KB of cache per token at FP16. Against that:
- At 8K context: ~2 GB of cache next to ~15 GB of 4-bit weights. Quantizing the cache here is rounding error — we measured this on real workloads and deprioritized it, correctly.
- At 256K context: the FP16 cache is ~70 GB. Nearly 5× the quantized weights. The "small model" is mostly cache.
So the first thing to know about KV-cache quantization is when it matters: at short context it's rounding error, at long context it's the dominant term — and it buys you twice, because attention re-reads the cache on every generated token. Halving cache bytes halves that bandwidth too, which is why KV quant shows up in decode speed at long context, not just in whether the model fits.
The formats mirror the weight story, one step behind: FP16/BF16 cache, then 8-bit integer cache formats in the open runtimes and FP8 in the datacenter frameworks, then the 4-bit cache formats that are to attention what NF4 was to weights. The trade-off is different in kind, though: weight quantization blurs knowledge; cache quantization blurs attention over your actual document — and long-context retrieval (needle-in-haystack behavior) degrades before short-answer quality notices anything. That is why "1M context" claims need scrutiny: a 4-bit 27B model is 15 GB of weights, but a million tokens of its FP16 cache is ~266 GB. Nobody is serving that on one box; anyone advertising it is quantizing the cache hard, and the question is whether their gates can tell the difference.
So the production answer has to be boring on purpose: pick the conservative cache format, measure behavior on long-context retrieval sets rather than perplexity alone, and treat every cache-policy change as a new variant that re-earns its place. Which is not a flag-flipping exercise — it's the same governance problem as the weights, so it belongs in the same platform.
How CID handles it
CID treats quantization as a pipeline event, not a post-processing script — because that's what it is.
Quantized artifacts are first-class, registered, and attested. Quantization metadata is reserved schema in CID's model registry — format, KV-cache quantization, and quantization method ride alongside the model as structured fields, and quantized checkpoints enter the registry through the same commitment mechanism as trained ones (hf_quantized_checkpoint_commitment, gguf_quantized_checkpoint_commitment). A quant is a committed, auditable artifact with a lineage back to the base weights — not a side file someone scp'd to a server.
Registered identity is pinned. When a model is registered to CID's serving fleet, its identity binds to an exact, containment-checked artifact path. A checkpoint — quantized or otherwise — cannot be silently swapped after registration. If someone re-quantizes with different calibration data, that's a new version that re-enters the pipeline, not an edit in place.
Quality gates apply to quants like any other candidate. A quantized variant runs the same mandatory stages — eval, benchmark, validate — before deploy. Golden evaluation sets measure behavior; the benchmark stage measures throughput per quantization variant, so the speed/quality trade is a measured number on the model's governance card rather than a vibes-based claim from a forum thread. A 4-bit quant that regresses on golden evals doesn't get promoted. One that passes ships with its numbers attached.
The KV cache is governed, not configured. Cache quantization is reserved schema on the registered model version alongside the weight format — the cache policy is part of the artifact's identity, not a serve-time flag flipped in production. Cache-quantized variants run the same eval/benchmark/validate gates, with long-context retrieval behavior measured explicitly, because that is where cache error shows up first. And because spend is tracked per model variant, the platform answers the question cache formats are actually for: what does this checkpoint cost to serve, per million tokens, at the context length you actually run?
The fleet runs the modern formats in production. CID's own inference serving runs quantized checkpoints with quantized caches — and the registry pins their identity the same way it pins weights, so a cache-policy change is a new version that re-enters the pipeline, not a silent swap.
And the recovery loop is built in. When a quant costs more quality than you wanted to pay, the fix on CID is the same platform: an Elixir LoRA pass on top of the quantized base, attested at every stage, re-evaluated through the same gates, redeployed through the same registry. Quantize, measure, recover, attest — as a loop, not an incident.
The takeaway
Quantization spent fifty years as a compression detail and the last five becoming a model-governance problem. The format wars will keep moving — 4-bit floats today, something stranger tomorrow — but the discipline doesn't change: every quantized checkpoint is a new artifact that needs evaluation, provenance, and a measured cost before it earns production. Build that into the pipeline and quantization is free performance. Bolt it on afterward and it's a silent behavior change waiting for the wrong Tuesday.