Concept card

Quantization Strategies — INT8, INT4, AWQ, GPTQ, GGUF

As of 2026-07-23

Edited by Cédric-Olivier Mâry, sole editor and publication director — 27 years in the SAP ecosystem. How these pages are produced, and where automation is used.

What is Quantization Strategies — INT8, INT4, AWQ, GPTQ, GGUF?

INT4 quantization shrinks a 70B model's footprint from 140GB to 35GB — the difference between needing two H100s and just one A100 — for only 1-3% accuracy loss on MMLU/HumanEval.

What it is

Quantization shrinks a model's weights from 16-bit floats (FP16/BF16, the training format) down to 8-bit (INT8) or 4-bit (INT4) integers, cutting GPU memory 2-4x and inference latency 1.5-3x — at the cost of small, controlled accuracy loss. For a 70B-parameter model, INT4 turns a 140 GB FP16 footprint into 35 GB, the difference between needing two H100s and one A100 — a cost gap that decides whether a use case is economically viable for on-premises or edge deployment.

Four families dominate. AWQ (Activation-aware Weight Quantization, MIT 2023) measures which weight channels carry the most signal on a calibration dataset, leaves those in higher precision, and aggressively quantizes the rest; preserves accuracy within 0.5-1 perplexity point of FP16 at 4-bit. GPTQ (Frantar et al. 2022) is the older one-shot quantizer using second-order information from a calibration set; very fast to apply, slightly worse accuracy than AWQ on instruction-tuned models. INT8 SmoothQuant moves activation outliers into the weight matrix so both can be 8-bit cleanly; the conservative choice when accuracy must be near-FP16. GGUF (GPT-Generated Unified Format, llama.cpp ecosystem) is a container format with multiple quantization variants (Q2_K, Q4_K_M, Q5_K_M, Q8_0); the de-facto format for llama.cpp and edge deployment.

Why it matters

AWQ preserves accuracy within 0.5-1 perplexity point of FP16 at 4-bit by leaving high-signal weight channels in higher precision, edging out the older GPTQ on instruction-tuned models.
INT8 SmoothQuant is the conservative choice with under 1% degradation, the default for data-centre serving where accuracy can't be compromised.
Below INT4 (3-bit, 2-bit) degradation becomes task-dependent — math and code reasoning collapse first, conversational chat degrades last.

Key points

Quantization shrinks weights from FP16 to INT8 (2x memory cut) or INT4 (4x memory cut) — at small accuracy cost.
AWQ (activation-aware, MIT 2023) — best 4-bit accuracy, 0.5-1 perplexity point of FP16; default for instruction-tuned models.
GPTQ — older one-shot quantizer, fast to apply, slightly worse than AWQ on chat models.
INT8 SmoothQuant — conservative 2x compression, <1% accuracy hit; default for production data-centre serving.
GGUF — container format with Q2_K to Q8_0 variants; canonical for llama.cpp edge + laptop deployments.
Below INT4 (3-bit, 2-bit) — math and code reasoning collapse first; chat degrades last.
Quantization Strategies — INT8, INT4, AWQ, GPTQ, GGUF is mastered only when it changes a named buyer decision.
Start with the semantic contract and control model before demonstrating the tool.
Use current SAP, analyst, study, KG, and news signals as evidence, not decoration.
Separate verified facts from directional trends and modeled assumptions.

Terms used on this page

Quantization: Reducing the numerical precision of a model's weights (and optionally activations) from FP16/BF16 down to INT8 or INT4 to shrink memory and accelerate inference.
AWQ: Activation-aware Weight Quantization (MIT 2023); identifies salient weight channels from a calibration dataset and preserves them in higher precision while aggressively quantizing the rest.
GPTQ: One-shot post-training quantizer (Frantar et al. 2022) using approximate second-order information from a small calibration set; fast but slightly less accurate than AWQ on instruction-tuned models.
GGUF: Container format for quantized models in the llama.cpp ecosystem; carries weight tensors at variants from Q2_K (2-bit) to Q8_0 (8-bit) with metadata.
Decision owner: The accountable person who accepts the trade-off and funds the next action.
Semantic contract: The shared definition of business terms, metrics, entities, and access rules used by tools and teams.
Control plane: The layer that applies policy, access, lineage, monitoring, and escalation across the operating model.
Evidence grade: A label that separates verified fact, directional signal, modeled assumption, and field observation.

Sources

Full card available to members. What the full card adds: the full decision framework · the SAP vs Snowflake / Databricks / Fabric comparison · the common pitfalls and their fix · the cheat sheet · the architecture schemas · the code blocks.

Open in the app →