How much quality do you lose when you quantize a local LLM?

The shape of the curve matters more than any single number

Quantization stores each model weight in fewer bits. A 27B-parameter model at BF16 needs roughly 55 GB on disk; the same model at about 4.5 bits per weight is roughly 17 GB. The obvious question is what you paid for that reduction.

The measured answer is not a straight line. In the benchmark run cited below there is a long flat region where the quantized model matches the full-precision model within measurement noise, then a modest decline, then a collapse. Treating quantization as a smooth dial where 8-bit is a bit better than 4-bit and 2-bit is a bit worse than 4-bit will lead you to the wrong file.

That shape is why the practical advice is a threshold rather than a preference. You are not trading quality against size continuously. You are staying on one side of a cliff.

• 8-bit and 4-bit: no reliably measurable difference on the benchmarks tested.
• 2-bit: instruction-following largely intact, agentic coding measurably worse, more tokens spent per solved task.
• 1-bit: around random-guess performance on hard reasoning, and tool calling breaks.

What the published measurements actually show

The most detailed public run available today tested a 27B model at BF16, 8-bit Q8_0 at 29 GB, 4-bit Q4_K_M at 17 GB, 2-bit UD-Q2_K_XL at 10.7 GB, and 1-bit UD-IQ1_S at 6.2 GB, across three benchmarks: GPQA Diamond for graduate-level science reasoning, IFBench for instruction following, and Terminal-Bench 2.1 for agentic coding.

The result worth remembering is that the 4-bit build replicated the full-precision score on Terminal-Bench 2.1, the hardest of the three. On GPQA Diamond the author first replicated the vendor's published BF16 number, which is the credibility check that most informal comparisons skip, and then found little difference down to 4-bit with only the 2-bit build scoring lower. On IFBench there was no change at all down to the 2-bit build.

The 2-bit result is more interesting than a single score suggests. It fell noticeably on agentic coding but stayed at a level the author compared to earlier frontier models, and on the tasks it did solve it used a similar number of turns while writing roughly a quarter more output tokens. That is a cost you feel as latency and token spend rather than as a wrong answer.

At 1-bit the numbers stop being a tradeoff. Scores landed around the random-guessing line on GPQA Diamond, and the smallest build fell below it. A detail worth internalising: at 1-bit, giving the model more reasoning effort made it worse, because it reasoned until the token budget ran out and returned nothing.

Why one benchmark run is not the whole truth

This is a single model family, tested by one author, on three benchmarks, with one set of quantization files from one vendor. It is unusually well documented for this kind of comparison, including a replication of the official full-precision numbers and a disclosure of roughly 3,000 USD in rented GPU cost, but it is still one data point about the shape of the curve rather than a constant that transfers to every model.

Separate factors move the result. The benchmark used an F16 KV cache regardless of weight quantization, at roughly 2.3 GB per 32k tokens, so it does not tell you what happens when you also quantize the KV cache; llama.cpp exposes that as its own setting with values from f32 down to q4_0. Quantization quality also differs by producer, because the calibration data and the per-layer bit allocation are choices, not a fixed formula.

Reasoning effort turned out to matter more than the quantization level in some of the tests. That is a useful warning: if you change two settings at once and the model gets worse, do not automatically blame the quantization.

A decision rule you can apply today

Start from memory, not from the quantization label. Pick the largest model whose weights and your intended context window both fit in the memory you have, then choose roughly a 4-bit build of that model. On current evidence that gets you most of the capability of the full weights.

Go above 4-bit only when you have a specific reason and the memory is free anyway. The evidence does not show a reliable benefit for 8-bit on task benchmarks, so spending memory there usually costs you context length or model size that would have helped more.

Go below 4-bit only for non-agentic work. A 2-bit build is a reasonable choice for chat, summarising, and general knowledge on hardware that cannot hold the 4-bit file. Do not put a 2-bit or smaller build behind a tool-calling agent.

Treat 1-bit files as experiments. The vendor that produces them says directly that they should not be used for tool calling, that they need a presence penalty of 1.5 or higher to stop looping, and that thinking must stay enabled or they may return nothing at all. That is not a quality tradeoff; it is a different kind of artifact.

• Fit the model and the context first, then choose the quantization.
• 4-bit is the default. Deviating needs a reason.
• Anything below 2-bit is out of scope for agents and tool use.
• KV-cache precision is a separate setting; change one thing at a time.

How to check it yourself in under an hour

Public benchmarks answer the general question. Your own prompts answer yours. Download two builds of the same model, one at roughly 4-bit and one a step smaller, and keep everything else identical: same runtime, same context length, same sampling settings, same KV-cache type.

Assemble twenty prompts drawn from work you actually do, with an answer you can grade quickly as right or wrong. Run both builds on all twenty, record the score and the wall-clock time, and only then decide. Twenty items will not give you a statistically tight result, but it will reliably catch a collapse, which is the failure mode that matters.

If you use tools or function calling, include at least five prompts that require a tool call. That is where low-bit builds fail first and where a chat-only test will not warn you.

What this answer does not claim

RepoRadar has not run these benchmarks. Every number here is attributed to a named published source and is reproduced rather than reproduced-and-verified by us.

The figures describe one model family at one point in time. A different architecture, a different quantization producer, or a mixture-of-experts model with a different sensitivity profile can move the cliff. The direction of the finding, that damage is nonlinear and that the collapse is below 2-bit, is more transferable than the exact scores.

Nothing here covers quantization-aware training or quantization for fine-tuning, both of which have different tradeoffs from the post-training quantization used to produce downloadable GGUF files.

A useful next action

Open the model page for whatever you are running locally, look at which quantization you actually downloaded, and check its bits per weight against the format table in the Hugging Face documentation. Many people are running a legacy or an unnecessarily large build without knowing it.

If you are below 4-bit and you use tool calling, move up one step and re-run your twenty prompts. If you are above 4-bit, try moving down one step and spending the freed memory on a longer context or a larger model instead.