Why 4-Bit Quantization Is Changing Local AI Computing

The past two years of large language model development have been dominated by a race for ever-larger parameter counts. But VRAM has become the most tangible constraint on practical deployment in self-hosted computing environments. 4-bit quantization techniques such as AWQ and GPTQ offer an efficiency leap that was previously thought impossible.

Why memory, not speed, is the bottleneck

The most common misconception about language model inference is that it is a question of compute speed. In reality, when generating tokens one at a time, the GPU spends most of its time waiting for model weights to move from VRAM to the compute units. This operation is called memory-bound: the number of arithmetic operations per byte moved is very low.

The consequence matters. Shrinking weight size does not just save storage space, it directly speeds up inference, because every byte that does not have to be moved is waiting time eliminated. This is why quantization delivers a speed gain, not merely a memory saving.

How selective quantization works

Naive quantization — rounding all weights uniformly to 4-bit — destroys model accuracy. Modern approaches start from the observation that not all weights matter equally. A small fraction of weights, often under one percent, has an outsized influence on the output because it corresponds to high-magnitude activations.

AWQ identifies those important weights by analyzing activation statistics on a calibration sample, then protects them with scaling before quantization. GPTQ takes a different route: it quantizes weights column by column while compensating for the error accumulated in columns not yet processed, using second-order information from the loss function.

Both share the same principle — quantization error is not avoided, but steered to where it does the least damage.

The realistic numbers

Compressing weights from 16-bit precision to a 4-bit representation cuts memory bandwidth requirements by around 70 percent. A 70-billion-parameter model that once demanded more than 140 GB of VRAM can be squeezed to roughly 40 GB, enough for two high-end consumer graphics cards, with inference speeds in the range of 20 to 25 tokens per second.

Perplexity evaluations show a drop of under two percent compared with the base model. But perplexity hides something: the degradation is not spread evenly. Chain-of-thought reasoning, multi-step arithmetic and structured function calling degrade far more noticeably than summarization or everyday conversation.

What rarely gets mentioned

Quantization is not free. Calibration requires representative data; a model calibrated on English text often shows greater degradation on other languages, Indonesian among them. Kernel support is also uneven — some formats only run optimally on certain GPU architectures, and running them on unsupported hardware is actually slower than full precision.

For developers who put data sovereignty first, the breakthrough still opens a real opportunity: internal assistants with no dependence on third-party APIs, at a one-time hardware cost instead of a per-token bill. The practical advice is simple — measure quality on your own tasks, not on public leaderboards, because that is where the two percent difference shows its true form.