AWQ or GPTQ for 4-bit weights?
目录
Answer #
Use AWQ as the default for production W4A16 when you want a short calibration and you do not want a reconstruction loop. Keep GPTQ if the pipeline already emits it, or if you are pushing toward 3-bit and AWQ is the less stable of the two. At 2-bit, the Chinese note says both fall over and you look at methods like QuIP#.
GPTQ admits the quantization error and spreads it onto columns that are not quantized yet, using the inverse Hessian. That rebuild is why a 175B run is described as hours, not seconds. AWQ leaves the stored weights alone and scales channels by activation magnitude. The diagnostic in the note: on OPT-13B INT3-g128, keeping about 1% of channels in FP16 by activation size returns WikiText-2 perplexity from 46.04 to 10.51 (FP16 baseline 10.13); picking those channels by weight size does not.
What it does not decide #
Neither method quantizes activations or the KV cache. GGUF is a file format for llama.cpp, not a third algorithm in this pair. Which engine can run the file is a separate choice.