Quantization in Machine Learning and Deep Learning

Floating-point precision (FP32) is the most common precision used for training new models, while FP16, INT8, and others are employed for quantization. Quantization involves converting higher-precision values to lower-precision values, resulting in a loss of precision that may degrade model prediction quality. However, quantization offers several benefits:

Just to do inference on a large language model (LLM) such as BLOOM-176B, you would need to have 8x NVIDIA 80GB A100 GPUs¹. Currently the NVIDIA A100 80GBs have an average price of $20,640.64 on GPU Poet.

To fine-tune BLOOM-176B, you'd need 72 of these GPUs!

Quantitizing has the following benefits:

• Reduced Memory Usage: For example, FP16 requires half the memory footprint of FP32 and INT8 half the memory of FP16. This enables training of larger models or larger mini-batches on the same hardware.
• Faster Computation: On many GPUs or TPUs (Tensor Processing Units), using FP16 or INT8 can speed up arithmetic operations over FP32, reducing training time.
• Improved Resource Utilization: Certain GPUs are optimized for FP16 operations, allowing for more efficient utilization of these resources.

Hardware-Native FP8 and FP4

Newer GPU generations go a step further than software-driven quantization: recent tensor/matrix cores natively execute FP8 and FP4 matrix multiplication in hardware, rather than emulating it on top of higher-precision units. NVIDIA's Hopper and Blackwell architectures and AMD's CDNA 3/4 datacenter parts publish dedicated FP8 and FP4 TFLOPS figures for exactly this reason — hardware-native low-precision inference is now a first-class performance metric, not just a memory-savings trick.

Because hardware FP8/FP4 support varies significantly by GPU — even within the same product generation — see GPUs ranked by $ / FP8 TFLOP and GPUs ranked by $ / FP4 TFLOP to compare cost-performance across cards that support these precisions in hardware.

Is Quantization Necessary?

FP16 is increasingly being used right from the start of training new models, particularly in deep learning. This approach is known as mixed precision training. It combines the use of FP32 and FP16 to optimize the training process.

Quantization is rapidly evolving. For example, in 2023 the 'LLM.int8() paper'² demonstrated that quantitizing down to 8-bit from 16-bit while keeping a couple of dimensions in high precision maintains zero-shot LLM performance. In their 2023 study, Liu et al. demonstrated that 4-bit floating-point quantization could effectively reduce model size without substantially compromising performance ³. I'm sure there are others, but I am aware of these.

To gain a more intuitive understanding of quantization, I recommend Tim Dettmers' blog posts:

• LLM.int8() and Emergent Features
• A Gentle Introduction to 8-bit Matrix Multiplication for transformers at scale using Hugging Face Transformers, Accelerate and bitsandbytes by Hugging Face.