Quantization — smaller, faster models
Quantization reduces the numerical precision of a model's weights (and sometimes activations) — e.g. from 16-bit floats to 8-bit or 4-bit integers. Effects:
- Less memory — a 4-bit model is ~4× smaller than 16-bit, so it fits on smaller/cheaper GPUs.
- Faster, cheaper inference — less data to move and compute.
- Small quality loss — modern methods (GPTQ, AWQ, bitsandbytes) keep degradation minor, especially at 8-bit; 4-bit trades a bit more.
Two flavors: post-training quantization (quantize a trained model — easy, common) and quantization-aware training (train with quantization simulated — more work, better low-bit quality). For serving, post-training 4-bit/8-bit is the common lever to cut inference cost (Cost course).
Training at scale (know the vocabulary)
When a model or dataset is too big for one GPU:
- Data parallelism — replicate the model across GPUs, each processes a different batch shard, then sync gradients. Scales throughput; each GPU still holds the whole model.
- Model / tensor parallelism — split the model itself across GPUs when it doesn't fit on one.
- FSDP (Fully Sharded Data Parallel) / ZeRO — shard the model weights, gradients, and optimizer state across GPUs so each holds only a slice — the standard way to train large models memory-efficiently.
- Pipeline parallelism — split layers across GPUs in a pipeline.
Most application teams don't do large-scale pretraining, but you should recognize these terms and know FSDP/ZeRO is how big models are trained without any single GPU holding everything.
The efficiency toolkit
Putting it together: PEFT (LoRA/QLoRA) to fine-tune cheaply, quantization to serve cheaply, mixed precision and FSDP/ZeRO to train big models. Each trades a little quality or complexity for large memory/cost savings — and you validate the trade against your eval set.