3 hints available

MediumML EngineeringTheoryModel Training Distributed Training Quantization

FSDP Fine-Tuning of a 70B Parameter Model

Your team needs to fine-tune a 70B parameter language model. A single H100 GPU has 80GB of memory.

(1) At fp16 (2 bytes per parameter), how much memory does the model alone require? Does it fit on one H100? Show your math.

(2) Given this constraint, explain why pure data parallelism (DDP) is insufficient for this problem. Be specific about which step of DDP would fail.

(3) Describe the FSDP approach that would allow fine-tuning across 8 H100 GPUs. Walk through what gets sharded, when parameters are gathered, and why this works when DDP does not.

(4) Beyond model parameters, what other memory consumers must you account for during training? List at least three, and explain how each scales with model size.

Sign in to attempt this problem

Free account gives you full access to community problems with the complete solution reveal: golden answer, senior walkthrough, and score breakdown, after submission.

Start free →Already have an account? Sign in