Distributed fine-tuning of an 8-billion-parameter transformer
I profiled and optimized distributed training on dual NVIDIA A100 GPUs, focusing on communication overhead, memory fragmentation, and scaling limits under heavily constrained hardware.
What I did
- Analyzed communication overhead for ZeRO-2 and ZeRO-3 optimizer stages
- Identified scaling bottlenecks through NCCL trace analysis
- Quantified how memory fragmentation limits batch size
- Improved total GPU utilization through configuration tuning
Result
Showed the tradeoffs between parameter sharding and throughput, and presented the findings to the University of Basel HPC research group.