0:00 Multi-GPU Training
2:15 Cyclic Learning Rate Schedules
3:07 Mixup: Beyond Empirical Risk Minimization
3:44 Label Smoothing
4:28 Deep Double Descent
5:55 Transfer Learning
6:18 Mixed Precision Training (Theory)
10:00 Mixed Precision Training (Tutorial)
@CodeEmporium (Ajay's channel)
/ @codeemporium
Code for the video
gist.github.com/ajhalthor/140...
Code behind the DCGAN with Apex
github.com/NVIDIA/apex/blob/m...
Mixed Precision Training
arxiv.org/abs/1710.03740
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
arxiv.org/abs/1706.02677
Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
arxiv.org/abs/1807.11205
Don't Decay the Learning Rate, Increase the Batch Size
openreview.net/pdf?id=B1Yy1BxCZ
On the Variance of the Adaptive Learning Rate and Beyond
arxiv.org/abs/1908.03265
Cyclical Learning Rates for Training Neural Networks
arxiv.org/abs/1506.01186
The 1cycle policy
sgugger.github.io/the-1cycle-...
Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates
arxiv.org/pdf/1708.07120.pdf
Bag of Tricks for Image Classification with Convolutional Neural Networks
arxiv.org/pdf/1812.01187.pdf
mixup: Beyond Empirical Risk Minimization
arxiv.org/abs/1710.09412
Deep Double Descent
openai.com/blog/deep-double-d...
Deep Double Descent: Where Bigger Models and More Data Hurt
arxiv.org/abs/1912.02292
Reconciling modern machine learning practice and the bias-variance trade-of
arxiv.org/abs/1812.11118