[Concept] Scaling Laws of ML Models
TL;DR: Modern LLM training is long and costly. To plan the best configuration of compute, model size \(N\), and dataset size \(D\) for a test-loss target, scaling laws are fit by training small models and modeling the relationship between compute, \(N\), \(D\), and test loss. That relationship is then extrapolated to bigger models. This is my summary of Scaling Laws, Carefully Rosenfeld Et Al Scaling Laws (2020) Modeling the error as a joint function of both model size and data size, across a diverse set of architectures (ResNet, WRN, LSTM, Transformer) and optimizers (Adam, SGD variants)....