[Concept] Scaling Laws of ML Models

TL;DR: Modern LLM training is long and costly. To plan the best configuration of compute, model size \(N\), and dataset size \(D\) for a test-loss target, scaling laws are fit by training small models and modeling the relationship between compute, \(N\), \(D\), and test loss. That relationship is then extrapolated to bigger models. This is my summary of Scaling Laws, Carefully Rosenfeld Et Al Scaling Laws (2020) Modeling the error as a joint function of both model size and data size, across a diverse set of architectures (ResNet, WRN, LSTM, Transformer) and optimizers (Adam, SGD variants)....

November 13, 2026 · 4 min · 758 words

[Summary] Training Vision Transformers with Only 2040 Images

TL;DR Vision Transformers (ViTs) outperform Convolutional Neural Networks (CNNs) with sufficient data but are data-hungry, limiting their use with small datasets. The authors propose a method to train ViTs with limited data by pre-training with label smoothing, lower resolution images, and parametric instance discrimination, followed by fine-tuning on the target task. Method Training a Vision Transformer on small datasets involves two steps Self-supervised pretraining: Parametric instance discrimination: Classify each image as its own class....

February 15, 2025 · 2 min · 217 words