[Summary] Tabular Foundation Models: From TabPFN to TabFM
TL;DR Decision tree models have dominated machine learning for tabular data, and previous attempts to apply transformers to tabular data ended with no success. Recently, a new paradigm has emerged: train a large transformer model on synthetic datasets, then apply in-context learning to a new dataset to predict the entire test set labels. Multiple recent papers (TabPFN, TabICL, TabFM) show this approach outperforms tuned decision-tree models. However, per-prediction inference is actually slower than trees....