Foundation Models

TL;DR Decision tree models have dominated machine learning for tabular data, and previous attempts to apply transformers to tabular data ended with no success. Recently, a new paradigm has emerged: train a large transformer model on synthetic datasets, then apply in-context learning to a new dataset to predict the entire test set labels. Multiple recent papers (TabPFN, TabICL, TabFM) show this approach outperforms tuned decision-tree models. However, per-prediction inference is actually slower than trees....

TL;DR TimesFM is a 200M-parameter decoder-only transformer trained on ~100B timepoints. It treats time-series patches the way LLMs treat tokens. In zero-shot, it matches or beats supervised SOTA on standard benchmarks while costing a fraction of LLM-based approaches like LLMTime. Motivation Classical methods (ARIMA, ETS) fit per-series and cannot transfer across datasets. LLMTime repurposes GPT-3/LLaMA-2 as zero-shot forecasters but is expensive and underperforms supervised models. NLP and CV have foundation models, but time series is harder: no discrete vocabulary, variable context/horizon/granularity, and far less public data....