Tabular Data

TL;DR Decision tree models have dominated machine learning for tabular data, and previous attempts to apply transformers to tabular data ended with no success. Recently, a new paradigm has emerged: train a large transformer model on synthetic datasets, then apply in-context learning to a new dataset to predict the entire test set labels. Multiple recent papers (TabPFN, TabICL, TabFM) show this approach outperforms tuned decision-tree models. However, per-prediction inference is actually slower than trees....

TL;DR Many machine learning systems deal with multimodal data, however there’s no study examining different design choices across modalities. The paper surveys common “tricks” for multimodal systems and found the most effective techniques are: (i) basic strategies such as gradient clipping and learning-rate warmup, (ii) late fusion using pretrained unimodal encoders, (iii) auxiliary cross-modal alignment objectives, (iv) simple input-level augmentation, and (v) modality dropout and learnable embeddings for handling missing inputs....