Reinforcement learning

TL;DR Current AI systems do not learn autonomously after deployment. The usual process includes fine-tuning, in which a human carefully selects data for a specific task and tunes the model on it. This paper proposes combining learning from observation (System A), learning from action (System B), and a meta-controller (System M) that decides what and how to learn. Why Current AI Systems Do Not Learn The current AI paradigm focuses on scaling LLMs....

TL;DR Direct Preference Optimization is a method of fine-tuning Large Language Models (LLM) to better align their outputs with human preference. It’s used as a simpler alternative to RLHF since it can be directly applied to the model without needing a reward function nor reinforcement learning optimization. Method The authors propose to re-parameterize the reward model of RLHF to obtain the optimal policy in closed form. This enables to solve the standard RLHF problem using a simple classification loss....

TL;DR Machine learning models require a loss function to tune their parameters. Designing a loss function to reflect ambiguous human values poses a challenge, e.g., it’s not clear how to formulate a loss function to represent what is funny or ethical. To this end, a reward model is trained via human feedback. This reward model takes the model’s output and predicts a reward score that is then used by the model to optimize its parameters....