VARIFOLD / LEARNING
Mathematics
for AI Safety
Based on Professor Yevgeny Liokumovich's Mathematics for AI Safety course at the Fields Institute. Varifold's edited notes bring the material together by topic and reading order.
Fields Institute course ↗YouTube recordings ↗Source archive
Explore the mathematical ideas behind model behavior, AI development, and alignment. Follow a reading path or begin with the question you want to understand.
Open the reading guide →Browse the papers
Choose a starting point
Understand how models learnStart with probability and loss, then move to scaling and generalization.Understand safety risksBegin with capabilities, incentives, and research feedback.Find a research problemConnect open questions to the concepts and papers they need.
Build the foundations
Read these topics in order to connect model predictions to learning behavior.
Language models and lossWhat does a model predict, and what does its training objective measure?Background: Probability and logarithmsScaling and computeHow do data, parameters, and compute shape prediction loss?Background: Language models and lossGeneralization and learning dynamicsWhat explains behavior on new data and changes in learned features?Background: Loss, gradients, and linear algebra
Explore safety questions
Choose a question and follow its connections to the foundations.
Capabilities and AI research feedbackHow could automation, incentives, and research feedback change AI development?Background: Entry point for safety motivationRepresentations and interpretabilityWhat features does a model use, and how can we understand or change its behavior?Background: Learning dynamics and linear algebraAgency and interactionHow do goals, corrections, and interactions shape agent behavior?Background: Probability and expected reward
Find a research question
The research guide groups nine open directions by learning theory, interpretability, AI research feedback, objectives, and multi-agent behavior.