Agency and Decision Theory

Planned topics include mathematical models of agency and goal-directed behavior, utility theory, sequential decision-making, causal models of incentives, instrumental goals, power-seeking, and corrigibility.

Add lecture notes, readings, and transcripts here as the course progresses.

Further reading

• Quantifying stability of non-power-seeking in artificial agents — preprint by Yevgeny Liokumovich, E. R. Gunter, and V. Krakovna, listed on the instructor's page.
• The Off-Switch Game — a decision-theoretic model of corrigibility and shutdown incentives.
• Appendix B: Utility Functions — background on utility representations.
• Mathematics for AI Safety — lecture notes with additional references on incentives and agency.