Related Safety Papers

An annotated reading list on capabilities, learning, interpretability, oversight, guarantees, and agency. Add reading notes or project ideas alongside the primary papers and preprints.

Capabilities and scaling

• Scaling Laws for Neural Language Models — studies empirical scaling relationships between model size, data, compute, and language-model loss.
• Training Compute-Optimal Large Language Models (Chinchilla) — shows that compute-optimal language-model training requires scaling training tokens along with model size. Compare its compute-allocation conclusions with Kaplan et al.
• Forecasting time to automated superhuman coders (AI 2027 Timelines Forecast) — April 2025 forecast of timelines for superhuman AI coders, including the milestone of 30 times as many researcher copies working 30 times faster. The milestone combines capability, speed, and scale.
• Takeoff Forecast (AI 2027) — companion April 2025 forecast of AI research speedups.
• ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks? — a provisional match for the source example about scoring and flag validation. The caption does not name the paper; identification is based on the incident and benchmark context, so verify against the recording.

Generalization and learning theory

• Understanding Deep Learning Requires Rethinking Generalization — Zhang et al. (2017); demonstrates that neural networks can fit randomized labels, challenging classical intuitions about generalization.
• Deep Learning is Not So Mysterious or Different — Andrew Gordon Wilson (2025); develops an account of deep learning in terms of soft inductive biases. This appears to be the “soft inductive bias” reading mentioned in the source material.

AI risk and development trajectories

• Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development — Jan Kulveit, Raymond Douglas, Nora Ammann, Deger Turan, David Krueger, and David Duvenaud (2025); analyzes how competitive pressures during incremental AI development could reduce human influence.

References needing identification

• The source material describes a recent paper measuring the neural tangent kernel spectrum during training and tracking learned eigenfunctions. The auto-caption does not preserve the title or authors, so the identification remains unresolved; add it once the source reference is checked.

Representations and interpretability

• Sky sphere representation in language models — Yevgeny Liokumovich and A. Berdnikov (2026 preprint); geometric representations in language models.
• Toy Models of Superposition — studies how neural networks represent more features than they have dimensions.
• Towards Monosemanticity: Decomposing Language Models With Dictionary Learning — sparse feature decomposition in language models.
• A Mathematical Framework for Transformer Circuits — foundations for mechanistic interpretability.
• Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet — large-scale feature extraction and analysis.
• Golden Gate Claude — Anthropic's 24-hour research demo of steering Claude 3 Sonnet by amplifying an interpretable Golden Gate Bridge feature; a control example, not a formal safety proof.
• Evaluating feature steering: A case study in mitigating social biases — evaluates potential benefits and limitations of steering features in Claude 3 Sonnet.

Multi-agent interaction

• Cooperative and uncooperative institution designs: Surprises and problems in open-source game theory — Andrew Critch, Michael Dennis, and Stuart Russell; studies strategic interaction when agents can inspect one another's programs.

Scalable oversight

• Measuring Progress on Scalable Oversight for Large Language Models — proposes empirical ways to study human supervision with model assistance.
• AI Safety via Debate — introduces debate as a method for eliciting supervision signals.
• Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision — studies supervision of stronger models using weaker supervisors.

Safety guarantees

• TorchLean: Formalizing Neural Networks in Lean — a Lean 4 framework for neural-network specification, execution, and verification; demonstrated on certified robustness, PINN residual bounds, and neural-controller verification. See the TorchLean project and its verification overview for implementation details and trust boundaries.
• Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks — an early example of formal verification for neural networks.

Agency and decision theory

• Quantifying stability of non-power-seeking in artificial agents — Evan Ryan Gunter, Yevgeny Liokumovich, and Victoria Krakovna (2024 preprint); studies when non-shutdown-avoiding behavior remains stable under changes in an environment.
• The Off-Switch Game — a decision-theoretic model of corrigibility and shutdown incentives.