Physics Wiki

Bibliography and Paper Map — Neural Networks and Language Models

4 min read

Bibliography and Paper Map

The single place where identifiers live for the neural-networks course. A weekly note cites through this page and never from memory.

The existing identifiers were imported from the companion notes. The six records for the linearized-learning supplement were checked against arXiv on 2026-09-19 and linked below. An identifier check establishes the reference metadata; it does not independently validate every claim in the paper.

Unit 0 — Language as a statistical system

WorkIdentifierUsed in
Shannon, A Mathematical Theory of Communication (1948)journalWeek 1
Shannon, Prediction and Entropy of Printed English (1951)journalWeek 2
Lin and Tegmark, Critical Behavior in Physics and Probabilistic Formal Languages (2017)journalWeek 2
Gage, A New Algorithm for Data Compression (1994)journalWeek 2
Sennrich, Haddow and Birch, Neural Machine Translation of Rare Words with Subword Units (2016)arXiv:1508.07909Week 2

Unit 1 — Networks as many-body systems

WorkIdentifierUsed in
Cover, Geometrical and Statistical Properties of Systems of Linear Inequalities (1965)journalWeek 3
Gardner, The Space of Interactions in Neural Network Models (1988)journalWeek 4
Gardner and Derrida (1988)journalWeek 4
Engel and Van den Broeck, Statistical Mechanics of Learning (2001)bookWeeks 3–4
Mézard, Parisi and Virasoro, Spin Glass Theory and Beyond (1987)bookWeek 4
Hopfield (1982)journalWeek 5
Amit, Gutfreund and Sompolinsky (1985, 1987)journalWeek 5
Krotov and Hopfield, Dense Associative Memory (2016)arXiv:1606.01164Week 6
Demircigil and collaborators (2017)journalWeek 6
Ramsauer and collaborators, Hopfield Networks is All You Need (2021)arXiv:2008.02217Week 6
Lucibello and Mézard (2024)journalWeek 6
Hoover and collaborators, Energy Transformer (2023)arXiv:2302.07253Week 6
Krotov, A New Frontier for Hopfield Networks (2023)journalWeek 6, further reading
Krotov and collaborators, Modern Methods in Associative Memory (2025)arXiv:2507.06211Week 6, further reading

The linearized regime and the interpolation peak

Optional study after Weeks 3–4: nn-llm-linearized-learning. No lecture hours assigned.

WorkIdentifierNote
Jacot, Gabriel and Hongler, Neural Tangent Kernel (NeurIPS 2018)arXiv:1806.07572
Chizat, Oyallon and Bach, On Lazy Training in Differentiable Programming (NeurIPS 2019)arXiv:1812.07956
Belkin, Hsu, Ma and Mandal, Reconciling Modern Machine-Learning Practice and the Classical Bias–Variance Trade-off (PNAS 116, 2019)arXiv:1812.11118
Hastie, Montanari, Rosset and Tibshirani, Surprises in High-Dimensional Ridgeless Least Squares Interpolation (Ann. Statist., 2022)arXiv:1903.08560the asymptotics used in the text
Mei and Montanari, The Generalization Error of Random Features Regression (CPAM, 2022)arXiv:1908.05355
Advani, Saxe and Sompolinsky, High-dimensional dynamics of generalization error in neural networks (Neural Networks, 2020)arXiv:1710.03667
Negri and collaborators, Storage and Learning Phase Transitions in the Random-Features Hopfield Model (2023)journalthe bridge to Unit 1’s memory half
Bahri and collaborators, Statistical Mechanics of Deep Learning (Annu. Rev. Condens. Matter Phys. 11, 2020)journalsurvey
Roberts, Yaida and Hanin, The Principles of Deep Learning Theory (2022)arXiv:2106.10165book

Unit 2 — Building the transformer

WorkIdentifierUsed in
Vaswani and collaborators, Attention Is All You Need (2017)arXiv:1706.03762Weeks 7–8
Su and collaborators, RoFormer (rotary position embeddings)arXiv:2104.09864Week 7
Ba, Kiros and Hinton, Layer NormalizationarXiv:1607.06450Week 8
He and collaborators, Deep Residual LearningarXiv:1512.03385Week 9
Chen and collaborators, Neural Ordinary Differential EquationsarXiv:1806.07366Week 9
Dehghani and collaborators, Universal TransformersarXiv:1807.03819Week 9
Fan and collaborators, Addressing Some Limitations of Transformers with Feedback MemoryarXiv:2002.09402Week 9
Geiping and collaborators, recurrent-depth language model (2025)arXiv:2502.05171Week 9
Zhang, Recurrent Looped Transformer technical note(technical note; read 2026-09-13)Week 9
Mandt, Hoffman and Blei, stochastic gradient descent as approximate inferencearXiv:1704.04289Week 10
Radford and collaborators, Language Models are Unsupervised Multitask Learners (2019)institutional reportWeeks 11–13 laboratories

Unit 3 — Physics of what emerged

WorkIdentifierUsed in
Geshkovski and collaborators, interacting-particle view of transformersarXiv:2312.10794Week 11
Dong, Cordonnier and Loukas, Pure Attention Loses Rank Doubly Exponentially with DeptharXiv:2103.03404Week 11
Rende, Gerace, Laio and Goldt, factored attention and the inverse Potts problemarXiv:2304.07235Week 12
Cui and collaborators, positional–semantic transition in a solvable attention modelarXiv:2402.03902Week 12
Zhai and collaborators, attention entropy and training instabilityarXiv:2303.06296Week 13
Kaplan and collaborators, scaling lawsarXiv:2001.08361Week 13
Hoffmann and collaborators, compute-optimal trainingarXiv:2203.15556Week 13
Power and collaborators, grokkingarXiv:2201.02177Week 13
Schaeffer, Miranda and Koyejo, on emergent abilities and the choice of metricarXiv:2304.15004Week 13

Unit 4 — Diffusion and non-equilibrium thermodynamics

WorkIdentifierUsed in
Sohl-Dickstein and collaborators, Deep Unsupervised Learning using Nonequilibrium ThermodynamicsarXiv:1503.03585Week 14
Ho, Jain and Abbeel, Denoising Diffusion Probabilistic ModelsarXiv:2006.11239Week 14
Song and collaborators, score-based generative modelling through SDEsarXiv:2011.13456Week 14
Jarzynski, Nonequilibrium Equality for Free Energy DifferencesarXiv:cond-mat/9610209Week 14

Background texts

WorkIdentifier
Mehta and collaborators, A high-bias, low-variance introduction to machine learning for physicistsjournal
Prince, Understanding Deep Learning (2023)book

Maintenance

The six formerly pending arXiv identifiers for the optional learning supplement are now linked. Other journal and book entries retain their imported metadata and have not received a complete independent bibliographic audit in this revision. The companion repository’s bibliography is maintained separately and was not edited by this wiki revision.

Graph View

Backlinks