A Large-D Closure for Asymmetric Kinetic Vector-Spin Systems
Account for fluctuations missing from naive mean field on the sphere in high dimensions.
Last updated
Nonequilibrium Dynamics in Spin-Model Transformers
Connect dynamical mean-field descriptions for vector spins to nonequilibrium concepts and provide a full mapping to a parallel transformer block.
Last updatedBuilds on
Spin-Model Transformers
Move beyond symmetric equilibrium models to asymmetric, driven vector-spin models evolving in time.
Last updatedCul-de-sac: Equilibrium methods and symmetric couplings
Equilibrium methods assume symmetric couplings and detailed balance. Softmax attention is asymmetric, so we need to come up with a dynamical description for vector spins that supports asymmetric couplings.
Transformers Are Secretly Collectives of Spin Systems
Distill the emerging physical picture into a unifying perspective.
Last updated
Transformers from Spin Models: Approximate Free Energy Minimization
Recover transformer-like structure from a vector-spin free energy.
Last updated
Deep Implicit Attention: A Mean-Field Theory Perspective on Attention Mechanisms
Treat attention as a self-consistent mean-field response of vector spins.
Last updatedCul-de-sac: Energy-based perspective
Energy-based methods are unable to recover the full structure of the transformer module architecture (residual, normalization, feed-forward, ...), so we need to jump to another idea basin. Focus on vector-spin stat-mech approaches following a simple data-probes-system-responds intuition.
Attention as Energy Minimization: Visualizing Energy Landscapes
Explore the geometry of attention as energy minimization.
Last updated
Transformer Attention as an Implicit Mixture of Effective Energy-Based Models
Derive effective energy models behind the attention update.
Last updated
An Energy-Based Perspective on Attention Mechanisms in Transformers
Review connections between softmax attention, associative memories, and Hopfield networks.
Last updated