Entropy Production in Non-Equilibrium Neural Networks
An exercise in cybernetics
Spin-Model Transformers
A non-equilibrium statistical mechanics perspective on transformers
Transformers Are Secretly Collectives of Spin Systems
A statistical mechanics perspective on transformers
Transformers from Spin Models: Approximate Free Energy Minimization
How far can we push the idea of transformers as physical systems?
Deep Implicit Attention: A Mean-Field Theory Perspective on Attention Mechanisms
Can we model attention as the collective response of a statistical-mechanical system?
Attention as Energy Minimization: Visualizing Energy Landscapes
Can we swap softmax attention for energy-based attention?
Transformer Attention as an Implicit Mixture of Effective Energy-Based Models
Where does the energy function behind Transformers' attention mechanism come from?
An Energy-Based Perspective on Attention Mechanisms in Transformers
Can an energy-based perspective shed light on training and improving Transformer models?