Blog posts

Spin-Model Transformers

A non-equilibrium statistical mechanics perspective on transformers

Last updated on Dec 7, 2023 51 min read

Spin-Model Transformers

Transformers Are Secretly Collectives of Spin Systems

A statistical mechanics perspective on transformers

Last updated on Nov 29, 2021 15 min read

Transformers Are Secretly Collectives of Spin Systems

Transformers from Spin Models: Approximate Free Energy Minimization

How far can we push the idea of transformers as physical systems?

Last updated on Oct 12, 2021 25 min read

Transformers from Spin Models: Approximate Free Energy Minimization

Deep Implicit Attention: A Mean-Field Theory Perspective on Attention Mechanisms

Can we model attention as the collective response of a statistical-mechanical system?

Last updated on May 11, 2021 20 min read

Deep Implicit Attention: A Mean-Field Theory Perspective on Attention Mechanisms

Attention as Energy Minimization: Visualizing Energy Landscapes

Can we swap softmax attention for energy-based attention?

Last updated on Mar 18, 2021 19 min read

Attention as Energy Minimization: Visualizing Energy Landscapes

Transformer Attention as an Implicit Mixture of Effective Energy-Based Models

Where does the energy function behind Transformers' attention mechanism come from?

Last updated on Dec 30, 2020 14 min read

Transformer Attention as an Implicit Mixture of Effective Energy-Based Models

An Energy-Based Perspective on Attention Mechanisms in Transformers

Can an energy-based perspective shed light on training and improving Transformer models?

Last updated on Dec 3, 2020 25 min read

An Energy-Based Perspective on Attention Mechanisms in Transformers