Blog posts

Last updated on Feb 4, 2026 4 min read

Exploring applications of spin-model transformers

Last updated on Dec 7, 2023 51 min read

A non-equilibrium statistical mechanics perspective on transformers

Last updated on Nov 29, 2021 15 min read

A statistical mechanics perspective on transformers

Last updated on Oct 12, 2021 25 min read

How far can we push the idea of transformers as physical systems?

Last updated on May 11, 2021 20 min read

Can we model attention as the collective response of a statistical-mechanical system?

Last updated on Mar 18, 2021 19 min read

Can we swap softmax attention for energy-based attention?

Last updated on Dec 30, 2020 14 min read

Where does the energy function behind Transformers' attention mechanism come from?

Last updated on Dec 3, 2020 25 min read

Can an energy-based perspective shed light on training and improving Transformer models?

Matthias Bal