The Transformer: How 'Attention Is All You Need' Reshaped AI
Some papers arrive and are filed away by Friday. A handful quietly become the default setting for an entire field. "Attention Is All You Need" โ posted to arXiv in June 2017 as paper 1706.03762 and presented that December at the 31st Conference on Neural Information Processing Systems (NeurIPS 2017) in Long Beach โ belongs in the second group. Authored by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, ลukasz Kaiser, and Illia Polosukhin, all working at Google Brain or Google Research, it proposed a network architecture that did away with recurrence altogether and called it the Transformer.
The problem it solved
By 2017, the workhorse of machine translation was the sequence-to-sequence model: a recurrent network that read an input sentence token by token, and a second one that generated a translation. Attention, refined since the 2014 work of Bahdanau, Cho, and Bengio, helped these models align source and target words. But the underlying recurrent and LSTM machinery had structural limits.
Recurrence processes a sequence in order, so each position depends on the one before it. That serial dependency makes parallelism hard, which is a problem as training moves to fast multi-GPU hardware. It also makes long-range information fade: by the time a recurrent net reaches the end of a long sentence, its memory of the start is thin. Convolutional alternatives parallelized better but did not touch those distant connections directly.
What the Transformer did differently
The Transformer dropped recurrence and convolution entirely. It relies solely on an attention mechanism to draw global dependencies between input and output. The core is self-attention: for every position in a sequence, the model weighs how relevant every other position is, so any two words can relate no matter how far apart they sit.
Two refinements made this work well. Multi-head attention runs several attention computations in parallel โ the paper uses eight heads โ letting the model capture different kinds of relationships at once, rather than averaging them into one blurred picture. And because the architecture has no notion of word order thrown in by recurrence, the authors added positional encodings: signals that tell each position where it sits in the sequence.
On the two translation tasks they tested, the results were striking. On WMT 2014 English-to-German, the base Transformer scored 27.3 BLEU โ a single model beating the previous best ensemble result โ and the larger big configuration reached 28.4, a new state of the art more than two BLEU points ahead of everything that had come before. On English-to-French, the big model set a new single-model record of around 41 BLEU, ahead of the prior best single model's 39.9. Crucially, all of this came while being substantially more parallelizable and requiring far less time to train โ the very knobs that had been blocking earlier models.
The impact
The Transformer did not wait to be noticed. A year later, in June 2018, researchers at OpenAI, led by Alec Radford, published "Improving Language Understanding by Generative Pre-Training": a decoder-only Transformer trained as a language model and then fine-tuned for many NLP tasks. It was the seed of the GPT line.
Four months after that, in October 2018, Google's Jacob Devlin and colleagues published "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" (arXiv:1810.04805). BERT kept the Transformer encoder, added bidirectional context via masked language modeling, and reset the state of the art on a broad set of benchmarks.
The two papers made opposite bets โ GPT read left to right and could generate; BERT read both directions and could not. Their shared breakthrough was the same: pretrain a Transformer on unlabeled text, then fine-tune it cheaply. That recipe became the foundation of modern deep learning. Every large language model you have used in recent years is a direct descendant of the architecture this paper proposed in a few pages. Recurrence is gone, and attention, the idea that could once be added onto an RNN, is all we need.