Transformer Models Explained Without the Scoreboard

Transformer models explained, in the 2017 paper that named them, are a sequence model built from attention. They are not a product, and they are not a way to make a model ignore its rules. This page describes the architecture the paper describes and points at the paper for every number. It is not advice about a medical, legal, or hiring decision.

If you want to call a model you already exported, that is a separate job, written up as how to implement AI into Java. This page does not train anything.

What transformer models explained refers to

The paper is Vaswani and coauthors, Attention Is All You Need. The abstract says the dominant sequence transduction models of that time were recurrent or convolutional networks with an encoder and a decoder, and that the best of them also connected encoder and decoder with attention. The authors propose a network, the Transformer, based solely on attention, dispensing with recurrence and convolutions entirely. They say experiments on two machine translation tasks showed the models to be higher quality, more parallelizable, and faster to train than the models they compared. This page does not repeat the scores, the hardware, or the hours. Those figures are in the paper. Quoting them here would turn a description into a leaderboard this site did not run.

The introduction says recurrent models compute step by step along the sequence, which makes parallel work inside one example hard, especially for long sequences. Attention, the paper says, models dependencies without regard to their distance. The Transformer uses attention instead of recurrence to draw those dependencies between input and output.

Encoder, decoder, and self-attention

Section 3 says most competitive sequence models have an encoder-decoder shape. The encoder maps the input symbols to a sequence of continuous representations. The decoder then generates output symbols one at a time, using what it has already generated as extra input. The paper calls that auto-regressive. The Transformer keeps that shape and fills both stacks with self-attention and position-wise feed-forward layers.

Self-attention, the background section says, relates different positions of a single sequence in order to build a representation of that sequence. In the encoder, each position can attend to the other positions of the encoder’s previous layer. In the decoder, self-attention is masked so a position cannot look at later positions. The paper says that mask, plus output embeddings shifted by one position, keeps the prediction at a position dependent only on outputs already known. Encoder-decoder attention is the third use: queries come from the decoder, and keys and values come from the encoder, so each output position can look across the input.

The paper describes attention as mapping a query and a set of key-value pairs to an output, computed as a weighted sum of the values. It then defines a scaled dot-product form and a multi-head form that runs several of those maps in parallel and combines them. The reason the paper gives for multiple heads is that one head averages, and averaging makes it harder to keep different kinds of relation at once. The exact widths are in the paper. The shape is the point of this page, not a copy of the table.

Why the paper adds a position signal

Section 3.5 says the model has no recurrence and no convolution, so it has to be given the order of the tokens some other way. The authors add positional encodings to the input embeddings at the bottom of both stacks, with the same width as the embeddings so the two can be added. They use sine and cosine functions of the position. They also say they tried learned positional embeddings and saw nearly the same result, and they chose the sine and cosine version because they thought it might extend to longer sequences than the ones in training. That is a hypothesis in the paper, not a guarantee for a model you train later.

Section 4 compares self-attention with recurrent and convolutional layers on three ideas: work per layer, how much of that work can run at once, and the path length between distant positions. The paper’s point is that a shorter path makes long-range dependencies easier to learn. The table of costs is in the paper. This page does not reprint it. If you need the comparison, open the table rather than a summary that drops the conditions.

What the experiments are, without the numbers

The paper trains on machine translation and also tries English constituency parsing, to see whether the same architecture is useful beyond translation. It describes the data, the optimizer, dropout, and label smoothing in the training section. Those settings are part of the experiment, not a recipe this page is handing you. Follow the paper, or a library’s current guide, if you are actually training. A blog that copies a learning-rate formula from 2017 without the rest of the setup is how people waste a run.

The conclusion says the Transformer is a sequence transduction model based entirely on attention, replacing the recurrent layers common in encoder-decoder models with multi-head self-attention. It also says the authors planned to try other modalities and restricted attention for long inputs. That is a plan in the paper, not a claim about every later system that uses the word transformer. The definition stops at what that paper defined. Transformer models explained should not grow a product name in the retelling.

Do not treat a chat product as this diagram. A product may use a related network, a different training set, and extra rules. The diagram does not tell you those rules, and it does not tell you how to get around them. This page will not describe a bypass.

How to read the paper next

Read the abstract, then section 3, then the figure captions. When a sentence in a later article gives a score, check whether it matches the table or whether someone rounded it. If you cannot find the sentence in the paper, do not repeat it. This page is a map of the architecture, plus the link.

If your job is to fit a small table model with a held-out split, you do not need this paper yet. If your job is to serve an HTTP API from Java, you do not need this paper yet. Come back when someone asks what the word means. You should be able to say: encoder, decoder, self-attention, a position signal because there is no recurrence, and the numbers live in the paper.

Keep a copy of the link in the notes for the page you publish. The authors, the title, and the arXiv abstract page are enough for a reader to audit you. A paraphrase that adds a product name or a score you did not check is worse than a short page. The code link in the paper points at the tensor2tensor repository they used. That repository is historical context, not a current install guide. Prefer the paper for the definition and a maintained library’s own docs if you write code.

When you explain this to someone else, draw three boxes: the encoder stack, the decoder stack, and the attention that connects them. Then say the decoder cannot see future tokens while it generates. If the drawing needs a number, stop and open the paper. That habit matters more than memorizing a diagram from a slide.

The paper also shows attention maps in an appendix and says some heads appear to track syntactic relations. That is an inspection of their trained models, not a promise that every head in every later model means a grammar rule. Mention it as their observation, or leave it out. Do not turn an appendix figure into a claim about your own system.

Tomorrow, open the abstract page, write four sentences in your own words, and add the link. If one of the sentences contains a score, a layer count, or a training time, delete it and point at the paper instead. That is the whole explainer. Transformer models explained is done when a reader can find the source for anything you were tempted to quote.