Lesson · ⏱ 8 min read

🔦 Transformers and attention: how LLMs read

Q: What does attention help a transformer do?

Use the context of other words to understand each word. Attention mixes in information from relevant words, so each word's meaning depends on context.

🏠 Real-life example

Read: 'The trophy didn't fit in the suitcase because it was too big.' What does 'it' mean? You automatically look back at 'trophy'. Attention lets a model do the same: for every word, it decides which other words to focus on. A transformer is built from many layers of this attention.

💡 In simple words

Text is first split into tokens (word pieces) and each token becomes a vector (an embedding). In each transformer layer, self-attention lets every token gather information from every other token, weighted by relevance, so meaning depends on context ('bank' of a river vs a bank account). Stacking dozens of layers builds a deep understanding. Large language models (GPT, Claude, Llama) are giant transformers trained to predict the next token. Transformers also power modern image, speech and code models.

🎬 See how it works

When the model reads 'it', attention links it to every other word. The strongest link is to 'trophy', so the model understands 'it' means the trophy.

🪜 Step by step

Split text into tokens and turn each token into a vector (embedding), plus information about its position.
Self attention: each token compares itself with every other token (query vs keys) to decide which ones matter for its meaning.
It then mixes in information from the important tokens (values). 'Bank' near 'river' becomes different from 'bank' near 'loan'.
Stack many such layers, each with attention and a small feed forward network, connected by residuals and normalisation.
For an LLM, the final layer predicts the next token. Generating text repeats this one token at a time.

🔑 Key terms

Token: A piece of text, roughly ¾ of an English word on average.
Self-attention: Each token weighing how relevant every other token is.
Context: The surrounding text that shapes a word's meaning.

👀 See it in action

text

Sentence: "I deposited cash at the bank"
Attention for 'bank' → strongest links: 'deposited' (0.41), 'cash' (0.33)
→ model understands: bank = financial institution, not a river bank

Attention weights (illustrative numbers) show which words shaped the meaning.

⚠️ Common mistakes to avoid

Thinking attention cost is small. It grows with the square of the text length, which is why long contexts are expensive.
Confusing training (learning weights) with prompting (using a trained model). Most product work is the second.

✅ Quick check

What does attention help a transformer do?

Store files on disk
Use the context of other words to understand each word
Make training data smaller

Show answer

B. Use the context of other words to understand each word: Attention mixes in information from relevant words, so each word's meaning depends on context.

🙋 Questions learners ask

What is multi head attention?

Several attention mechanisms in parallel, each free to focus on different relationships, like grammar in one head and meaning in another.

Are transformers only for text?

No. They power image, audio, video and multimodal models too, by turning those inputs into tokens as well.

📌 Remember

Tokens → vectors → many attention layers → next-token prediction.

📖 Words from this lesson (AI dictionary)

Learn this as part of a full course

Track your progress, take the quick checks and follow the lessons in the right order. This lesson is part of:

ML Engineer · AI Research Engineer

Continue in the course