Language Model Architectures: Transformers, Attention, and the Path from GPT-1 to GPT-4

All modern LLMs are Transformers: neural networks with self-attention. From GPT-1's decoder-only stack to GPT-4's multi-expert mixture, the core building blocks remain the same. Understanding attention, positional encoding, and scaling laws explains why Transformers scale better than RNNs and CNNs.

Self-Attention Mechanism

Self-attention computes relationships between all tokens: how much should each token attend to every other token?

import torch
import torch.nn.functional as F

def scaled_dot_product_attention(Q, K, V, mask=None):
    """
    Compute self-attention.
    Q, K, V: (batch, seq_len, d_model)
    """
    d_k = Q.shape[-1]
    
    # Compute attention scores
    scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(d_k)
    
    # Apply mask (to prevent attending to future tokens)
    if mask is not None:
        scores = scores.masked_fill(mask == 0, -1e9)
    
    # Softmax
    attention_weights = F.softmax(scores, dim=-1)
    
    # Apply to values
    output = torch.matmul(attention_weights, V)
    return output, attention_weights

Self-attention's parallelizability (vs. RNNs' sequential nature) enabled the scaling that made LLMs possible.

Conclusion

Transformers and attention are the foundation of modern LLMs. Understanding the mechanism explains why LLMs work and scales so well. The evolution from attention-is-all-you-need to modern LLMs has been about making attention more efficient and adding scale. Next: we'll explore training methodologies—pretraining, SFT, and RLHF.