Deep Learning Fundamentals 2026: Neural Networks from Scratch

Deep learning is function approximation with trainable parameters, optimized using gradients on large datasets. In practice, it is a workflow: define a model (layers + activations), define an objective (loss + regularization), feed batches of data, and update parameters with an optimizer. It is not a guarantee of intelligence, and it is not a substitute for clean data, correct labels, or a stable evaluation protocol.

In 2026, the “baseline” expectation is that you can reproduce an experiment, monitor training, and diagnose failures. Many real-world failures are mundane: train/validation leakage, incorrect normalization, silent dtype promotions, or metrics computed on the wrong axis. You’ll avoid these by adopting a tooling checklist early: a pinned Python environment (e.g., uv/conda/poetry), a known PyTorch version, GPU drivers that match, and a single command to run training.

• Tooling checklist: Python 3.11+, PyTorch, CUDA toolkit (or CPU-only for learning), a package lockfile, and a simple project layout (data/, src/, runs/).
• Common mistake: updating packages mid-project and comparing results across different library versions without noticing.
• Practical outcome: you can run a toy model today and get the same learning curve tomorrow.

We’ll use PyTorch because it exposes tensors, gradients, and computation graphs clearly. The goal is not to memorize APIs; it’s to learn the mental model: tensors flow forward, losses summarize error, gradients flow backward, and optimizers adjust parameters.

Training begins with data, and most “model issues” are actually data pipeline issues. A clean pipeline has three parts: (1) a dataset that returns a single example, (2) a dataloader that batches and shuffles, and (3) splits that reflect how the model will be used.

Think in terms of contracts. A Dataset must implement __len__ and __getitem__, returning (x, y) in consistent types and shapes. A DataLoader collects examples into batches, often producing x with shape [B, ...] and y with shape [B] or [B, C]. Your code should treat batch size as a variable, not a constant—because the last batch may be smaller, and distributed training changes batching behavior.

• Splits: train/validation/test should be decided before training. The validation set is for model selection; the test set is for final reporting.
• Shuffle: shuffle training data; usually do not shuffle validation/test. Ensure shuffling is seeded for reproducibility.
• Transforms: apply data augmentation only on training data. Apply normalization consistently across splits.

Common mistakes include leakage (the same user or near-duplicate images appearing in both train and validation), and mismatched preprocessing (e.g., normalization computed on all data, including validation/test). Practical judgement: spend time verifying a few batches visually or statistically—print shapes, min/max values, and class counts per split. If you can’t describe your data distribution, you can’t trust your model’s metrics.

A tensor is a typed, shaped array: dtype (float32, int64), device (CPU/GPU), and shape (dimensions). Deep learning is tensor programming with gradients. The fastest way to improve is to become fluent in shapes and broadcasting.

Start with batching: instead of a single example x shaped [D], a batch is [B, D]. A linear layer then maps [B, D] to [B, H] via y = x @ W + b, where W is [D, H] and b is [H]. Broadcasting adds b to every row automatically; knowing when broadcasting happens (and when it should not) prevents subtle bugs.

• Shape checks: print x.shape, y.shape, and verify axes match your intent (batch first is a common convention).
• Dtypes: losses typically expect float logits and integer class indices; mixing float64 and float32 can slow training and cause device mismatches.
• Devices: move both model and data to the same device; inconsistent device placement triggers runtime errors.

PyTorch builds a computation graph dynamically during the forward pass when tensors have requires_grad=True. Calling loss.backward() traverses that graph to compute gradients for parameters. Debugging intuition: if a parameter’s gradient is None, it wasn’t used in the forward pass, it was detached, or you disabled gradient tracking (e.g., torch.no_grad()). If gradients are finite but training doesn’t move, check the learning rate, the loss scale, and whether you’re accidentally zeroing gradients at the wrong time.

A model’s behavior is determined by its objective. The loss function is the training signal; if it’s mis-specified, optimization will faithfully optimize the wrong thing. For classification, a common setup is linear outputs (“logits”) followed by a cross-entropy loss. For regression, mean squared error is typical, but robust alternatives (Huber, MAE) can be better when outliers matter.

Objective design includes more than the primary loss. You may add regularization terms (weight decay), constraints, or auxiliary losses. The key is to keep units and scales in mind: if you add two losses with wildly different magnitudes, one will dominate unless you weight them. Practical tip: log each component separately so you can see what drives learning.

• Logits vs probabilities: many PyTorch losses expect raw logits for numerical stability (e.g., CrossEntropyLoss combines softmax + NLL internally).
• Label formats: check whether your loss expects class indices ([B] int64) or one-hot vectors ([B, C] floats).
• Common mistake: applying softmax in the model and then using a loss that also applies softmax, causing training instability.

In your first model, you’ll implement a linear layer and minimize loss with gradient descent. Even in this simple case, good habits matter: verify the loss decreases on a small subset (overfit a tiny batch) before scaling up. If you can’t overfit 32 examples, something is wrong—often shapes, labels, or learning rate.

The training loop is where deep learning becomes engineering. A minimal loop has: set model to train mode, iterate over batches, compute predictions, compute loss, backpropagate, update parameters, and track metrics. You also need evaluation: switch to eval mode, disable gradients, compute validation metrics, and decide whether to checkpoint.

Order matters. In PyTorch, the standard pattern is: optimizer.zero_grad(), forward pass, loss, loss.backward(), then optimizer.step(). If you forget to zero gradients, they accumulate across steps and can explode. If you compute metrics under gradient tracking, you waste memory. If you evaluate with dropout or batch norm still in train mode, validation metrics will be noisy and misleading.

• Batching: ensure the loop works for any batch size; treat B as dynamic.
• Metrics: track loss and at least one task metric (accuracy, MAE). Compute metrics on detached tensors.
• Checkpoints: save model state, optimizer state, and epoch/step. Always keep the “best validation” checkpoint.
• Optimization: start with SGD for intuition, then use momentum/Adam for speed. Add learning-rate schedules once the baseline is stable.

Debugging basics are part of loop design. Add guards: check for NaNs/Infs in loss and gradients; clip gradients if values explode; print a single batch’s tensor stats (min/max/mean). Many “mysterious” failures are shape errors hidden by broadcasting—so assert expected shapes, especially for logits and labels.

If you can’t reproduce a run, you can’t trust improvements. Reproducibility is not only setting a seed—it’s controlling sources of randomness and recording the full experiment context. Start by seeding Python, NumPy, and PyTorch RNGs, and ensure dataloader workers are seeded as well. Then decide how strict you need to be: full determinism can reduce performance on GPU, but it’s valuable when debugging.

At minimum, log: code version (git commit), configuration (hyperparameters, model sizes), dataset version, random seeds, and hardware/software details (PyTorch/CUDA versions). Store learning curves (train/val loss, metrics), and keep checkpoints that match those logs. The point is to make comparisons fair: if two runs differ in both learning rate and data augmentation, you can’t attribute changes confidently.

• Determinism tradeoff: enable deterministic algorithms when diagnosing bugs; relax later for speed once the pipeline is stable.
• Experiment logs: use a simple JSON/YAML config plus a run directory with metrics CSV and checkpoint files.
• Common mistake: changing preprocessing silently (e.g., normalization constants) and believing the model change caused the metric shift.

A reproducible setup also makes debugging NaNs and exploding values easier: once you can rerun the same failing step, you can instrument it—print intermediate tensor stats, inspect gradient norms, and bisect changes. That discipline will pay off as models get deeper, optimizers more complex, and training runs longer.

Section 2.1: Linear models as neural networks

The simplest “neural network” is a linear model: a weighted sum of inputs plus a bias. Given an input vector x ∈ ℝ^d, a single neuron computes z = w·x + b. If you use a step function (output 1 if z ≥ 0 else 0), you recover the classic perceptron: it draws a linear decision boundary (a hyperplane) and separates the space into two halves. This is the core perceptron intuition—linear boundaries are powerful but limited: XOR-type problems cannot be solved by one hyperplane.

Replace the step with a smooth function, and you get logistic regression: ŷ = σ(z) where σ is the sigmoid. Logistic regression is still linear in the inputs; the nonlinearity only converts a score into a probability. That distinction matters: a single neuron cannot create curved boundaries in input space. The boundary is still w·x + b = 0.

Engineering judgement: treat “linear as a baseline.” When a dataset is linearly separable (or close), linear models train fast, are easy to debug, and often provide a strong reference to beat. A common mistake is to jump to a deep model when the problem is primarily feature engineering or data quality. Another mistake is forgetting the bias term—without b, the hyperplane is forced through the origin, reducing expressiveness and often harming accuracy.

• Forward pass: compute z, apply activation (identity for regression, sigmoid/softmax for classification).
• Loss: compute a scalar measuring mismatch between predictions and targets.
• Backward pass: compute gradients ∂Loss/∂w and ∂Loss/∂b, update weights.

This workflow—forward, loss, backward—is identical for deep networks; only the number of layers grows.

Section 2.2: MLP architecture: depth, width, and capacity

An MLP stacks multiple linear layers with nonlinear activations between them. A two-layer MLP (one hidden layer) looks like: h = φ(W₁x + b₁), then ŷ = g(W₂h + b₂). The key upgrade from logistic regression is the hidden nonlinearity φ. With it, the model can represent non-linear decision boundaries by composing simple transforms. In practice, even small MLPs can carve complex regions in input space.

Depth (number of layers) increases compositional power: deeper networks can reuse intermediate features, often representing complex functions more parameter-efficiently than a single extremely wide layer. Width (neurons per layer) increases capacity by providing more feature channels at each stage. Capacity is not “free”: too much capacity relative to data can overfit, memorizing training examples but failing to generalize.

Practical design heuristics for tabular or low-dimensional inputs:

• Start with 2–4 hidden layers and moderate width (e.g., 64–256 units) before trying very deep stacks.
• Prefer consistent widths (e.g., 128→128→128) for simplicity unless you have a reason to funnel or expand.
• Track parameter count: dense layers scale as O(in×out). A jump from 256 to 2048 units can explode memory and overfit risk.

Common mistakes: (1) forgetting to normalize inputs, causing some features to dominate early layers; (2) using an output activation that conflicts with the loss (e.g., applying sigmoid then using a “with logits” loss); (3) assuming deeper is always better—when optimization becomes unstable, a smaller model that trains cleanly will usually outperform a larger one that “should” be better but isn’t learning.

Section 2.3: Activation functions and their tradeoffs

Activations determine where nonlinearity enters the network and how gradients flow. Historically, sigmoid and tanh were popular because they are smooth and bounded. Their downside is saturation: for large |z|, the derivative approaches zero, so gradients vanish as they propagate backward. This often makes deep sigmoid/tanh networks train slowly or not at all without careful initialization and normalization.

ReLU (Rectified Linear Unit), φ(z) = max(0, z), is the modern default for MLP hidden layers because it is simple, fast, and does not saturate on the positive side. It tends to preserve gradient magnitude better than sigmoid. However, ReLU can “die”: if a neuron’s pre-activation becomes negative for most inputs, its output is always 0 and its gradient is 0, so it may never recover. This is more likely with high learning rates or poor initialization.

Practical alternatives in the ReLU family:

• Leaky ReLU: allows a small negative slope, reducing dead neurons.
• ELU/GELU/SiLU (Swish): smooth alternatives that can improve accuracy, especially in deeper networks, at slightly higher compute cost.

Engineering judgement: choose activations based on training stability first, then accuracy. For most dense networks, ReLU or GELU are strong defaults. If you see many activations stuck at zero, try Leaky ReLU, lower the learning rate, or revisit initialization. Another common mistake is adding an activation after the final layer without thinking: for regression, the output is often linear; for multi-class classification, you typically output logits (no activation) and let the loss handle softmax numerically stably.

Section 2.4: Backpropagation mechanics (manual to autograd)

Backpropagation is the chain rule applied efficiently across a computation graph. In the forward pass you compute intermediate values (z, h, ŷ). In the backward pass you reuse those intermediates to compute gradients with respect to each parameter. Conceptually, you push an “error signal” backward through each operation, multiplying by local derivatives.

Start with a tiny example: one hidden layer MLP for a single sample. Let z₁ = W₁x + b₁, h = φ(z₁), z₂ = W₂h + b₂, and loss L(ŷ, y) where ŷ is derived from z₂. Backprop computes: ∂L/∂W₂ = (∂L/∂z₂) h^T, ∂L/∂b₂ = ∂L/∂z₂. Then propagate to hidden: ∂L/∂h = W₂^T(∂L/∂z₂), and through activation: ∂L/∂z₁ = (∂L/∂h) ⊙ φ′(z₁). Finally: ∂L/∂W₁ = (∂L/∂z₁) x^T, ∂L/∂b₁ = ∂L/∂z₁.

Two practical habits make you effective at debugging backprop:

• Shape checking: every gradient must have the same shape as its parameter. If W is (out, in), then ∂L/∂W must be (out, in).
• Autograd checks: implement the manual backward for a small network and compare against an autograd framework using the same forward pass and loss. If they disagree, reduce the example (single sample, small dimensions) until you find the first mismatch.

Common mistakes: mixing up transposes, forgetting to average gradients over the batch, applying the derivative of an activation to the wrong variable (use pre-activation z for ReLU masks), and accidentally detaching tensors in code so autograd stops tracking operations.

Section 2.5: Weight initialization (Xavier/He) and gradient flow

Training fails in surprisingly quiet ways when gradients are unhealthy. If gradients shrink layer by layer, you get vanishing gradients: early layers learn extremely slowly. If gradients grow, you get exploding gradients: loss becomes NaN, or updates oscillate wildly. Both are strongly influenced by initialization because initial weights determine the scale of activations and derivatives throughout the network.

Initialization aims to keep the variance of activations (forward) and gradients (backward) roughly stable across layers. Two widely used schemes:

• Xavier/Glorot: designed for tanh/sigmoid-like activations. A common form samples W from a distribution with variance ≈ 2/(fan_in + fan_out).
• He/Kaiming: designed for ReLU-family activations. Variance ≈ 2/fan_in, compensating for half the activations being zeroed by ReLU.

Engineering judgement: match initialization to activation. Using Xavier with deep ReLU networks often under-scales activations, causing weak gradients; using He with tanh may over-scale and saturate. If you’re unsure, use He for ReLU/Leaky ReLU and Xavier for tanh.

How to detect gradient issues in practice:

• Log gradient norms per layer; look for layers with near-zero gradients or rapidly increasing norms.
• Monitor activation statistics (mean, std, fraction of zeros for ReLU). A very high zero fraction suggests dying ReLUs.
• Use gradient clipping if you see occasional spikes (common in recurrent/transformer setups, but sometimes helpful in MLPs too).

Common mistakes: initializing all weights to zero (symmetry prevents learning), using an excessively large learning rate and blaming initialization, and ignoring input scaling—if inputs vary by orders of magnitude, even perfect initialization cannot keep activations stable.

Section 2.6: Choosing losses for classification vs regression

The loss function defines what “good” means and determines the gradient signal your model learns from. A correct architecture with a mismatched loss can train slowly, converge to a wrong solution, or appear numerically unstable. The first decision is whether your target is continuous (regression) or categorical (classification).

Regression: For real-valued targets, common choices are MSE (mean squared error) and MAE (mean absolute error). MSE penalizes large errors more strongly and has smooth gradients; MAE is more robust to outliers but has a non-smooth point at zero (most frameworks handle it fine). If you need probabilistic regression, a Gaussian negative log-likelihood is often better than MSE because it can model uncertainty (predicting both mean and variance).

Binary classification: Use binary cross-entropy. In implementation, prefer the numerically stable “with logits” form: feed raw logits z (no sigmoid in the model output) into a loss that internally applies sigmoid in a stable way. A common mistake is applying sigmoid in the model and then using a logits-based loss, effectively applying sigmoid twice and weakening gradients.

Multi-class classification: Use softmax cross-entropy (categorical cross-entropy). Again, prefer the “with logits” variant: output a vector of logits and let the loss compute log-softmax stably. For label encoding, know whether your framework expects class indices (sparse) or one-hot vectors (dense), and avoid mixing them.

Practical outcome: when you wire the output layer and loss correctly, your forward pass produces logits or predictions with the right shape, and your backward pass delivers gradients of the right scale. When training “does nothing,” check this pairing first—loss/activation mismatches are among the highest-frequency bugs in from-scratch implementations.

Section 3.1: Gradient descent variants and batch mechanics

Gradient descent updates parameters \(\theta\) by stepping opposite the gradient: \(\theta \leftarrow \theta - \eta \nabla_\theta \mathcal{L}\), where \(\eta\) is the learning rate (LR). The big choice is how you estimate the gradient. Full-batch gradient descent uses the entire dataset to compute \(\nabla\mathcal{L}\); it is stable but expensive and often slow to improve because each step costs so much. Stochastic gradient descent (SGD) uses one example at a time; it is noisy but cheap and can escape flat regions. Mini-batch SGD is the practical default: use batches (e.g., 32–4096) to balance compute efficiency and gradient noise.

Batch size affects both the statistics of gradients and the hardware utilization. Larger batches reduce gradient variance, often allowing a larger LR, but they can also make training “too deterministic,” sometimes harming generalization and making sharp minima more likely. Smaller batches inject noise, which can act like regularization but may require a smaller LR to avoid exploding updates. A common mistake is changing batch size without re-tuning LR; as a rough starting heuristic, if you multiply batch size by k, try multiplying LR by k (linear scaling) and then validate stability. This heuristic breaks when optimization becomes curvature-limited, so treat it as a starting guess, not a rule.

• Gradient accumulation: If your GPU can’t fit a large batch, accumulate gradients over multiple micro-batches before stepping. This approximates a larger batch without changing model code.
• Shuffling: Shuffle each epoch; poor shuffling can create correlated gradients and unstable learning curves.
• Loss scaling by batch: Ensure your implementation averages the loss over the batch (or adjust LR accordingly). Summing vs averaging silently changes effective step size.

From a loss-landscape viewpoint, mini-batch noise is not purely bad. In narrow valleys, noisy gradients can prevent the optimizer from bouncing between steep walls. But if the noise is too high (tiny batches, heavy augmentation, high LR), the model may never settle and validation loss will oscillate or degrade.

Section 3.2: Momentum methods and acceleration

Momentum addresses a common curvature pattern: gradients point in consistently useful directions along some axes, but oscillate along others (the classic “ravine” picture). With plain SGD, you may zig-zag across the ravine and make slow progress. Momentum keeps a velocity vector \(v\) as an exponential moving average of gradients: \(v \leftarrow \mu v + g\), \(\theta \leftarrow \theta - \eta v\). Here \(g\) is the current gradient and \(\mu\in[0,1)\) is the momentum coefficient (often 0.9 or 0.99).

Intuitively, momentum is a low-pass filter: it suppresses high-frequency gradient noise and amplifies persistent directionality. This often lets you use a higher LR or reach a good loss faster. A practical benefit is smoother training curves—loss and accuracy improve with fewer sharp spikes.

Nesterov momentum (Nesterov Accelerated Gradient) is a small but useful tweak: it computes the gradient after “looking ahead” along the velocity direction. In many libraries, you enable it with a flag (e.g., SGD + nesterov=True). In practice, classic momentum and Nesterov are close; the main win is usually “use momentum at all,” not which variant.

• When momentum helps most: deep MLPs with ill-conditioned curvature, CNNs trained from scratch, and regimes where SGD is stable but slow.
• Common mistake: increasing both LR and momentum simultaneously without checking stability. High \(\eta\) and high \(\mu\) can create runaway velocity and divergence.
• Debug signal: if training loss decreases at first but then explodes, try lowering LR first; if it decreases but plateaus early, try adding momentum or increasing LR gradually with a schedule.

In modern deep learning, momentum-SGD remains a strong baseline, especially for vision models, because it can generalize well and behaves predictably under learning-rate schedules. Even if you ultimately use AdamW, keeping an SGD+momentum baseline is valuable for ablations and sanity checks.

Section 3.3: Adaptive methods (RMSProp, Adam, AdamW)

Adaptive optimizers scale updates per-parameter based on historical gradient magnitudes. RMSProp maintains an exponential moving average of squared gradients \(s\): \(s \leftarrow \beta s + (1-\beta)g^2\), then updates \(\theta \leftarrow \theta - \eta \frac{g}{\sqrt{s}+\epsilon}\). Parameters that consistently see large gradients get smaller effective steps; rarely-updated parameters get larger steps. This can be a big win when features are sparse or gradients differ greatly across layers.

Adam combines momentum (first moment) and RMSProp-style scaling (second moment): it tracks \(m\) and \(v\), bias-corrects them early in training, then uses \(\frac{m}{\sqrt{v}+\epsilon}\) for updates. Adam typically trains quickly and is forgiving when you’re prototyping architectures. However, a common pitfall is treating Adam’s default weight decay like L2 regularization inside the gradient update. In classic Adam, “weight decay” implemented as L2 penalty interacts with adaptive scaling in a way that is not equivalent to true decay.

AdamW fixes this with decoupled weight decay: it applies weight decay directly to parameters (a separate shrink step) rather than mixing it into the gradient. This small change is critical for modern transformer and large-scale training recipes, and it usually improves tuning consistency. If your library offers AdamW, prefer it over Adam when you want weight decay.

• Practical defaults: AdamW with \(\beta_1=0.9\), \(\beta_2=0.999\), \(\epsilon=10^{-8}\) is common; tune LR and weight decay.
• Weight decay scope: often exclude biases and normalization parameters (e.g., LayerNorm/BatchNorm scales) from decay.
• Common mistake: using a large weight decay with AdamW while also using heavy dropout and strong augmentation, leading to underfitting.

Optimizer choice is partly about the loss landscape and partly about workflow. If you need fast iteration and stable early learning, AdamW is a strong first choice. If you care about maximal generalization for some vision tasks, momentum-SGD is still competitive. The key is to pair the optimizer with an appropriate learning-rate schedule and to validate with controlled ablations.

Section 3.4: Learning-rate scheduling strategies

The learning rate is the single most influential hyperparameter for successful training. Schedules change LR over time to match different phases of optimization: early exploration, mid-training progress, and late-stage refinement. Many “mysterious” training wins are simply better LR schedules.

Warmup gradually increases LR from a small value to the target LR over the first N steps (often 100–5000 steps). Warmup reduces early instability when weights are random, activations can be poorly scaled, and gradients are volatile—especially with large batch sizes or transformers. Without warmup, you may see immediate divergence or loss spikes.

Cosine decay smoothly decreases LR from the peak value toward a minimum using a cosine curve. It avoids abrupt drops and often produces good final accuracy. A practical pattern is: warmup → cosine decay. Pick a minimum LR that is small but nonzero (or zero) depending on how long you train.

Restarts (cosine with restarts) periodically reset LR to a higher value, then decay again. This can help the optimizer jump out of shallow local basins and explore new regions. Restarts can be useful when you have long training runs and want robust performance without carefully placing manual step drops.

• Step decay: historically common (drop LR by 10× at set epochs). Works, but can be brittle if your epoch budget changes.
• Plateau-based: reduce LR when validation metric stops improving. Helpful when training time is unknown, but can react to noise in validation.
• Scheduling in steps, not epochs: prefer step-based schedules when batch size changes, so the schedule tracks optimization progress, not dataset passes.

One engineering rule: treat LR and schedule as part of the optimizer, not an afterthought. If you switch from SGD to AdamW but keep the same LR curve, you may be running an unfair comparison. Align your experiments: optimizer + schedule + batch size form a coupled system.

Section 3.5: Gradient clipping, mixed precision, and stability

Even with good optimizers, training can fail due to numerical or dynamical instability. Two common symptoms are exploding gradients (loss becomes NaN/Inf) and erratic spikes that derail progress. Stability tools are not “cheats”; they are standard engineering controls.

Gradient clipping limits the magnitude of gradients before the optimizer step. The most common form is global norm clipping: if \(\lVert g \rVert\) exceeds a threshold \(c\), scale all gradients so the norm becomes \(c\). This preserves direction while preventing rare catastrophic steps. Transformers and RNN-like architectures often benefit from clipping (e.g., clip norm 0.5–1.0), but it can also help dense networks when LR is aggressive.

Mixed precision (FP16/BF16) speeds training and reduces memory usage, enabling larger batches or models. However, reduced precision increases risk of underflow/overflow. Use automatic mixed precision (AMP) with dynamic loss scaling where available: it scales the loss up during backprop to keep gradients representable, then scales gradients back down before the optimizer step. BF16 is generally more stable than FP16 when supported, because it has a wider exponent range.

• Monitor: gradient norm, update norm, and loss for NaNs/Infs. Catch issues early.
• Initialization and normalization: poor initialization or missing normalization can amplify instability; don’t blame the optimizer first.
• Batch size and LR: if you see spikes, first lower LR or add warmup; if convergence is slow, consider increasing batch or using momentum/AdamW.

Stability is also about reproducibility. Fix random seeds when diagnosing, log exact optimizer settings, and record effective batch size (including accumulation). When a run “randomly” diverges, it is often because one of these details changed.

Section 3.6: Hyperparameter tuning workflow and ablations

Optimization improves fastest when you adopt a disciplined tuning workflow. Start by defining what “works” means: stable loss decrease, reasonable time-to-accuracy, and validation improvement without obvious overfitting. Then tune in layers, from most impactful to least.

Step 1: establish a baseline. Pick one optimizer family (often AdamW for quick iteration, or SGD+momentum for a strong classic baseline). Choose a simple schedule (warmup + cosine) and a modest batch size your hardware handles comfortably. Train for a short run that is long enough to see trend (e.g., 5–20% of full budget).

Step 2: sweep learning rate. LR dominates. Run a small log-scale sweep (e.g., 1e-5 to 3e-3 for AdamW, 1e-3 to 1 for SGD depending on setup). Look for: (a) divergence at high LR, (b) painfully slow learning at low LR, (c) a wide “good” region. Prefer settings with a margin of stability, not just the absolute best single run.

Step 3: tune batch size and schedule. If you increase batch size, re-check LR (often higher works) and consider longer warmup. Adjust total steps if compute changes. If late-stage progress is slow, lower the final LR (stronger decay) or extend training.

• Ablations: change one factor at a time (optimizer, decay, clipping, schedule). Otherwise you won’t know what helped.
• Track effective regularization: weight decay, dropout, augmentation, and early stopping interact. If validation lags far behind training, increase regularization; if both are bad, focus on optimization (LR/schedule/model).
• Use diagnostic plots: training vs validation curves, LR over time, gradient norms. These reveal whether you are underfitting, overfitting, or unstable.

The practical outcome of this workflow is confidence: you can tell whether an architecture change improved representational power or whether a training tweak merely changed optimization dynamics. That skill—separating modeling from optimization—is what lets you build neural networks from scratch that train reliably in 2026-scale workflows.

Generalization problems show up first in curves. If training loss keeps falling while validation loss bottoms out and rises, you have classic overfitting (high variance). If both training and validation losses plateau at a high value, you are underfitting (high bias): the model, features, or optimization are insufficient. The trick is to make sure your curves are telling the truth.

Start with splits that reflect deployment. Random splits are fine for i.i.d. data, but for time series you usually need chronological splits; for user-centric problems you want user-disjoint splits; for medical imaging you may need patient-disjoint splits. If the same identity appears in both train and validation, the model can “memorize” style rather than learn signal.

• Three-way split: train for gradient updates, validation for model selection, test for a final one-time estimate. Avoid “peeking” at test results during iteration.
• Leakage checks: ensure preprocessing (normalization statistics, vocabulary building, imputation) is fit on the training set only, then applied to validation/test.
• Baselines: compare to simple heuristics (majority class, logistic regression, small MLP). A strong baseline can reveal that your deep model’s gains are illusory.

Common mistake: using the validation set repeatedly to choose architectures and hyperparameters, then reporting that same validation score as “final.” Treat validation as a development tool, and keep an untouched test set (or use cross-validation if data is scarce). When data is small, variance is high; confidence intervals or repeated runs become part of honest evaluation.

Regularization is any technique that reduces effective model capacity or discourages brittle solutions. In practice, you will reach for three tools often: weight decay, dropout, and (for deep residual-style networks) stochastic depth. They are conceptually different and behave differently under modern optimizers.

Weight decay (L2 regularization) adds a penalty proportional to the squared weight magnitude, encouraging simpler functions. With SGD, it behaves like “shrink weights a little every step.” With Adam-family optimizers, prefer decoupled weight decay (AdamW) so the decay is not entangled with adaptive learning rates. Practical guidance: start with 1e-4 to 1e-2, and tune alongside learning rate; too much causes underfitting.

Dropout randomly zeroes activations during training, effectively training an ensemble of subnetworks. It can help most in fully-connected layers and smaller data regimes, but it can also slow convergence and interact with normalization. Typical dropout rates: 0.1–0.5 in MLPs; lower for CNNs; often unnecessary in large transformer models where other regularizers dominate. Remember to disable dropout at evaluation time (or use model.eval()).

Stochastic depth randomly drops entire residual branches during training (common in ResNets and modern vision backbones). It regularizes very deep models without the same noise pattern as dropout. You set a drop probability that increases with depth; at inference, all blocks are kept but scaled appropriately. This is particularly effective when depth is the main driver of capacity.

• Label smoothing is another practical regularizer for classification: replace hard one-hot targets with slightly softened targets (e.g., 0.9 for the correct class, remaining 0.1 spread over others). It can reduce overconfident predictions and improve calibration, but may slightly hurt top-1 accuracy in some regimes.
• Engineering judgement: if your training accuracy is near 100% quickly, add regularization; if training accuracy is low, regularization is not the bottleneck—fix optimization, data quality, or model capacity first.

Data augmentation is “regularization by generating plausible variation.” Unlike weight penalties, augmentation changes the training distribution to encode invariances you want the model to learn. Done well, it improves robustness and reduces reliance on spurious cues. Done poorly, it creates label noise and teaches the wrong invariances.

For vision, start with augmentations that match the task. For natural images, random crops/resized crops, horizontal flips (when semantics allow), color jitter, and mild rotations are common. For detection/segmentation, preserve geometry carefully and apply transforms consistently to images and labels. Stronger methods (MixUp, CutMix, RandAugment) can be powerful but can also shift calibration; monitor both accuracy and confidence behavior.

• Rule of thumb: if the augmentation could plausibly occur at inference time, it is usually safe; if it changes the label (e.g., flipping digits 6/9), it is not.
• Where leakage happens: do not augment validation/test. Your validation set should represent the real evaluation distribution, not an expanded training distribution.

For text, augmentation is more delicate because small edits can change meaning. Light-touch techniques include synonym replacement with constraints, back-translation, random deletion of stopwords, or span masking for self-supervised pretraining. In classification, you can also use paraphrasing models, but validate that label semantics remain stable. Another “augmentation” is simply more diverse sampling: balancing classes, adding hard negatives, or including adversarial examples that are realistic for your domain.

Practical workflow: introduce augmentation gradually. Train a baseline with minimal augmentation to establish a reference, then add one augmentation at a time. If performance improves but error types change (e.g., fewer false positives but more false negatives), incorporate that into thresholding and metric selection rather than assuming “higher accuracy” means “better model.”

Normalization layers are often the difference between “trains reliably” and “diverges mysteriously.” They stabilize activations and gradients, enabling higher learning rates and faster convergence. But each normalization method makes assumptions about batch structure and feature geometry, which affects both training dynamics and deployment behavior.

Batch Normalization (BatchNorm) normalizes activations using batch statistics (mean/variance) during training, and uses running averages at inference. It works extremely well in many CNNs, but it is sensitive to small batch sizes and distributed training details (synchronization). Common mistakes include evaluating in training mode (using batch stats at inference) and forgetting that tiny batch sizes can make BatchNorm noisy; in that case, consider SyncBatchNorm, freezing BN, or switching to GroupNorm/LayerNorm.

Layer Normalization (LayerNorm) normalizes across features within each example, independent of batch size. This makes it the standard in transformers and many sequence models. It interacts differently with dropout and residual connections: the placement (pre-norm vs post-norm) changes stability, especially in deep stacks.

Group Normalization (GroupNorm) is a compromise: normalize within groups of channels, giving BatchNorm-like benefits without batch dependence. It is popular in detection/segmentation where batch sizes are often small due to high-resolution inputs.

• Training dynamics tip: if loss spikes early or gradients explode, normalization plus a smaller learning rate warmup often fixes it.
• Debugging tip: track activation/gradient statistics. Saturated activations, huge variance, or NaNs often point to normalization misconfiguration, mixed-precision issues, or an out-of-range learning rate.

Normalization is not a free lunch: it can mask bugs (a model “sort of trains” even with poor initialization), and it changes how weight decay behaves (especially in transformers where you often exclude bias and norm parameters from decay). Treat normalization as part of the model’s design, not a bolt-on.

Accuracy answers “how often are we right?” but ignores which errors matter. In imbalanced datasets, a model can achieve high accuracy by predicting the majority class. Better evaluation starts by matching metrics to decisions: what happens if you miss a positive case, or if you raise a false alarm?

Confusion matrices are the first upgrade: they show false positives and false negatives per class. From them you derive precision, recall, and F1. Use F1 when you need a balance between precision and recall, especially under class imbalance. For multi-class problems, specify whether you use macro, micro, or weighted averaging; each tells a different story about minority classes.

AUROC measures ranking quality across thresholds, useful when you can adjust the decision threshold later. For heavily imbalanced data, also consider AUPRC (precision-recall curve area), which is often more sensitive to improvements on rare positives. Always report the operating threshold used in production; “threshold-free” metrics do not replace a chosen decision point.

Calibration answers: “when the model says 0.8 confidence, is it correct about 80% of the time?” Deep nets are often miscalibrated (overconfident), which matters for risk-sensitive applications. Practical checks include reliability diagrams and expected calibration error (ECE). If calibration is poor, try temperature scaling on the validation set, label smoothing, or revisiting augmentation strength (some augmentations can distort confidence).

• Common pitfall: tuning thresholds on the test set is leakage. Tune thresholds on validation, then evaluate once on test.
• Robustness evaluation: test across slices (device types, lighting conditions, demographic groups, text domains). Aggregate metrics can hide severe failures in a subgroup.

A model that is slightly less accurate but better calibrated and more robust can be the correct engineering choice, especially when predictions drive automated actions.

Once you have trustworthy evaluation, the question becomes: what do we do next? Error analysis turns metrics into an iteration plan. Start by sampling misclassified or high-loss examples from validation (not training). Categorize them: label errors, ambiguous cases, rare subtypes, domain shifts, or systematic confusions between specific classes. A small, structured review (even 50–200 examples) often reveals the highest-leverage fixes.

Use early stopping to prevent over-training: monitor a validation metric and stop when it stops improving for a patience window. Early stopping is a form of regularization and also saves compute, but it must be paired with clean model selection: if you run many experiments and pick the best validation score, you are implicitly overfitting to validation. Counter this with fewer, more principled sweeps, or use nested validation/cross-validation when stakes are high.

• Model selection discipline: decide the primary metric ahead of time (e.g., AUROC, macro-F1, or calibrated recall at a fixed precision). Avoid switching metrics after seeing results.
• Iteration menu: if errors are noisy labels, improve data; if errors are rare subtypes, collect targeted samples; if errors are domain shift, add augmentation or domain-specific data; if errors are overconfidence, apply calibration and consider label smoothing.
• Reproducibility: log seeds, data version, preprocessing, hyperparameters, and commit hash. Many “gains” vanish when you cannot reproduce them.

Finally, keep a clear separation between research curiosity and production decisions. In production, robustness and stability often dominate small average improvements. A disciplined loop—clean splits, regularization and normalization suited to your architecture, evaluation beyond accuracy, and deliberate error analysis—makes generalization a controllable engineering outcome rather than a hope.

Section 5.1: Convolution operations, padding, stride, dilation

A 2D convolution layer takes an input tensor shaped (C_in, H, W) (or (N, C_in, H, W) with batches) and produces (C_out, H_out, W_out). Each output channel has a learnable kernel (C_in, K_h, K_w). The same kernel weights are applied at every spatial location, which is the parameter-sharing property that makes CNNs efficient and translation-aware. Conceptually, each output value is a dot product between the kernel and a local patch of the input.

Padding controls what happens at the borders. With “valid” convolution (no padding), spatial size shrinks because border pixels have fewer full neighborhoods. With “same” padding, you add zeros (or another padding mode) so that H_out and W_out match the input size when stride=1. Engineering judgment: for early layers, “same” padding often preserves information and simplifies shape reasoning; shrinking too early can discard fine details and hurt small-object performance.

Stride is the step size of the sliding window. Stride=2 halves spatial resolution (approximately), acting like downsampling. This is common in modern CNNs where you replace pooling with strided convolutions, giving the network learnable downsampling. The trade-off is that aggressive stride can cause aliasing: the model may miss high-frequency details. A practical rule is to downsample gradually (e.g., every 2–3 conv blocks) unless you have very high-resolution inputs.

Dilation spaces out kernel elements, increasing the effective receptive field without increasing parameter count. A 3×3 kernel with dilation=2 covers a 5×5 area of the input (with gaps). Dilation is useful in segmentation and tasks where context matters, but it can introduce “gridding artifacts” if overused. When experimenting, change one thing at a time: stride affects resolution, dilation affects context, and padding affects border behavior—confusing them leads to shape bugs and silent performance drops.

• Common mistake: forgetting that convolution output size depends on (H, W, K, stride, padding, dilation), leading to mismatched tensor shapes in the head.
• Practical outcome: you should be able to design conv stacks on paper and predict output shapes before writing code.

Finally, remember why convolution works: locality reduces the number of connections, translation handling comes from weight sharing, and fewer parameters improves sample efficiency compared to dense layers on raw pixels.

Section 5.2: Receptive fields and hierarchical features

The receptive field of a unit is the region of the input image that can affect that unit’s value. In CNNs, receptive fields grow with depth: stacking small kernels (like 3×3) repeatedly increases the effective receptive field while keeping parameters manageable. This is one reason “many 3×3 layers” became a standard pattern: two 3×3 convolutions approximate a 5×5 receptive field with fewer parameters and more nonlinearities in between.

Hierarchical features emerge because early layers can only “see” local patches, so they tend to learn edges, color blobs, and simple textures. Middle layers combine those into motifs (corners, repeated patterns), and deeper layers build object parts and category-level abstractions. You can think of this as a learned feature pyramid: as spatial resolution shrinks (through pooling or strided conv), semantic richness increases (channels often increase).

Engineering judgment shows up when deciding where to downsample. If you downsample too early, deeper layers may never recover fine spatial details, which matters for small objects or thin structures. If you downsample too late, compute and memory grow quickly. A practical workflow is to define “stages”: each stage keeps resolution constant while increasing channels, then downsamples once to begin the next stage.

Receptive field is not only about geometry; it’s also about optimization. Even if the theoretical receptive field is large, the effective receptive field can be smaller due to how gradients distribute through many paths. Residual connections (covered later) help gradients flow so deeper layers can actually use broader context. When debugging poor performance, ask: does the model have enough context (receptive field) for the task, and is it trained well enough to use it?

• Common mistake: assuming “deeper is always better” without checking whether the task needs more context or more resolution.
• Practical outcome: you should be able to reason about why a classifier might confuse visually similar classes (needs better features) versus miss large-scale structure (needs larger receptive field or different downsampling schedule).

Interpretability ties in here: feature maps from early layers often look like oriented edges; later maps respond to textures or parts. Visualizing activations per stage is a concrete way to verify that the hierarchy you intended is actually forming.

Section 5.3: Common CNN architectures (LeNet to ResNet overview)

CNN design evolved from simple stacks to sophisticated systems, but the core building blocks are stable: conv → nonlinearity → normalization (often) → downsampling → head. LeNet-5 (1990s) popularized conv + pooling for digit recognition. It used relatively few layers, average pooling, and a small fully connected head—perfect for low-resolution images.

As datasets grew, VGG-style networks showed that depth plus consistent 3×3 convolutions can work well, at the cost of heavy computation. In parallel, Inception-style designs explored multi-branch modules to capture multiple receptive field sizes at once. These ideas matter conceptually, but modern practice converged on something simpler: staged networks with residual connections.

ResNet’s key innovation is the residual block: instead of learning a direct mapping F(x), the block learns a residual F(x) and adds the input back: y = x + F(x). This makes optimization easier because gradients can flow through the identity path, reducing vanishing-gradient problems and enabling much deeper networks. In practical terms, residual connections are a “default” for CNNs beyond a small baseline.

Normalization choices also became central. Batch Normalization (BN) stabilizes training by normalizing activations across the batch. It works very well when batch sizes are reasonably large and consistent. When batch size is small (common with high-resolution images), BN can become noisy; Group Normalization or Layer Normalization variants may be more stable. The engineering choice depends on hardware constraints and data regime, not just accuracy.

• Baseline CNN template: 3–4 stages; each stage has 2–3 conv blocks (Conv → Norm → ReLU), then downsample; end with global average pooling and a linear classifier.
• Common mistake: keeping a large fully connected head; global average pooling usually reduces parameters and overfitting.

Modern “heads” are often minimal. For classification, global average pooling followed by a linear layer is hard to beat. For small datasets, a smaller head can prevent the classifier from memorizing. For interpretability, a pooling-based head also makes class activation techniques easier to apply later.

Section 5.4: Regularization and augmentation for vision

Vision models overfit easily because images contain many spurious cues: background textures, lighting, camera artifacts, and annotation biases. Regularization is not a single trick; it is a system of choices that constrain learning and improve generalization. Weight decay (L2 regularization) is a strong default for CNNs. Dropout is sometimes helpful in the head, but in convolutional trunks it can be less effective than weight decay plus augmentation.

Augmentation is especially powerful for CNNs because it teaches invariances that convolution alone does not guarantee. Convolution provides translation equivariance (shifting input shifts feature maps), but classification often needs invariance to crop, scale, lighting, and mild rotation. Practical augmentations include random resized crops, horizontal flips (when label-preserving), color jitter, and mild blur/noise. Stronger policies (RandAugment, MixUp, CutMix) can further improve robustness, but they can also destabilize training if applied too aggressively early on.

Class imbalance is common in real image datasets. If one class dominates, accuracy can look good while minority recall collapses. Practical fixes include class-weighted loss, focal loss (for hard examples), balanced sampling, and reporting per-class metrics. Choose the fix that matches the failure: if the model never sees minority examples, sampling helps; if it sees them but ignores them, loss weighting may help more.

• Monitoring: track training/validation loss, but also track calibration (confidence), per-class precision/recall, and a small “visual audit” of predictions every epoch.
• Common mistake: adding heavy augmentation without verifying label preservation (e.g., flipping text, rotating directional signs).

The practical outcome is a training pipeline that generalizes: augmentation policy documented, regularization settings consistent, and evaluation robust enough to detect overfitting before you waste compute on bigger models.

Section 5.5: Transfer learning and fine-tuning workflows

For most real-world vision tasks, you should not train a CNN from scratch unless you have a large dataset and a strong reason. Transfer learning leverages a backbone pretrained on a large corpus (often ImageNet or a domain-specific dataset). The early layers learn generic features (edges, textures), and later layers learn more task-specific combinations. This matches the hierarchical feature story: reuse the hierarchy, then adapt the top.

A practical workflow starts with freezing the backbone and training only a new head. This tests whether the representation is already sufficient. If performance plateaus below your target, progressively unfreeze: first the last stage, then more stages. Use a smaller learning rate for pretrained layers (discriminative learning rates) and a larger one for the new head. A common recipe is: head LR = 10× backbone LR, then reduce both with a schedule.

Normalization layers deserve special care. With BatchNorm, you must decide whether to keep running statistics frozen (eval mode) or update them. If your dataset is small or batch sizes are tiny, freezing BN statistics can be more stable. If your dataset distribution differs strongly from pretraining, updating BN may help—if you can afford consistent batch statistics. This is a frequent source of “it trains but validation is chaotic” issues.

• Common mistake: fine-tuning everything immediately with a high learning rate, which can erase pretrained features (“catastrophic forgetting”).
• Practical outcome: you should be able to get a strong baseline quickly: pretrained backbone + simple head + light augmentation, then iterate deliberately.

Transfer learning also supports interpretability: when predictions fail, you can compare saliency maps or feature responses between your fine-tuned model and the frozen-backbone baseline to see whether fine-tuning improved attention to the right regions or merely amplified dataset biases.

Section 5.6: Debugging vision models: data issues and failure modes

Debugging CNNs is rarely about a single bug; it is about identifying which subsystem is responsible: data, model, loss, optimization, or evaluation. Start with data. Visualize a batch after all preprocessing and augmentation. Confirm shapes, color channels (RGB vs BGR), normalization ranges, and label alignment. A shocking number of “mysterious” failures come from wrong label files, broken resizing (aspect ratio distortion), or normalization mismatched to the pretrained backbone.

Next, verify that the model can overfit a tiny subset (e.g., 50–200 images). If it cannot reach near-zero training loss, suspect implementation issues: wrong loss (e.g., using sigmoid vs softmax incorrectly), wrong label encoding, frozen layers unintentionally, or learning rate far off. If it overfits tiny data but fails to generalize, suspect augmentation, regularization, class imbalance, or dataset shift between train/validation.

Monitoring should include more than scalar metrics. Inspect confusion matrices to find systematic confusions. Sample “highest confidence wrong” predictions; these often reveal spurious correlations (watermarks, backgrounds) or mislabeled data. For interpretability basics, compute saliency maps (gradients of the class score w.r.t. input) to see what pixels influence decisions, and visualize intermediate feature maps for a few layers to confirm that edges and textures activate meaningfully. If saliency highlights borders or irrelevant backgrounds, you likely have shortcut learning.

• Failure mode: class imbalance leads to high accuracy but poor minority recall; fix with weighted loss or sampling and track per-class metrics.
• Failure mode: augmentation mismatch (train too distorted, val clean) causes a train/val gap; tune augmentation strength and ensure it preserves labels.
• Failure mode: learning rate too high during fine-tuning destroys pretrained features; use smaller backbone LR and warmup.

The practical outcome is a repeatable debugging playbook: validate the input pipeline visually, prove learnability on a tiny subset, instrument training with the right metrics, and use simple interpretability tools to check whether the network is attending to the signal you intended.

Section 6.1: Sequence modeling recap and limitations of recurrence

Before attention, the standard approach to sequences was recurrence: an RNN/LSTM/GRU reads tokens one at a time, updating a hidden state that acts like a summary of the past. In teacher-forced language modeling, you feed the ground-truth previous token and predict the next; in sequence-to-sequence, an encoder compresses an input sequence into a context (or final hidden state), and a decoder generates outputs conditioned on that context.

This works, but it has two practical limitations. First, recurrence is inherently sequential: time step t depends on t−1, so you cannot fully parallelize training across positions. On modern accelerators, this bottleneck dominates. Second, long-range dependencies are hard: even with LSTMs, information and gradients must travel through many steps, and the model tends to forget or blur distant details. You can see this in tasks like copying, long document modeling, or translation where early input matters late in output.

Another engineering pain point is representational “compression.” If the whole past is squeezed into a fixed-size hidden state, you are asking a small vector to store everything relevant. Attention reframes the problem: instead of compressing the past into one state, keep the past states (or token representations) and let the model learn to retrieve what it needs, when it needs it.

When deciding whether to use recurrence today, the usual reason is not quality but constraints: tiny models on microcontrollers, streaming with strict latency, or legacy systems. Otherwise, transformer-style attention typically wins on both speed (parallel training) and modeling power.

Section 6.2: Self-attention and multi-head attention concepts

Self-attention is a content-based lookup. Each position builds a new representation by taking a weighted average of other positions’ representations, where the weights are learned from the data. Concretely, start with an input matrix X of shape (sequence_length T, model_dim d). We create three projections: Q = XW_Q, K = XW_K, V = XW_V. Queries ask “what am I looking for?”, keys advertise “what do I contain?”, and values are the information to aggregate.

The core score matrix is S = QKᵀ, shape (T, T). After scaling by 1/√d_k (critical for stable gradients when d_k is large), we apply softmax row-wise to get attention weights: A = softmax(S / √d_k). The output is Y = AV, giving each token a mixture of value vectors from other tokens. A common implementation mistake is applying softmax over the wrong axis; the correct interpretation is “for each query position, distribute probability mass over key positions.”

Multi-head attention (MHA) repeats this mechanism in parallel with smaller subspaces. Instead of one big attention, split into h heads, each with dimension d_k = d/h. Each head can specialize: one head may track local syntax, another long-range coreference, another punctuation or formatting. Practically, MHA improves expressiveness without increasing the quadratic (T×T) nature of attention beyond a constant factor.

Finally, transformers wrap attention with two stabilizers: residual connections (add the input back) and layer normalization. Residuals keep gradient paths short; layer norm stabilizes activations per token regardless of batch statistics. In modern “pre-norm” blocks, you normalize before attention/MLP, which usually makes training deep stacks easier.

Section 6.3: Positional information (sinusoidal and learned)

Self-attention alone is permutation-invariant: if you shuffle tokens, attention still computes relations based on content, not order. But language and many sequence problems require order. Transformers inject position information into token representations so that “dog bites man” differs from “man bites dog.”

The classic approach is sinusoidal positional encoding. For position p and channel index i, define alternating sine/cosine functions at different frequencies. You then add this positional vector to the token embedding. Two practical benefits: (1) it has no learned parameters, and (2) it can generalize to sequence lengths beyond training (up to numeric limits) because positions are defined by a formula.

The most common approach in modern small models is learned positional embeddings: a table P of shape (max_len, d) where position p selects row P[p], and you add it to the token embedding. This is simple and often performs well when train and inference lengths match. The trade-off is that extrapolation to longer sequences is not guaranteed; if you trained with max_len=256 and try 2048 at inference, you cannot even index the table.

Engineering judgement: if you expect variable or longer-than-trained contexts, prefer sinusoidal encodings or modern alternatives (relative position bias, RoPE). If you are training a toy model to learn fundamentals, learned embeddings reduce conceptual overhead and are perfectly fine. Common mistakes include forgetting to add position information at all, adding it with the wrong shape/broadcast, or applying dropout inconsistently between embeddings and positional vectors.

Section 6.4: Decoder-only vs encoder-decoder architectures

Transformer blocks can be assembled into two major families. Encoder-decoder models (like the original Transformer for translation) have an encoder that reads the input sequence and produces contextual representations, and a decoder that generates the output sequence. The decoder uses two attention modules: (1) masked self-attention over already-generated output tokens, and (2) cross-attention over encoder outputs (unmasked), allowing the decoder to “look at” the source sequence while generating.

Decoder-only models (common in modern language models) remove the encoder and train a single stack with causal masking: each position can attend only to earlier positions. Training is typically next-token prediction over a single stream of tokens. This simplicity is practical: one attention pattern, one loss, and easy scaling. It also matches many deployment scenarios where you generate continuations from a prompt.

Choosing between them is task-driven. If you have a clear input/output separation (translation, summarization with explicit source), encoder-decoder can be more compute-efficient because the encoder processes the source once and the decoder attends to it. If your tasks are “predict next token given previous,” multi-task instruction data, or general text generation, decoder-only is the standard baseline.

A common confusion is where masking applies. Encoder self-attention is typically unmasked (bidirectional) because the whole input is known. Decoder self-attention must be masked causally to prevent leakage from future tokens during training. If you accidentally remove or misapply the causal mask, training loss may look suspiciously good while generation fails because the model learned to cheat.

Section 6.5: Practical training: batching, masking, and sampling

Training a small transformer is a workflow problem as much as a modeling problem. Start with tokenization. For a fundamentals project, you can use a simple character-level tokenizer or a small subword tokenizer (BPE/Unigram). The key is that your model sees integer token IDs, which you map to embeddings. Decide on a fixed context length ctx (e.g., 128), and build training examples as sliding windows: input tokens x[0:ctx] predict targets x[1:ctx+1].

Batching requires careful tensor shapes: typical input is (batch B, time T). Your embedding layer produces (B, T, d). The attention mask should broadcast to (B, heads, T, T) (or similar) depending on your implementation. You usually need two masks: a causal mask (upper triangular set to −∞ before softmax) and an optional padding mask if sequences are variable-length. A classic bug is mixing up “masked positions are 0 vs −∞” semantics; for softmax masking, you want large negative logits (e.g., −1e9) so probabilities become ~0.

Loss is cross-entropy over vocabulary for each position. In practice you reshape logits from (B, T, vocab) to (B·T, vocab) and targets to (B·T). If you have padding, ensure padded target positions are ignored (loss mask), otherwise the model learns to predict pad tokens and metrics become misleading.

Sampling is part of training evaluation. Teacher-forced loss measures next-token prediction under ground-truth context; generation quality depends on sampling strategy. Implement greedy decoding first, then add temperature, top-k, or nucleus (top-p). If your model repeats or collapses, check: (1) training data quality, (2) context length too small, (3) sampling too deterministic (temperature too low), or (4) model too small/undertrained.

Section 6.6: Efficiency fundamentals: KV cache, distillation, quantization

Transformers are powerful but can be expensive. Two costs dominate: attention’s O(T²) compute for long sequences, and autoregressive generation’s step-by-step decoding. Even for small models, you should build efficiency habits early.

During decoder-only generation, a key optimization is the KV cache. Without caching, at every new token you recompute keys/values for all previous tokens in every layer, which is wasteful. With a KV cache, you store past keys and values per layer and append new ones each step. Then attention for the new token only computes query for the new position and attends over cached K/V. This reduces per-step compute from O(T²) to roughly O(T) for attention (though total generation remains O(T²) across steps, it’s much faster in practice).

Distillation compresses a large “teacher” model into a smaller “student” by training the student to match teacher outputs (logits or hidden states) in addition to ground-truth labels. Practically, distillation is one of the best tools when you need smaller latency or memory without starting from scratch. It also tends to smooth training because teacher probabilities provide richer targets than one-hot labels.

Quantization reduces weight/activation precision (e.g., FP16, INT8, INT4) to save memory bandwidth and increase throughput. Engineering judgement matters: post-training quantization is easiest but can hurt quality; quantization-aware training is harder but preserves accuracy better. Always evaluate after quantization with the same decoding settings you plan to deploy, because small numerical changes can alter sampling behavior. Finally, track real metrics: tokens/sec, memory usage, and task-relevant accuracy—not just training loss.