Neural Networks from Scratch in Python: Build & Train

Section 1.1: What a neural network is (and isn’t)

A neural network (in the form we’ll build) is a parameterized function: it maps an input vector x to an output y_hat using a sequence of linear transformations and nonlinear activations. For a dense layer, the core operation is z = XW + b, followed by an activation such as ReLU or sigmoid. Stacking layers increases the function’s capacity—its ability to represent complex mappings.

What it is not: it is not “intelligence,” not a database of training examples, and not automatically robust. A network does not discover truth; it minimizes a loss function on the provided dataset. If your labels are noisy, your features leak information, or your evaluation split is flawed, training can look successful while the model is unusable in the real world.

In practice, think of the network as a flexible curve-fitting tool with constraints you choose: architecture (layers/width), activation functions, and regularization. Your job is to pick a function class that can fit the underlying pattern but not memorize irrelevant noise. Early in the course we’ll start with a tiny baseline model (even logistic regression) to establish a reference point. Baselines are valuable because they tell you whether the dataset is learnable at all and how much a neural network actually improves results.

• Engineering judgment: if a simple linear baseline performs nearly as well as a deeper model, prefer the simpler model until you have a reason not to.
• Common mistake: increasing model size to fix poor data quality; this often increases overfitting and instability.

This perspective will guide every implementation decision later: forward propagation is just computing the function; backpropagation is computing how to adjust parameters to reduce loss; training is repeating that adjustment while checking generalization.

Section 1.2: Datasets, features, labels, and splits

Training starts with data shaped into features and labels. Features (X) are the numeric inputs the model can use; labels (y) represent the target you want to predict. For a classification toy dataset, X might be an (N, D) matrix (N examples, D features) and y might be integer class ids of shape (N,) or one-hot vectors of shape (N, C).

A “tiny dataset pipeline” in this course means: generate or load data, normalize it, shuffle it, and produce batches. Even when you later use real datasets, the same responsibilities apply. A minimal pipeline often includes (1) deterministic shuffling, (2) normalization based on training statistics, and (3) a batch iterator. A simple baseline model—like a single dense layer with softmax—should train quickly and reveal whether your pipeline and labeling are correct.

Splits matter because training performance is not the goal; generalization is. Use three splits when possible:

• Train: used to compute gradients and update parameters.
• Validation: used to choose hyperparameters (learning rate, layer sizes, regularization).
• Test: used once at the end for an unbiased estimate.

Common mistakes include: leaking normalization statistics from validation/test into training (compute mean/std on train only), splitting without shuffling (time-ordered data can bias results), and tuning repeatedly on the test set (turning it into a validation set). In your first experiment, keep the dataset small enough that you can rerun training in seconds; speed enables iteration, and iteration is how you learn to diagnose behavior.

Section 1.3: Loss functions vs metrics

Loss functions and metrics answer different questions. The loss is the objective the optimizer minimizes; it must be differentiable (or almost everywhere differentiable) with respect to model parameters. A metric is how you report performance in terms humans care about, and it does not need to be differentiable.

For example, in multi-class classification you might minimize cross-entropy loss while reporting accuracy. Cross-entropy provides a smooth gradient signal: it rewards not just correct classes but also calibrated probabilities. Accuracy is intuitive, but it is flat with respect to small probability changes—making it poor for gradient-based optimization.

In regression, mean squared error (MSE) is both a loss and a metric, but you might also report mean absolute error (MAE) for interpretability. When you add regularization, the loss typically becomes: data_loss + reg_loss. Your metric, however, usually reflects only predictive performance (exclude the regularization term) so you can compare models fairly.

• Common mistake: optimizing one quantity and claiming success on another (e.g., decreasing loss while validation accuracy stagnates due to class imbalance or threshold effects).
• Practical workflow: log both loss and at least one metric for train and validation each epoch; if they diverge, investigate.

During your first end-to-end run, explicitly print (or log) train loss, validation loss, and accuracy. This creates the habit of separating “optimization progress” from “real-world performance,” which is essential when you later encounter overfitting, label noise, and distribution shift.

Section 1.4: Training loop anatomy (fit/eval/predict)

A clean training script usually exposes three modes: fit (update parameters), eval (measure without updates), and predict (produce outputs for downstream use). Even in a from-scratch NumPy project, keeping these responsibilities separate prevents subtle bugs—like accidentally applying dropout during evaluation or updating running statistics when you shouldn’t.

The anatomy of a basic loop is:

• Initialize parameters (weights/biases) and optimizer settings (learning rate).
• For each epoch: shuffle training data, iterate mini-batches.
• For each batch: forward pass → compute loss → backward pass → parameter update.
• After epoch: evaluate on validation split; record learning curves.

This chapter’s “first end-to-end training experiment” should be intentionally small: a two-layer dense network on a toy classification dataset (e.g., blobs or spirals). The point is to validate the plumbing: data pipeline feeds batches, forward pass produces logits, loss is finite, gradients are nonzero, and parameters change. If loss is nan on the first batch, don’t continue—debug immediately (learning rate too high, numerical instability in softmax, bad initialization, or incorrect labels).

Learning curves are your diagnostic tool. If training loss decreases but validation loss increases, you’re likely overfitting (or the split is flawed). If neither decreases, suspect underfitting, too-small learning rate, a bug in gradients, or features that contain little signal. If the curves are wildly noisy, mini-batches may be too small, learning rate too large, or data not shuffled. The habit you build here—observe, hypothesize, change one variable, rerun—will carry through to backpropagation and regularization chapters.

Section 1.5: Numerical computing with NumPy arrays

From-scratch neural networks are mostly about correct and efficient array math. NumPy lets you express whole-batch computations as matrix operations, which is both faster and less error-prone than looping over examples. The price is that you must be disciplined about shapes.

Adopt a consistent convention early: represent a batch as X with shape (N, D). A dense layer maps (N, D) to (N, H) using W of shape (D, H) and b of shape (H,) (broadcast across the batch). Activations preserve the batch dimension: ReLU, sigmoid, and tanh apply elementwise.

Common failure modes are shape mismatches that “work” due to broadcasting but compute the wrong thing. For example, using b with shape (N, H) can silently bake batch size into parameters. Another frequent issue is mixing row/column vectors for labels: if your cross-entropy expects one-hot labels (N, C) but you pass integer labels (N,), you’ll either get incorrect indexing or accidental broadcasting.

• Practical tip: assert shapes at boundaries (after loading, after each layer, before loss computation).
• Numerical stability: implement softmax with a shift (logits - logits.max(axis=1, keepdims=True)) to avoid overflow.

Your baseline model plus the first training run should use fully vectorized operations. This discipline is not just about speed; it makes backpropagation derivations match the code: gradients become matrix expressions you can implement directly and test with finite differences later.

Section 1.6: Reproducibility (seeds, determinism, logging)

Neural network training has randomness: parameter initialization, data shuffling, and sometimes stochastic regularizers. If you can’t reproduce results, you can’t reliably compare experiments or debug regressions. Reproducibility is not a bureaucratic extra—it is a core engineering tool.

At minimum, set and record seeds. In NumPy, that typically means creating a generator (e.g., rng = np.random.default_rng(seed)) and using it for all randomness: initializing weights, shuffling indices, and sampling mini-batches. Keep the seed in your run configuration so you can rerun the exact experiment later. Determinism can be harder on GPU frameworks, but in a NumPy-only chapter you can get very close to identical runs if you control the random number generation and avoid non-deterministic parallelism.

Logging turns a run into an artifact you can reason about. Log: hyperparameters (learning rate, batch size, layer sizes), dataset split sizes, seed, and per-epoch metrics (train loss/accuracy, val loss/accuracy). Save learning curves to disk (CSV/JSON) so you can compare runs without rerunning everything. A simple “experiment directory” structure—one folder per run with config and logs—prevents confusion when you start tuning.

• Common mistake: changing code and rerunning without noting what changed; later you cannot explain why results differ.
• Practical outcome: you can spot failure modes faster because you can reproduce the exact state that produced them.

By the end of this chapter, you should have a small, repeatable training script that you trust. That trust is what will let you move into forward propagation, backpropagation, and optimization with confidence: when something breaks, you’ll know it’s the math or the implementation—not the experiment setup.

In NumPy, a “vector” is typically a 1D array (shape (D,)), while a “column vector” is a 2D array (shape (D, 1)). Neural network code is easiest to maintain when you minimize ambiguous 1D shapes. A practical rule: represent a batch of examples as a 2D matrix X with shape (N, D), where N is batch size and D is feature dimension. Then everything is consistently “row-major examples.”

Broadcasting is NumPy’s way of aligning shapes for elementwise operations. It’s extremely useful (e.g., adding a bias vector to every row), but it can also silently hide bugs if you’re not disciplined. Broadcasting works by comparing dimensions from the right; dimensions match if they are equal or one of them is 1. For example, if X is (N, D) and b is (D,) or (1, D), then X + b produces (N, D) by repeating b across rows.

• Good use: Z = X @ W + b where b is (M,) (or (1, M)) and Z is (N, M).
• Common mistake: using b shaped (N,) by accident, which broadcasts down columns and produces nonsense that “looks” the right shape.

Engineering judgment: prefer explicit 2D shapes for parameters (W as (D, M), b as (M,) or (1, M)) and add assertions in development code. Shape bugs waste hours; one assert X.ndim == 2 can save you a day.

A dense layer computes an affine transformation: Z = XW + b. “Affine” means linear plus a shift (bias). This is the workhorse of forward propagation: every dense layer is essentially a matrix multiply plus a bias add, followed by a nonlinearity (activation) in later chapters.

Shape discipline makes the formula actionable. If X is (N, D) (N examples, D features) and the layer outputs M features, then W must be (D, M) and b must be broadcastable to (N, M), typically (M,). The output Z is (N, M). This is the vectorized version of computing each neuron’s weighted sum for every example in the batch—no Python loops required.

Two practical notes about bias terms:

• Bias matters: without b, a layer can only represent functions that pass through the origin in its input space (too restrictive).
• Don’t “fold” bias into W unless you have a strong reason. Some textbooks append a column of ones to X to absorb bias into W, but in NumPy code it usually complicates debugging and regularization.

Implementation pattern (forward pass): store X for the backward pass later. For now, get comfortable writing a function like affine_forward(X, W, b) that returns Z and a cache (X, W, b). This is the first step toward a clean, modular neural network implementation.

Most “mysterious” NN bugs are shape bugs. The fix is to adopt a consistent convention and enforce it. In this course, we’ll use:

• X: (N, D) input batch
• W: (D, M) weights
• b: (M,) bias
• Z: (N, M) pre-activation outputs

This convention scales: if you stack layers, the output dimension of one layer becomes the input dimension of the next. You can verify an entire network by walking through shapes. For example, (N, 784) → (N, 128) → (N, 10) for a simple classifier on flattened images.

Batch dimensions are not optional. Even if you train with a single example, keep it as (1, D), not (D,). Mixing these leads to subtle differences in NumPy behavior (especially with transpose: x.T does nothing to a 1D array). A practical workflow is to add tiny utilities:

• as_2d(X) to coerce a vector into (1, D).
• check_shape(name, arr, expected_ndim) during development.

Refactoring for clean tensor shape handling is not “extra polish.” It is a speed multiplier once you start implementing backprop. If your forward pass caches consistent shapes, your backward pass can be written once and reused everywhere.

Forward propagation produces predictions; training requires a loss that scores how wrong those predictions are. Two losses cover most introductory use cases: Mean Squared Error (MSE) for regression and Cross-Entropy (CE) for classification.

MSE (regression). Suppose y_pred and y are both (N, 1) (or (N,) if you’re careful). MSE is typically (1/N) * sum((y_pred - y)^2). In vectorized form, you compute the residual r = y_pred - y, then reduce. Common mistake: forgetting to average over N, which makes gradients scale with batch size and complicates learning-rate choice.

Cross-Entropy (classification). If your model outputs logits scores of shape (N, C), you convert them to probabilities with softmax, then compute CE against labels. In practice, you should implement “softmax + cross-entropy” in a numerically stable way by subtracting the per-row max before exponentiating. Labels may be class indices ((N,)) or one-hot vectors ((N, C)); be consistent and document which format your functions accept.

• Numerical stability rule: compute shifted = scores - scores.max(axis=1, keepdims=True).
• Shape rule: keep keepdims=True during reductions so broadcasting stays predictable.

Practical outcome: by the end of this section you should be able to compute loss values for a linear model (XW + b) and prepare the gradient of the loss with respect to the model outputs, which is the entry point for backprop.

Backpropagation is just the chain rule applied to vectorized computations. The reason linear algebra matters is that we want gradients for entire batches and entire weight matrices at once.

Start with the affine layer Z = XW + b. Assume you already have dZ, the gradient of the loss with respect to Z, shaped (N, M). Then the matrix-form gradients are:

• dW = X.T @ dZ giving shape (D, M)
• db = dZ.sum(axis=0) giving shape (M,)
• dX = dZ @ W.T giving shape (N, D)

These formulas are worth memorizing because they appear everywhere. They also demonstrate why we stored X and W in the forward cache: the backward pass needs them.

Common mistakes and how to catch them:

• Forgetting the transpose: if dW isn’t (D, M), something is wrong.
• Summing bias over the wrong axis: db must match b shape.
• Not averaging gradients when your loss is averaged over the batch. If your loss is mean over N, then dZ (or earlier) should include 1/N.

Engineering judgment: implement backward functions that return gradients with exactly the same shapes as their parameters, and assert that explicitly. When you later add regularization (like L2), you’ll add terms such as dW += reg * W; this only works cleanly if shapes are consistent and predictable.

Analytic gradients are fast and exact (up to floating-point), but they’re also easy to implement incorrectly. Gradient checking uses finite differences as a debugging tool: perturb a parameter slightly and measure how the loss changes. If your backprop is correct, the numeric gradient and analytic gradient should agree closely.

The basic idea for one parameter element theta is:

• Compute loss_plus with theta + eps
• Compute loss_minus with theta - eps
• Numeric gradient ≈ (loss_plus - loss_minus) / (2*eps)

Use a small eps like 1e-5. Too large and the approximation is crude; too small and floating-point noise dominates. Compare using relative error: rel = |g_num - g_ana| / max(1e-8, |g_num| + |g_ana|). For well-implemented layers, you often see relative errors around 1e-7 to 1e-5 depending on the operation and dtype.

Practical workflow:

• Test one layer at a time (e.g., affine forward/backward) with random small inputs.
• Check a handful of random indices instead of every parameter to keep it fast.
• Turn off sources of randomness (like dropout) when checking gradients.

Refactor utilities here: write a small function that flattens parameters and gradients into 1D views for sampling indices, then unflattens back. Clean shape handling makes gradient checking straightforward instead of painful, and it builds confidence before you start training full networks where bugs can hide behind “it sort of learns.”

Neural networks are best understood as computational graphs: nodes represent operations (matrix multiply, add, max), and edges carry tensors (arrays). A parameterized layer is a subgraph with learnable parameters—typically weights and biases—that will be updated during training. Forward propagation is simply evaluating this graph from inputs to outputs.

In code, treat each layer as a reusable module with (1) parameters, (2) a forward(X) method, and (3) a place to cache inputs needed later for backprop. Even though we are not implementing gradients yet, design now as if you will. For example, a dense layer will later need the original X to compute dW and db, so it should store self.X during forward.

Engineering judgment: keep modules minimal and explicit. Don’t hide shape changes or implicit broadcasting. When a bug happens, you want to locate it quickly by printing shapes and summary statistics at module boundaries.

• Common mistake: mixing row-vector and column-vector conventions. This course uses rows as examples, columns as features.
• Common mistake: relying on NumPy broadcasting for bias addition without confirming the bias shape. Make bias shape (1, n_units) to broadcast across batches.
• Practical outcome: you’ll be able to assemble a network as a list of modules and run a forward pass like a pipeline.

Conceptually, forward propagation is function composition: y = f3(f2(f1(X))). Each module should be testable in isolation, because later you’ll debug training by checking whether intermediate activations look sensible (not all zeros, not all NaNs, not extremely large).

A dense layer (also called fully connected) computes an affine transform: Z = XW + b. Here X is (B, D), W is (D, H), and b is (1, H). The output Z is (B, H). This is the fundamental building block for MLPs.

Implement it as a small class. Initialize parameters with small random values and zeros for bias. In later chapters you’ll improve initialization, but for now start with something deterministic and inspectable (e.g., np.random.randn(D, H) * 0.01). A practical trick: accept a random seed or RNG object so your tests are reproducible.

Example (forward only, caching for later):

class Dense: def __init__(self, n_in, n_out, rng=None, weight_scale=0.01): self.rng = np.random.default_rng() if rng is None else rng self.W = self.rng.standard_normal((n_in, n_out)) * weight_scale self.b = np.zeros((1, n_out), dtype=float) self.X = None def forward(self, X): self.X = X return X @ self.W + self.b

Edge cases matter. Ensure X is 2D; a single example should be reshaped to (1, D). Validate that X.shape[1] == W.shape[0] early with a clear error message. Silent shape errors can “work” via broadcasting but produce nonsense outputs.

• Common mistake: bias shape (H,) sometimes broadcasts unexpectedly; prefer (1, H).
• Common mistake: using integer arrays; forward should run in floating point to avoid truncation.

Once dense layers are modular, you can swap sizes, stack multiple layers, and later attach gradients without rewriting the forward logic.

Without activation functions, a stack of dense layers collapses into a single affine transform, no matter how deep it is. Activations introduce nonlinearity, enabling the network to model complex functions. Implement activations as separate modules with forward(Z), and plan to cache outputs for backprop later.

Sigmoid maps values to (0, 1). It is historically common for binary outputs, but it saturates: large positive inputs push output near 1, large negative near 0, and gradients become tiny. Implement it carefully to reduce overflow risk. A stable approach uses conditional forms, but at minimum clip inputs or use np.exp cautiously.

Tanh maps to (-1, 1), is zero-centered, and often behaves better than sigmoid, but it still saturates at extremes.

ReLU is max(0, z). It is simple and typically trains well, but it can create “dead” neurons: if a unit outputs 0 for all inputs (because its pre-activation stays negative), it may never recover.

GELU (Gaussian Error Linear Unit) is a smoother alternative popular in transformers. For scratch implementations, an approximate formula is common: 0.5*z*(1 + tanh(sqrt(2/pi)*(z + 0.044715*z^3))). GELU is not required for basic MLPs, but implementing it teaches you to handle more complex elementwise operations.

Practical edge-case testing: feed very large positive/negative numbers to sigmoid/tanh and confirm you do not produce NaN or overflow warnings. For ReLU, check that negative values become exact zeros (useful later when detecting dead units). For GELU, confirm output is roughly linear for moderate positive inputs and near-zero for negative inputs.

• Engineering judgment: choose ReLU (or GELU) for hidden layers by default; reserve sigmoid for final binary probability outputs.
• Common mistake: applying sigmoid before softmax for multi-class classification—this breaks probability normalization.

By implementing activations as modules, you make it trivial to instrument and swap them, which is crucial when debugging saturation and training dynamics.

For multi-class classification, the network typically outputs logits: unconstrained real numbers, one per class. Softmax converts logits into a probability distribution per example: p_k = exp(z_k) / sum_j exp(z_j). The problem is that exp can overflow for large logits, producing inf and then NaN probabilities.

The standard fix is the “max trick”: subtract the maximum logit per row before exponentiating. This does not change the resulting probabilities because softmax is shift-invariant. In vectorized NumPy:

def softmax(logits): shifted = logits - np.max(logits, axis=1, keepdims=True) exp_vals = np.exp(shifted) return exp_vals / np.sum(exp_vals, axis=1, keepdims=True)

Cross-entropy loss for one-hot labels y and predicted probabilities p is -sum(y * log(p)). In practice, labels often come as integer class indices. Then you compute -log(p[range(B), y]). For stability, never take log(0); clip probabilities or compute via log-softmax. A pragmatic approach is:

eps = 1e-12 p = np.clip(p, eps, 1.0) loss = -np.mean(np.log(p[np.arange(B), y]))

Even better (and commonly used) is to compute stable cross-entropy directly from logits without explicitly forming p, using logsumexp. But if you implement softmax with shifting and add a small eps before log, you’ll already avoid most numerical disasters in small projects.

• Common mistake: computing exp(logits) without shifting; it may work on tiny values and then fail unpredictably.
• Common mistake: averaging loss incorrectly (sum over classes and batch vs mean over batch). Decide a convention and stick to it.

Stable softmax and cross-entropy are non-negotiable. If you see NaN losses during training, this is one of the first places to check.

Forward propagation is easy to “implement” and surprisingly hard to trust without tests. Before you build backprop, you want high confidence that every module respects shape contracts and produces numerically reasonable outputs.

Start with shape tests. For a dense layer, assert output shape is (B, H) for various batch sizes, including B=1. For activations, assert they preserve shape exactly. For softmax, assert output shape matches logits and that each row sums to ~1 (within floating-point tolerance).

Next, test value ranges and invariants:

• Sigmoid: output in (0, 1); check extreme inputs like -1000 and +1000.
• Tanh: output in (-1, 1); check saturation at extremes.
• ReLU: output is never negative; count zeros to detect dead behavior.
• Softmax: probabilities are non-negative; rows sum to 1; the argmax of softmax matches argmax of logits.
• Cross-entropy: loss is non-negative; if logits strongly favor the true class, loss should be near 0.

Instrument activations to detect saturation. A simple approach is to log summary statistics per layer: mean, standard deviation, min/max, and the fraction of values in “saturated” regions. For sigmoid/tanh, saturation can be measured as fraction with output near 0/1 or -1/1 (e.g., p<1e-3 or p>1-1e-3). For ReLU, measure the fraction of zeros; if it’s ~1.0, the layer is dead.

Engineering judgment: don’t wait until training diverges. Run a single forward pass on random inputs and inspect these stats. If logits are extremely large at initialization, your weight scale is too big. If everything is near zero, learning may be slow. This “activation telemetry” becomes your early-warning system.

With dense layers, activations, and softmax in place, you can build a multi-layer perceptron forward pass as a sequence of composable blocks. A typical classification MLP looks like: Dense → ReLU → Dense → ReLU → Dense → logits, then softmax for probabilities and cross-entropy for loss.

A clean pattern is to represent the network as a list of modules, each exposing forward. The MLP forward simply loops through modules:

class MLP: def __init__(self, layers): self.layers = layers def forward(self, X): out = X for layer in self.layers: out = layer.forward(out) return out

Then define your model:

model = MLP([ Dense(n_in=784, n_out=128, rng=rng), ReLU(), Dense(128, 64, rng=rng), ReLU(), Dense(64, 10, rng=rng) ]) logits = model.forward(X_batch) probs = softmax(logits)

Practical workflow: start with a tiny synthetic batch (e.g., B=4) and verify every intermediate shape. Add instrumentation hooks—either inside each module or in the MLP loop—to capture activation stats layer-by-layer. If you detect heavy saturation (e.g., sigmoid outputs pinned near 0/1 or ReLU producing mostly zeros), adjust initialization scale, consider a different activation, or reduce depth until behavior looks healthy.

• Common mistake: applying softmax inside the model and then again in the loss function. Decide whether your model outputs logits (recommended) or probabilities, and keep it consistent.
• Common mistake: forgetting that the last layer usually has no activation for softmax-based classification; the nonlinearity is effectively in the softmax.

By the end of this chapter you have a forward pipeline that is modular, testable, and numerically stable. This sets you up to derive backpropagation next: every cached input and every stable operation you implemented here will directly simplify and stabilize your gradient computations.

Backpropagation is just the chain rule applied repeatedly, but the key mental model is “local gradients.” Each layer takes an input X and produces an output Y. During the backward pass, you do not recompute the whole network; you only need the upstream gradient dY (how the loss changes with respect to the layer output) and the layer’s local derivative dY/dX (how the layer output changes with respect to its input). The layer then returns dX to the previous layer and stores parameter gradients like dW and db.

In practice, this means your forward pass must cache the minimal set of tensors required to compute local derivatives later. For a dense layer, you typically cache X, W, and sometimes the pre-activation Z. For an activation function, you cache either its input (Z) or its output (A) depending on which makes the derivative cheaper or more stable. Caching is not an academic detail; it prevents subtle bugs (using the wrong tensor for the derivative) and avoids expensive recomputation.

A good implementation pattern is: each layer has forward(X) that returns output and stores a cache; and backward(dout) that uses the cache to compute gradients. When vectorizing, ensure all operations work on the full batch: avoid Python loops over samples. A common mistake is mixing shapes (e.g., treating b as (H,) in forward but expecting (N, H) in backward). Decide on a consistent broadcasting approach early and stick to it.

• Rule of thumb: if you can’t write down the shapes of X, W, b, Z, and A at every layer, pause and do that first—most backprop bugs are shape bugs.
• Workflow: implement one layer’s forward/backward, verify with gradient checking, then stack layers into an MLP.

Finally, remember what gradients represent: dW tells you how to change weights to reduce the loss. If your gradients are zero everywhere, your model can’t learn; if gradients explode to huge values, the training step becomes unstable. The rest of this chapter shows how to compute gradients correctly and how to diagnose their behavior with simple statistics.

Activations provide nonlinearity; their derivatives determine how gradients flow. In vectorized backprop, you compute an elementwise derivative and multiply by the upstream gradient (Hadamard product). Suppose an activation takes Z and outputs A = f(Z). Given upstream gradient dA, the gradient with respect to pre-activation is dZ = dA * f'(Z) (elementwise).

ReLU: A = max(0, Z). Derivative: f'(Z) = 1 where Z > 0, else 0. Implementation uses a mask: dZ = dA * (Z > 0). Cache Z or cache a boolean mask from forward. Common mistake: using A > 0 is usually fine for ReLU (since A is zero when Z is non-positive), but if you later switch to leaky ReLU or other variants, you’ll want Z.

Sigmoid: A = 1 / (1 + exp(-Z)). Derivative: A * (1 - A). Cache A from forward because it is reused directly: dZ = dA * A * (1 - A). Engineering note: sigmoid saturates for large |Z|, making A*(1-A) near zero, which can contribute to vanishing gradients in deep networks.

Tanh: derivative is 1 - A^2 if you cache A = tanh(Z). Like sigmoid, tanh can saturate; it is often less problematic than sigmoid but still can vanish in deep stacks without good initialization.

Practical outcomes: implement each activation as a small module with forward/backward, and test them in isolation. Many training failures trace back to activation derivatives computed from the wrong cached tensor, or using integer masks that accidentally upcast/downcast types. Keep everything in floating point (e.g., float32 or float64) and ensure masks are multiplied as floats.

• Tip: for numerical stability, prefer caching the activation output A for sigmoid/tanh; it avoids recomputing exp in backward and reduces opportunities for overflow.
• Tip: if you later add dropout, your activation backward will also apply the dropout mask; cache masks explicitly so the backward pass matches forward behavior.

A fully connected (dense) layer computes Z = XW + b with shapes X:(N,D), W:(D,H), b:(H,), Z:(N,H). The dense layer backward pass is the backbone of your MLP: it converts the upstream gradient dZ into gradients for parameters (dW, db) and for the previous layer (dX).

Start from differentials and matrix calculus results. With Z = XW + b:

• Gradient w.r.t. weights: dW = X^T dZ, shape (D,H). Intuition: each weight connects an input dimension to a hidden unit, and gradients aggregate over the batch.
• Gradient w.r.t. bias: db = sum(dZ, axis=0), shape (H,). Bias affects every sample equally, so you sum over samples.
• Gradient w.r.t. inputs: dX = dZ W^T, shape (N,D). This is what you pass to the previous layer.

If your loss is defined as an average over the batch (common), be consistent: either average the loss and let gradients naturally include 1/N, or explicitly divide dW and db by N. Many “my learning rate is wrong” issues are actually “my gradients are scaled inconsistently.” A clean approach is: compute the data loss as mean, and ensure dZ from the loss is already scaled by 1/N. Then dense backward formulas stay simple without extra scaling.

Implementation detail: cache X and W in the dense layer’s forward pass. In backward, compute dW, db, and dX as above. Then store dW and db in a dictionary keyed by parameter names. This makes parameter updates straightforward and keeps your training loop readable.

Common mistakes: transposes in the wrong place (often X dZ^T instead of X^T dZ), summing db over the wrong axis, and accidentally modifying cached arrays in-place. When debugging, print shapes at runtime and assert them. For example, assert dW.shape == W.shape and dX.shape == X.shape—these assertions catch most issues early.

For multi-class classification, the standard head is softmax followed by cross-entropy loss. Separately, softmax has a Jacobian and cross-entropy has a log; together, they simplify beautifully, producing a stable and efficient gradient. This is one of the most valuable “from scratch” derivations because it removes both complexity and numerical risk.

Let logits be Z with shape (N,C). Softmax produces probabilities P where P[i,c] = exp(Z[i,c]) / sum_k exp(Z[i,k]). For stability, compute softmax with a shift: Z_shift = Z - max(Z, axis=1, keepdims=True) before exponentiating. Cross-entropy loss with one-hot targets Y is L = -mean(sum_c Y_c * log(P_c)).

The key result: if L is the mean cross-entropy, then the gradient w.r.t. logits is

dZ = (P - Y) / N

This means you do not need to explicitly form the softmax Jacobian. You compute P in forward, cache it (and Y or class indices), and in backward compute dZ with one subtraction. If your labels are integer class indices y shape (N,), you can build dZ by copying P and subtracting 1 from the correct class positions: dZ[np.arange(N), y] -= 1, then divide by N.

Engineering judgement: always combine softmax and cross-entropy into a single “loss layer” in code. It reduces bugs (mismatched scaling), improves numerical stability (log of tiny probabilities), and simplifies gradient checking. Also, watch for the common pitfall of computing log(P) without clipping; instead, rely on stable softmax or clip P with a small epsilon when logging.

Once you have dZ from the loss, the rest of backprop is just dense backward and activation backward repeated layer-by-layer. This is where vectorization shines: your entire network backward pass becomes a handful of matrix multiplications and elementwise products.

With gradients computed, training becomes an optimization loop: forward pass → loss → backward pass → parameter update. The simplest update rule is gradient descent: W -= lr * dW, b -= lr * db. Your implementation should keep parameters and gradients in dictionaries (e.g., params['W1'], grads['W1']) so updates are uniform across layers.

Batch gradient descent uses the full dataset each step. It gives a smooth loss curve but can be slow and memory-heavy. Mini-batch SGD uses small batches (e.g., 32–256), giving noisier but faster updates and often better generalization. In NumPy, mini-batching is just slicing: shuffle indices each epoch, then iterate in chunks. Make sure your shuffling keeps features and labels aligned.

Learning rate is the most sensitive knob. Too large: loss may diverge or oscillate, gradients may explode. Too small: training crawls. Practical workflow: start with something like 1e-2 for small MLPs on standardized inputs, then adjust by factors of 2–10. If your loss decreases initially then plateaus high, you may be underfitting or using too small a model; if loss decreases then blows up, reduce learning rate or add gradient clipping (Section 4.6).

To train an MLP classifier end-to-end, stack layers: Dense → ReLU → Dense → ReLU → Dense → SoftmaxCrossEntropy. The training loop should report at least: loss, accuracy, and gradient norms (or parameter update magnitudes). Accuracy alone can be misleading early; loss is a better signal for whether gradients are correct.

• Common mistake: forgetting to zero/overwrite gradients each step. In a from-scratch NumPy setup, you typically compute new grads each iteration; don’t accidentally accumulate unless you intend to.
• Common mistake: mixing “sum loss” and “mean loss” across epochs; it changes effective learning rate as batch size changes.

As you move toward more robust training (later chapters), this same gradient/update interface will allow you to add L2 regularization, momentum, Adam, and dropout. For now, keep it minimal and correct: correctness beats cleverness when building from scratch.

When training fails, assume your gradients are wrong until proven otherwise. Gradient debugging is an engineering skill: you use targeted checks to isolate whether the issue is math, shapes, scaling, or numerical instability. The two highest-value tools are gradient checking (finite differences) and gradient statistics (norms, mins/maxes, percent zeros).

Gradient checking: for a small network and tiny batch, numerically approximate dW by perturbing one parameter at a time: (L(W+eps)-L(W-eps))/(2*eps). Compare to backprop’s dW using relative error: abs(a-b)/max(1e-8, abs(a)+abs(b)). Use eps ~ 1e-5 and float64. Only check a random subset of parameters (e.g., 50 elements) to keep it fast. If relative error is ~1e-6 to 1e-4, you’re usually fine; if it’s 1e-2 or worse, something is off (often missing 1/N scaling or a transpose).

Gradient norms and activation stats: to diagnose exploding/vanishing gradients, log per-layer statistics each iteration or every few iterations:

• ||dW|| and ||dX|| (e.g., L2 norm)
• mean/std of pre-activations Z and activations A
• fraction of zeros after ReLU (dead units)

If norms grow rapidly layer-to-layer or step-to-step, you may have exploding gradients (too large learning rate, poor initialization, or deep network). If norms shrink toward zero in earlier layers, you have vanishing gradients (saturating activations, poor initialization, or overly deep architecture). These stats connect directly to the course outcomes: they help you interpret training stability and guide practical fixes.

Clipping: gradient clipping is a pragmatic stabilization technique: scale gradients down when their norm exceeds a threshold. For global norm clipping, compute g_norm over all parameter gradients, and if g_norm > clip, multiply all gradients by clip / (g_norm + 1e-12). Clipping can prevent catastrophic steps while you tune learning rate and initialization; it should not be used to hide persistent gradient bugs.

Common mistakes: running gradient check on a network with dropout or batch norm in training mode (stochasticity breaks the finite difference assumption), forgetting to fix a random seed, or checking with too-large eps (poor approximation) or too-small eps (floating-point noise). Keep the check deterministic and small, then scale back up to real training once correctness is established.

Section 5.1: Weight initialization (Xavier/He) and why it matters

Initialization is not a cosmetic detail; it sets the starting scale of activations and gradients. If your weights are too large, pre-activations grow with depth, saturating sigmoids/tanh or causing ReLU activations to explode. If weights are too small, signals shrink layer by layer and gradients vanish. Either way, you may see loss stuck near chance, NaNs, or extreme sensitivity to learning rate.

The key idea is variance preservation: you want the variance of outputs of each layer to be roughly similar to the variance of its inputs, and similarly for backpropagated gradients. For a dense layer with fan_in inputs and fan_out outputs, two widely used schemes are:

• Xavier/Glorot (good for tanh / linear): initialize W with variance ~ 2/(fan_in + fan_out). Commonly: W ~ N(0, sqrt(2/(fan_in+fan_out))) or uniform in [-sqrt(6/(fan_in+fan_out)), +sqrt(6/(fan_in+fan_out))].
• He/Kaiming (good for ReLU/LeakyReLU): initialize W with variance ~ 2/fan_in. Commonly: W ~ N(0, sqrt(2/fan_in)).

In NumPy, implement this inside your Dense layer constructor by computing fan_in from the input dimension and sampling W accordingly; keep biases at zero. A common mistake is to reuse a single global scale (like 0.01) across all layers; it may “work” for shallow nets but becomes brittle as depth grows.

Practical debugging tip: log the mean/std of activations per layer for a single forward pass before training. If early layers output std ~ 0.001 and later layers ~ 1e-6, you are shrinking signal; if std balloons to 50, you’re amplifying. Fixing initialization often turns a “broken” training run into a normal-looking loss curve without changing anything else.

Section 5.2: Learning rates, schedules, and warmup basics

The learning rate (LR) is the highest-leverage knob in training. Too high: loss oscillates, spikes, or becomes NaN. Too low: loss decreases smoothly but painfully slowly and may plateau early. With mini-batches, the gradient is noisy, so “stable” often means “stable on average,” not every step.

A systematic way to choose an LR is to run a short LR range test: start very small (e.g., 1e-5) and multiply by a constant each iteration until the loss blows up. The largest LR that still produces a consistent downward trend is a good upper bound; choose something 3–10× smaller for full training. This is faster than guessing and often reveals that your current LR is off by orders of magnitude.

Schedules adjust LR over time. Two simple, practical options you can implement from scratch:

• Step decay: every K epochs, multiply LR by gamma (e.g., 0.1). Good when you see early fast improvement then a plateau.
• Cosine decay: gradually reduces LR to a small value; often smoother than steps.

Warmup (increasing LR from near-zero to the target over the first few hundred/thousand steps) can prevent early divergence, especially with Adam, batch normalization, or large batch sizes. Warmup is easy: for the first T steps, use lr = lr_target * step/T, then switch to your normal schedule.

Batch size interacts with LR. Larger batches reduce gradient noise, which allows larger LR, but also reduces the “regularizing” effect of noise. A practical heuristic: if you double batch size, try increasing LR by ~1.5–2×, then verify with the validation curve. Common mistake: changing batch size and optimizer simultaneously; change one variable at a time so you can attribute improvements correctly.

Section 5.3: Regularization: L2, dropout, and early stopping

Regularization is how you bias the model toward solutions that generalize. When training loss keeps falling but validation loss starts rising, you are seeing overfitting: the network is memorizing patterns specific to the training set. Regularization methods intentionally make fitting harder in exchange for better out-of-sample performance.

L2 regularization (weight decay) adds a penalty proportional to the squared weights. In code, if your data loss is L and you add 0.5 * lambda * sum(W^2), then the gradient becomes dW = dW_data + lambda * W. The 0.5 is optional but convenient because it cancels the 2 when differentiating. A common bug is to regularize biases; typically you regularize weights only. Another bug is to add the penalty to the loss but forget to add lambda * W to dW, which makes your metrics inconsistent.

Dropout randomly zeros a fraction of activations during training, forcing the network to not rely on any single feature pathway. Use inverted dropout: during training, sample a mask M = (rand > p) and compute A_drop = (A * M) / (1-p) so the expected activation stays the same. During inference/validation, do not drop; just use A. In backprop, multiply the upstream gradient by the same mask and scaling factor. The most common mistake is applying dropout at test time or forgetting the /(1-p) scaling, which changes activation magnitudes and makes training/inference mismatch.

Early stopping is often the most cost-effective regularizer. Track validation loss each epoch, keep a copy of the best parameters, and stop after “patience” epochs without improvement. Early stopping pairs well with LR schedules: decay LR when validation plateaus, then stop if it still doesn’t improve. Practically, these tools reduce variance; they also make training curves interpretable when comparing experiments.

Section 5.4: Optimizers: momentum, RMSProp, Adam (from scratch)

Plain SGD updates parameters with W -= lr * dW. It’s simple but can be slow in ravines (steep in one direction, flat in another) and noisy with mini-batches. Modern optimizers modify the update using running statistics of gradients.

Momentum accumulates a velocity that averages gradients over time: v = beta * v + (1-beta) * dW, then W -= lr * v. With beta around 0.9, momentum damps oscillations and accelerates consistent directions. Implementation detail: you need one v array per parameter tensor (each layer’s W and b). Forgetting to initialize or persist these buffers across steps is a common source of “momentum that does nothing.”

RMSProp rescales updates by an exponential moving average of squared gradients: s = beta * s + (1-beta) * (dW*dW), then W -= lr * dW / (sqrt(s) + eps). This helps when different parameters have very different gradient magnitudes. Always add eps (e.g., 1e-8) to avoid division by zero.

Adam combines momentum (first moment) and RMSProp-like scaling (second moment), plus bias correction: m = beta1*m + (1-beta1)*dW, v = beta2*v + (1-beta2)*(dW*dW), m_hat = m/(1-beta1^t), v_hat = v/(1-beta2^t), then W -= lr * m_hat / (sqrt(v_hat)+eps). Track timestep t globally per update. Most from-scratch Adam bugs come from forgetting bias correction or using t per epoch rather than per parameter update.

Engineering judgment: start with Adam for fast iteration and sensitivity reduction, but if you can afford tuning, SGD+momentum sometimes yields slightly better final generalization. Regardless of optimizer, keep gradient checks and sanity metrics (loss decreases on a tiny subset) before you chase hyperparameters.

Section 5.5: Normalization concepts (feature scaling, batchnorm overview)

Normalization is about controlling scale. If your input features vary wildly (one feature in [0, 1], another in [0, 10,000]), the network wastes capacity and the optimizer struggles because gradients inherit those scales. The simplest fix is feature scaling: standardize each input dimension using training-set statistics: x_scaled = (x - mean) / (std + eps). Save mean/std and reuse them for validation/test. A common leakage mistake is computing mean/std on the full dataset including validation/test, which inflates metrics.

Batch normalization (BatchNorm) normalizes intermediate activations per mini-batch, then learns a scale and shift. Conceptually, it reduces internal covariate shift and allows higher learning rates, often making training less sensitive to initialization. Practically, it introduces two modes: training (use batch mean/var) and inference (use running averages). Even if you don’t implement BatchNorm fully yet, you should understand its workflow because it affects how you structure your training loop (mode flags, running stats) and why warmup and LR tuning can change.

Where normalization fits with other tools: good input scaling is non-negotiable; it makes all optimizers behave better. BatchNorm can reduce the need for dropout in some settings, but it is not a replacement for validation discipline or proper regularization when the dataset is small.

Practical outcome: if training is unstable, check input scaling and per-layer activation stats before changing the optimizer. Normalization issues often masquerade as “bad learning rate.”

Section 5.6: Hyperparameter tuning workflow and ablations

Hyperparameters are interconnected, so you need a workflow that produces trustworthy conclusions. Start by defining a baseline: fixed architecture, fixed data split, fixed preprocessing, and a single optimizer choice. Make runs reproducible with random seeds and consistent shuffling. Log training/validation loss, accuracy, learning rate, and (if possible) gradient/activation stats.

A practical tuning order that avoids wasted effort:

• Data and splits first: verify scaling, no leakage, and that a small model can overfit a tiny batch (sanity check).
• Learning rate next: do a short LR range test; then choose a schedule (or constant LR) and confirm stable descent.
• Batch size: choose based on hardware and noise needs; smaller batches often generalize better but are slower.
• Regularization: add L2, then dropout if needed, then early stopping for robust selection.
• Optimizer variants: compare SGD+momentum vs Adam only after the above is stable.

Use ablations to understand what helped: change one factor at a time and rerun. For example, if Adam “improved” results, confirm it wasn’t actually the higher effective step size by matching training loss curves or retuning LR for SGD. Keep a simple experiment table (run id, changes, best val metric, epoch of best, notes). This discipline prevents accidental “progress” driven by randomness.

Validation strategy matters. Use a dedicated validation set for tuning and keep the test set untouched until the end. If data is scarce, use k-fold cross-validation or repeated splits to estimate variance. Finally, select the model based on validation performance with early stopping, then retrain (optionally) on train+val using the chosen settings for a final model, documenting exactly what changed and why.

Section 6.1: Clean project structure and API design

A framework begins with a folder layout that separates concerns. A practical structure is: nn/ (core modules), data/ (datasets and transforms), train/ (loops, callbacks), experiments/ (configs and entrypoints), and tests/. This keeps “library code” stable while experiments change rapidly.

Design three tiny APIs: Parameter, Module, and Optimizer. A Parameter is a container for a NumPy array (.data) plus a gradient buffer (.grad). This prevents the common mistake of passing raw arrays everywhere and losing track of what should be updated. A Module defines forward(x), backward(dout), and a parameters() iterator that yields all nested Parameter objects. Your Dense layer should store W and b as parameters and cache inputs needed for backprop.

Include train() and eval() modes on Module. Even in a “minimal” framework, this is critical because dropout and batch norm behave differently during training. Forgetting to switch modes is a subtle bug that looks like “my validation is random.”

Finally, implement Optimizer.step(params) and Optimizer.zero_grad(params). Keep the first version simple (SGD, optional momentum, optional weight decay). Your framework should treat L2 regularization as a deliberate choice: either add weight_decay in the optimizer update or include L2 in the loss; don’t accidentally do both.

• Practical outcome: you can define a model as a composition of modules, call loss.backward() (or your manual backward), then let an optimizer update every parameter consistently.
• Common mistake: mixing “logits” and “probabilities.” Define one standard: your classifier outputs logits, and your loss function applies softmax internally for stability.

With these conventions, your capstone won’t collapse under bookkeeping. You’ll spend your time on modeling decisions (activation choice, hidden width, regularization) instead of chasing shape errors.

Section 6.2: Datasets, dataloaders, and mini-batch iteration

Training code should not care where data comes from. Create a Dataset interface with __len__ and __getitem__(idx) returning (x, y). For a real dataset capstone, choose something accessible and meaningful, such as the UCI Wine dataset, Fashion-MNIST (via a simple download script), or a CSV classification dataset from Kaggle. The key is: split into train/validation/test, and standardize features using statistics from the training split only.

Next, implement a DataLoader that yields mini-batches. It should accept batch_size, shuffle, and drop_last. Shuffling matters for SGD because it reduces correlations between consecutive batches; without it, your training curve may oscillate or overfit to ordering artifacts. For small tabular datasets, shuffling every epoch is usually enough; for large datasets, you may also want a random seed per epoch for reproducibility.

Implement batching using an index array: at the start of each epoch, build indices = np.arange(n), shuffle if needed, then iterate over slices. Use vectorized stacking so each batch has consistent shapes: X_batch should be (B, D) and y_batch either (B,) for class indices or (B, C) for one-hot labels. Decide early which label format your loss expects and stick to it to avoid silent broadcasting errors.

• Engineering judgement: if your dataset is small, full-batch gradient descent can work, but mini-batches typically converge faster in wall-clock time and act as mild regularization due to gradient noise.
• Common mistake: leaking normalization statistics from validation/test into training. Always fit mean/std on training only, then apply to all splits.

With a clean Dataset/DataLoader boundary, your capstone training loop can swap datasets or batch sizes without rewriting model code, and you can reproduce results by fixing seeds for shuffling and initialization.

Section 6.3: Training utilities: callbacks, logging, and timing

A training loop is where most “framework value” appears. You want a loop that is short, readable, and instrumented. Start with a function like fit(model, optimizer, train_loader, val_loader, epochs, callbacks). Inside each iteration: forward pass → loss computation → backward pass → optimizer step. Around it, measure metrics, timing, and learning-rate schedules.

Use a simple callback system to keep the loop clean. Define callback hooks such as on_train_begin, on_epoch_end, and on_batch_end. Then implement small utilities as callbacks: (1) progress logging (loss/accuracy every N batches), (2) learning-rate decay, (3) early stopping on validation loss, and (4) gradient norm tracking. This modularity helps you add features without turning fit into a 200-line function.

For metrics, separate computation from aggregation. Compute per-batch loss and accuracy, then maintain running sums to report epoch-level metrics. Track both training and validation. A common bug is averaging accuracies incorrectly (averaging per-batch accuracy rather than counting correct predictions across all samples), which matters when batch sizes vary.

Timing is an underrated tool for debugging. Record time per epoch and optionally time per batch. If time suddenly spikes, you may be doing unintended work (e.g., computing a full confusion matrix every batch). In NumPy, also watch out for accidental Python loops over samples; most performance issues trace back to missing vectorization.

• Practical outcome: you can compare experiments fairly because each run reports the same metrics with the same definitions.
• Common mistake: evaluating validation metrics with dropout still enabled. Ensure model.eval() for validation, then model.train() again afterward.

These utilities turn your capstone from “it trains” into “it trains predictably,” which is the difference between learning and guessing.

Section 6.4: Saving/loading weights and experiment tracking

Checkpointing is how you protect your work and enable iteration. Implement state_dict() on Module to return a nested mapping of parameter names to NumPy arrays. Also implement load_state_dict(state) to restore weights. Keep it strict: verify shapes match and fail loudly if a key is missing. Silent partial loading creates confusing “it runs but accuracy is worse” situations.

For the optimizer, store any internal buffers (e.g., momentum velocity). If you want to resume training exactly, checkpoint both model and optimizer state, plus the epoch number and RNG seeds. Save checkpoints at least when validation improves (“best.ckpt”) and optionally every N epochs (“epoch_10.ckpt”). Use a consistent directory structure such as runs/2026-03-21_14-30-00/ containing config.json, metrics.csv, and checkpoints.

Experiment tracking does not require a heavy tool. A CSV log with columns (epoch, train_loss, val_loss, train_acc, val_acc, lr, time_sec) is enough. Plot learning curves from this file so results are decoupled from your training process. When you review a run, you should be able to answer: Did we overfit? Did we underfit? Did a learning-rate choice destabilize training?

• Engineering judgement: checkpointing “best validation loss” is usually better than “best validation accuracy” for multi-class problems because loss is more sensitive to confidence calibration.
• Common mistake: saving only weights but not the preprocessing parameters (mean/std, label mapping). Your model is incomplete without its input contract.

For the capstone, treat each run as a small scientific experiment: fixed config, logged outcomes, and artifacts you can reload to reproduce evaluation and error analysis.

Section 6.5: Error analysis: confusion matrix and failure cases

A final accuracy number is not a model report. Error analysis tells you what to fix next. Start by computing a confusion matrix on the test set: for each true class, count predicted classes. Implement this in NumPy by accumulating counts into a (C, C) array. From it, compute per-class precision and recall; a model can have strong overall accuracy while failing a minority class.

Then inspect failure cases. For tabular data, look at feature ranges and whether misclassified samples are outliers after normalization. For images, save a small grid of misclassified examples with predicted/true labels and the model’s top-k probabilities. You are looking for patterns: systematic confusion between similar classes, sensitivity to lighting/background, or predictions that are overconfident on ambiguous inputs.

Connect failures to concrete interventions. If the confusion matrix shows two classes are often swapped, ask whether your network capacity is too small (underfitting) or whether features are insufficient. If training accuracy is high and validation accuracy is much lower, prefer regularization and data augmentation (if applicable): L2 weight decay, dropout in hidden layers, or earlier stopping. If both training and validation are low, increase capacity, improve initialization, or tune learning rate.

• Practical outcome: you can propose improvements backed by evidence, not intuition.
• Common mistake: performing error analysis only on the validation set and then tuning based on it repeatedly. Reserve a test set for final reporting; otherwise you slowly overfit your evaluation.

Your capstone report should include at least: learning curves, a confusion matrix, and 5–10 representative failure cases with a brief hypothesis for each category of error.

Section 6.6: Packaging your work: tests, docs, and reproducible runs

To make your framework usable beyond this course, add lightweight tests and documentation. Tests should target the most failure-prone parts: gradients, shape conventions, and serialization. A practical gradient test is a finite-difference check on a tiny network: perturb one parameter element by ±ε, measure the change in loss, and compare to backprop. You do not need to test every layer exhaustively; test one representative layer and your loss function, and add regression tests for bugs you actually hit.

Document the “user path” with a short README: how to install dependencies, how to run training, and how to reproduce the capstone results. Put hyperparameters in a config file (JSON/YAML) or a dataclass so runs are not hidden in code edits. Ensure every run records: random seeds, dataset split seed, model architecture, optimizer settings, and preprocessing parameters. Reproducibility is not perfection—NumPy can still differ across platforms—but it should be close enough that rerunning yields similar curves and the same qualitative conclusions.

Create a single entrypoint script, e.g. python -m experiments.train --config configs/wine_mlp.json, that trains, validates, saves the best checkpoint, and writes plots. This makes your capstone “push-button” and reduces human error (like forgetting to switch to eval() for validation).

• Practical outcome: your project becomes a small toolkit you can reuse for new datasets.
• Common mistake: letting notebooks become the only source of truth. Notebooks are great for exploration; the final training pipeline should run from scripts with versioned configs.

When you submit or share your capstone, include a model report: problem statement, dataset and preprocessing, architecture, training setup, results with plots, error analysis, and the next three improvements you would try. That’s the workflow used in real teams—and your minimal framework is the foundation.