Deep Learning for Beginners: Build a Photo Classifier

Humans learn concepts from examples. A child learns “cat” by seeing many cats: different colors, poses, backgrounds, and lighting. No one gives the child a strict list of rules like “if there are two triangles on top of a circle then it is a cat.” Machine learning follows the same philosophy: instead of hand-coding rules, we show the computer labeled examples and let it discover patterns that help it make good guesses on new inputs.

This difference—rules vs. learning from examples—matters most when the problem is messy. Photos are messy. Lighting changes, cameras differ, objects are partly hidden, and backgrounds vary. Rule-based systems break quickly because you can’t anticipate every case. With learning, you accept that you won’t describe the world perfectly; you’ll let the model fit itself to the evidence in your dataset.

That brings us to the three parts of any ML project:

• Data: the photos and their labels (what the correct answer is).
• Model: the function that turns an input photo into a predicted label.
• Evaluation: how you measure whether the predictions are good (accuracy, confusion matrix, etc.).

A common beginner mistake is spending 90% of the time on the model and 10% on the data. In practice, image projects often live or die by dataset quality: wrong labels, inconsistent classes, duplicates, or data leakage (the same photo appearing in train and test) can make results look great while the model learns nothing useful. In this course you’ll practice making the dataset small, clean, and well-structured first—then scaling up.

Deep learning is a type of machine learning that uses neural networks: flexible systems made of layers of simple math operations that can approximate complex patterns. A useful plain-language view is: a neural network is a stack of pattern detectors that can be tuned by data. Early layers tend to notice simple things (edges, corners, textures). Later layers combine those into more meaningful features (fur patterns, wheel shapes, faces), and the final layer turns those into a decision like “cat” vs. “dog.”

Why “deep”? Because the network has many layers, and each layer transforms the representation a little. Instead of you manually designing features like “count the whiskers,” the network learns internal features automatically. This is especially powerful for images because the number of possible visual variations is huge.

For your first toy classifier, you’ll likely use a small convolutional neural network (CNN) or a prebuilt model from a library. The engineering judgment here is to start simple and verify the pipeline end-to-end. If a tiny model cannot learn a tiny dataset, something is wrong: labels may be mixed, folders may be misread, or the input size may be incorrect. A toy model is a diagnostic tool, not just a learning exercise.

Another beginner trap is expecting the model to “understand” the image semantically. The network is learning statistical patterns, not meaning. It can still be extremely useful, but you must evaluate it honestly and watch for shortcuts (for example, it might learn that “dogs” photos were taken outdoors more often, and use background grass as a clue).

A model is a function: it takes an input and produces an output. For photo classification, the input is an image (really, a grid of pixel values), and the output is a set of numbers that represent the model’s confidence in each class. If you have two classes—say cats and dogs—the model might output something like [0.20, 0.80], meaning “20% cat, 80% dog.” The predicted label is usually the class with the highest score.

At the beginning of training, those guesses are almost random because the network weights start near random. Training is the process of adjusting weights so that the scores align with labels on the training images. Importantly, the model is not “storing” images; it is learning a set of numerical parameters that generalize patterns across images.

To make this concrete, you will train a tiny classifier on a tiny dataset—just enough images per class to see learning happen. The practical goal is to make the workflow real: load data, train for a few epochs, and then run the model on a photo it hasn’t seen. When it predicts correctly, you’ll know the pipeline works. When it predicts incorrectly, you’ll practice reading signals from evaluation rather than blaming the model blindly.

Common mistakes at this stage include mismatched label order (e.g., the folder order differs from the label list), images being read in the wrong color format, or the model output layer not matching the number of classes. A good habit is to print (1) the class names the loader discovered, (2) one batch of images with their labels, and (3) the output shape of the network. Small checks prevent hours of confusion.

Computers do not see objects; they see numbers. A photo is a rectangular grid of pixels, and each pixel is typically three numbers for red, green, and blue (RGB). A 224×224 color image contains 224×224×3 ≈ 150,000 numbers. “Recognizing a photo” means learning a function that maps that big pile of numbers to a label.

This is why dataset preparation and folder structure matter. Most beginner-friendly tools assume a simple layout like:

• dataset/
  • cats/ (images of cats)
  • dogs/ (images of dogs)

The folder name becomes the label. If you accidentally put a dog photo in cats/, you are teaching the model incorrect associations. With small datasets, even a few mislabeled images can significantly hurt learning.

Two more practical realities: images come in different sizes, and pixel values can have different ranges. Your notebook workflow will resize images to a consistent size and normalize pixel values (often to 0–1 or to a standardized distribution). Resizing is not “cheating”—it is a necessary step so the model sees consistent input shapes.

Finally, note that “recognition” is not perfect identification of every object in the scene. In classification, you provide one label per image. If an image contains both a cat and a dog, your label choice is ambiguous and will confuse training. Early projects work best with clean examples where the class is obvious and consistent.

All neural network training can be summarized as a loop:

• Guess: run the model on a batch of images to get predicted scores.
• Measure: compute a loss that penalizes wrong guesses (for classification, commonly cross-entropy).
• Adjust: use backpropagation and an optimizer (like SGD or Adam) to nudge weights to reduce the loss.
• Repeat: do this over many batches and multiple passes through the dataset (epochs).

In a notebook, you’ll see this as a few lines of code, but it helps to keep the loop in your head. If the model is not improving, ask which part of the loop is broken. Are the labels correct? Is the loss decreasing but accuracy not changing (could be class imbalance)? Is the learning rate too high (loss explodes) or too low (no progress)?

Evaluation is not optional; it is how you keep the model honest. You will measure accuracy, but you will also use a confusion matrix to see what kinds of mistakes the model makes. A confusion matrix is especially useful when classes are similar. For example, a model might be great at recognizing “cat” but frequently mislabel “small dog” as “cat.” Accuracy alone hides that pattern.

Even in Chapter 1, you should start thinking in terms of data splits: training vs. validation (and later test). If you evaluate on the same images you trained on, you are measuring memorization, not learning. With tiny toy datasets, you may still see high training accuracy; the key lesson is to keep a separate validation set so you can detect overfitting early.

As you progress, you’ll improve results with augmentation (random flips, crops, color jitter) to simulate new examples, and with basic tuning (learning rate, model size, number of epochs). The engineering judgment is to change one thing at a time and re-measure, so you can tell what actually helped.

Deep learning is easiest to learn in a notebook environment because you can run code in small pieces, inspect outputs, and visualize images and metrics. Your workflow for this course will follow a consistent pattern: open the notebook, set the dataset path, verify class names, train, then evaluate with accuracy and a confusion matrix.

You can train small models on a CPU, but training is often much faster on a GPU. A practical rule: if your notebook offers a GPU option (for example, in a cloud environment), use it once you move beyond toy datasets. However, don’t let hardware become a blocker. For Chapter 1, the objective is correctness and understanding, not speed.

Organize your project like an engineer, even when it’s small. A clean structure reduces mistakes and makes your work reproducible:

• project/
  • notebooks/ (your training notebooks)
  • data/
    • train/
    • val/
  • models/ (saved model files or checkpoints)
  • results/ (plots, confusion matrices, notes)

Common mistakes this structure helps prevent: accidentally training on validation images, losing track of which dataset version produced which result, and overwriting model files. Name outputs with timestamps or brief descriptors (e.g., cnn_baseline_aug1) so you can compare experiments.

Finally, treat your first toy classifier as a “hello world” for deep learning: small dataset, fast training, immediate feedback. If you can load images, train, and evaluate end-to-end in a notebook, you’ve built the foundation for everything that follows in the course—larger datasets, better models, and systematic improvements.

Section 2.1: Pixels are numbers: how a photo becomes input

An image file (like JPG or PNG) is a compressed representation. When you “load” it in Python, it becomes an array: a grid of pixel values. For a color image, each pixel typically has three channels—red, green, and blue—so the data looks like (height, width, channels). A common shape is (224, 224, 3), meaning 224 pixels tall, 224 pixels wide, with 3 color channels. Grayscale images often look like (height, width) or (height, width, 1).

In a notebook workflow, you should always inspect shapes early. If one file loads as (480, 640, 3) and another as (1024, 768, 4), you’ve already learned something important: your dataset is not uniform. The “4” channel case is often an alpha (transparency) channel. Many beginner pipelines break here because models expect a fixed channel count. You either convert all images to RGB (3 channels) or you consistently keep RGBA and build the model accordingly (usually not needed for beginners).

• Width/height vary: Cameras produce different resolutions; your model will require a fixed input size, so you will resize later.
• Channel count varies: RGB vs RGBA vs grayscale; convert deliberately.
• Value range matters: Pixels may load as integers 0–255 or floats 0–1 depending on the library.

A good practical habit: load 5–10 random images from each class, print their shapes and data types, and display them. You are looking for surprises: rotated images, corrupted files, odd color formats, and huge resolution differences. Catching these before training saves hours of confusion later.

Finally, remember that the network isn’t reading “pixels” as a photograph. It sees structured numbers. If your pipeline accidentally changes those numbers (e.g., inconsistent color conversion), the model will learn the wrong patterns. Treat image loading as part of your model, not just a pre-step.

Section 2.2: Labels and classes: what the model is trying to predict

Supervised image classification needs two things for each example: the input image (pixel array) and the target label (the “answer”). A label is typically the class name (e.g., "cat", "dog"), which is then encoded as an integer index (e.g., cat=0, dog=1). The neural network’s job is to output a probability distribution over classes and choose the most likely one.

Clear labels are more important than lots of data at this stage. If you accidentally label some dogs as cats, the model will be punished for correct behavior and rewarded for incorrect behavior. In practice, mislabeled examples often show up as stubborn errors that don’t improve with training. If you see the validation accuracy “stuck” or the confusion matrix heavily skewed, labeling quality is a top suspect.

Define your classes in plain language before you collect images. Ask: what counts as the class? If your class is “pizza,” do you allow photos of pizza slices, pizza boxes, or only whole pizzas? If “sports car,” do toy cars count? These decisions affect the boundary the model is asked to learn. A beginner-friendly dataset should have classes that are visually distinct and consistently defined.

• One image, one label: This chapter focuses on single-label classification. If an image can belong to multiple classes, you need multi-label setup (different output and loss).
• Don’t mix tasks: “Cat vs dog” is classification; “where is the cat?” is detection/segmentation and needs different labels.
• Keep a mapping: Save the class-to-index mapping used by your loader so predictions stay interpretable.

Practical outcome: by the end of this section, you should be able to explain, in one sentence, what the model is predicting (“Given an image, output which of N classes it belongs to”), and you should be able to point to where labels come from in your dataset structure.

Section 2.3: Dataset structure: folders, filenames, and consistency

For beginner projects, the simplest reliable dataset format is “one folder per class.” Many deep learning libraries can automatically infer labels from folder names. A common layout looks like:

• dataset/
  • cats/ (images of cats)
  • dogs/ (images of dogs)

This sounds trivial, but consistency here is what makes your entire training pipeline reproducible. Folder names become labels. If you later rename a folder, you have effectively renamed a class. Be deliberate: use lowercase, no spaces (or be consistent about spaces), and avoid special characters that may behave differently across operating systems.

Filenames matter less than folders, but good naming helps you debug. If you can, include a source identifier or a numeric id. Avoid duplicates that differ only by filename. Also avoid mixing non-image files (like .DS_Store or thumbs.db) into class folders; some loaders will error, others will silently skip, and both can confuse beginners.

Engineering judgment: decide whether you will create separate train/, val/, and test/ folders now or start with a single pool and split later. If you are collecting data manually, it can be safer to keep a single “raw” dataset folder and generate split folders programmatically so you can re-run the split in a controlled way.

• Rule: every image must belong to exactly one class folder.
• Rule: class folders must contain only images for that class.
• Tip: keep a small “quarantine” folder for questionable images you’re not sure about.

Practical outcome: you can point your notebook to a dataset directory and reliably load images with labels inferred from folder names, and you can explain how a mislabeled folder instantly becomes mislabeled training data.

Section 2.4: Train/validation/test: why we split and how

A model must be evaluated on data it has not seen during training. That is the purpose of splitting into three sets:

• Training set: used to fit the model parameters.
• Validation set: used during development to tune choices (learning rate, number of epochs, augmentation, model size).
• Test set: used once at the end to estimate real-world performance.

If you tune on the test set, you are effectively “studying the answers.” Your reported accuracy will look better than it should, and the model may disappoint on new images. This mistake is extremely common in first projects because the test set feels like “just more evaluation.” Treat the test set as locked until you are ready to finalize.

How to split: for a small beginner dataset, a typical split is 70–80% train, 10–15% validation, 10–15% test. The key is that the split must be random but reproducible (use a fixed seed), and it must be done at the right level. If your dataset includes near-duplicates (burst photos, multiple crops of the same scene), keep those variants in the same split. Otherwise you create data leakage, where the model sees almost the same image in training and evaluation and appears to generalize when it does not.

Practical notebook workflow: generate three folder trees (train/, val/, test/), each containing the same class subfolders. Many loaders expect exactly this. Then confirm the counts per class in each split and sample-view a few images from each split to ensure the split worked as intended.

Practical outcome: you can explain why three splits exist, create them consistently, and avoid leaking evaluation images into training by accident.

Section 2.5: Class imbalance and duplicates: easy checks

Two dataset issues can quietly ruin results: class imbalance and duplicates. Class imbalance means one class has many more images than another (e.g., 2,000 “dogs” vs 200 “cats”). A model trained on imbalanced data can achieve deceptively high accuracy by predicting the majority class most of the time. This becomes obvious when you look at a confusion matrix later: one row or column dominates.

Start with simple counting. In your notebook, count images per class for the overall dataset and for each split. If the imbalance is severe, you have options: collect more images for the minority class, downsample the majority class, use class weights, or use augmentation more heavily on the minority class. For beginners, the most reliable fix is “get more balanced data,” because it improves both training and evaluation.

Duplicates and near-duplicates are equally dangerous because they create accidental memorization. If the same photo (or a resized version) appears in both train and test, your test score stops measuring generalization. Easy checks include:

• Exact duplicates: compute file hashes (MD5/SHA) and look for repeated hashes across splits.
• Near-duplicates: compare image perceptual hashes (pHash) or check for repeated filenames/sources.
• Human spot-check: scroll a grid of random samples; repeated backgrounds and identical poses are clues.

Also watch for “label leakage” through artifacts: if one class was scraped from a website that adds a watermark, the model may learn the watermark instead of the object. Your checks should include looking for systematic patterns that correlate with labels (borders, text overlays, consistent backgrounds).

Practical outcome: you can run quick counts, detect imbalance, remove duplicates across splits, and recognize when high accuracy might be an illusion caused by data issues.

Section 2.6: Preprocessing basics: resizing and normalization

Neural networks require a fixed input size. Real photos do not. Preprocessing is the controlled conversion from “whatever images you have” to “uniform tensors the model expects.” Two basics are resizing and normalization.

Resizing: Choose a target size such as 128×128 or 224×224. Smaller sizes train faster but may lose detail. When resizing, you must decide how to handle aspect ratio. The simplest approach is to resize directly to the target shape, but this can distort images. A more careful approach is to resize while preserving aspect ratio and then center-crop or pad. For a beginner classifier, distortion is often acceptable, but be consistent across all splits.

Normalization: Pixel values are often 0–255 integers. Many models train better when inputs are scaled to 0–1 floats (divide by 255). Some pretrained models expect specific normalization (mean/std). The important rule is: apply the same normalization to training, validation, and test images. Inconsistent preprocessing is a classic cause of “my validation accuracy is terrible even though training looks fine.”

• Convert color consistently: ensure all images are RGB (3 channels) if that’s what your model expects.
• Keep preprocessing in the data pipeline: do it during loading so it’s repeatable, not manually in an image editor.
• Validate with a batch: after preprocessing, print one batch shape (e.g., (32, 224, 224, 3)) and confirm labels align.

Practical outcome: you can load images, inspect their shapes, convert them into a consistent size and numeric range, and produce clean batches ready for training in the next chapter’s classifier workflow.

At the smallest scale, a neural network is built from “neurons,” but a beginner-friendly way to think about a neuron is: a calculator that multiplies inputs by weights, adds them up, and produces an output. If your image is 64×64 RGB, it has 64×64×3 = 12,288 pixel values (numbers). A single neuron could take those 12,288 numbers as input and produce one number as output. That sounds extreme, but it shows the core idea: the model has many adjustable knobs (weights).

Mathematically, a neuron does a weighted sum: output = w1·x1 + w2·x2 + … + wn·xn + b, where w are weights, x are inputs, and b is a bias term. In code, this is just a dot product plus a bias. During training, the model changes w and b to reduce mistakes.

Practical workflow: when you define a model in a notebook (for example, in Keras), you aren’t manually creating each weight. You declare layer types and shapes; the framework initializes weights for you (usually small random values). Common mistake: assuming the model “starts smart.” It doesn’t—random initialization means early predictions may look like guessing, and that’s normal.

Engineering judgement: weights are where learning lives. If your model cannot improve at all (accuracy stuck near random), one of the first checks is whether anything is preventing weights from updating: wrong labels, training loop not running, learning rate too small/large, or layers accidentally “frozen.”

Outcome to aim for in this section: you can explain a neuron as “weighted mixing of inputs,” and you understand that training is mostly about changing those weights so the same input image maps to the correct class more often.

A single neuron is limited. A layer is simply many neurons working in parallel, producing a vector of outputs. Layers let you build step-by-step transformations: pixels → simple patterns → more abstract patterns → class prediction. In image tasks, you’ll usually use convolutional layers later, but your first model can start simpler just to learn the mechanics.

Why can’t we stack only weighted sums? Because without a non-linear step, multiple layers collapse into one big linear transformation. That means the model can only draw simple decision boundaries. Activations—functions like ReLU—introduce non-linearity. ReLU (“rectified linear unit”) is popular because it’s simple: it keeps positive values and zeros out negatives, helping the network represent complex patterns.

Beginner-friendly baseline architecture for images often looks like: Flatten (turn the 2D/3D image into a 1D vector) → Dense (a fully connected layer) → Dense (output layer). Your slightly better model adds at least one hidden Dense layer with ReLU. For multi-class classification, the output layer typically uses softmax, which converts raw scores (“logits”) into probabilities that sum to 1.

• Baseline: Flatten → Dense(num_classes, softmax)
• Slightly better: Flatten → Dense(128, ReLU) → Dense(num_classes, softmax)

Common mistakes: forgetting to normalize pixel values (e.g., keeping 0–255 instead of scaling to 0–1) can slow training; using the wrong output activation (e.g., sigmoid for multi-class softmax problems) causes confusing results; mismatching labels (integer vs one-hot) with the wrong loss function leads to shape errors or silent underperformance.

Outcome to aim for: you can point at each layer in your notebook and describe it as a transformation step, and you know activations are what allow the network to learn complex mappings rather than only linear ones.

Accuracy tells you how often the model is correct, but accuracy is not what training directly optimizes. Training needs a smoother signal than “right/wrong,” especially early on when the model is mostly wrong. That smoother signal is the loss: a number that is small when predictions match labels and large when they don’t.

For multi-class classification, the most common choice is cross-entropy loss. In plain language: it rewards the model for assigning high probability to the correct class, and it heavily penalizes confident wrong predictions. If the true label is “cat” but the model says 0.99 “dog,” the loss is large. If the model says 0.55 “cat,” the loss is smaller, even if the prediction is still uncertain.

In your notebook workflow, you’ll typically “compile” the model with a loss and metrics. For example, if your labels are integers like 0, 1, 2, you’d use sparse categorical cross-entropy. If your labels are one-hot vectors like [0,1,0], you’d use categorical cross-entropy. Choosing the wrong one is a classic beginner bug and can produce either errors or mysteriously bad learning.

Loss is also your best early debugging tool. If loss does not decrease at all after a few batches, something is wrong: labels may not match images, the learning rate may be unstable, or the model may be too simple for the task. If loss decreases but accuracy stays low, you may have class imbalance or a metric/label mismatch.

Practical outcome: after your first training session, you should look at the training curves (loss vs epoch, accuracy vs epoch). You’re not just hoping for a high number—you’re checking whether learning behavior makes sense and whether the model is improving steadily rather than erratically.

Once you can compute loss, you need a method to reduce it. That method is the optimizer. Conceptually, the optimizer asks: “If I change each weight slightly, will the loss go up or down?” It then adjusts weights in the direction that reduces loss. This process relies on gradients, which are computed automatically in modern frameworks (automatic differentiation).

The simplest optimizer is gradient descent. In practice, you’ll almost always use a variant like SGD (stochastic gradient descent) or Adam. Adam is a strong beginner default because it adapts learning rates for different parameters and often converges faster with less tuning. The key hyperparameter you will hear constantly is the learning rate: how big a step the optimizer takes when updating weights.

• Learning rate too high: loss may bounce wildly or diverge (training becomes unstable).
• Learning rate too low: loss decreases painfully slowly; model seems “stuck.”

Practical workflow: start with Adam’s default learning rate (commonly 0.001) and confirm that loss decreases within the first epoch. If it doesn’t, make one change at a time: lower the learning rate, confirm input normalization, and verify label mapping. Avoid changing multiple knobs at once—you’ll lose the ability to explain what helped.

Engineering judgement: optimizers don’t fix broken data. If your dataset folders are mislabeled or your train/validation split is leaking (same image in both), an optimizer can still produce deceptively “good” numbers. Always pair optimizer tuning with basic data sanity checks: visualize a batch, print a few labels, and confirm class counts.

Outcome to aim for: you can explain that the optimizer updates weights using gradients to reduce loss, and you know learning rate is the first knob to consider when training behaves oddly.

Training does not happen all at once. It happens in repeated passes over the dataset. One full pass is an epoch. Within an epoch, the data is processed in chunks called batches. Batch size is how many images the model sees before it updates weights once.

Why batches? Because updating weights after every single image is noisy and slow, while updating after the entire dataset requires too much memory and can be inefficient. Batches are a practical middle ground. Typical beginner batch sizes for small image datasets might be 16, 32, or 64 depending on your machine.

What should you expect during your first run? Early epochs often show a quick loss drop, then slower improvement. If training accuracy rises but validation accuracy stalls or drops, that’s a sign of overfitting—the model is memorizing training examples rather than learning general patterns. In later chapters you’ll address this with better splits, augmentation, and architecture improvements, but you can already watch for the symptoms now.

Common mistakes: setting epochs too high and trusting the final epoch blindly; not shuffling the training data (batches become biased); comparing runs with different batch sizes without noting that the learning dynamics change. Also, beginners sometimes assume “more epochs always means better.” In reality, once validation performance stops improving, more epochs can make things worse.

Practical outcome: you can choose a reasonable batch size, run a short training schedule (e.g., 5–10 epochs) for your baseline, and interpret the learning curves as evidence about whether the model is learning, overfitting, or not learning at all.

A baseline is a deliberately simple approach that gives you a reference point. In this chapter, your baseline model should be easy to implement and fast to train—often a Flatten + single Dense output layer. It will likely be weak, but it answers critical questions quickly: does the data pipeline work, do labels match images, does training run end-to-end, and does loss decrease?

After the baseline, you build a slightly better model by adding one improvement—typically one hidden Dense layer with ReLU, or a small convolutional block if you’re ready. The point is not to jump to a complex architecture immediately. When you change one thing at a time, you can attribute improvements correctly and build intuition.

How do you compare “weak” vs “better”? Use the same train/validation split, the same number of epochs, and the same metric (accuracy). Then look beyond a single accuracy number: examine a confusion matrix to see which classes get confused. A baseline might do fine on obvious classes but collapse on similar-looking ones. That information guides your next steps: more data, augmentation, or a model that can learn spatial features (convolutions).

• Sanity checks before celebrating: confirm class distribution, visualize a batch, and ensure no label leakage.
• Useful baseline outcome: “My pipeline trains and reaches X% validation accuracy; confusion matrix shows class A vs B is hardest.”

Common mistake: skipping the baseline and starting with a complicated model. If results are poor, you won’t know whether the issue is data, training settings, or architecture. A baseline acts like a diagnostic tool: it narrows the problem space so you can iterate with confidence.

Practical outcome: you finish the chapter with two trained runs—baseline and improved—plus a clear comparison that explains why the second model performs better (or, if it doesn’t, what evidence suggests your next debugging step).

Dense layers connect every input number to every neuron. If you flatten a 128×128 RGB photo, you get 128×128×3 = 49,152 input values. A single dense layer with 128 hidden units would need about 49,152×128 ≈ 6.3 million weights (plus biases). That’s already large for a beginner dataset, and it gets worse as image size grows. More weights mean more memory, slower training, and a higher chance of overfitting—memorizing training images rather than learning general rules.

Dense layers also ignore spatial structure. In an image, nearby pixels are related: edges and textures are local. Flattening destroys that “nearness” information. A dense model has to rediscover locality from scratch, which is inefficient. In practice this shows up as learning curves where training accuracy climbs but validation accuracy stalls or drops, especially when your dataset is small.

This is why your Chapter 3 baseline is a baseline: it’s a reference point, not the destination. Keep its metrics (accuracy, confusion matrix, learning curves) because you will compare them with the CNN. A common mistake is to discard the baseline and lose your ability to tell whether the CNN actually helped or whether you just changed random seeds or data preprocessing.

Engineering judgment: if your model has millions of parameters but you only have a few hundred images, expect instability. You can reduce image size, add regularization, or use transfer learning (Section 4.5). But the first big improvement is switching from dense-only to convolutional layers, which are designed to exploit image structure.

A convolution layer learns a small set of filters (also called kernels). Each filter is a tiny grid (for example 3×3) that slides across the image. At every position it computes a weighted sum of the pixels under it. The result is a new image-like output called a feature map. One filter might respond strongly to vertical edges, another to horizontal edges, another to simple textures. Early layers learn “simple” patterns; deeper layers combine them into more meaningful ones.

The key advantage is weight sharing: the same filter weights are used at every location. This massively reduces parameters and bakes in a useful assumption: a pattern can appear anywhere in the image. That assumption matches most photo classification tasks (a dog can be left or right in the frame), and it is why CNNs generalize better than dense layers on images.

In your notebook workflow, you can replace the baseline with a compact CNN such as: Conv(32, 3×3) → ReLU → MaxPool → Conv(64, 3×3) → ReLU → MaxPool → Flatten → Dense → Softmax. Keep it small at first so you can iterate quickly and compare fairly. Use the same train/validation split and the same preprocessing so the comparison is meaningful.

To visualize what filters and feature maps do, pick one image, pass it through the first convolution layer, and plot a few resulting feature maps. You are not “reading the model’s mind,” but you can confirm basic behavior: edge-like activations often appear early, and different filters highlight different structures. Common mistake: expecting feature maps to look like clean outlines. Often they look noisy; that’s normal. Your goal is to build intuition, not to diagnose every pixel.

After a convolution layer, it’s common to apply pooling, most often max pooling (e.g., 2×2). Pooling reduces the spatial size of feature maps by summarizing small neighborhoods. Max pooling keeps the strongest activation in each neighborhood. Conceptually, it answers: “Did this pattern appear somewhere in this region?” This makes the network more tolerant to small shifts in the image and reduces computation.

Pooling is not mandatory, but it’s a practical default in small CNNs. It helps you build deeper networks without exploding memory usage. However, too much pooling too early can throw away information—especially for small objects or fine details. A beginner-friendly rule: after each convolution block, pool once, and watch whether validation accuracy improves or worsens when you add more pooling.

Engineering judgment shows up here in learning curves. When you run training again with the CNN, compare the curves to your baseline: does validation accuracy rise earlier? Does the gap between training and validation widen (overfitting) or remain reasonable? If training accuracy becomes very high while validation lags, reduce model capacity (fewer filters), add dropout, use augmentation, or stop earlier with early stopping.

Common mistakes: (1) changing too many things at once (new model plus new preprocessing plus new split), which makes results hard to interpret; (2) forgetting to shuffle data when creating splits; (3) pooling repeatedly until feature maps become tiny, leaving the model unable to represent useful detail. Track each change and re-run training so you can explain why results improved (or didn’t).

Convolutions and pooling produce stacks of feature maps—tensors that still have a “height × width × channels” structure. To make a final class decision, you need a classifier head that turns those feature maps into class probabilities. The simplest approach is Flatten → Dense → Softmax. Flatten converts the feature maps into a long vector. Dense layers then learn combinations of detected patterns to separate classes.

Softmax produces a probability distribution across classes (values between 0 and 1 that sum to 1). During training, you typically minimize cross-entropy loss, which encourages the probability of the correct class to be high. Practical tip: confirm your label encoding matches the loss function (e.g., sparse labels with sparse categorical cross-entropy). A mismatch can lead to confusingly low accuracy even when the model is learning.

When you re-run training with the CNN, log both accuracy and loss for training and validation. Loss is often a more sensitive early indicator: accuracy can plateau while loss still improves. Compare the learning curves against your baseline. You want to see the CNN learn faster and reach better validation accuracy, not just memorize the training set.

After training, repeat the evaluation tools you already know: accuracy and confusion matrix. The confusion matrix is especially useful here because CNNs can shift which errors they make. For example, a baseline might confuse two flower types frequently, while a CNN might reduce that confusion but introduce a different one. This tells you where to collect more data, which classes may need clearer labels, or whether augmentation should focus on certain variations (lighting, rotation, background clutter).

If your dataset is small, transfer learning often beats training a CNN from scratch. The idea: start from a model pretrained on a large dataset (commonly ImageNet). Early layers have already learned general-purpose features like edges and textures. You then attach a new classifier head for your labels and train only that head at first. This typically yields strong results quickly and reduces overfitting.

Safe workflow: (1) choose a lightweight backbone (e.g., MobileNetV2/EfficientNet-B0) so training is fast; (2) freeze the backbone weights; (3) train a small head (global average pooling → dense → softmax); (4) evaluate; then optionally (5) unfreeze the last few backbone layers and fine-tune with a very small learning rate. Fine-tuning can help, but it can also overfit rapidly on small datasets, so watch validation curves closely.

Common mistakes: using the wrong preprocessing for the pretrained model (many backbones expect a specific normalization); unfreezing too much too soon; and inadvertently leaking data by applying augmentation or preprocessing differently between train and validation. Keep your train/validation/test splits consistent, and ensure augmentation is applied only to training images.

Transfer learning is not “cheating”; it is standard engineering. Your practical outcome is a model that performs well with less data and less compute. Still, you must validate honestly: compare to your scratch CNN and baseline, and use the confusion matrix to ensure improvements are real and not just a shift in which classes dominate predictions.

Once your CNN (or transfer-learned model) is trained, you will use it to predict new images. In code, you load an image, apply the same resizing and normalization used in training, and run a forward pass. The model outputs a vector of class probabilities. The “top class” is simply the index with the highest probability; the “confidence” is that probability value.

Interpret confidence carefully. A model can be highly confident and still wrong, especially when the new image is unlike the training data (different camera, unusual lighting, novel background, partial objects). Softmax confidence is not a guarantee of correctness; it is the model’s internal scoring. A practical habit is to look at the top-3 probabilities, not just the top-1, and to manually inspect failures. Many real improvements come from noticing a pattern in bad predictions: labels are ambiguous, images are mislabeled, or one class has fewer examples.

Practical outcome: create a small “prediction folder” of fresh images (not seen in training), run batch predictions, and record results. When a prediction is wrong, ask: is the image truly in one of the known classes? Is it too blurry? Is the object too small? This helps you define limits: what the model is intended to handle and what it should reject or flag for human review.

Finally, connect prediction back to evaluation. If the model often confuses two classes, you can tune by adding data, applying targeted augmentation, or adjusting the model capacity. The goal is not just a single accuracy number; it is a classifier whose behavior you understand well enough to use responsibly in a simple application.

Section 5.1: Metrics you can trust: accuracy, precision, recall (simple)

Accuracy is the most common metric: the fraction of predictions your model gets right. If your model correctly classifies 90 out of 100 images, accuracy is 90%. Accuracy is useful, but it can lie to you when classes are imbalanced. For example, if 95% of your photos are “cat” and only 5% are “dog,” a model that always predicts “cat” gets 95% accuracy while being useless for dogs.

That’s why you also want precision and recall (kept simple). Pick one class as the “target” to reason about—say “dog.” Precision answers: When the model predicts dog, how often is it correct? If precision is low, your “dog” predictions include many cats (false alarms). Recall answers: Out of all real dog photos, how many did the model catch? If recall is low, the model misses many dogs (false negatives).

Practical workflow:

• Start with accuracy on a held-out validation/test split (not training).
• If one class matters more (e.g., “defective” vs “ok”), track precision/recall for that class.
• When results change after an improvement, compare the same metrics on the same split so you’re not fooling yourself.

Common mistakes include reporting training accuracy as “final performance,” and changing the data split between runs (making metrics incomparable). In the next sections, you’ll add a confusion matrix to see what’s behind these numbers.

Section 5.2: Confusion matrix: seeing exactly what’s misclassified

A confusion matrix turns evaluation into something you can inspect. For a classifier with N classes, it’s an N×N table: rows are the true labels, columns are the predicted labels. The diagonal cells are correct predictions; off-diagonal cells are the mistakes. Instead of “82% accuracy,” you can say, “Most errors are birds predicted as planes when the background is sky.” That statement is actionable.

How to use it in practice:

• Look for one or two dominant error patterns. If “Class A” is often predicted as “Class B,” those two classes may be visually similar or your dataset may have labeling issues.
• Click into examples. For the biggest off-diagonal cell, review a handful of misclassified images. Are they blurry, too small, wrongly labeled, or unusually lit?
• Decide the fix type. If the images are genuinely hard, you may need augmentation or more data variety. If labels are wrong, fix labels first—training on errors teaches the model errors.

Engineering judgment: a confusion matrix helps you avoid “guessing improvements.” If most mistakes are concentrated in one pair of classes, changing the learning rate probably won’t help as much as improving the data that separates those classes (cropping, cleaning, or adding varied examples).

Also watch for a “collapsed” model: if one column is unusually full, the model is predicting one class too often. That often signals an imbalanced dataset, a label mapping bug, or a data preprocessing mismatch between training and validation.

Section 5.3: Overfitting vs underfitting: how to tell from curves

Two curves tell you more than almost any single metric: training performance vs validation performance over epochs. If you plot training loss and validation loss (or training accuracy and validation accuracy), you get a diagnostic tool.

Overfitting happens when the model learns the training data too specifically. Typical pattern: training loss keeps going down (or training accuracy keeps rising), but validation loss stops improving and may rise. The model is “memorizing” instead of generalizing. You might also see a big gap between training accuracy and validation accuracy.

Underfitting happens when the model can’t learn the task well even on the training data. Typical pattern: both training and validation metrics are poor and improve slowly, and the gap between them is small. This can happen when the model is too small, training is too short, learning rate is wrong, or the images are too low quality for the labels.

Practical steps when you see these patterns:

• If overfitting: stop training earlier (early stopping), add augmentation, add dropout, or simplify the model.
• If underfitting: train longer, try a slightly larger model, check preprocessing, and verify labels and image sizes.
• If curves are noisy: your batch size may be small or the dataset may be tiny; focus on stable evaluation (consistent split, same seed) before reacting.

Common mistake: “fixing” overfitting by training even longer because training accuracy looks great. Your goal is validation performance, because validation better predicts how the model will perform on new photos.

Section 5.4: Data augmentation: creating variety without new photos

Data augmentation improves generalization by showing the model varied versions of your images during training. You’re not inventing new labels; you’re simulating natural variation: small rotations, flips, crops, zoom, brightness shifts, and slight color changes. This is especially powerful for beginner datasets where you may only have dozens or hundreds of photos per class.

How to apply augmentation correctly:

• Augment training only. Validation/test data must remain “real” so metrics reflect real performance.
• Choose realistic transforms. For animals, horizontal flips may be fine. For digits or text (like “6” vs “9”), flipping/rotating can change the meaning and harm learning.
• Start small. Mild rotation (e.g., ±10°), slight zoom, and brightness jitter are good first choices. Extreme distortions can create unrealistic samples and confuse the model.

Augmentation also interacts with your data split. If you accidentally place near-duplicate images (same object, same photo burst) across training and validation, you get inflated validation scores. A better split groups similar shots together (e.g., by time, by folder, by source device) so validation is truly “unseen.” That one change—better splitting—often produces a more honest metric and guides better improvements.

Practical outcome: after adding augmentation, it’s normal to see training accuracy drop slightly (training got harder) while validation accuracy rises (generalization improved). That’s a good trade.

Section 5.5: Regularization basics: dropout and early stopping

Regularization means reducing overfitting by preventing the model from relying too heavily on specific features. Two beginner-friendly tools are dropout and early stopping.

Dropout randomly “turns off” a fraction of neurons during training. This forces the network to build multiple redundant pathways instead of memorizing a narrow set of cues. In practice, dropout is often added near the end of the model (e.g., before the final dense layer). Typical values are 0.2 to 0.5. Too much dropout can cause underfitting, so treat it like a dial: increase gradually and watch validation curves.

Early stopping stops training when validation performance stops improving. Instead of guessing the right number of epochs, you let the curve tell you. A common pattern is: validation loss improves for several epochs, then plateaus, then rises. Early stopping captures the best checkpoint before the rise. Use a small “patience” (e.g., wait 3–5 epochs without improvement) to avoid stopping due to noise.

Practical workflow:

• Enable early stopping that monitors validation loss and restores best weights.
• If overfitting remains, add dropout (or increase it slightly) and retrain.
• Re-check the confusion matrix: regularization may reduce certain brittle mistakes (e.g., background-driven errors).

Common mistake: applying these fixes while changing multiple other things at once. If you also change augmentation, learning rate, and model size, you won’t know what helped. Section 5.6 shows how to avoid that.

Section 5.6: Experiment tracking: keeping notes like a scientist

Improving a model is an engineering process, not a guessing game. Experiment tracking is simply writing down what you changed, why you changed it, and what happened. You don’t need fancy tools to start—just a consistent template in a notebook cell, a markdown table, or a text file in your project.

Track these minimum fields for every run:

• Dataset version and split method: number of images per class, how you split (random, by folder, by source), and any cleaning you did.
• Model details: base architecture, input size, whether you used dropout, and where.
• Training settings: epochs, batch size, learning rate, augmentation on/off, early stopping patience.
• Results: validation accuracy, precision/recall (if relevant), and a saved confusion matrix screenshot or values.
• Notes: what errors dominate, what you plan to try next.

A practical rule: change one major thing per experiment (or at most one major and one minor). For example, “Add augmentation” is a major change; “increase epochs from 10 to 15” is minor. This discipline lets you attribute improvements to causes. It also prevents you from “chasing noise”—small metric swings due to randomness rather than real progress.

Finally, keep your best model checkpoint and the code/config that produced it. When someone asks, “How did you get this performance?” you can answer with evidence: metrics, confusion matrix, curves, and a clear timeline of decisions. That is what makes your improvements reliable and repeatable.

When you “save a model,” you’re deciding what information to persist so you can reuse it later. In deep learning, there are two common approaches: saving weights only or saving the full model. They sound similar, but they solve different problems.

Weights-only saving stores the learned numbers (the parameters) but not the architecture code that describes how layers connect. In practice, that means you must recreate the exact same model structure in code before loading the weights. This is a great choice when you control the codebase (you’ll keep the notebook or a Python file that defines the model) and you want smaller files. Typical filenames look like model_weights.h5 or weights_epoch10.keras.

Full-model saving stores the architecture, the weights, and often training configuration. This is the best beginner-friendly option for sharing, because someone can load the model without rebuilding the network by hand. In Keras, the modern recommended format is .keras (for example, photo_classifier_v1.keras). A full model is also less error-prone: you avoid “shape mismatch” mistakes when someone accidentally changes an input size or layer order.

• Common mistake: saving weights, then later changing the model definition (even slightly) and wondering why loading fails.
• Common mistake: saving a full model but forgetting that your preprocessing (resizing, normalization) is not automatically included unless you built it into the model graph.

Practical workflow: after training, save a full model for easy reuse and sharing, and optionally save weights checkpoints during training for recovery and experimentation. Then immediately test a “cold start” reload: restart the runtime, load the saved model, and run one prediction. If that works, you have a reproducible artifact—not just a one-time training run.

A model is only half of a photo classifier. The other half is the prediction pipeline: the steps that turn a user’s image into the exact numeric format your model expects. Your pipeline should be boring, predictable, and identical every time.

A clean pipeline usually has these stages: (1) load image from a file path, (2) convert to RGB (to avoid surprises with grayscale or transparency), (3) resize to your training size (for example, 224×224), (4) convert to a tensor/array, (5) normalize pixel values the same way you trained (common: divide by 255.0), (6) add batch dimension so the shape is (1, H, W, C), and (7) predict and map output scores back to a label name.

Engineering judgment: keep preprocessing in one place. If you copy/paste normalization logic into three notebooks, it will drift over time and you’ll get confusing results. Instead, put the preprocessing function in a small Python file (for example predict.py) and import it. If your training used Keras preprocessing layers (like Rescaling or RandomFlip), keep only the deterministic parts for inference—augmentation should be disabled at prediction time.

• Common mistake: training with normalized inputs but predicting with raw 0–255 pixels, leading to wildly wrong outputs.
• Common mistake: mixing up label order. Always save the class_names list used during training and reuse it in prediction.

Practical outcome: you can build a “predict a photo” workflow anyone can run: a command like python predict.py --image path/to/photo.jpg prints the top label and confidence. This is the bridge from your notebook to a usable tool.

Single-image prediction is useful for demos, but real work often needs batch predictions: score an entire folder of images, produce a CSV, and quickly spot failure patterns. This is how you move from “it seems to work” to “I know where it fails.”

A simple batch workflow looks like this: collect all image paths in a directory (including nested folders if needed), run your preprocessing function on each image, stack them into batches (for speed), and call model.predict once per batch. For each image, store the filename, predicted label, and confidence (such as the max softmax probability). Save results to predictions.csv so you can sort by confidence and manually inspect edge cases.

Engineering judgment: batching is about performance and stability. Predicting one image at a time can be slow due to overhead; predicting thousands at once can run out of memory. A reasonable batch size (like 16 or 32) is a safe starting point on CPU. Also, handle failures gracefully: if one image file is corrupted, skip it and log the error rather than crashing the whole run.

• Common mistake: assuming high confidence means “correct.” Confidence can be misleading when the model sees out-of-distribution images (different lighting, new backgrounds, different camera types).
• Common mistake: forgetting to keep the same resize/crop behavior used in training, producing systematic errors on tall or wide images.

Practical outcome: batch scoring helps you create a small “review set” of the worst predictions. That, in turn, tells you what data to collect next and whether augmentation or label cleanup is the right fix.

If you plan to share your project—or even revisit it next week—you need model versioning. Versioning is not complicated; it’s mostly about disciplined file naming and saving the right metadata so results are reproducible.

Start with a simple convention: every saved model gets a version ID and a short description. For example: models/photo_classifier_v1_2026-03-28.keras. Alongside the model file, save a small JSON (or YAML) “model card” with: training date, dataset name or snapshot, image size, class names (in order), final validation accuracy, and any key settings (augmentation on/off, optimizer, learning rate). This prevents the classic confusion of “Which model did I email?” and “Why does this one predict differently?”

Packaging your project folder becomes straightforward when you standardize structure. A beginner-friendly layout might be:

• models/ (saved models)
• data/ (or a README pointing to where data lives, if it’s too large)
• notebooks/ (training notebook)
• src/ (predict and utility code)
• requirements.txt (dependencies)
• README.md (how to run training and prediction)

Engineering judgment: do not share trained models without also sharing the exact preprocessing steps and class label order. Many “broken model” reports are actually “broken assumptions” about inputs. Also avoid putting huge datasets inside a shared zip; instead, provide a download link, a script, or instructions to recreate the folder structure.

Practical outcome: you can zip the project, send it to a friend, and they can reproduce a prediction with minimal setup. That’s the difference between a private experiment and a usable deliverable.

Photo classifiers feel harmless because they’re “just images,” but real-world use raises serious questions. Responsible practice starts early, even in beginner projects, because habits formed now scale into bigger systems later.

Bias: your model learns patterns from your dataset, including unintentional shortcuts. If your “cat vs dog” set has cats mostly indoors and dogs mostly outdoors, the model may learn “background” instead of “animal.” The confusion matrix you used earlier is a great starting tool, but you should also test on a small, diverse set of photos: different lighting, camera quality, backgrounds, and subjects. If performance collapses for a particular subgroup (for example, dark images or certain environments), that’s a sign your training data is not representative.

Privacy: photos often contain faces, addresses, license plates, or other sensitive details. Don’t upload private images to shared repos, and don’t ask others to contribute images without explaining how they’ll be used and stored. If you must store images, limit access and keep only what you need. Consider anonymizing by cropping or blurring where appropriate.

Consent and licensing: only use images you have the right to use. “Found on the internet” is not consent. If you’re collecting your own dataset, tell participants what you’re building, whether the images will be shared, and whether they can request removal. Document the source of each dataset and any restrictions in your README.

• Common mistake: evaluating only on your validation split and assuming the model is “ready.” Real-world photos often differ sharply from curated datasets.
• Common mistake: sharing a project zip that accidentally includes personal photos in data/.

Practical outcome: you develop a habit of testing generalization, protecting privacy, and documenting consent—skills that matter more as you move toward deployment.

You now have an end-to-end beginner workflow: prepare labeled images, train a classifier, measure accuracy and a confusion matrix, save the model, and run predictions. The next steps depend on your goal—better performance, broader coverage, or real-world deployment.

If you want better results, focus on data first. Collect more diverse examples per class, clean labels, and address the failure cases you saw during batch scoring. Then revisit simple tuning: adjust image size, train longer with early stopping, and try a pretrained backbone (transfer learning) if you haven’t yet. You’ll often get the biggest gain by improving the dataset rather than changing the network.

If you want bigger datasets, learn about efficient input pipelines: streaming from disk, caching, prefetching, and using TFRecord (or equivalent) formats. This keeps training fast and avoids memory bottlenecks. Also learn more robust evaluation: stratified splits, cross-validation (when feasible), and separate “test sets” that you do not touch until the end.

If you want deployment, start small and safe. Wrap prediction in a simple CLI or lightweight web app, then add guardrails: input validation, logging, and clear error messages. Learn about exporting and serving models, CPU vs GPU inference, and measuring latency. If your app will affect people (even indirectly), learn about monitoring, drift detection, and rollback—this is where model versioning becomes essential.

• Common mistake: chasing complicated architectures before you’ve fixed dataset quality and pipeline consistency.
• Common mistake: deploying without a clear statement of what the model can and cannot do.

Practical outcome: you have a realistic roadmap: improve data, strengthen evaluation, adopt transfer learning, and only then move toward deployment with versioning and responsibility built in.