Reproducible AI for Beginners: Replicate a Study & Report Changes

An “AI study,” in beginner terms, is any documented attempt to answer a question using a model and evidence. The question might be “Can a classifier detect spam?” or “Does model A outperform model B on dataset X?” A study usually includes (1) a dataset (or a way to collect it), (2) a method (preprocessing + model + training/evaluation procedure), and (3) a reported result (metrics, plots, examples, or a table). The “paper” could be a formal academic publication, a conference workshop report, a blog post with code, or even a well-structured GitHub repository with a README and an experiment table.

For this course, you want a study that is small enough to run on a laptop (or free cloud notebook) and concrete enough that you can compare your outcome with the author’s. Good beginner studies often use classic datasets (MNIST, CIFAR-10 subset, IMDb reviews) or small tabular datasets (UCI). They usually report one or two main metrics (accuracy, F1, RMSE) and include a baseline model that trains quickly.

Practical test: if you can describe the study in one sentence—“Train a simple CNN on MNIST and report test accuracy”—you’re likely in the right scope. If you need specialized hardware, huge datasets, proprietary APIs, or complex data access agreements, it is not beginner-friendly. Your target is a study with clear steps and publicly accessible resources.

• Study artifact: paper/blog/README that states the goal and the result.
• Runnable method: code, notebook, or enough detail to implement.
• Comparable output: metric(s) you can reproduce and record.

Think of a study as a recipe plus a taste test. Your job is to see whether the recipe works in your kitchen, and to document what changed if it doesn’t.

These terms are often used loosely, so you need a practical set of meanings you can apply while working. In this course, treat them as different levels of “same-ness,” from easiest to hardest.

Repeatable means you can run the same code, in the same environment, with the same inputs, and get the same outputs (or statistically indistinguishable outputs if randomness is expected). Example: you rerun a notebook in the same Python environment with fixed seeds and the accuracy matches within 0.1%.

Reproducible means someone else can do that too using your shared artifacts and instructions. Example: a classmate clones your repo, installs dependencies, runs one command, and gets the same evaluation table. This requires documentation, pinned versions, and recorded parameters—not just working code on your machine.

Replicable (or replication) means an independent re-implementation can reach the same scientific conclusion, even if the codebase is different. Example: you implement the described model from the paper from scratch (or using different libraries) and still observe the claimed performance trend (e.g., model A > model B) on the stated dataset.

• Repeatable: same person, same setup, rerun.
• Reproducible: different person, same shared setup.
• Replicable: different implementation, same conclusion.

Engineering judgment: beginners should aim first for repeatability, then reproducibility, and only then attempt deeper replication. If you skip straight to “replicate the paper,” you may not know whether a mismatch is due to your new implementation, a missing preprocessing step, or a library difference.

Milestone 1 in this chapter is simply being able to label what you are doing at each step. When you run the baseline code as provided, that is usually repeatability/reproducibility work. When you intentionally change elements (re-implement, swap libraries), that is replication work.

Results change because “the same thing” is rarely fully specified. AI pipelines have hidden degrees of freedom—random seeds, data ordering, nondeterministic GPU kernels, and preprocessing defaults. Even small shifts can move a metric enough to matter, especially on small datasets or near decision boundaries.

Start with the most common cause: randomness. Training often uses random initialization, shuffled minibatches, dropout, and data augmentation. If the original study ran five seeds and reported the mean, but you run one seed, your number can legitimately differ. Fixing seeds helps, but note that seeds do not guarantee identical results across hardware or library versions.

Data differences are next. You might download a newer dataset version, use a different train/test split, or apply slightly different tokenization. “Same dataset” can still vary if the study used a curated subset, filtered examples, or a particular preprocessing script. A common beginner mistake is to rely on dataset names rather than verifying checksums, record counts, label distributions, and split logic.

Environment drift is subtle but powerful. A minor change in Python, NumPy, PyTorch/TensorFlow, CUDA, or even BLAS libraries can change numerical behavior and therefore training dynamics. Another common failure: dependency installation silently picks newer versions than the author used, because versions were not pinned.

Choices not written down also matter: early stopping patience, learning-rate schedules, batch size, number of epochs, and evaluation details. Two people can “train the same model” but evaluate on different checkpoints, compute metrics differently, or include/exclude preprocessing in the evaluation pipeline.

• Randomness: seeds, shuffling, augmentation, dropout.
• Data: versions, splits, preprocessing, filtering.
• Environment: library versions, hardware, nondeterministic ops.
• Procedure: hyperparameters, stopping rules, metric calculation.

Milestone 3 is learning to see mismatches as diagnostic signals, not personal failure. When your results differ, your task is to identify which moving part changed, then record it. Your final report can be valuable even if you do not match the original metric exactly—because you will explain why it shifted.

To replicate safely and systematically, you need a “reproducibility map” that breaks an AI study into three buckets: inputs, process, outputs. This is Milestone 2 (identify moving parts) expressed as a practical diagram you can keep in your project README.

Inputs include data files, labels, splits, and any external resources (pretrained embeddings, checkpoints). Also include configuration inputs: hyperparameters, random seeds, and command-line arguments. A good habit is to treat every run as a function call: if you can’t list the function’s inputs, you can’t expect the same output later.

Process includes preprocessing code, model architecture, training loop, evaluation procedure, and the environment in which it runs. For beginners, you can track the environment without advanced tools by recording: OS, Python version, core library versions, and whether you used CPU or GPU. Also record the exact command used to run training and evaluation.

Outputs include metrics, plots, logs, and saved artifacts (model weights). Outputs must be stored in a predictable place and named so you can compare runs. Avoid overwriting results; instead, create a new run folder per attempt.

• Minimal folder structure: project/ → data/, src/ (or notebooks/), runs/, reports/.
• Run naming: runs/2026-03-28_seed42_baseline/ with config.txt, metrics.json, stdout.log.
• Experiment log: a simple experiment_log.md table: date, goal, changes, command, result, notes.

Milestone 4 (choose a target study) and Milestone 5 (success criteria/time box) become easier when you can point to what you will hold constant (inputs/process) and what you will compare (outputs). This map also protects you from “accidental improvements” that are really untracked changes.

This checklist is designed for beginners who are not using containers or heavy experiment platforms. It is intentionally lightweight: a handful of files and habits that prevent most confusion. Use it before you claim a run “matches” or “doesn’t match.”

• Study reference captured: save the paper/blog link and commit hash (if code). Copy key claims (metric, dataset, model) into reports/notes.md.
• Data pinned: record dataset source URL, download date, file counts, and (when possible) checksums. Record exact split procedure or store split files.
• Environment recorded: write down OS, Python version, and key package versions. Keep a requirements.txt (even if not perfect) generated after installation.
• Randomness controlled: set seeds where possible (Python, NumPy, framework). Note if GPU nondeterminism remains.
• Commands saved: store the exact command(s) used for training and evaluation in the run folder.
• Configs preserved: save hyperparameters and any config files alongside outputs.
• Outputs not overwritten: each run gets its own directory with metrics and logs.
• Single source of truth log: maintain experiment_log.md with what changed and why.

Common mistakes this checklist prevents: reusing the same output folder (so you compare the wrong results), “fixing” a bug but forgetting to note the change, and changing two variables at once (e.g., new seed and new learning rate). Engineering judgment here is simple: change one thing per run unless you are intentionally doing a larger migration, and then write it down clearly.

This checklist is also your bridge from repeatable to reproducible: once your log is clean and your run folders are consistent, another person can follow your steps with far fewer questions.

Your project study should be safe (ethically and legally), small (computationally), and clear (method and metric). Milestone 4 is choosing the target; Milestone 5 is setting success criteria and a time box so you finish. Aim for something you can run end-to-end in 1–3 hours once the environment is set up, and iterate on in short cycles.

Safety and ethics first. Avoid studies involving private user data, scraping that violates terms of service, medical/biometric identification, or anything requiring access-controlled datasets. Prefer widely used public datasets with clear licenses. Also avoid “harmful capability” targets (e.g., malware generation). Your goal is academic skill-building, not deploying a high-risk system.

Beginner-friendly targets often include: a baseline classifier on a standard dataset; a simple sentiment model; a small image model; or a tabular regression. Choose a study that reports a baseline and provides either code or enough detail to re-create it. If the study’s result depends on extensive hyperparameter search, it’s harder to time-box.

• Good target signals: public data, single GPU not required, clear metric, clear preprocessing, a stated baseline.
• Red flags: “we tuned extensively” with no details, proprietary data, heavy compute, unclear evaluation protocol.

Set success criteria: define what “success” means before you run anything. For example: “I can run the author’s code and get within ±1–2 percentage points of the reported accuracy,” or “I reproduce the ranking between two models even if absolute numbers differ.” Also define what you will deliver: a filled experiment log, a baseline run folder, and a short report describing differences.

Time box: commit to a fixed window (for example, two evenings or one weekend). If you hit a blocker (missing data, broken code), record it and choose a backup study rather than sinking the whole project. Replication work is real research work: knowing when to stop and document is part of the skill.

Start with a single project folder dedicated to the replication. Do not mix it into “Downloads,” a class folder with unrelated assignments, or a drive full of old experiments. Confusion is the number-one enemy of repeatability because it leads to accidental overwrites and missing context. Your project folder should separate: (a) inputs you obtained, (b) code you wrote or adapted, (c) outputs you generated, and (d) notes describing what happened.

Use a small, predictable structure. Here is a beginner-friendly layout that works for most replications:

• 01_readme/ (what this project is, links to the study, how to run)
• 02_setup/ (environment notes, install notes, version snapshots)
• 03_data/
  • raw/ (downloaded or original data—treat as read-only)
  • processed/ (your cleaned/filtered data)
• 04_code/ (scripts, notebooks, small utilities)
• 05_runs/ (one subfolder per run, with outputs and run log)
• 06_reports/ (figures you’ll include, write-up drafts, final report)

This structure supports Milestone 1: creating a clean folder structure. The key design choice is the “raw vs processed” split. If raw data is ever edited in place, you can’t reliably re-create the processed version. Treat 03_data/raw as read-only: if you must change something, save a new file in processed and document the transformation.

Common mistake: storing outputs directly next to code (“results.png” next to “train.py”). That invites overwriting and makes it unclear which settings produced which file. Instead, each run gets its own folder under 05_runs. This single habit prevents many replication headaches.

File names are part of your reproducibility system. Good names let you compare runs without opening files and guessing. Aim for names that answer three questions: what is it, when was it produced, and under which configuration (or run ID).

A practical naming scheme for runs is a timestamp plus a short label:

• 2026-03-28_1530_baseline/
• 2026-03-28_1610_seed42/
• 2026-03-29_0905_lr-0p001/

Inside each run folder, keep a consistent set of filenames so you can diff runs quickly:

• run_log.md (your filled-in template)
• metrics.json or metrics.csv (accuracy, loss, etc.)
• stdout.txt (captured terminal output, if relevant)
• fig_*.png (plots with meaningful prefixes)
• model.bin or checkpoint/ (only if needed)

For individual files, avoid spaces and ambiguous names like “final,” “new,” or “results2.” Prefer lowercase with underscores, and encode key distinctions: train_split_v1.csv, train_split_v2_stratified.csv. If you export a figure, name it for the comparison it supports, not its aesthetics: fig_loss_curve_baseline.png is better than plot1.png.

Engineering judgement: don’t over-encode everything in the filename. If you find yourself creating names like run_seed42_lr0p001_bs32_augStrong_dropout0p3_v7.png, move those details into the run log instead. The filename should help you navigate; the run log should preserve the full truth.

Common mistake: reusing the same output path each time (“outputs/”). That silently mixes artifacts from multiple runs. If you must keep a single output folder, force it to include a run ID subfolder (even manually).

Milestone 2 is a “setup note”: a simple text file that records your environment. This matters because small version differences can change results—sometimes slightly (floating-point behavior, default seeds) and sometimes dramatically (model implementations, preprocessing defaults). Your goal is not to freeze the world; it is to leave a clear trail so another person (or future you) can approximate your environment and understand discrepancies.

Create 02_setup/setup_note.md and include the following, filled in as best you can:

• Date created and your name/initials
• Operating system (e.g., Windows 11 23H2, macOS 14.3, Ubuntu 22.04)
• Hardware (CPU model; GPU model if used; RAM optional but helpful)
• Python/R version (or “browser only” if that’s the tool)
• Key libraries with versions (e.g., numpy, pandas, torch, tensorflow, scikit-learn)
• How you installed (pip, conda, system packages) and where (system, venv)
• Browser version if you rely on web tools or hosted notebooks

Keep it simple: copy-paste version outputs from your terminal or “About” pages. If you use Python, it’s usually enough to save a list of installed packages at the time of the run (for example, exporting a “requirements” style list). If that feels advanced, record only the top 5–10 libraries you directly import. The point is coverage of the likely causes of changes.

Common mistake: recording versions once and never updating them. Instead, update the setup note when you notice something changed (you upgraded Python, your library auto-updated, you moved machines). Reproducibility is a living record, not a one-time ceremony.

Practical outcome: when your replication differs from the paper, you can rule in/out environment drift as a cause rather than guessing.

Milestone 3 is about data provenance: documenting where your data came from, which exact version you used, and what you did to it. In replications, “same dataset” is often the hidden variable—especially when data is hosted online and can be updated, reprocessed, or mirrored. Provenance is your defense against invisible data drift.

Create 03_data/data_provenance.md (or put this in 01_readme if you prefer) and record:

• Source: URL(s), paper citation, or repository link
• Date accessed: when you downloaded or copied it
• Identity: filename(s), dataset version tag, commit hash (if available)
• Integrity: checksum if you can (optional), or at least file size and row counts
• License/terms: usage permissions and any restrictions
• Transformations: what you changed to create processed data

Keep raw data untouched in 03_data/raw. Put any cleaning scripts in 04_code and write outputs to 03_data/processed. Then, in your provenance note, describe transformations at the level a careful peer would need: “Removed rows with missing label; lowercased text; tokenized with X; split 80/10/10 using stratified sampling; random seed 42.” If you used a tool that does preprocessing automatically (some notebooks and libraries do), write that down too—defaults count as choices.

Common mistakes include: renaming raw files without recording the original name, copying data into a notebook cell (hard to track), or using “latest” links that change over time. Prefer stable links and archived releases when possible. If the study used a dataset snapshot, try to find that snapshot; if you can’t, record that mismatch explicitly because it may explain differences in results.

Practical outcome: you can re-run the same preprocessing months later and know whether a difference came from data content, data cleaning decisions, or model training.

Milestone 4 is the heart of beginner reproducibility: a run log template you reuse every time. Many “replications” are not reproducible because the researcher cannot reconstruct what they did between attempt #1 and attempt #6. A run log turns that fog into a sequence of testable steps.

Create 05_runs/run_log_template.md and copy it into each run folder as run_log.md. Keep it short enough that you will actually fill it in. Here is a practical template:

• Run ID: (folder name)
• Date/time:
• Goal: (baseline replication, change seed, match paper preprocessing, etc.)
• Inputs: dataset files used (paths), counts (rows, classes), any downloaded resources
• Code: script/notebook name(s), plus commit hash if you use git (optional)
• Settings: key hyperparameters, random seed, number of epochs, batch size, learning rate
• Environment: reference to setup note; note any deviations
• Command: exact command or steps to run (copy-pasteable)
• Outputs: paths to metrics/figures/models; headline numbers
• Notes: warnings, errors, surprises, interpretation, “what to try next”

This log directly supports the course outcomes “Run a baseline replication and record inputs, settings, and outputs” and “Track experiments with a clear log so someone else can repeat your steps.” It also prepares you for Chapter 3’s work of explaining changes. When results differ, you can compare run logs line-by-line: was the seed different, the split different, the library version different, or did you change preprocessing?

Common mistake: writing narrative notes but not recording the exact command or the exact data file paths. If a step cannot be copied and executed by a stranger, it is not yet reproducible. Another mistake is recording metrics without recording how they were computed (e.g., macro vs micro F1). Put evaluation choices in “Settings.”

Milestone 5 is a dry run: confirm that everything is accessible and you could repeat your own steps. Before that, you need one more protection: simple backups and snapshots. Beginners often postpone backups until after something breaks. In reproducible AI, backups are not just safety—they’re part of the evidence.

You can do snapshots without fancy tools:

• Copy the whole project folder to a second location (external drive or cloud) at key moments: after baseline run, after matching paper settings, after writing the report draft.
• Export important artifacts (metrics files, final figures, run logs) into 06_reports so they survive cleanup in 05_runs.
• Use “read-only” protection for raw data and for baseline outputs once confirmed, to avoid accidental changes.

Now do the dry run. Pick your baseline run folder (or create one), then verify:

• You can locate the paper/study link from 01_readme.
• Your setup note contains enough detail to reinstall the main tools.
• Your data provenance note identifies the exact raw files present.
• Your run log includes a copy-pasteable command or step list.
• Outputs are stored inside the run folder, not scattered elsewhere.

If the dry run fails, treat it as a gift: it found a reproducibility gap early. Fix the gap immediately (update the note, rename the file, move the output). This is the practical mindset shift: you are not “done” when the model runs; you are done when the work can be rerun on purpose.

Common mistake: backing up only code. In replication work, the data snapshot and the run logs are often more valuable than the scripts, because they capture the conditions that produced your observed results.

The method section in a study is rarely written like a tutorial. Your job is to translate it into an actionable to-do list that you can execute and later prove you executed. Start by scanning for anything that sounds like a setting, a choice, or a measurable outcome. Then rewrite those details into steps you can check off.

A practical way is to create a “baseline checklist” in a plain text file (for example, runs/2026-03-28_baseline/checklist.md). Use short, testable statements such as “Download dataset X version Y,” “Use model architecture Z,” “Train for N epochs,” “Evaluate on test split,” “Report metric M.” If the paper provides a command or pseudo-code, copy it verbatim into your checklist and mark what must be substituted (paths, GPU/CPU, etc.).

• Inputs: dataset name, source URL, version/date, preprocessing rules, train/val/test split rules
• Model: architecture, initialization, pretrained weights, tokenizer/vocabulary, loss function
• Training: optimizer, learning rate schedule, batch size, epochs/steps, early stopping, augmentation
• Evaluation: which split, which metric(s), averaging method, thresholding rules
• Hardware/software: OS, Python version, library versions, GPU/CPU notes

Then run the baseline exactly as described—no extra regularization, no “better” split, no updated hyperparameters. If a detail is missing, make the smallest reasonable assumption and write it down as an explicit decision (e.g., “Paper does not state random seed; defaulted to 0”). Common mistake: silently “fixing” things (like changing a deprecated API or using a newer dataset mirror) and forgetting to record it. A baseline is not about perfection; it is about establishing a documented reference run you can compare against later.

Most AI experiments contain randomness: shuffled data order, random weight initialization, dropout, data augmentation, and sometimes non-deterministic GPU operations. A seed is a number that initializes the random number generator so that “random” choices become repeatable. If you use the same code, same data, same environment, and the same seed, you often (not always) get the same result.

For a beginner baseline replication, you should do two things: (1) set the seed wherever the framework allows; (2) record the seed value and any determinism settings. In practice this may include Python’s random, NumPy, and your ML framework (PyTorch/TensorFlow/JAX). Some frameworks also need flags for deterministic behavior, and some GPU kernels can still vary slightly run-to-run. That’s okay—your goal is to reduce avoidable randomness and make remaining randomness visible.

Engineering judgment: if the paper reports a mean and standard deviation across multiple runs, you should ideally match that later, but your baseline can start with a single run if time or compute is limited. Just label it clearly as “1-run baseline.” If the paper reports the best of several seeds, note that too—“best-of” results are usually higher than a single typical run.

• Record: seed values, whether deterministic mode was enabled, number of runs
• Watch for: different results when switching CPU vs GPU, or when changing library versions
• Don’t panic: tiny metric differences can happen even with the same seed

Common mistake: setting a seed in one place but not others, then assuming the run is deterministic. Another mistake: re-running until you “get the paper’s number” and calling that replication. That approach hides the variability you are supposed to understand.

During the baseline run, you are collecting evidence. Evidence should answer: “What exactly did I run?” and “What did it output?” The simplest approach is to capture three categories: configuration, environment, and results. You do not need to record everything; you need to record the right things so someone else can reproduce your baseline without guessing.

Configuration includes command-line arguments, config files, and any defaults that matter (learning rate, batch size, epochs, dataset path, model name). If the code uses a config file, save a copy of the resolved config (the final config after defaults are applied). Environment includes OS, Python version, and library versions; also note CPU/GPU type if relevant. If you can, save a small environment snapshot: a pip freeze output or equivalent is usually enough for beginners.

Results includes the final metric(s), plus any training curves or intermediate evaluations that help diagnose issues later. A run log is valuable evidence; save the console output to a text file (redirect stdout/stderr) so you can prove what happened without relying on memory.

• Record (minimum): date/time, git commit or code version, command run, seed(s), dataset version, final metrics
• Record (helpful): training/validation curves, runtime, warnings, and any failed attempts with brief notes
• Ignore (usually): every single batch-level printout, massive debug dumps, or duplicative logs that you will never read

Milestone 3 (evidence screenshots) fits here: take screenshots only when they add trust—e.g., the terminal showing the command and final metric, the dataset version page, or a configuration screen. Screenshot discipline matters: name them predictably (e.g., evidence_terminal_final-metric.png) and capture the smallest area that proves the point.

To compare baseline vs changed runs later, you need outputs in formats that are stable, diff-friendly, and easy to load. Think in layers: (1) raw metrics in a machine-readable file; (2) a human-readable summary table; (3) visualizations; (4) small qualitative samples when applicable.

Metrics: Save a single metrics.json (or metrics.csv) containing the final numbers you will compare: accuracy/F1/loss, plus any key settings (seed, epochs, dataset version). Keep it one row per run for easy later aggregation. Tables: If the paper reports a table, recreate it with the same columns and save it as CSV so you can compare values precisely. Plots: Save training curves as PNG or PDF with consistent axes labels and units; also save the underlying data (CSV) if possible so you can re-plot later without rerunning.

Samples: For tasks like text generation or classification, save a small set of representative predictions (e.g., 20 examples) as a CSV with input, model output, and ground truth. Choose a fixed sample set (same IDs) so comparisons are meaningful. This is especially helpful when metrics are close but behavior differs.

• Compare-ready rule: prefer .json/.csv for numbers, .png for plots, and small, fixed-size sample files for qualitative checks
• Name outputs clearly: include run ID and content (e.g., baseline_metrics.json, baseline_confusion-matrix.png)
• Avoid: screenshots of tables as the only record—screenshots are evidence, not data

This section completes Milestone 2: once outputs are in stable formats, you can compare without reinterpreting pictures or re-reading logs.

Artifacts are everything your run produced that you might need later: logs, metrics files, plots, model checkpoints, configs, and evidence screenshots. Without organization, you will mix files from different runs, overwrite outputs, and lose the ability to explain differences. Your goal is to make each run self-contained.

A beginner-friendly structure is “one folder per run,” with a short run ID that encodes date and purpose:

• runs/2026-03-28_baseline/
  • config/ (resolved config, command used)
  • logs/ (console output, warnings)
  • metrics/ (metrics.json, metrics.csv)
  • plots/ (curves, confusion matrix)
  • samples/ (predictions on fixed examples)
  • evidence/ (screenshots)

Keep checkpoints only if they are needed for later analysis; large model files can bloat your project. If you do keep them, note the file size and why it’s kept (e.g., “needed for error analysis in Chapter 5”). Add a short README.md inside the run folder stating: what this run is, how to reproduce it, and where the key results are stored.

Milestone 5 is basically an organization test: hand the run folder to someone else (or pretend you are that person one week later). If they cannot locate the command, the config, and the final metric in under two minutes, your artifacts are not organized enough. Common mistake: storing everything in the project root or relying on “latest.log” files that get overwritten every run.

After the baseline run, you will write a plain-language summary that captures both the result and your confidence in it. This is Milestone 4, and it should be understandable to a beginner who has not read the paper. Keep it short but specific: what you tried, what you matched, what you could not match, and what you observed.

A useful template is:

• Goal: “Replicate Table 1 accuracy on dataset X using model Y.”
• What I ran: “Used authors’ code commit abc123, dataset version 1.2, seed 0, trained 10 epochs on GPU.”
• What I got: “Test accuracy 0.842 (paper reports 0.850).”
• Differences/assumptions: “Paper did not specify tokenizer version; used default from repository.”
• Confidence: “Medium—metric is close; warnings about non-deterministic ops present.”

Confidence is not a feeling; it’s a judgment based on evidence. You can be confident even when numbers differ if you can explain why differences are plausible (seed variance, minor version differences, dataset mirror changes). Conversely, you should be low-confidence even when numbers match if you relied on undocumented tweaks or reran until it matched.

End the chapter by re-checking Milestone 1–5: you ran the baseline as described, saved outputs in compare-ready formats, captured minimal evidence, wrote a plain summary, and confirmed another person could repeat your steps. With that baseline locked in, you are ready to explore changes intentionally in later chapters—without losing track of what “baseline” really means.

In reproducible AI, a “controlled experiment” simply means you change something on purpose and keep everything else as close to the baseline as possible. You’re trying to answer questions like: “If I change the random seed, do my results wobble a little or a lot?” or “If I shuffle the dataset differently, does accuracy change in a meaningful way?” You do not need advanced statistics to do this well; you need discipline and good notes.

Start by defining your baseline as a snapshot: the exact code version, data files, environment details, and the command you ran. If you can’t recreate the baseline on demand, you can’t interpret any differences later. A good practical baseline includes: a saved configuration file (or documented parameters), the dataset version or checksum, and the resulting metrics plus any plots or example outputs.

• Control: what you keep constant (same dataset, same training steps, same evaluation script).
• Treatment: the one thing you change (seed, split ratio, library version).
• Outcome: what you measure (accuracy, loss curve, runtime, sample outputs).

Common mistake: “silent changes.” For example, you rerun the notebook and it downloads a newer dataset version, or your code uses the current date/time to name output folders and you later compare the wrong runs. Practical outcome: create a consistent run folder structure (e.g., runs/ with timestamps plus a short label), and always save the configuration and metrics in the same place for every run.

Milestone 1 is planning one-change-at-a-time experiments. This sounds obvious, but most replication failures come from changing multiple factors at once. If you change the seed, the data split, and the learning rate, then see a difference, you won’t know which change caused it. Your job is to make attribution easy.

A practical workflow is to write a tiny “experiment plan” before you run anything. Make a table in a text file called experiment_plan.md with rows like: baseline, seed change, data split change, preprocessing change, version change. For each row, include: the exact parameter you will alter, what you expect might happen, and what files you will save.

• Choose a small knob: a single config value or a single line edit.
• Keep runtime manageable: if training takes hours, reduce epochs consistently for all runs and say so.
• Use a fixed evaluation: same metric script, same thresholding, same test set.

Common mistake: accidentally changing two variables through a hidden dependency. Example: changing batch size can also change the number of optimization steps per epoch or memory usage, which can trigger different backend kernels. If you must change something that has side effects, document those side effects explicitly and consider it “more than one change.” Practical outcome: you end up with a sequence of runs you can line up and compare, each with a clear cause.

Milestone 2 is changing randomness settings and comparing results. Many ML pipelines contain randomness: weight initialization, data shuffling, dropout, augmentation, sampling, and sometimes nondeterministic GPU operations. Even with “the same code,” results can drift. Your job is to observe whether that drift is small (expected noise) or large (a warning sign).

First, locate where randomness is controlled. In beginner setups this usually means setting seeds for Python, NumPy, and your ML framework (such as PyTorch or TensorFlow). Also check for a shuffle=True in your data loader and any randomized augmentations. Then run a mini-seed sweep: baseline seed (e.g., 0), plus two or three other seeds (e.g., 1, 2, 3). Keep everything else identical.

• Record: seed value(s), whether determinism flags were used, and whether you ran on CPU/GPU.
• Compare: final metric, but also the training curve shape and qualitative outputs (a few predictions).
• Look for “small drift”: tiny metric differences that don’t change conclusions are normal.

Common mistake: believing a fixed seed guarantees identical results across machines. In practice, different hardware, different library versions, or parallelism can break bit-for-bit determinism. Practical outcome: you can report a “range” of outcomes under different seeds (for example, accuracy varies by about ±0.5 points), which is more honest than a single number.

Milestone 3 is changing data handling slightly and comparing. Data differences are one of the biggest causes of replication gaps, and they’re easy to introduce accidentally. Start with “safe” changes that teach you sensitivity without breaking ethics or invalidating the task: change the train/validation split seed, adjust a preprocessing step, or modify how you handle missing values—always one at a time.

Splits: If the original study specifies a split, match it. Then try a controlled variation: keep the same split ratio but change the split random seed. Some datasets are small enough that the specific split matters a lot. Save the exact indices (or a file listing the IDs) for each split so others can reproduce it.

Preprocessing: Change one preprocessing step in isolation—e.g., normalize inputs vs. not, lowercase text vs. keep case, resize images with a different method. Document the exact transformation and where it happens (before split vs. after split).

• Leakage warning: do not “learn” preprocessing from the full dataset if it should be learned from training only (e.g., scaling parameters, vocabulary, feature selection).
• Label hygiene: ensure labels aren’t accidentally included in features (common in CSV merges).

Common mistake: applying preprocessing using statistics computed on the entire dataset (train + test), then reporting test performance. That makes results look better and undermines reproducibility. Practical outcome: you learn which data handling choices are fragile and can explain differences responsibly.

Milestone 4 is changing environment/tool versions (or documenting constraints). Even if you never touch the code, results can change because the software stack changes. Library updates can alter default behaviors, numerical precision, or random number generators. Hardware differences (CPU vs GPU, different GPUs) can affect speed and sometimes determinism.

Begin by writing down what you have, not what you wish you had. Capture: operating system, Python version, key libraries (framework + NumPy/Pandas), and whether you used CPU or GPU. A lightweight approach is to save pip freeze (or a short curated list of important packages) into a file like environment_baseline.txt. If you’re using a hosted environment (Colab, Kaggle), note that it can change over time; include the date.

• Version bump experiment: upgrade exactly one library (e.g., your ML framework) and rerun the baseline.
• Hardware change experiment: run once on CPU if you used GPU (or vice versa) and compare metrics and runtime.
• Constraints: if you cannot change versions, document that limitation and treat it as part of your replication report.

Common mistake: “mystery upgrades” where a new environment is created and many packages change at once. If you do need a fresh setup, rebuild it stepwise and log each change. Practical outcome: you can explain whether a result difference might plausibly come from a version change, and you can help others rebuild your environment.

Milestone 5 is logging results and labeling changes clearly, then interpreting what you see. Not every difference matters. Your job is to separate expected noise from changes that alter the study’s conclusion. This is where engineering judgment shows up: you decide what comparisons are fair, what variability is normal, and what needs deeper investigation.

Create an experiment log that is easy to scan. A simple format works: one row per run in experiment_log.csv (or a Markdown table). Include: run ID, change label, command/config used, seed, data version, environment note, key metrics, and a link/path to artifacts (plots, model file, predictions). Use consistent naming like R03_seed2, R04_split_seed99, R05_torch2.2.

• Noise indicators: small metric shifts across seeds, similar curves, similar error types.
• Meaningful change indicators: large metric shifts, instability (diverging loss), or different qualitative failure modes.
• Fairness check: ensure evaluation data and metric code are identical across runs.

Common mistake: chasing the “best” run and reporting only that. Replication is about transparency, not leaderboard behavior. Practical outcome: you can write a clear statement like: “Changing only the seed produced minor variation; changing the data split produced larger swings; upgrading the framework changed results slightly and slowed training,” backed by your logged evidence.

Section 5.1: Metrics and outcomes: reading them as a beginner

Start by confirming you are comparing the same outcome. Many replication “failures” are metric misunderstandings. A paper might report test accuracy, while your notebook prints validation accuracy; a leaderboard might use macro F1, while you used micro F1; the reported number might be averaged over 5 seeds, while you ran once.

As a beginner, you can read metrics with a few simple questions: What is the unit (accuracy %, loss, BLEU points)? Which split (train/validation/test)? Which aggregation (single run, mean±std, best checkpoint)? Which decision threshold (0.5 by default, tuned on validation)? If any of these differ, you are not yet comparing like-with-like.

• Match the split: confirm that your evaluation is on the same dataset partition as the study.
• Match the definition: for F1, specify macro/micro/weighted; for AUC, specify ROC vs PR.
• Match the checkpoint rule: “last epoch” vs “best validation” can change results substantially.
• Match preprocessing: tokenization, normalization, resizing, and label mapping are part of the metric pipeline.

Practical outcome: write one short “Metric spec” paragraph in your log (or README) that states exactly what you computed. This becomes the anchor for all later comparisons and prevents you from chasing differences that are only bookkeeping.

Section 5.2: Comparing outputs side-by-side (practical approach)

Milestone 1 is a simple comparison table across runs. Keep it boring and explicit: one row per run, one column per variable or outcome you care about. The table is not just for metrics—it should include the key inputs and settings that could plausibly affect outcomes.

A practical minimum table for beginners includes: run ID, date/time, code version (commit hash or zip name), dataset version (file checksum or download URL + date), environment (Python and library versions), seed, training steps/epochs, and primary metric(s). Add 1–2 “sanity check” outputs such as number of training examples, label distribution, or a small sample prediction.

• Run IDs: use a consistent naming convention (e.g., run_003_seed42_augOff).
• Keep raw outputs: save evaluation JSON/CSV, not just screenshots.
• Include failure runs: crashes and partial runs often reveal environment issues.

Milestone 2 is the “what changed/what didn’t” list. After you fill the table, write two bullets lists: (A) fields that differ across runs (e.g., seed, GPU type, tokenizer version), and (B) fields that stayed constant (e.g., dataset split file, metric function). This list is your guardrail against over-explaining: you can’t attribute a change to something that did not change.

Common mistake: comparing only the final metric. If your accuracy dropped, but your number of training examples also changed because of a filtering step, the “result difference” is downstream of a data difference. Your side-by-side view should expose that immediately.

Section 5.3: Effect size in plain language (how big is the change?)

Once you’ve confirmed metrics match and you have a comparison table, you need to answer: is the change small noise or a meaningful shift? “Effect size” here does not require advanced statistics. For beginner replications, you can use clear, plain-language comparisons that still communicate magnitude responsibly.

Start with absolute difference and relative difference. Example: accuracy went from 0.84 to 0.82 (absolute −0.02, relative −2.4%). For loss, lower is better, so state direction explicitly. Then add one stability check: run multiple seeds (even 3 is helpful) or rerun evaluation if training is expensive but evaluation is cheap.

• Small change: within the typical run-to-run wiggle you observe across seeds (e.g., ±0.5% accuracy).
• Medium change: larger than seed variation but still plausible from minor preprocessing or library updates.
• Large change: clearly outside observed variability; likely a pipeline mismatch, data shift, or major environment change.

If the original study reports mean±std over multiple runs, compare your score to that range. If it reports only a single number, be careful: you don’t know its variability. In that case, your best move is to report your own variability (“Across 3 seeds, mean=…, std=…”) and interpret differences with humility.

Practical outcome: in your report, include one sentence that quantifies size and one sentence that interprets it. Example: “Our test F1 is 0.71 vs 0.74 reported (−0.03). Across 3 seeds our std is 0.01, so this gap is larger than our observed randomness and suggests a methodological difference.”

Section 5.4: Root-cause thinking for result differences

Milestone 3 is classifying causes: data, randomness, environment, and choices. This is engineering judgment: you won’t always “prove” the cause, but you can build a strong case by testing one change at a time and keeping careful notes.

Data causes include different dataset versions, different split files, missing examples, preprocessing differences (tokenization, normalization, resizing), label mapping, and leakage (train/test contamination). Data issues often show up as changed dataset counts, different class balance, or surprising examples when you spot-check.

Randomness causes include different seeds, nondeterministic GPU operations, data loader shuffling, and dropout. A telltale sign is that results move around with seeds but stay within a narrow band.

Environment causes include library version differences, different CUDA/cuDNN behavior, CPU vs GPU execution, and OS-level differences. These can change performance or even numerics. Record versions in every run so you can connect a score change to a version change.

Choices causes are your decisions: hyperparameters, early stopping criteria, batch size, learning rate schedule, augmentation settings, and evaluation thresholds. These are not “mistakes,” but they must be labeled as deviations from the original protocol.

• Make one controlled change: if you suspect preprocessing, rerun with only that preprocessing aligned.
• Use ablation logic: revert changes until the result returns, then reapply to confirm.
• Prefer simplest explanations first: metric mismatch, split mismatch, seed differences, version drift.

Common mistake: changing five things to “get closer” to the paper and then not knowing what mattered. Your comparison table plus controlled edits keeps the story traceable.

Section 5.5: When you can’t replicate: documenting blockers

Sometimes replication fails for reasons that have nothing to do with your skill. Data may be unavailable, code may not run, a dependency may be deprecated, or the study may omit key details. Milestone 4 is writing limitations and confidence statements that are honest and useful.

Document blockers as specific, testable statements. “Couldn’t reproduce” is vague; “Dataset download link returns 404 as of 2026-03-28; attempted mirrors X and Y; no checksum provided in paper” is actionable. For missing hyperparameters, list what you tried and why: “Paper did not specify max sequence length; tested {128, 256} based on GPU memory; results varied by 0.02 F1.”

• Blocker types: unavailable data, incomplete method description, unreleased code, licensing/ethics constraints, compute limits.
• Evidence to include: error logs, screenshots of missing links, environment files, minimal failing example.
• Confidence language: “We are confident the metric implementation matches; we are uncertain about preprocessing because …”

Your limitations section should separate what you verified from what you inferred. This protects you from overclaiming and makes your replication valuable even when it’s incomplete: future readers can pick up exactly where you left off.

Section 5.6: Creating a reproducibility “scorecard” for your project

Milestone 5 is a final reproducibility checklist for the study—your “scorecard.” This turns your work into something others can quickly trust and reuse. Keep it short enough to fit in a README, but concrete enough that someone can rerun your pipeline without a meeting.

A practical scorecard has two parts: (1) replication status (did you match the headline metric within an acceptable range?), and (2) reproducibility readiness (can a new person rerun your exact results?). Include both, because you can sometimes match results without being reproducible (e.g., manual steps), and you can be reproducible even if results differ (e.g., clear documented deviations).

• Runs logged: run table includes code version, data version, env versions, seeds, metrics.
• Outputs saved: raw predictions/metrics files stored with run IDs.
• Determinism: seeds set; nondeterminism noted; multi-seed variability reported.
• Data traceability: source URLs, checksums (or counts), and split method documented.
• Protocol match: deviations from the original study listed (choices) and justified.
• Limitations: blockers and uncertainties stated with evidence.

End your chapter deliverable with a brief “findings” paragraph that ties everything together: what stayed the same, what changed, how big the change was, and your best-supported cause category. When you can do that clearly, you have moved from “I tried it” to “I produced a reproducible research artifact.”

Section 6.1: Replication report template (beginner-friendly)

Milestone 1 is to draft a 1–3 page structure that you can fill in quickly. A replication report is different from a typical research paper: you are not introducing a new method, you are validating (or stress-testing) a published claim. Your structure should make it easy to see (1) what the original study did, (2) what you did, and (3) how close the outcomes are.

Use a template like this (keep headings, keep it short):

• Title + citation: “Replication of [Paper Title], [Year]” + full citation.
• What you tried to replicate: one paragraph describing the dataset/task/metric and the specific result/figure/table you targeted.
• Replication scope: what you did not replicate (e.g., “no hyperparameter sweep,” “smaller subset due to compute limits”).
• Environment summary: OS, Python version, key libraries, hardware basics (CPU/GPU), and where these details are recorded (appendix or files).
• Method: your pipeline steps, aligned with the original paper’s steps; note any necessary substitutions.
• Results: side-by-side comparison (original vs yours) with the metric definition.
• Discussion: reasons for matches/differences (random seeds, data versions, preprocessing, hardware, library changes).
• How to rerun: one command sequence; point to your packaged instructions.

Common beginner mistake: mixing “what I changed” into every paragraph. Instead, write the method as if it is the official procedure, then call out deviations explicitly in a short “Deviations from the original” bullet list. This reduces confusion and makes your engineering judgment visible.

Section 6.2: How to cite sources, data, and tools correctly

Replication work stands or falls on traceability. Milestone 2 (appendix with logs and artifacts) becomes more valuable when you cite every dependency precisely: the paper, the dataset, the codebase, and the tools. Proper citation is not just academic style—it is the reproducibility “address” that lets someone fetch the same inputs you used.

At minimum, cite:

• Original paper: include DOI/URL and the exact version you read (conference version vs arXiv preprint can differ).
• Datasets: name, official homepage, license, and version/date. If the dataset is dynamic, record the snapshot date or checksum.
• Code: repository URL plus commit hash/tag. If you used your fork, cite both upstream and your fork.
• Tools and libraries: key packages and versions (PyTorch/TensorFlow, NumPy, scikit-learn). Cite major tools if requested by their maintainers (some provide “Cite this software” text).

Include a short “Provenance” subsection listing where each artifact came from and how you verified it (e.g., checksum, official release). If you used a pre-trained model, treat it like data: cite the model card, version, and any constraints. Avoid vague statements like “used CIFAR-10” without specifying which source and split procedure.

Common mistake: citing only URLs. URLs rot. Add a date accessed and, when possible, a permanent identifier (DOI, Zenodo record, commit hash). This small habit prevents future confusion when an online file changes silently.

Section 6.3: Presenting comparisons: tables, bullets, and visuals

Milestone 3 is writing results and discussion so the reader can evaluate “same or different” quickly. Your job is not to overwhelm with plots; it is to present the core comparison clearly, then back it up with evidence.

Start with a compact comparison table. Include: metric name, original reported value, your value (mean ± std if you ran multiple seeds), difference (absolute and/or relative), and notes (e.g., “different tokenizer,” “dataset version mismatch”). Tables force precision and reduce the temptation to over-explain.

• Use bullets for deviations: one list labeled “Differences from original implementation,” each item starting with a noun (“Preprocessing: …”, “Training schedule: …”).
• Use visuals only when they add meaning: a learning curve plot can reveal underfitting; a confusion matrix can show class-specific drift. Don’t add charts that merely restate the table.
• Report uncertainty: if randomness matters, run at least 3 seeds when feasible and report variability. If you cannot, say so and explain why.

In the discussion, connect changes to likely causes. Good discussion uses testable language: “Accuracy is 1.8 points lower; likely due to data preprocessing mismatch (normalization differs). Future check: rerun with the paper’s mean/std.” Avoid claims like “the paper is wrong” unless you have ruled out common causes (seed control, data version, metric definition, evaluation code).

Common mistake: comparing against the wrong metric or split. Always restate: which dataset split, which thresholding, which averaging method (macro vs micro), and which evaluation script. If you used a different evaluation function, put the reasoning and code location in the appendix.

Section 6.4: Ethics basics: privacy, licenses, and responsible claims

Replication is not only technical; it is also about responsible sharing. Before you publish anything, verify you have the right to redistribute code, data, and outputs. This is where beginners often make avoidable mistakes—especially when datasets include people, text scraped from the web, or images with unclear rights.

Three practical checks:

• Privacy: do not share raw personal data, identifiers, or model outputs that could reveal private information. If you used sensitive data, share only aggregated metrics and a description of how access is controlled.
• Licenses: read dataset and code licenses. Some allow research use but forbid redistribution. If redistribution is restricted, provide scripts to download from the official source instead of bundling the files.
• Claims: phrase conclusions carefully. A replication with small compute is evidence, not a verdict. Say “In our setup…” and list constraints. Avoid overstating generality.

Also check whether the original paper has known issues or retractions; note them neutrally. If you discovered a potential bug, describe it precisely and respectfully, and provide a minimal reproduction. Responsible reporting improves the community; accusatory language shuts down collaboration.

Common mistake: uploading a full “results/” folder that includes copyrighted images or private text. A safer pattern is to share derived artifacts (plots, summary tables) and the code to regenerate them, plus a clear README explaining what is intentionally excluded.

Section 6.5: Packaging your work so others can rerun it

Milestone 4 is to create a shareable package: files plus instructions that allow a rerun with minimal guessing. Think of packaging as a product: if a stranger cannot run it in 15–30 minutes (excluding training time), your package is incomplete.

A beginner-friendly folder layout might look like this:

• README.md: what this is, what it replicates, prerequisites, and the exact commands to run.
• environment.txt or requirements.txt: pinned versions where possible.
• src/: code.
• configs/: JSON/YAML configs used for runs; name them with dates or experiment IDs.
• logs/: experiment log (CSV/Markdown) and console outputs.
• results/: final metrics tables and plots (avoid raw restricted data).
• appendix/: full method notes, command history, and references to artifacts.

Include a “Quickstart” section with copy-paste commands: create environment, download data (from official source), run training/evaluation, and reproduce the main table/figure. If the project requires a GPU, state it and provide a CPU fallback if possible (even if slower or lower accuracy). Record random seeds in config files and ensure your code actually uses them (set seeds for Python, NumPy, and your ML framework).

Common mistake: packaging only code, not configs. If someone cannot see your hyperparameters, preprocessing, and evaluation settings, they cannot replicate your replication. Treat configs as first-class artifacts.

Section 6.6: Final audit: completeness, clarity, and next steps

Milestone 5 is a final self-audit before you publish or submit. Your goal is to catch “silent gaps” that make reruns fail or make your conclusions ambiguous. Use a checklist and be strict—this is how you earn trust.

Completeness checks:

• One-click path: can you reproduce the main result from a clean folder using only the README?
• Versions recorded: OS, Python, libraries, and any external tools (CUDA version if relevant).
• Data provenance: dataset source, version/snapshot date, and any filtering steps.
• Experiment log: each run has an ID, config reference, seed, and output location.
• Appendix present: method details, deviations, and links to artifacts.

Clarity checks:

• Metric definition: explicitly state how the metric is computed and on which split.
• Comparison is readable: a single table or figure that answers “how close are we?”
• Discussion is constrained: you separate observed facts from hypotheses and note limitations.

Next steps: if you plan to share publicly, choose an appropriate venue (class submission, GitHub, OSF, or a simple zip). If you found a likely issue in the original work, consider contacting the authors with a short, courteous summary and a minimal reproduction package. Replication is a skill; every careful report you ship makes your future projects faster, safer, and more credible.