Hands-On AI Ethics: Bias Tests, Simple Policy, Clear Reports

In everyday terms, “AI” is software that produces an output (a label, score, ranking, or generated text/image) by learning patterns from examples or by following complex rules. You do not need to start with neural networks. If a system takes inputs (data about a person, event, or document) and returns a decision-like output that people rely on, it deserves ethical scrutiny.

From an engineering perspective, most AI products are pipelines: (1) collect data, (2) convert it into features (structured fields, embeddings, etc.), (3) run a model or heuristic, (4) apply business rules, (5) present results to a human or another system. Ethical risks can enter at every step. A model can be “accurate” and still be harmful if the data reflects historical inequities or if the output is used in a high-stakes way without safeguards.

Milestone: Map one real-life AI decision and its possible impacts. Choose one AI-assisted decision you can describe in one sentence: “The system recommends which applicants move to interview.” Then write the intended benefit in plain language: “Reduce recruiter workload and improve consistency.” This simple sentence becomes your anchor; when you later evaluate fairness or safety, you will test whether the system actually supports that benefit without creating unacceptable tradeoffs.

• Practical rule: If the output changes what happens to someone (money, access, time, reputation, safety), treat it as an AI decision worth reviewing.
• Common mistake: Describing AI as “the model.” Ethics issues often come from the surrounding workflow (data entry, thresholds, appeal process), not just the algorithm.

Set a clear project goal for the rest of the course: “By the end, I will run a simple bias check on our interview-screening score and write a one-page use policy plus a findings log.” A specific goal keeps ethics work actionable instead of turning into endless debates.

Most AI systems do one of three things: predict, recommend, or decide. A prediction estimates something uncertain (“chance of late payment”). A recommendation suggests an action (“show this video next” or “route to Tier 2 support”). A decision directly triggers an outcome (“deny,” “approve,” “suspend”). In practice, products blend these: a prediction becomes a decision when you attach a threshold, and a recommendation becomes a decision when humans must follow it.

Engineering judgment starts with identifying the “decision point.” Ask: Where does the output become an action? Who can override it? How often do they override it? If overrides are rare because the UI hides context or the team is overloaded, you effectively have automation, even if you claim “human in the loop.” Ethics work should reflect the reality of use, not the intended design.

To support your first milestone map, draw a simple flow on paper: input → model score → threshold/rules → action → user impact. Add two boxes: “appeal” and “monitoring.” This is where many harms hide. If people cannot challenge an outcome, small errors become persistent injustice.

• Common mistake: Treating scores as neutral. A score is a policy encoded as a number: it reflects what you chose to optimize and what you ignored.
• Practical outcome: You should be able to name one measurable success metric (e.g., reduced response time) and one “do no harm” constraint (e.g., no large approval-rate gaps by group).

This course will later have you run a no-code bias check with a tiny dataset. For now, note what “positive outcome” means in your system (approval, selection, access) and what a “negative outcome” means (denial, demotion, removal). Clear definitions are essential for fairness measures like rates and gaps.

In a product context, “ethics” is not a philosophical essay; it is a set of practical constraints and responsibilities around how your system affects people. Ethical work asks: Are we treating people fairly? Are we respecting their privacy and autonomy? Are we exposing them to avoidable risk? Are we communicating honestly about what the system can and cannot do?

Think of ethics as product quality in high-stakes areas. You already manage bugs, uptime, and security. Ethics adds categories of failure that are easy to miss if you only measure aggregate accuracy or revenue. A system can “work” for the average user while systematically failing for a subgroup, or it can create incentives that push users toward harmful behavior.

Milestone: Create your first ethics checklist for a single use case. Start small and concrete. Build a checklist that you can actually run in a meeting, with yes/no questions tied to evidence. For example: “Do we know what data sources feed the model?” “Is there a documented appeal path?” “Have we checked outcome rates by at least one relevant group?” “Do we log when the model is overridden?”

• Common mistake: Using vague checklist items like “Ensure fairness.” Replace with testable statements: “Approval-rate gap between Group A and Group B is under X percentage points, or we can explain why not and what mitigations exist.”
• Engineering judgment: Pick thresholds appropriate to harm. For low-stakes personalization, you may tolerate more error; for hiring or healthcare, you need tighter controls and better documentation.

Finally, ethics requires writing things down. You will later create an incident-style finding log. Start the habit now: whenever you identify a potential harm, capture what happened (scenario), who is affected, and why it matters. This is how ethics becomes operational rather than aspirational.

Ethical risks show up in recognizable patterns. Four of the most common are unfairness, privacy harms, safety harms, and deception or manipulation. You do not need perfect definitions to begin; you need the ability to spot how a system could fail in the real world and what evidence would confirm the risk.

Unfairness often comes from data and labels. If historical decisions were biased, the model may learn those patterns. If labels are noisy or subjective (e.g., “good employee,” “high risk”), different groups may be labeled differently for the same behavior. Even when inputs exclude protected attributes, proxies (zip code, school, device type) can recreate group differences.

Privacy harms include collecting more data than necessary, using data for unexpected purposes, leaking sensitive information through outputs, or enabling re-identification. A practical test is purpose limitation: could the product still work if we removed or minimized a particular data field?

Safety includes physical and psychological safety: unsafe recommendations, missed fraud signals, harmful medical or legal guidance, or escalation failures that put users at risk. Safety also includes over-reliance: users trusting outputs that should be treated as uncertain.

Deception includes misleading claims, unclear automation, or interfaces that hide uncertainty. Users deserve to know when AI is involved and what its limitations are, especially when the output influences important decisions.

• Common mistake: Only checking “bias” and ignoring privacy or deception. Many incidents are multi-factor: biased data plus opaque UI plus no appeal path.
• Practical outcome: Add at least one risk from each category to your map, even if you later discard it. This broad scan prevents blind spots.

As you continue, you will translate these harms into measurements (rates and gaps), documentation (policy and findings log), and mitigations (threshold changes, review queues, data fixes, clearer user messaging).

Ethical analysis fails when it only considers the “user.” Many AI decisions affect people who never touch your interface: applicants evaluated by a hiring model, bystanders recorded by a camera system, creators whose content is ranked down, family members impacted by a credit decision, or neighborhoods affected by resource allocation.

Milestone: Identify stakeholders and who could be harmed. Start by listing three stakeholder categories: (1) direct users (operators, customers), (2) subjects of the decision (people being scored or classified), and (3) indirect stakeholders (family, coworkers, communities, regulators, support staff). For each, write one plausible harm and one plausible benefit. This keeps the discussion balanced and specific.

Next, identify “affected groups” relevant to your context. In some domains you can use legally protected classes; in others, you may need operational groupings like language, region, disability access needs, device type, or new vs returning users. The goal is not to stereotype; it is to ensure you do not hide failures inside averages.

• Common mistake: Assuming a group is “not in our data” so it is “not affected.” People can be affected even if you do not track group membership (e.g., via proxies or downstream impacts).
• Engineering judgment: If group labels are sensitive, consider privacy-preserving evaluation approaches, sampling audits, or using consented research panels rather than collecting more personal data by default.

This stakeholder list feeds directly into your incident-style finding log later. A good finding log names the impacted stakeholders, describes the mechanism (how the system caused the harm), and notes what evidence you have versus what you still need to test.

To move from “concerns” to action, classify risks using a simple two-by-two grid: impact (how severe the harm would be) versus likelihood (how probable it is, given your current design and controls). This is your Milestone: Classify risks by severity and likelihood (simple grid). You do not need perfect numbers; you need a shared, documented judgment that guides what to do next.

Define impact levels in plain language. Example: Low (minor inconvenience), Medium (lost opportunity, moderate financial or emotional harm), High (major financial loss, illegal discrimination, physical safety risk). Define likelihood similarly: Unlikely (rare edge case), Possible (could occur in normal use), Likely (expected to occur without mitigation). Now place each risk you identified—unfairness, privacy, safety, deception—into the grid for your chosen decision.

• High impact + likely: Prioritize immediately. Add safeguards (human review, stricter thresholds, clearer notices), and plan evaluation.
• High impact + unlikely: Add monitoring and a response plan. Rare but catastrophic failures still require preparation.
• Low impact + likely: Fix if cheap; otherwise document and revisit.
• Low impact + unlikely: Track as background risk.

Milestone: Set a clear project goal for the rest of the course. Use your grid to pick one “top risk” you will measure and document. Example: “Our main risk is unfair denial of interviews for Group B; we will run a simple outcome-rate gap check, write a one-page policy for appropriate use, and maintain a findings log for any incidents.”

Common mistake: Treating the grid as a one-time exercise. The correct practice is iterative: update likelihood after mitigations, update impact after learning more about downstream use, and record changes in your checklist and findings log. This is how ethics becomes a routine part of building and operating AI systems.

In everyday conversation, “bias” often means “unfairness” or “prejudice.” In AI work, you need a more precise definition: bias is a systematic difference in outcomes, errors, or treatment across groups that is not justified by the stated goal and constraints of the system. The “system” includes the model, the data, the labeling process, and the decision workflow around it.

What bias is not: it is not simply “any difference between groups.” Some differences can be expected because of real-world base rates, job requirements, or measurement limits. Bias is also not automatically fixed by removing sensitive attributes like race or gender; models can learn proxies (Section 2.5). And bias is not only a machine learning problem—manual rules and human review processes can create or amplify the same harms.

Milestone: write a plain-language problem statement and success definition. For the hiring screener, a good starting problem statement is: “Help recruiters review applications by surfacing candidates likely to meet the job requirements, while minimizing unfair exclusion of qualified candidates.” Then define success in plain language and in a measurable way: time saved, interview yield, and constraints like “the tool must not reduce interview rates for any protected group compared with the current process without a documented, job-related reason.”

Common mistake: jumping to a fairness metric before the goal is clear. If you do not specify whether the model is advising humans, auto-rejecting candidates, or merely sorting queues, you cannot interpret fairness numbers responsibly. Your ethics work begins with engineering judgment: stating what the tool will do, what it must not do, and where humans remain accountable.

Historical bias happens when yesterday’s decisions reflect unequal opportunities, discrimination, or uneven access—and you treat those decisions as “ground truth.” In hiring, past interview and offer decisions may reflect networking access, school prestige preferences, biased performance reviews, or inconsistent standards across managers. If you train on that history, your model can reproduce it efficiently.

Feedback loops occur when model outputs change the world, and the new data you collect is shaped by those outputs. Example: if the screening tool ranks candidates from certain zip codes lower, fewer of them get interviews, fewer get hired, and your future “successful employee” dataset contains fewer examples from those communities. The model then “learns” that those communities are less successful—because it helped make it true.

Milestone: spot 5 bias sources in a sample scenario. In the hiring screener, you can often identify at least five: (1) historical hiring decisions used as labels, (2) performance reviews influenced by manager bias, (3) applicant pool shaped by prior company reputation, (4) referral-heavy hiring creating homogeneous pipelines, (5) downstream selection effects (who accepts offers) reflecting unequal bargaining power or location constraints.

Practical workflow: when you see a loop, treat it like a safety risk. Write it down as “mechanism + impact.” For example: “Lower ranking reduces interviews for Group X → fewer hires from Group X → future training data underrepresents Group X.” In your incident-style finding log later in the course, this becomes a clear narrative: what happened, who is affected, and why it matters.

Sampling bias is about who is in your dataset—and who is not. Even if your labels were perfect, a model trained on an unrepresentative sample will perform unevenly. In hiring, a dataset might contain only applicants who made it past an early filter (e.g., only people who submitted a portfolio, or only people who applied via a specific platform). That means the model never sees qualified candidates who were filtered out earlier.

A common “missing group” failure: the dataset has too few examples for a subgroup to learn meaningful patterns (for example, applicants with non-traditional career paths, career breaks, or international credentials). Another common failure: the dataset reflects the company’s current geography and role mix, but the tool is deployed globally or used for different roles. The model may be accurate on average while being unreliable for smaller groups.

Milestone: draft a “data questions” checklist for your use case. Start with practical questions you can answer with metadata and basic counts: What time period does the data cover? Which roles, locations, and seniority levels? How many rows per group? Who is missing due to prior filters? Are there duplicates (repeat applicants)? Are there changes in policy over time (new recruiter team, new job requirements) that make older data misleading?

Engineering judgment: decide whether to pause. If a protected group has extremely low representation, fairness testing might produce unstable metrics (tiny denominators). The right action may be to collect more data, widen the sampling frame, or narrow the deployment scope to match the training population. “We can’t measure it well” is itself an ethics finding that should appear in your report.

Labels turn messy reality into a training signal. Labeling bias happens when those labels encode subjective judgments, inconsistent standards, or unequal treatment. In hiring, labels like “good candidate,” “culture fit,” or even “successful employee” can be heavily subjective and shaped by structural factors (mentorship access, project assignments, evaluation style, and manager expectations).

Even seemingly objective labels can be biased. “Stayed at the company for 12 months” might reflect who felt included, who had caregiving constraints, or who received fair pay—not just job performance. “Sales quota achieved” may depend on territory assignment quality. If you train a model to predict these labels without understanding their drivers, you can formalize unfairness.

Practical techniques: (1) document label definition and who applied it, (2) check inter-rater consistency if multiple reviewers labeled data, (3) separate “can do the job” signals from “was rewarded in our environment” signals, and (4) create an escalation path for label disputes. For small projects, you can do a no-code audit by sampling 20 labeled examples per group and asking: do we see different reasons being used to justify the same label?

Common mistake: treating labels as neutral because they came from a system of record. A database field is not automatically objective; it is a record of past decisions. Your later fairness measures (rates and gaps) will only be meaningful if the labels represent what you actually care about. If not, your model can optimize the wrong thing very well.

Proxy variables are features that are not explicitly sensitive but correlate with sensitive attributes or reflect historical disadvantage. Common proxies include zip code (race and income correlations in many regions), school attended (socioeconomic status and race), employment gaps (caregiving and disability), and even time-of-day activity patterns (shift work and income).

Milestone: list possible sensitive attributes and proxies—without over-collecting. Start with a minimal set of protected or sensitive attributes relevant to your jurisdiction and context (often: gender, race/ethnicity, age, disability status). Then list plausible proxies already present in your data: zip code, commute distance, school, graduation year, name (as a proxy for ethnicity), and referral source. The goal is not to collect more sensitive data “just in case.” The goal is to know where risk can enter and what you might need to measure with appropriate governance.

Practical workflow: create a “feature risk table” with three columns: feature name, why it might be a proxy, and what you will do about it (keep, transform, restrict use, or monitor). For example: keep zip code only at a coarse level (e.g., region rather than full postal code), or exclude it if it is not job-related. If you must keep a feature for legitimate reasons, plan a fairness check specifically around it (e.g., compare selection rates by region and by protected groups).

Common mistake: removing sensitive attributes and assuming the model is now fair. Proxies can reintroduce the same patterns. Fairness work is not only “feature hygiene”—it is measurement plus controls: you measure impacts by group, and you set policy on what the system may use and how decisions are reviewed.

Not every disparity is evidence of bias, and not every fairness goal can be satisfied simultaneously. Real systems face constraints: limited data, small subgroup sizes, changing job requirements, and the need to manage false positives and false negatives differently depending on harm. In hiring, a false negative (rejecting a qualified candidate) harms applicants; a false positive (interviewing an unqualified candidate) mostly costs time. That asymmetry affects what you prioritize.

Some trade-offs are mathematical: you often cannot make all error rates equal across groups when base rates differ, especially with a single threshold. That is why you must choose what you will measure. Milestone: choose what you will measure for fairness in Chapter 3. For a beginner-friendly plan, pick two or three measures you can explain to non-experts: selection rate (who gets advanced), true positive rate (qualified candidates advanced), and false negative rate (qualified candidates rejected). Then define gaps: the difference or ratio between groups.

Engineering judgment: write down acceptable ranges and escalation triggers. Example: “If any group’s selection rate is below 80% of the highest group’s rate, we pause and investigate.” This is not a legal standard by default; it is an internal safety tripwire that forces review.

Also consider constraints that are legitimate and job-related: certification requirements, language proficiency for a role that truly requires it, or availability for specific shifts. The ethics question becomes: are these requirements necessary, consistently applied, and measured accurately? When constraints are real, transparency matters. You should be able to explain, in plain language, why the model behaves the way it does and how a person can contest or override the outcome.

Start by defining the test in one sentence: “For decisions made between start date and end date, compare how the system treated Group A vs Group B, using outcome as the ground truth.” This sentence forces three choices that determine whether your results are meaningful: the outcome definition, the group definition, and the time window.

Outcome (ground truth): Pick an outcome you can actually observe. For hiring, you might use “passed probation” rather than “manager liked candidate.” For lending, use “repaid within 90 days” rather than “credit score improved.” In a spreadsheet, you’ll typically encode outcome as 1/0 (Success/Failure). If outcomes are missing for many rows (e.g., loans not yet matured), document this and either filter to mature cases or clearly label the analysis as preliminary.

Groups: Choose a protected or relevant group attribute that is permitted for auditing (e.g., gender, age band, disability status), or a policy-relevant segment (e.g., region). Avoid grouping on something that is essentially the decision itself (e.g., “VIP customers”), because that bakes in the system’s logic and can hide disparity.

Time window: Use a stable period where the policy/model was consistent. If the decision rule changed mid-month, split the analysis. Seasonality matters: comparing December to January can reflect business cycles, not bias. A common practical choice is a 4–12 week window with enough volume per group to make rates less noisy.

Milestone: Load and clean basics. In your spreadsheet, ensure each row is one decision. Standardize values (e.g., “approved”, “Approved”, “APPROVE” → “Approved”). Remove duplicates, fix obvious typos, and verify the columns you will use: group, decision, and outcome. Do not “clean” by deleting inconvenient rows; instead, create a filtered view with documented criteria (e.g., remove rows with missing outcome). This protects integrity and makes your test reproducible.

Fairness metrics are built from simple counts. Before you compute anything, create a small summary table for each group: total cases, number approved, number denied, number with successful outcomes, and number with unsuccessful outcomes. These totals are your “sanity checks.” If they look off, stop and inspect the raw rows.

Confusion table (per group): If you have both a decision and an outcome, you can build a 2×2 table. Define “Positive decision” as Approved (or Selected, Allowed) and “Positive outcome” as Success (repaid, performed well, not actually fraudulent). Then count:

• TP (true positive): Approved and Success
• FP (false positive): Approved and Failure
• TN (true negative): Denied and Failure
• FN (false negative): Denied and Success

Milestone: Build a simple confusion table for two groups. In a spreadsheet, you can do this with pivot tables: rows = Decision, columns = Outcome, filter = Group. Or use COUNTIFS formulas such as COUNTIFS(Group,"A",Decision,"Approved",Outcome,"Success") to compute TP for Group A. Repeat for FP, TN, FN and for Group B.

Example: Suppose Group A has 100 cases with TP=30, FP=10, TN=40, FN=20. Group B has 100 cases with TP=20, FP=5, TN=55, FN=20. From here, you can compute:

• Approval (selection) rate = (TP+FP)/Total. Group A: (30+10)/100=0.40. Group B: (20+5)/100=0.25.
• Error rate (overall) = (FP+FN)/Total. Group A: (10+20)/100=0.30. Group B: (5+20)/100=0.25.
• Success rate (base rate) = (TP+FN)/Total. Group A: (30+20)/100=0.50. Group B: (20+20)/100=0.40.

These rates answer different questions. Approval rate describes how often the system says “yes.” Error rate describes how often the system is wrong given your ground truth. Success rate describes how outcomes are distributed in the population being evaluated. Mixing these up is one of the most common no-code fairness mistakes.

Accuracy and fairness are related but not identical. Accuracy asks, “How often is the decision correct?” Fairness asks, “Are errors or outcomes distributed in a way that creates unjustified harm across groups?” You can have a model with high overall accuracy that still concentrates mistakes on one group, especially when groups differ in size or base rates.

In practice, you will evaluate both because stakeholders care about both: operations teams care about performance (e.g., reducing defaults or catching fraud), while governance teams care about harm and compliance (e.g., equal access, non-discrimination, and consistent treatment).

Why accuracy can hide problems: If Group A is 90% of your data and Group B is 10%, a model can be “accurate” by doing well on Group A while performing poorly on Group B. That is why you compute rates per group, not just overall. Another issue is asymmetric cost: a false negative (denying a qualified applicant) might be more harmful than a false positive (approving an unqualified one), or vice versa, depending on context.

Milestone: Compute key rates. Along with approval rate and overall error rate, compute two directional error rates that often matter more in ethics discussions:

• False Positive Rate (FPR) = FP/(FP+TN): among truly negative cases, how often did we incorrectly approve?
• False Negative Rate (FNR) = FN/(FN+TP): among truly positive cases, how often did we incorrectly deny?

Directional rates help you connect metrics to real-world harm. For example, in lending, a high FNR means qualified applicants are being denied—an access-to-opportunity harm. In fraud detection, a high FPR means legitimate customers are being flagged—an unnecessary burden and potential reputational damage.

Engineering judgment: Decide which mistakes matter most for your use case and lead with those in your report. Don’t flood readers with every metric. Pick two or three that map clearly to the decision’s risk profile and the organization’s stated values.

Once you have per-group rates, you move from “rates” to “gaps.” A gap is simply a difference (or ratio) between Group A and Group B. Gaps are easier to discuss because they directly express disparity.

Selection (approval) rate gap: Compute each group’s selection rate, then compute:

• Difference: SelectionRate(A) − SelectionRate(B)
• Ratio: SelectionRate(B) / SelectionRate(A) (often used when A is the higher-rate group)

Using the earlier example: selection rate A=0.40, B=0.25. Difference = 0.15 (15 percentage points). Ratio = 0.25/0.40=0.625. Differences are intuitive; ratios are useful for some policy thresholds.

Error rate gaps: Compute overall error rate, and/or compute FPR and FNR per group. Then compute differences (A−B or B−A) in percentage points. A simple and readable set for no-code audits is:

• FNR gap (access harm): FNR(B) − FNR(A)
• FPR gap (burden harm): FPR(B) − FPR(A)

Milestone: Calculate group gaps and flag potential issues. In a spreadsheet, keep a clean “Metrics” table with one row per group and columns for TP, FP, TN, FN, Total, SelectionRate, ErrorRate, FPR, FNR. Then add a separate “Gaps” table that subtracts Group B minus Group A. This separation reduces mistakes where someone edits a formula and silently changes results.

What these checks do (and do not) tell you: A selection rate gap may indicate disparate impact, but it could also reflect different base rates or different opportunity structures upstream (e.g., unequal access to resources). An error rate gap suggests the system is less reliable for one group, which can be a fairness issue even if selection rates are similar. Your job in no-code testing is to surface these signals and recommend the next step, not to claim causality from metrics alone.

Fairness metrics rarely come with universal “pass/fail” lines. Interpretation depends on context, risk, and data quality. Still, you need practical thresholds to decide when to escalate. The key is to define “needs review” triggers that are conservative, easy to apply, and consistently documented.

Start with three checks:

• Material selection disparity: selection rate difference ≥ 10 percentage points or selection rate ratio < 0.80. (The “0.80 rule” is a common screening heuristic in HR contexts, not a legal conclusion.)
• Material error disparity: FNR or FPR difference ≥ 5–10 percentage points, especially in high-stakes decisions.
• Small-sample warning: any group with fewer than ~50 cases (or fewer than ~10 positives/negatives for FPR/FNR denominators) should be labeled “unstable estimates.”

Engineering judgment: A 6-point FNR gap might be urgent in medical triage and merely a watch item in low-stakes marketing. Also consider direction: if the higher error rate falls on a historically disadvantaged group, the same numeric gap may warrant faster action because the potential harm compounds existing inequity.

Milestone: Write a short findings note. Your note should include: (1) scope (time window, dataset size), (2) metrics with numbers, (3) plain-language meaning, (4) who may be affected, and (5) recommended next step. Example phrasing: “In Jan–Feb (n=200), Group B’s approval rate was 25% vs Group A’s 40% (−15 pp; ratio 0.63). Group B’s FNR was 50% vs Group A’s 40% (+10 pp), indicating more qualified applicants in Group B were denied. This meets our ‘needs review’ trigger; recommend reviewing features, thresholds, and upstream data quality for Group B.”

The most valuable outcome of no-code fairness testing is not the metric itself—it’s a decision-ready escalation package that connects numbers to potential harm and a concrete follow-up plan.

No-code testing is powerful, but it is easy to get wrong in ways that look “mathematical” while being logically broken. Avoiding these mistakes is part of responsible AI practice.

• Mixing populations: Don’t compare groups across different products, channels, or time periods. Fix by filtering to one consistent workflow and one stable time window.
• Using the wrong ground truth: If outcomes are delayed or missing, your confusion table may be biased toward older cases or easier-to-observe outcomes. Fix by restricting to matured cases and stating the limitation clearly.
• Denominator errors: FPR and FNR have different denominators. A frequent mistake is dividing by Total instead of (FP+TN) or (FN+TP). Fix by labeling denominators in your spreadsheet (e.g., “Actual Negative” and “Actual Positive”).
• Over-cleaning: Removing “messy” rows can remove evidence of harm (e.g., missing outcomes concentrated in one group). Fix by tracking exclusions with counts per group and reporting what was removed.
• Confusing correlation with discrimination: A gap is a signal, not proof of intent or illegal bias. Fix by recommending next-step analysis (feature review, policy review, threshold tuning, process changes) instead of making causal claims.
• Ignoring practical significance: Tiny gaps in huge datasets can be statistically “real” but operationally minor, while large gaps in tiny samples can be noise. Fix by combining thresholds with sample-size warnings and domain context.

Workflow guardrails: Keep a versioned spreadsheet, freeze your raw data tab, and put all formulas in a dedicated “Metrics” tab. Add a one-line “data dictionary” defining each column and allowed values. These small practices prevent accidental edits and make your work auditable.

By the end of this chapter, you have a repeatable, spreadsheet-based fairness test: clean dataset → confusion table by group → key rates → gaps → a findings note that a manager, auditor, or policy writer can act on. In the next chapter, you’ll build on this by translating findings into clear do’s and don’ts in an AI use policy and a lightweight incident-style log.

Section 4.1: Root-cause thinking for beginners

When your bias check shows a gap (for example, different approval rates across groups), avoid jumping directly to solutions. First diagnose the likely root cause. A beginner-friendly way is to categorize the cause into one or more buckets: data, labels, rules, or design. This aligns with an incident-style finding log: what happened, who is affected, and why it matters.

Data causes include missing coverage (one group underrepresented), historical imbalance (past decisions reflect unequal access), or measurement issues (fields recorded differently by group). Label causes include target variables that encode past bias (e.g., “prior performance” measured via biased evaluations) or inconsistent labeling standards. Rules causes include thresholds or business logic that disproportionately blocks certain groups (even if the model is neutral). Design causes include user flows that create unequal opportunity to succeed (e.g., extra steps that some users are less likely to complete).

A practical root-cause method is a “five whys” adapted for ML: (1) Where is the gap measured (which metric, which stage)? (2) Which inputs or steps differ by group? (3) Are differences explained by legitimate requirements, or by avoidable process artifacts? (4) What part is under your control? (5) What evidence would confirm the hypothesis?

• Common mistake: Treating the model as the only lever. Many bias issues sit in the pipeline around the model.
• Practical outcome: Write 2–4 root-cause hypotheses and the data you need to validate each. This becomes the plan for the next iteration.

By the end of this step you should be able to say, in plain language: “The gap is likely coming from X and Y, not just ‘the algorithm.’” That clarity is what allows targeted mitigation rather than random tuning.

Section 4.2: Data fixes: coverage, balance, and quality

Data mitigation is often the highest leverage, but it is also the easiest to do poorly. “Add more data” is not a plan. Your plan should specify coverage (do we have enough examples for each group?), balance (are outcomes and contexts comparable?), and quality (are fields accurate, consistent, and timely?). Start by reviewing your earlier fairness measures and break them down by subgroup and scenario (e.g., region, device type, or product tier). Bias can hide in intersections.

Coverage fixes include targeted data collection, partnerships, or sampling strategies that reduce underrepresentation. If you cannot collect new data quickly, you can at least label and audit what you already have: quantify missingness by group, check whether some groups have more “unknown” values, and confirm that your preprocessing doesn’t drop more rows for one group than another.

Balance fixes include reweighting or resampling to prevent the model from learning “majority-only” patterns. In no-code settings, you may not control algorithms directly, but you can often control the training set composition. Be careful: oversampling can increase overfitting; reweighting can affect calibration.

Quality fixes include label cleanup and feature hygiene. If the label is a proxy for historical decisions, consider alternative labels (e.g., later success rather than initial approval) or add context that reduces reliance on biased proxies. Track changes: if you change labels or data rules, write it down as part of your finding log so re-tests are meaningful.

• Trade-off to predict: Better group parity may reduce overall accuracy, especially short term. Your job is to estimate whether the accuracy loss is acceptable given the risk and stakes.
• Common mistake: Fixing imbalance without checking whether the label itself is biased, which can amplify harm.

A good data mitigation result is measurable: after changes, you re-run the same simple bias check and show whether gaps narrowed, and which new risks appeared (like increased false positives in one group). This sets you up to decide whether to ship, pause, or adjust the use case.

Section 4.3: Decision rule fixes: thresholds and guardrails

Many “AI decisions” are actually a model score plus a decision rule. If you found bias, you can often reduce harm by changing the rule even before retraining a model. The simplest lever is the threshold: the score above which you approve, flag, or route a case. A single global threshold may create uneven error rates across groups. Adjusting thresholds can change acceptance rates and false positive/negative trade-offs.

Start by clarifying what type of error is most harmful in your context. In a fraud screen, false positives can block legitimate users; in a safety setting, false negatives may be worse. Use your existing rate metrics to compare error rates by group. Then test “what-if” thresholds: if the threshold moves, how do group gaps shift? In no-code tools, you can often simulate this by sorting by score and recalculating outcomes.

Beyond thresholds, add guardrails: rules that constrain the model’s influence. Examples include: (1) never auto-deny; only auto-approve low-risk cases and send the rest to review, (2) require additional evidence before a negative action, (3) cap the number of adverse decisions per day until monitoring stabilizes, or (4) block use in scenarios the model was not trained for (out-of-scope detection).

• Trade-off to predict: Guardrails usually reduce automation and increase operational cost (more reviews), but they can meaningfully reduce harm while data fixes are in progress.
• Common mistake: Changing thresholds to “equalize” a metric without considering downstream impacts (e.g., shifting risk to another team or creating new inequities).

This milestone is about engineering judgment: pick 2–3 mitigations you can actually implement now, state the expected effect on fairness measures, and note what you are sacrificing (speed, accuracy, cost, or user friction). Record the decision rule in your policy and documentation so it is not silently changed later.

Section 4.4: Product fixes: user experience and informed choice

Bias is not only statistical; it is also experiential. Two users can receive the same model score but experience different burdens based on how the product is designed. Product mitigations change the user journey to reduce unequal impact, increase transparency, and provide alternatives when the model is uncertain or when the cost of an error is high.

Start by mapping the decision flow: where does AI influence the user, what options exist, and what happens on failure? Then look for “friction bias”: steps that disproportionately disadvantage certain users, such as requiring high-quality scans, long forms, stable connectivity, or knowledge of specific jargon. Reducing friction can reduce apparent performance gaps without changing the model at all.

Next, build informed choice into the experience. If AI is used to recommend or pre-screen, tell users what is happening in plain language and what they can do if it seems wrong. Provide meaningful recourse: a way to correct data, submit additional context, or request review. Avoid vague messages like “not eligible” without guidance; they increase harm and complaints while offering no path to resolution.

• Practical fixes: show key factors used (at a high level), provide a “check and edit your info” step, offer non-AI alternative routes, and design accessible flows (language, disability, device constraints).
• Common mistake: Treating transparency as a legal disclaimer. Good transparency helps users succeed and helps you detect problems faster.

Product fixes also shape the ship/pause decision. If the use case is high stakes and you cannot close gaps quickly, you may still deploy a limited version with strong user controls and no automated adverse actions. Make that limitation explicit in your one-page AI use policy.

Section 4.5: Human-in-the-loop review: roles and escalation

When bias is detected, a human oversight step is often the fastest safety improvement. But “add a human” only works if you design it: who reviews, when they review, what information they see, and how disagreements are handled. Oversight should be a defined operating procedure, not an informal promise.

Define roles clearly. A frontline reviewer handles routine cases using a checklist. A specialist reviewer (or small panel) handles escalations, edge cases, and suspected bias incidents. A system owner (product/ops lead) is accountable for the overall performance and for triggering pauses or rollbacks. If you have a compliance or ethics function, specify when they are notified.

Define timing: review can be pre-decision (human confirms before action), post-decision audit (human samples and reverses if needed), or exception-based (human intervenes when confidence is low or protected attributes are implicated). Exception-based review is a common compromise when full review is too costly.

• Escalation triggers: user complaint alleging unfairness, repeated errors for a subgroup, unusual spike in denials, or mismatch between model and human judgments.
• Common mistake: Reviewers are given the model output but not the context needed to challenge it, turning oversight into rubber-stamping.

Connect oversight to your incident-style finding log. Every escalated case should capture what happened, who was affected, the suspected cause, and the resolution. Over time, these logs become training data for process improvements and a defensible record if you must justify a ship/pause/change decision.

Section 4.6: Monitoring: drift, complaint signals, and periodic re-tests

Bias mitigation is not a one-time fix because systems change: users change, policies change, and the environment changes. Monitoring is how you detect drift and prevent yesterday’s “fair enough” model from becoming today’s problem. A practical monitoring plan includes signals, thresholds for action, and periodic re-tests using the same simple fairness measures you learned earlier.

Track data drift signals: changes in missingness rates, shifts in key feature distributions, and changes in subgroup proportions. Track performance drift signals: overall error rates and subgroup error rates where labels are available. Track outcome drift signals: approval rates and gaps by group, plus any high-stakes adverse action counts.

Also track complaint signals, which are often the earliest indicator of harm: volume of appeals, time-to-resolution, reversal rates after review, and qualitative tags from support tickets (e.g., “ID scan failed,” “unfair denial,” “language issue”). Complaint monitoring is especially important when ground truth labels are delayed or rare.

• Periodic re-tests: schedule a fairness re-check monthly or quarterly, and always after major changes (new data source, new threshold, new product flow).
• Decision gates: predefine what triggers a pause (e.g., gap exceeds X, complaints spike by Y%, or audit failure rate exceeds Z).

This closes the loop on the final milestone: decide whether to ship, pause, or change the use case. If monitoring shows stable performance and acceptable gaps with oversight and recourse, you can ship with constraints. If gaps persist in high-stakes contexts, pause and invest in deeper data/label fixes or redesign the use case. If the use case is inherently too sensitive for the available data and controls, changing or narrowing the use case is the responsible choice—and documenting that choice is a success, not a failure.

Section 5.1: What a policy is (and why it’s not a technical doc)

An AI use policy is a decision tool. It tells people how the system may be used, what boundaries apply, and what checks must happen to keep risk at an acceptable level. It is written for a mixed audience: product, engineering, operations, compliance, and sometimes customer support. That’s why it must be plain language and short.

A common mistake is to treat the policy like a model card, architecture diagram, or research report. Those technical documents are valuable, but they answer different questions (“How does it work?” “Which model version?”). A policy answers “What are we allowed to do with it, under what conditions, and who is accountable?” You can link to technical artifacts, but don’t bury the rules inside them.

Think in terms of behaviors. For your milestone in this chapter, you will define allowed and not-allowed uses for one AI feature. This is the heart of the policy because it prevents “scope creep,” where a tool built for low-stakes assistance slowly becomes a decision-maker for high-stakes outcomes.

• Policy language: “The AI may suggest draft replies. Humans must send the final message.”
• Not policy language: “We use Model X with temperature 0.3 and top_p 0.9.”

Engineering judgment matters here: if you can’t clearly state what the feature is for, you are not ready to ship it. Policies force clarity early, before “everyone assumes someone else checked.”

Section 5.2: Scope: where the AI is used and where it isn’t

Scope is where your one-page policy earns its keep. Write down exactly which workflow steps the AI touches, what inputs it can see, and what outputs it can influence. Then write the explicit “isn’t” list: scenarios that are out of bounds even if they seem convenient.

Start with a single sentence that names the feature and decision context, for example: “This policy covers the ‘Candidate Summary Generator’ used by recruiters to draft summaries of applicant resumes for internal review.” Immediately after, add boundaries: “The tool does not rank candidates, recommend hire/no-hire, or generate interview questions tied to protected attributes.”

When defining allowed vs not-allowed use (your first milestone), avoid abstract categories like “high risk.” Instead, name concrete actions:

• Allowed: drafting neutral summaries; extracting job-relevant skills already present in the resume; suggesting clarifying questions for a recruiter to ask.
• Not allowed: auto-rejection; scoring “culture fit”; inferring age, health, religion, or citizenship; using the output as the sole basis for a decision.

A frequent pitfall is leaving scope open-ended with phrases like “for recruiting purposes.” That invites downstream teams to reuse the same model for something harsher, like prioritizing candidates or predicting retention. If you anticipate future expansion, write a rule: “Any new use case requires a policy update and re-approval.” That one sentence prevents accidental repurposing.

Finally, define what “human in the loop” means in your context. Does a human merely see the output, or must they actively verify key facts? The policy should specify the minimum human action required before the output can affect someone.

Section 5.3: Roles and accountability (owner, reviewer, approver)

Policies fail when responsibility is distributed but accountability is not. Your one-page policy should name three roles, even in a small team: an owner, a reviewer, and an approver. One person can hold multiple roles in a startup, but the roles must be explicit.

Owner is the person accountable for safe operation day-to-day. They ensure the minimum documentation exists, tests are run, incidents are handled, and changes are tracked. The owner is typically the product owner or engineering lead for the feature.

Reviewer is the person who checks the owner’s work with some independence. They validate that fairness checks were performed correctly, that privacy rules are followed, and that changes don’t expand scope silently. Reviewers can be someone from data, security, compliance, or a senior engineer not building the feature.

Approver is the final sign-off authority for launch and major updates. This role exists to stop the “we were in a rush” dynamic. Approver authority should include the ability to delay release until the policy requirements are met.

• Owner must maintain: change log, test evidence, incident/finding log, and the current policy version.
• Reviewer must verify: fairness test results and gaps, documentation completeness, and that prohibited uses are not enabled by design.
• Approver must confirm: risk acceptance, launch readiness, and review cadence is scheduled.

Common mistake: making the “AI team” the owner. A team is not accountable; a named person is. Another mistake is failing to define what triggers re-approval. Write it plainly: “Any change to training data, decision threshold, target population, or user-facing messaging requires reviewer sign-off and approver confirmation.”

Section 5.4: Minimum checks: bias, safety, and data handling

This section turns ethics into a repeatable workflow. Keep the checks minimal but non-negotiable. Your milestones here are to add minimum documentation requirements and to add a fairness testing requirement with a review cadence, plus privacy rules in plain language.

Minimum documentation (must be recorded) should include: (1) the stated purpose and prohibited uses, (2) the input data sources and what fields are used, (3) the training/evaluation dataset description (even if small), (4) the model version or vendor configuration, (5) the decision point where humans intervene, and (6) links to the most recent bias/safety test results. Don’t overcomplicate it—this can live in a single page in your ticketing system.

Fairness testing requirement: specify what you will measure, on what groups, and what counts as a meaningful gap. For a beginner-friendly policy, you can require rate comparisons (selection/approval rates, error rates) and “gap” reporting (difference or ratio) across relevant groups. Write: “Before launch and every quarter thereafter, run the bias check on the current evaluation set. Record selection rates and false negative/false positive rates by group. If any gap exceeds the agreed threshold, the owner must open a finding and remediation plan before expansion.”

Review cadence must be explicit (monthly, quarterly, after any data change, and after incidents). Without cadence, checks happen once and then decay.

Safety and data handling: define what data is allowed. For example: “Do not input sensitive personal data unless explicitly approved. Do not store prompts containing personal data beyond X days. Mask or remove identifiers where possible.” Include operational rules like access controls (“Only authorized staff can view raw inputs”), retention, and deletion. These are simple statements, but they prevent the most common privacy failures: logging everything forever, and sharing data widely ‘for debugging.’

Common mistake: writing a fairness requirement without specifying who acts on it. Your policy should tie failures to action: create a finding log entry, pause expansion, and schedule a review.

Section 5.5: User-facing transparency and support channels

Ethical AI is not only internal governance; it also shows up in how you talk to users and how you handle problems. This section defines what you will tell users, when you will tell them, and where they can go for help. Keep the language simple enough to paste into a UI tooltip or FAQ.

Transparency rules should answer three questions: (1) Is AI involved? (2) What is it used for? (3) What are its limits? Example: “We use AI to suggest draft responses. A human reviews and sends the final message. The AI may be incorrect; please contact support if something looks wrong.” If the AI influences a decision that affects a person, include a clear description of the role it plays and what a person can do to contest or correct information.

Support channels are part of safety. Your policy should specify where feedback goes (support ticket category, email alias, in-product report button) and who monitors it. If you already created an incident-style finding log in earlier work, connect it here: “All user complaints involving potential bias, privacy, or harmful output must be recorded as a finding within 2 business days.”

• Provide a way to report errors and harms.
• Provide a way to request correction or deletion of personal data where applicable.
• Provide an escalation path for urgent issues (e.g., safety or discrimination concerns).

Common mistake: “We disclose AI use” without specifying the exact wording or location. Make it operational: “Disclosure appears in the UI next to the AI-generated text and in the help center article.” Clarity reduces user surprise, which reduces trust failures and escalations later.

Section 5.6: Exceptions, waivers, and when to stop using the model

No policy survives contact with reality unless it has a controlled way to handle exceptions. Teams will face time pressure, novel edge cases, and urgent incidents. Your one-page policy should allow exceptions, but only with friction and documentation.

Waivers: define what can be waived (for example, delaying a quarterly review by two weeks) and what cannot be waived (for example, logging requirements, prohibited uses, or privacy constraints). Require a short written justification, a risk note, and a time limit: “Waivers expire after 30 days and must be re-approved.” This prevents “temporary” shortcuts from becoming permanent.

Stop-use triggers are the most important part of accountability. Write explicit conditions that require pausing the feature or reverting to a safe baseline. Examples: (1) evidence of discriminatory impact above threshold with no immediate mitigation, (2) repeated privacy violations or sensitive data leakage, (3) a safety incident where output could cause material harm, (4) model behavior changes after an update without review, or (5) inability to produce required documentation during an audit or incident response.

Connect this to your incident-style finding log: “If a stop-use trigger occurs, the owner must file a critical finding within 24 hours, notify the approver, and disable the feature for the affected workflow until review is complete.” Make the workflow concrete—who flips the switch, who communicates to users, and how you record what happened.

Common mistake: relying on intuition (“We’ll stop if it gets bad”). Your policy should define “bad” in observable terms: measured fairness gaps, confirmed complaints, verified data leakage, or safety severity levels. This is how you turn ethics into a predictable operational standard rather than a debate during a crisis.

When you finish, read the whole page and check: could a new teammate follow it without additional meetings? If yes, you have a policy ready to share.

An ethics report has one job: make responsible decision-making possible. That means two audiences at once. First, non-technical readers (product owners, legal, operations, leadership) need a one-page executive summary that answers: What system is this? What did you test? What did you find? Who is affected? What do you recommend? Second, reviewers (analysts, data scientists, auditors) need enough detail to verify the work.

Write with accountability in mind. If your organization must later explain why it deployed (or paused) an AI feature, your report should show a reasonable process: defined scope, consistent metrics, stated thresholds (even if provisional), and a documented decision. This is how you prevent “we didn’t know” from becoming the default story.

Practical rule: separate observations from decisions. Observations are test outputs (rates, gaps, examples). Decisions are what you will do (ship, fix, monitor, stop) and why. Many beginner reports mix these, which makes it unclear whether the result is a fact or an opinion.

• Executive summary (one page): problem, system, key risk, top results, recommendation, and next steps.
• Evidence section: how you tested, the data you used, and the fairness measures you calculated.
• Action plan: specific fixes, owners, deadlines, and monitoring checks.

Engineering judgment matters: you are not trying to prove the system is “ethical.” You are trying to reduce avoidable harm, surface uncertainty, and create a clear record of responsible steps taken.

Non-technical readers won’t remember your formulas, but they will remember your story. A good testing story is a short narrative that connects the AI use case to the ethical risks and to the checks you ran. Start with scope: what model or rule-based decision you tested, which version, and which decision outcome you evaluated (for example: “approved/denied,” “flagged/not flagged,” “priority score above threshold”).

Next, state the groups you compared and why those groups matter in this context. Use plain language: “We compared outcomes across Group A and Group B because the system impacts access to a benefit, and unequal error rates could unfairly block eligible people.” Avoid jargon like “protected classes” unless your organization uses that term; instead, describe the attribute (e.g., age band, region, disability status) and how it is collected.

Then list what you checked, in the order a reviewer would follow:

• Data sanity: missing values, obvious labeling errors, and whether each group has enough examples to interpret results.
• Outcome rates: approval/flag rates by group (a basic disparity signal).
• Error rates: false positives/false negatives by group if you have ground truth (or a proxy).
• Threshold choices: if a score cutoff was used, record it and why.

Common mistake: describing tools instead of decisions. “We used a spreadsheet” is less important than “We calculated selection rate and false negative rate because these capture who gets access and who is wrongly denied.” Document methods without jargon by writing steps as actions a careful colleague could repeat.

Results should be readable in 60 seconds. Use one primary table that includes counts, rates, and gaps. Counts prevent misleading conclusions from tiny samples. Rates show the practical size of a difference. Gaps (difference or ratio) show comparison at a glance.

A simple table structure that works for beginners:

• Group
• N (cases)
• Outcome rate (e.g., approval rate)
• False negative rate (if available)
• False positive rate (if available)
• Gap vs. reference (difference in percentage points, and/or ratio)

Under the table, add a short narrative that explains the plain meaning. Example pattern: (1) what is higher/lower, (2) who is impacted, (3) why it matters, (4) how confident you are. Keep it concrete: “Group B is approved 12 percentage points less often than Group A (38% vs 50%) on this sample. If this pattern holds, Group B may receive fewer benefits even with similar eligibility.”

Charts are optional, not required. If you include one, use a bar chart of rates by group with counts labeled. Avoid complex multi-axis visuals. A common mistake is presenting only a fairness metric without context. A ratio like 0.76 means little unless the reader sees the underlying rates and sample sizes.

Finally, align results to decisions. If you have a provisional threshold (for example: “gap larger than 10 percentage points triggers investigation”), state it clearly. Even a beginner report benefits from a rule-of-thumb trigger, as long as you label it as provisional and revisit it later.

Limitations are not a confession of failure; they are a map of uncertainty. A professional report states what could change the conclusion and what you will do about it. The goal is to prevent overconfidence and to guide the next iteration of testing.

Write limitations in plain language, tied to impact. Useful categories:

• Sample size: “Group C has only 18 cases, so rates may swing widely. We cannot draw strong conclusions yet.”
• Label quality: “Ground truth comes from manual review, which may be inconsistent across reviewers.”
• Coverage: “This test covers last month’s data only; seasonal changes may affect results.”
• Missing attributes: “We cannot test outcomes by disability status because it is not collected.”
• Proxy risks: “Region may indirectly reflect socioeconomic status; interpretation requires care.”

Common mistake: hiding limitations in vague language (“data constraints”). Be specific about what you did not check (e.g., intersectional groups like “Group A + older age”), what you could not measure (e.g., long-term harm), and what assumptions you made (e.g., that labels are correct).

Engineering judgment: not every limitation blocks deployment. Some limitations are acceptable with monitoring; others require a stop or a redesign. Your job is to explain which is which, based on severity and likelihood of harm.

After results and limitations, make a clear recommendation. Beginner reports often avoid a decision (“needs more study”). Instead, choose one of four outcomes and justify it: ship, fix before shipping, ship with monitoring, or stop/pause. The decision should connect to your findings, not to optimism.

Then write an action plan with owners and dates. This is where the report becomes operational. Use a small table or bullets that answer: what action, who owns it, when it will be done, and how you will verify success.

• Fix: “Rebalance training data for Group B; Owner: Data Lead; Due: 2026-04-15; Verify: rerun fairness table on holdout set.”
• Policy: “Add a human review step for borderline denials; Owner: Ops Manager; Due: 2026-04-05; Verify: weekly audit of 30 cases.”
• Monitoring: “Track approval rate gap monthly; Owner: Product Analyst; Due: start 2026-04-01; Verify: dashboard with alert if gap > 10pp.”

Include a short “why this matters” note for each action (impact). Common mistake: listing generic recommendations (“improve data quality”) without specifying what “improve” means or how you will measure it.

Finally, document the decision meeting: date, attendees/roles, and what was approved. This is lightweight governance: it makes accountability real without adding bureaucracy.

Packaging matters. A report that cannot be found, reproduced, or reviewed is functionally useless. Keep a simple, repeatable structure so future you (or another team) can re-run the checks and compare versions over time.

Use this reusable beginner template (copy/paste and fill in):

• Title + version: System name, model/rule version, report version, date.
• One-page executive summary: purpose, key risks, top results, decision, top actions.
• System description: what it does, where used, who is impacted.
• Data used: source, timeframe, size, known issues.
• Methods: steps you followed, metrics computed, group definitions, thresholds.
• Results: primary table, short narrative interpretation, any charts.
• Limitations: specific uncertainties and their implications.
• Recommendations + action plan: owners, due dates, verification plan.
• Appendix: calculation notes, screenshots, or links to artifacts.

Suggested file organization (keep it boring and consistent):

• /report: final PDF or doc export, plus the editable source file.
• /data: the exact dataset used (or a pointer + hash if restricted).
• /analysis: spreadsheet or notebook with calculations.
• /figures: charts used in the report.
• /logs: incident-style finding log entries (what happened, who affected, why it matters).

Common mistake: updating the analysis but not the report, or changing data without recording it. Add a short changelog at the top of the report and name files with dates or version numbers. When your report is review-ready, you’ve completed the final milestone: it can be shared, questioned, and improved—without relying on memory or tribal knowledge.