Business Analyst to AI Decision Scientist: Causal & Experiments

Section 1.1: The BA-to-decision-scientist mindset shift

Reporting is descriptive: it summarizes observed data. Decision science is prescriptive: it estimates what would happen under different actions. The mindset shift begins when you stop treating metrics as “facts about the business” and start treating them as outcomes of a system influenced by choices, incentives, and hidden variables.

Milestone 1—spot the gap: a dashboard spike rarely answers the stakeholder’s real question. Suppose conversion rose after a UI redesign. Correlation says “conversion up,” but the decision is “should we roll this out?” The gap is the counterfactual: what would conversion have been without the redesign, at the same time, for the same users? If seasonality, marketing spend, or a competitor outage changed simultaneously, the spike may not be attributable to the redesign.

Engineering judgment matters here: you’re not trying to be philosophically pure; you’re trying to prevent expensive mistakes. A practical heuristic: if the decision is reversible and low-risk, weaker evidence may be acceptable. If the decision is irreversible, high-cost, or affects customers’ welfare, you need stronger causal identification and explicit guardrails.

Common mistakes in the transition include: treating a KPI movement as proof of causality, ignoring selection effects (who shows up in your data), and confusing optimization with understanding (a model can predict well and still mislead about what to do). Your new default is to ask: “What action are we considering, what outcome do we care about, and what would have happened otherwise?”

Section 1.2: Causal questions, counterfactuals, and why stakeholders care

Stakeholders rarely ask causal questions directly. They ask, “Does feature X work?” “Is channel A better?” “Will discounts increase retention?” Your job is to translate these into a causal estimand: the effect of a specific treatment on a specific outcome for a defined population over a defined time window.

Milestone 2—turn asks into causal questions and hypotheses: take “Should we add free shipping?” and rewrite it as: “Among eligible customers, what is the average change in 30-day contribution margin if we offer free shipping versus not offering it?” Now you can state a hypothesis (e.g., margin increases due to higher conversion and repeat purchase, but may decrease due to subsidy costs). Notice how this forces tradeoffs into the open.

Counterfactual thinking is the core skill. For any unit (a user, account, store), there are two potential outcomes: one if treated and one if not. We only observe one. Causal methods are about recovering the missing outcome using design (randomization) or assumptions (observational identification). When you frame work this way, stakeholders care because it directly supports decisions: rollout, budget allocation, policy thresholds, and roadmap prioritization.

Practical tip: write the causal question using “if we do X instead of Y, what happens to Z?” and insist on specifying Y (the baseline). Many failures come from an implicit baseline that later changes (“business as usual” isn’t stable), making results hard to interpret.

Section 1.3: Units, treatments, outcomes, and interference basics

Before selecting a method, you must define the evaluation object clearly. Milestone 3 is operational: specify units, treatment, outcome, and time windows so the analysis is computable and the result is decision-relevant.

Units: who or what receives the treatment—users, accounts, merchants, stores, or regions. Unit choice affects feasibility and bias. For example, treating individual users in a marketplace can cause spillovers (one user’s treatment affects another’s outcome). In that case, the appropriate unit may be a region or time block.

Treatment: the actionable change. Define it as something that can be turned on/off or varied. Avoid vague treatments like “improve onboarding.” Instead: “show step-by-step checklist on first session” or “require identity verification at signup.” Include treatment intensity if relevant (e.g., discount size).

Outcome: the business or customer measure the decision is optimizing. Be precise: “7-day retention” must define the event, the window, and eligibility rules. Write the outcome as a function of logs/events so an engineer can implement it.

Time windows: specify exposure timing, measurement timing, and any washout/lag. A retention change may not show until weeks later; a pricing change may have immediate effects but longer-run churn impacts.

Interference basics: many causal tools assume one unit’s treatment doesn’t affect another unit’s outcome (no spillovers). In practice, interference is common: referrals, social feeds, ads auctions, inventory constraints, fraud rings. Your job is to detect it early and adjust design (cluster randomization, geo experiments) or interpret results with limits.

Section 1.4: Metrics hierarchies: North Star, inputs, and guardrails

A decision needs a metrics hierarchy so teams don’t optimize the wrong thing. Think in three layers: (1) a North Star outcome aligned to value, (2) input/leading metrics that move earlier and help diagnose mechanisms, and (3) guardrails that prevent harmful tradeoffs.

For example, if the decision is to simplify checkout, a North Star might be “completed purchases per eligible session” or “net revenue per visitor.” Input metrics could include page load time, payment success rate, or add-to-cart rate. Guardrails could include refund rate, customer support contacts, or fraud loss. This structure prevents a common failure mode where conversion increases but returns and complaints explode.

Metrics hierarchy also helps you plan experiments. Guardrails become stopping criteria (if complaints exceed threshold, pause). Inputs help you detect instrumentation problems and understand why an effect occurred. A/B tests and observational studies both benefit from this discipline because it clarifies what must be measured and what risks must be monitored.

Engineering judgment: choose metrics that are (a) sensitive enough to detect change, (b) hard to game, and (c) stable in definition. Document exact computation rules and version them. Many “failed analyses” are actually metric drift: event names change, eligibility logic shifts, or backfills alter historical values.

Section 1.5: Common stakeholder traps (vanity metrics, proxy goals, moving targets)

Decision scientists earn trust by preventing predictable mistakes without slowing teams down. Three stakeholder traps show up repeatedly.

Vanity metrics: measures that look good but don’t reflect value (raw signups, app opens, impressions). The practical fix is to tie the metric to an economic or customer-value outcome: active retention, conversion to paid, contribution margin, or verified task completion. If a vanity metric must be tracked, demote it to an input metric and keep the North Star anchored to value.

Proxy goals: when the true goal is hard to measure, teams use a proxy (e.g., “time in app” for engagement, “click-through rate” for relevance). Proxies can invert incentives. CTR can rise with clickbait while satisfaction drops. Your job is to validate proxies against downstream outcomes and include guardrails that capture the missing dimension (e.g., long-click, survey satisfaction, churn).

Moving targets: stakeholders change the question midstream (“Now also optimize for enterprise users,” “Actually focus on Q4 revenue”). Prevent this by freezing the estimand, population, and primary metric before data collection. When change is necessary, treat it as a new decision with a new measurement plan, not a post-hoc rewrite.

Milestone 5 begins here: capture these risks in a one-page memo so alignment is explicit: what we’re deciding, what success means, what we will not sacrifice, and what would change our recommendation.

Section 1.6: Evidence ladder: experiments, natural experiments, and modeling

Milestone 4—choose the right evaluation approach: decide whether you can randomize. If you can run an A/B test ethically and operationally, it is usually the most credible way to estimate causal effects because randomization breaks confounding by design. But “can we randomize?” includes more than tooling: eligibility, spillovers, legal constraints, customer fairness, and whether the organization can tolerate short-term risk.

When experiments aren’t possible, you move down the evidence ladder to quasi-experiments: difference-in-differences (policy changes with comparison groups), regression discontinuity (threshold-based rules), instrumental variables (a source of exogenous variation), and matching (balancing observed covariates). These require stronger assumptions and more diagnostics. A practical rule: the less you control assignment, the more you must invest in design critique—draw a causal DAG, identify confounders and selection mechanisms, and pre-register what you’ll check.

Modeling (predictive ML) is valuable, but it answers a different question by default: “given what we observe, what is likely?” It does not automatically answer “what if we intervene?” You can adapt modeling for causal use (uplift modeling, doubly robust estimators), but only when the identification assumptions are justified and measurement is sound.

Close the chapter with a concrete artifact: a one-page decision memo and measurement brief. Include: decision to be made; treatment and baseline; population and unit; primary outcome (North Star) and time window; key guardrails; proposed method (A/B, DiD, etc.) with rationale; risks (interference, selection, metric drift); and what result magnitude would change the decision. This brief is your bridge from reporting to causal decisions—and the foundation for the rest of the course.

Section 2.1: Potential outcomes and average treatment effects (ATE/ATT)

Causal inference starts with a simple idea: every unit (a user, account, store, or shipment) has potential outcomes. If we apply a treatment (say, a new onboarding flow), the unit would have outcome Y(1). If we do not, it would have Y(0). The causal effect for that unit is Y(1) − Y(0). The catch is fundamental: you never observe both for the same unit at the same time. All practical causal work is about recovering average effects despite that missing counterfactual.

This is where Milestone 1—write the estimand before choosing a method—becomes non-negotiable. The most common estimands in business are:

• ATE (Average Treatment Effect): average impact if everyone in the target population received the change versus not.
• ATT (Average Treatment Effect on the Treated): average impact on the subset who actually received the change (common in observational rollouts or opt-in programs).

These are not interchangeable. Example: a retention team tests proactive outreach only for high-risk churn customers. If you estimate ATT, you learn the effect on high-risk customers who were contacted. If leadership asks “Should we roll this out to all customers?”, you need something closer to ATE (or at least conditional effects by risk group), because the effect can differ across segments.

Write the estimand in business terms and math terms. Business: “Effect of enabling auto-renewal on 90-day revenue per user for new subscribers in Q2.” Math: “ATE of treatment T on outcome Y among cohort C.” Also specify the time window, unit of analysis, and what counts as treatment compliance (e.g., assigned to new flow vs actually completed the new steps). This clarity prevents method-driven drift, like using a convenient dataset that only supports short-term conversion while the decision depends on long-term retention.

Section 2.2: DAGs for business problems: how to draw them quickly

A causal DAG (directed acyclic graph) is a compact way to document assumptions about what causes what. It’s not a statistical model; it’s a communication tool. Milestone 2 is to map assumptions with DAGs and identify adjustment sets—i.e., which variables you need to condition on to estimate the effect without opening bias paths.

To draw a DAG quickly in a business setting, use a four-step routine:

• Start with the decision lever (T) and the outcome (Y): e.g., T = “offer 10% discount,” Y = “30-day gross margin.” Draw T → Y.
• Add common causes of T and Y (candidate confounders): e.g., customer price sensitivity affects whether they receive/accept a discount and also affects margin and churn.
• Add measurement/selection nodes: e.g., “is observed in dataset,” “opens email,” “eligible for offer.” These often create selection bias.
• Add post-treatment variables separately: things that happen after treatment (like “usage”) might be mediators or colliders; do not auto-adjust for them.

Keep the first DAG deliberately coarse. Your goal is not completeness; it’s to surface disagreement early. In a stakeholder review, ask: “What makes us decide to treat someone?” and “What else drives the outcome?” Write answers as nodes. Then ask: “Does this happen before or after treatment assignment?” This timing question is the fastest way to prevent accidental mediator/collider control.

Once the DAG is sketched, identify an adjustment set: a set of pre-treatment variables that blocks all backdoor paths from T to Y. In practice, you often aim for “good enough” adjustment, prioritizing variables that materially influence both treatment assignment and outcome. Document why each variable is included, not just that it exists. That documentation becomes input to your assumption checklist later.

Section 2.3: Confounding, selection bias, and omitted variable intuition

Confounding happens when a variable influences both treatment and outcome, creating a spurious association. The classic business version: a sales team targets outreach (T) to accounts showing buying signals. Buying intent (U) also drives revenue (Y). If you compare contacted vs not contacted, you may attribute the effect of intent to the outreach.

The “omitted variable” intuition is useful but incomplete: it’s not that leaving out any variable causes bias; leaving out a common cause of T and Y causes bias. That distinction matters because analysts often over-correct by controlling for everything they can measure, which can create new bias (next sections).

Selection bias is different: it occurs when your dataset includes only a selected subset, and selection depends on variables related to treatment and outcome. Example: you want the effect of a new checkout UI on purchase completion, but your analysis dataset includes only users who reached the checkout page. If the UI change also affects whether users reach checkout, conditioning on “reached checkout” can bias the estimate and can even flip the sign.

Milestone 4—diagnose bias risks and decide what data is “good enough”—means you explicitly evaluate (a) unmeasured confounding risk, (b) selection mechanisms, and (c) measurement quality. Practical heuristics:

• If treatment assignment is strongly driven by human judgment (sales reps, support agents), assume unmeasured confounding until proven otherwise.
• If your analysis conditions on “engaged users,” “approved applicants,” or “active accounts,” treat it as a selection node and test whether treatment affects selection rates.
• Prefer pre-treatment covariates that are stable and well-instrumented (account tenure, prior spend) over noisy behavioral signals that might already be influenced by treatment rollout timing.

When “good enough” is not attainable, be explicit. Your credibility increases when you can say: “We can estimate ATT among eligible accounts with these assumptions, but we cannot generalize to all accounts without stronger design or additional data.”

Section 2.4: Colliders and why “controlling for more” can hurt

A collider is a variable caused by two other variables. Conditioning on a collider (controlling for it, stratifying on it, filtering by it) can create a false association between its causes and open a backdoor path that didn’t exist before. This is why “controlling for more” can harm causal validity even if it improves predictive fit.

Concrete example: You’re estimating the effect of a new recommendation algorithm (T) on revenue (Y). Suppose “number of sessions” (S) is influenced by both the algorithm (it changes engagement) and by latent user intent (U). The structure is T → S ← U and U → Y. If you control for sessions, you open the path T ↔ U → Y, inducing bias. You may conclude the algorithm hurts revenue after “adjusting for sessions,” even if it truly helps.

Colliders show up constantly in business analytics because many operational metrics are downstream of multiple causes: “support tickets,” “time on site,” “eligibility,” “approval,” “exposure,” “being in the dashboard,” “being seen by the model.” In experimentation platforms, a common collider is exposure: if only some assigned users actually see a feature due to logging or rollout gates, conditioning on “saw the feature” can reintroduce confounding through the factors that determine exposure.

Milestone 3 (distinguish confounders, colliders, mediators) is operationalized here with a rule: only adjust for variables you are confident are pre-treatment common causes of assignment and outcome. If a variable can plausibly be affected by treatment, treat it as unsafe by default until the DAG makes it safe.

A practical safeguard: maintain two covariate lists in your analysis plan—(1) “allowed pre-treatment adjusters,” (2) “do-not-adjust (post-treatment / colliders / selection).” Review those lists with stakeholders and data engineers before analysis so you don’t discover late that your KPI dashboard is conditioned on a collider.

Section 2.5: Mediation vs moderation: what you can and can’t claim

Mediation is about mechanism: treatment affects a mediator, which then affects the outcome (T → M → Y). Example: a faster page load (T) increases engagement (M), which increases conversion (Y). Leaders often ask “Did it work because engagement increased?” That is a mediation question.

Moderation is about heterogeneity: the treatment effect differs across groups or contexts. Example: faster page load helps mobile users more than desktop users. That is a moderation (effect modification) claim, typically assessed by interaction terms or subgroup analysis.

These are commonly confused, leading to overclaims. If you adjust for engagement while estimating the total effect of page speed on conversion, you may remove part of the true effect (because engagement is on the causal pathway). You end up estimating a direct effect (effect not through engagement), not the total effect the business cares about. If the decision is “Should we ship faster pages?”, total effect is usually the right estimand. If the decision is “Should we invest in speed even if it doesn’t change engagement?”, you might care about the direct effect—but that must be stated upfront as an estimand choice (back to Milestone 1).

Mediation analysis also requires stronger assumptions than total-effect estimation: no unmeasured confounding for T→M and M→Y relationships, and careful handling of post-treatment confounders. In many business settings, it’s better to communicate mechanism as supporting evidence rather than a definitive decomposition, unless the study is designed for it.

For moderation, be disciplined: pre-register which segments matter (e.g., platform, new vs returning, region) and why. Otherwise, you risk “finding” differences through multiple comparisons. Practical outcome: use moderation to inform targeting and rollout strategy, not as a post-hoc justification for ambiguous average effects.

Section 2.6: Identification: when effects are estimable vs unknowable

Identification asks: given the data you have and the assumptions you are willing to defend, is the causal estimand uniquely recoverable? This is where causal inference becomes an engineering judgement discipline. Some effects are estimable with careful adjustment or experiments; others are fundamentally unknowable without new design, new data, or narrower questions.

Use a simple identification triage:

• Can we randomize? If yes, default to an experiment for the primary decision, because randomization severs backdoor paths. If not, ask what “as-if random” variation exists (policy threshold, staggered rollout, natural shocks).
• Are key confounders measured pre-treatment? If the main drivers of treatment assignment are unobserved (e.g., rep persuasion skill, undocumented risk flags), regression or matching won’t identify the effect reliably.
• Is there problematic selection? If inclusion depends on post-treatment behavior, you may be estimating a distorted effect unless you redesign logging or define the estimand on the selected population explicitly.

This is also where Milestone 5—create an assumption checklist stakeholders can sign off on becomes practical. A good checklist includes: target population, treatment definition (assignment vs exposure), outcome window, causal contrast (ATE/ATT), adjustment variables and rationale, known unmeasured confounders, selection/filtering rules, interference risks (spillovers), and what would change your mind (sensitivity checks, negative controls, placebo tests).

When identification fails, don’t “pick a method anyway.” Narrow the estimand (e.g., ATT among eligibles), propose an experiment or quasi-experiment, improve instrumentation, or switch to decision-support outputs that don’t pretend to be causal (forecasting, scenario bounds). The practical outcome of this chapter is that you can explain, in plain language, why an effect estimate is credible, what assumptions it rests on, and what work is needed to make it more credible—before anyone bets a roadmap on it.

An A/B test is a controlled intervention designed to estimate a causal effect: the difference in outcomes if the same population were exposed to treatment versus control. In practice, “treatment” is a bundle (new UI, new model, new latency) and your first job is to define it precisely enough that engineers can implement it and analysts can measure it. Write down: what changes, who is eligible, when exposure starts, and what constitutes exposure (assignment vs actually seeing the feature).

For causal interpretation, stakeholders need a clear estimand. For most product experiments, the estimand is the average treatment effect (ATE) on a primary metric over a time window among eligible units. That sentence forces clarity: average over whom (units), under what assignment (randomized), measured when (window), and on which metric (primary). If you cannot articulate the estimand in one line, you do not yet have an experiment—just a hope.

Milestone connection: defining the experimental unit and randomization strategy is not administrative; it is how you protect causal meaning. If assignment is random and outcomes are measured consistently, the difference in means is an unbiased estimate of the intention-to-treat effect (ITT): the impact of being assigned to treatment, regardless of whether the user fully “complied.” ITT is usually what stakeholders should act on because it reflects the real operational effect of shipping.

Common mistake: letting the treatment definition drift during the run (“we hotfixed the model,” “we changed the ramp rules,” “we disabled it for iOS”). Treat those as new experiments or document them as deviations; otherwise, the causal statement becomes ambiguous and trust erodes.

Milestone 1 is choosing the experimental unit and randomization scheme. The unit is the “thing” you randomize (user, session, account, store, region). Choose it based on interference risk (one unit affecting another), practical implementation, and how the decision will be deployed. If you will ship per-user, randomize per-user. If the experience is inherently shared (marketplace liquidity, social feeds), you may need cluster randomization or a geo test.

User-level randomization is the default for consumer products: stable assignment, clean interpretation, and good power. Beware: users can have multiple devices; define identity resolution early or you will contaminate control with treated exposures.

Session-level randomization is tempting when you cannot persist assignment, but it increases variance and creates “within-user” contamination: the same user may see both variants across sessions, changing behavior and diluting effects. Use it when the treatment is ephemeral and interference within a user is unlikely.

Geo randomization (or store/region tests) is used when treatments spill over (pricing, delivery times, ads marketplaces). It is operationally realistic but statistically harder: fewer units (regions), correlated outcomes, and more sensitivity to time trends. Plan longer durations and use methods that respect clustering.

Cluster randomization (by team, school, employer, household) addresses interference but requires cluster-aware power calculations and analysis. Clusters must be large enough and sufficiently many; otherwise, randomization will not balance confounders well.

Engineering judgement: implement assignment at a single authoritative layer (feature flag service) and log both “assignment” and “exposure.” Assignment is for causal analysis; exposure is for debugging and compliance estimates. Also decide ramp strategy (e.g., 1%→10%→50%) while keeping randomization stable so early ramps do not become a different population.

Milestone 3 is power: how many units you need to detect an effect worth caring about. Stakeholders do not want “statistical significance”; they want decisions that are unlikely to be wrong in costly ways. Power analysis links business value to statistical design by forcing you to set a minimal detectable effect (MDE) and acceptable error rates (typically 5% false positive rate and 80–90% power).

Start with practical significance. For a conversion metric, ask: what lift changes the decision? A 0.1% relative lift may be meaningless unless you have huge volume or high margin. Translate MDE into dollars or risk: “We need to detect at least +0.3pp conversion because below that the engineering cost is not justified.” Then compute sample size using historical baseline rate and variance. For continuous metrics (revenue per user), variance is often large; consider transformations, winsorization, or using a more stable proxy metric as the primary outcome.

Do not ignore clustering. If you randomize by geo or cluster, the effective sample size is much smaller due to intra-cluster correlation (ICC). Your formula must include a design effect; otherwise you will underpower and later rationalize an inconclusive result as “no effect.”

Workflow tip: power is iterative. You may adjust (a) the unit (user vs session), (b) the metric (binary vs continuous), (c) the duration (one week vs four), or (d) the allocation ratio (50/50 vs 90/10 for risk) to reach feasible sample sizes. Document trade-offs: longer duration increases exposure to seasonality; smaller treatment allocation reduces power; switching metrics can change what the experiment truly optimizes.

Common mistake: picking MDE after seeing early results. That is reverse-engineering certainty and is a subtle form of p-hacking. Choose MDE upfront in your experiment brief.

Milestone 2 is metric design: pick a primary metric, guardrails, and explicit success criteria. The primary metric should align with the decision you’re enabling (ship/no-ship) and be sensitive to change. Guardrails protect against local optimization that harms users or the business (latency, error rate, refunds, customer support contacts, churn, fairness indicators). A trustworthy experiment declares both: “We win if primary improves by at least X and guardrails do not degrade beyond Y.”

Guardrails are risk management. They also reduce stakeholder anxiety because they show you anticipated failure modes. For example, a ranking change might increase clicks (primary) but also increase returns (guardrail) or degrade long-term retention (guardrail). If long-term outcomes are slow, use leading indicators (e.g., repeated visits) and plan follow-up analyses.

Ethics is not separate from design. Consider who bears the cost of the test, whether vulnerable populations are disproportionately exposed, and whether consent or disclosure is needed. In regulated domains (finance, health), randomization may require additional review; the “fastest” experiment is the one that will survive legal and compliance scrutiny.

Practical technique: define alert thresholds and operational playbooks. Example: “If payment failure rate increases by 0.2pp at any time, auto-disable treatment.” This is a stopping rule for harm, distinct from stopping for success. Ensure stakeholders agree on these thresholds before launch so you are not negotiating ethics in the middle of an incident.

Trustworthy experiments anticipate failure modes and bake in detection. Novelty effects occur when users respond to something new (curiosity, confusion), creating short-term lifts that fade. Mitigation: run long enough to observe stabilization, segment by tenure, and avoid over-interpreting day-1 spikes. If the decision is long-term, design the duration to match it.

Interference and spillovers break the “no interference” assumption: one unit’s treatment affects another’s outcome. Marketplaces, social products, and ad auctions are especially vulnerable. Symptoms include inconsistent effects across geos, time-varying impacts, and weird cross-group correlations. Mitigation: choose cluster or geo randomization, or explicitly model network effects. At minimum, warn stakeholders when interference risk is high so they interpret results cautiously.

Sample Ratio Mismatch (SRM) is an early warning signal that randomization or logging is broken: the observed allocation differs from expected (e.g., 50/50 becomes 48/52). SRM often indicates filtering after assignment, platform-specific bugs, or eligibility logic differences between variants. Milestone 5 should include an SRM check using a chi-square test and, more importantly, a root-cause investigation path. Do not “power through” SRM; it undermines the randomization guarantee and therefore the causal claim.

Other practical failure modes: missing data (events not firing), metric definition drift, bot traffic, and ramp-up bias (early ramp includes mostly low-activity users). A pre-flight checklist should validate event coverage, deduping, time zones, and identity joins before the experiment reaches meaningful traffic.

Milestone 4 is committing to an analysis plan and stopping rules that prevent p-hacking. The core principle: decide how you will compute the effect before you see the results. This is what makes the result credible when it is politically inconvenient.

Use intention-to-treat (ITT) as the default estimand: analyze users by assigned group, not by who “actually used” the feature. ITT preserves randomization. Per-protocol or “complier” analyses can be useful diagnostics (e.g., to estimate effect among exposed users), but they reintroduce selection bias because exposure is often behavior-driven.

Define exclusions narrowly and mechanically. Good exclusions: units that were ineligible by definition (employees, test accounts), or corrupted instrumentation (known outage window). Bad exclusions: “users with extreme spend” discovered after you saw variance. If you must handle outliers, pre-specify the rule (winsorize top 0.1%, use robust metrics) and apply it symmetrically to both groups.

Stopping rules need particular care. If you check results daily and stop when p<0.05, your false positive rate inflates. Options: commit to a fixed horizon (run N days), use alpha-spending/sequential methods, or use Bayesian decision thresholds—any is acceptable if agreed upfront. Separately, define safety stops based on guardrails, and define what happens when results are inconclusive (e.g., iterate, increase power, or deprioritize).

Pre-registration can be lightweight: an experiment brief stored in your ticketing system or wiki capturing hypotheses, units, metrics, MDE, duration, analysis method, and decision criteria. The practical outcome is repeatability: another analyst can reproduce your result, and a stakeholder can see that the decision followed the contract you set at launch.

Matching and propensity scores aim to make treated and untreated groups comparable by aligning them on observed covariates. This is the right tool when stakeholders ask, “Can’t we compare customers who used the feature to similar customers who didn’t?” Your milestone is to translate that into an estimand (often ATT: effect on those treated) and decide whether selection into treatment is plausibly captured by measured variables.

What it does: reduces confounding from observed differences (e.g., tenure, prior spend, industry). Propensity scores (the probability of treatment given covariates) help you match, weight (IPTW), or stratify when direct matching on many variables becomes hard. What it does not do: fix unobserved confounding. If “customer urgency” drives both adoption and outcome but is not measured, your estimate can still be biased.

Assumption validation (Milestone 2): start with balance tests. After matching/weighting, check standardized mean differences for all covariates used (and ideally some “extra” covariates). Don’t accept “p>0.05” as your criterion; use effect-size balance (e.g., SMD < 0.1) and examine overlap/positivity: do treated units have comparable untreated units at similar propensity scores? A falsification test is a pre-treatment outcome: if you “find an effect” of the treatment on last month’s outcome, you have residual confounding or time-varying selection.

Estimation and uncertainty (Milestone 3): estimate the treatment effect on the matched/weighted sample and use robust standard errors (and cluster if outcomes are correlated within accounts, regions, etc.). Report the effective sample size under weighting; executives need to know when the estimate is driven by a small slice of data.

Common mistakes: (1) using post-treatment variables in the propensity model (collider/mediator contamination), (2) “perfect prediction” propensity scores that create extreme weights, and (3) presenting a matched estimate without showing balance and overlap plots.

Practical outcome: a short table of covariate balance, an overlap plot, and a clear statement: “This adjusts for measured differences X, Y, Z; if there are unmeasured drivers like U, results may still be biased.” That sets you up for sensitivity analysis later.

Difference-in-differences (DiD) is often the most business-friendly quasi-experiment because it maps cleanly to “before vs after” comparisons with a control group. You use it when treatment is rolled out to one segment (regions, cohorts, accounts) while another segment remains untreated for a while. Milestone 1 here is choosing DiD only when you have enough pre-period data to argue the groups would have evolved similarly absent treatment.

Core assumption: parallel trends. The treated and control groups can have different levels, but their trends should be similar before treatment. Your first job is not to run the regression; it’s to draw the time series and look at pre-trends. Then formalize it with an event study: estimate coefficients for leads and lags relative to treatment. Leads (pre-treatment effects) should be near zero; if they aren’t, you may be capturing anticipatory behavior, selection, or unrelated shocks.

Assumption validation (Milestone 2): do falsification tests. Use outcomes you expect not to change (a “placebo” metric), or run the same DiD on a period where no rollout occurred. Also check for differential seasonality: if treated regions have different holiday peaks, your DiD may attribute seasonal effects to treatment unless you include time fixed effects and possibly group-specific seasonality controls.

Estimation and uncertainty (Milestone 3): at minimum, use group and time fixed effects and cluster standard errors at the treatment assignment level (e.g., region). If treatment timing is staggered, be careful: older two-way fixed-effect DiD can be biased when effects vary over time. Use modern estimators designed for staggered adoption (e.g., group-time ATT frameworks) and report how you handled heterogeneous effects.

Common mistakes: (1) choosing a control group that is “convenient” but exposed to spillovers, (2) ignoring anticipation (marketing announcements cause behavior to change before launch), and (3) reporting a single post coefficient without showing the event-study plot.

Practical outcome: an event-study chart with confidence intervals, a narrative about parallel trends, and a clear timeline of external events (pricing changes, outages) that could confound interpretation.

Regression discontinuity (RD) is your go-to when treatment assignment hinges on a cutoff: credit score thresholds, risk bands, SLA tiers, eligibility rules, or “accounts with >$X spend get assigned a CSM.” Milestone 1 is verifying that the threshold truly determines treatment (sharp RD) or at least strongly shifts treatment probability (fuzzy RD). If the cutoff is real and enforced, RD can be highly credible because units just above and below the threshold are often comparable.

Key idea: compare outcomes for observations narrowly around the cutoff. The identifying assumption is continuity: absent treatment, the outcome would vary smoothly with the running variable at the cutoff. You estimate the discontinuity at the threshold as the causal effect (or, in fuzzy RD, use the jump in treatment probability as an instrument to get a local effect).

Assumption validation (Milestone 2): manipulation checks are non-negotiable. If people can game the running variable (sales reps pushing deals over a threshold, customers timing applications), RD breaks. Use a density test around the cutoff and inspect the histogram: a suspicious pile-up suggests sorting. Also test continuity of pre-treatment covariates at the cutoff; large jumps imply the groups are not comparable.

Estimation and uncertainty (Milestone 3): use local linear regression with robust bias-corrected confidence intervals, and be explicit about bandwidth selection (data-driven methods are common). Show sensitivity to bandwidth choices and polynomial order—high-degree polynomials can hallucinate curvature and create fragile effects. Always visualize: plot binned means and fitted lines on both sides of the cutoff with the discontinuity highlighted.

Common mistakes: (1) using RD when the cutoff is advisory rather than enforced, (2) treating RD estimates as global effects (they are local near the threshold), and (3) failing to check for other policy changes at the same cutoff (e.g., different messaging or service levels bundled with eligibility).

Practical outcome: a plot, a manipulation/balance checklist, and a stakeholder-ready statement: “This effect applies to accounts near the eligibility boundary; it may not generalize to very small or very large accounts.”

Instrumental variables (IV) are for the hard cases: treatment is confounded by unobservables, but you have a source of quasi-random variation that nudges treatment without directly affecting the outcome. Examples include random-ish assignment to sales reps with different persuasion rates, distance to a facility that changes service take-up, or queue positions that influence whether a customer receives an intervention. Milestone 1 is deciding if you truly have an instrument—not just a correlated feature.

Three requirements: (1) Relevance: the instrument changes treatment (first stage is strong). (2) Exclusion: the instrument affects the outcome only through treatment, not through other channels. (3) Independence: the instrument is as-if random with respect to unobserved confounders. In practice, exclusion is the hardest and most debated; it must be argued with domain knowledge and process details, not just statistics.

Assumption validation (Milestone 2): show first-stage strength (e.g., F-statistics, treatment uptake by instrument values). Run balance tests: are baseline covariates similar across instrument groups? Use falsification outcomes that should not be affected. And tell the operational story: why would instrument assignment be unrelated to customer risk, seasonality, or channel mix?

Estimation and uncertainty (Milestone 3): two-stage least squares (2SLS) is standard. Interpret the estimand correctly: IV typically identifies the LATE—the effect for “compliers” whose treatment status is changed by the instrument. This is often exactly what decision-makers care about (e.g., people persuadable by a nudge), but it is not the average effect for everyone. Report wide intervals honestly; IV estimates can be noisy, especially with weak instruments.

Common mistakes: (1) using a weak instrument (results become unstable and biased), (2) ignoring that LATE may not generalize, and (3) hand-waving exclusion (“seems unrelated”) without documenting the mechanism.

Practical outcome: a diagram of the instrument mechanism, a first-stage table, and an executive sentence: “This estimates the effect for customers whose adoption is influenced by rep assignment; it may differ for customers who would adopt regardless.”

When you have one (or a few) treated units and many potential controls—like a product launching in one country first, a new pricing model in one business line, or a policy change in a single marketplace—synthetic controls can outperform simple DiD. The method constructs a weighted combination of control units that best matches the treated unit’s pre-treatment trajectory, creating a “synthetic twin.” Milestone 1 is choosing this approach when the treated unit is unique and pre-period fit can be made very tight.

Workflow: (1) define the treated unit and intervention date, (2) select a donor pool of unaffected units, (3) choose predictors (pre-outcome history and key covariates), (4) fit weights to minimize pre-treatment error, (5) estimate post-treatment gaps. The credibility hinges on pre-period fit: if you cannot reproduce the treated unit’s history, you should not trust the post-period gap.

Assumption validation (Milestone 2): do placebo tests by re-running the method treating each control unit as if it were treated. If your treated effect is not unusually large relative to placebo gaps, your result may be noise. Also test for contamination: were donor pool units indirectly affected (spillovers, shared marketing, macro shocks)? If so, the synthetic control can understate the effect.

Estimation and uncertainty (Milestone 3): uncertainty is often communicated via placebo distributions rather than classical standard errors. Present the gap plot, the pre-treatment fit, and a ratio such as post/pre RMSPE to show whether the treated deviation is exceptional. If you have many treated units over time, consider panel methods that generalize synthetic controls (matrix completion, interactive fixed effects) but keep the story anchored in “we built a counterfactual trajectory from similar histories.”

Common mistakes: (1) a donor pool that includes units with hidden exposure to treatment, (2) overfitting predictors that improve pre-fit but worsen interpretability, and (3) treating placebo p-values as definitive rather than as one robustness lens.

Practical outcome: a one-page graphic: pre-fit, post-gap, donor weights, and placebo comparison—ideal for rollout decisions and retrospectives.

Quasi-experiments rarely end with a single “best estimate.” Stakeholders need to know how fragile the conclusion is, and what new evidence would change your recommendation. This section ties together Milestone 4 (stress-test with sensitivity analyses) and Milestone 5 (write the “what would change my mind” section) in an executive-friendly reporting style.

Sensitivity analyses to standardize: (1) Spec sensitivity: alternative model forms (with/without covariates; different fixed effects; alternative functional forms). (2) Window/bandwidth sensitivity: time windows in DiD/event studies; bandwidth in RD. (3) Placebos: fake intervention dates, fake thresholds, or untreated outcomes. (4) Donor pool sensitivity: synthetic controls with different donor restrictions. (5) Unobserved confounding bounds: for matching, use quantitative sensitivity tools (e.g., Rosenbaum bounds or “how strong would an unmeasured confounder have to be” style metrics) to translate hand-wavy concerns into a threshold.

Robustness reporting pattern: present a “robustness table” that shows the estimate across key variants, and highlight which assumptions drive changes. Don’t bury the lede: if the sign flips under a reasonable alternative, say so and downgrade confidence.

How to interpret uncertainty responsibly (Milestone 3): report effect size with intervals and business translation (e.g., incremental revenue per 1,000 users) but keep the interval visible. Avoid false precision like “+1.3%” without context; instead: “Estimated lift 1–3% with moderate confidence; most uncertainty comes from pre-trend instability.”

“What would change my mind” section (Milestone 5): list concrete triggers: (a) evidence of non-parallel pre-trends in the next cohort, (b) detection of manipulation at RD threshold, (c) first-stage weakening below a defined threshold for IV, (d) spillover confirmation into donor units, (e) a placebo test producing effects as large as the main result. This turns critique into a forward plan: what to monitor, what data to collect, and when to revisit the decision.

Practical outcome: executives leave with a decision and a risk register. You leave with a repeatable standard: every quasi-experimental analysis ships with assumptions, falsifications, sensitivity, and explicit conditions for reversal—protecting trust even when results are nuanced.

Section 5.1: Estimation vs decision: thresholds, utility, and ROI framing

Estimation answers “what is the effect?” Decision-making answers “what should we do next?” They are related but not identical. A model can estimate a small positive effect with high confidence, yet the correct decision may still be “do nothing” if implementation cost, risk, or opportunity cost dominates. Your job is to connect estimands to decision thresholds.

Start by building a causal KPI tree (Milestone 1). Put the business action at the root (e.g., “launch personalized onboarding”). Under it, list proximal causal mechanisms (e.g., “reduces time-to-first-value,” “increases feature discovery”), then intermediate product KPIs (activation rate, week-1 retention), and finally the outcome that matters (net revenue, churn, margin). For each node, write the causal link explicitly: “If we change X, we expect Y to change via mechanism M.” This forces clarity on whether a metric is a mediator, a proxy, or a true outcome.

Next, translate this into a decision frame: define a minimum detectable decision threshold (MDD-T) that reflects utility, not just statistical power. Example: “Ship if expected incremental profit per user > $0.12 over 90 days AND the probability of harming support tickets by >2% is <10%.” This is a utility statement. It combines effect size, uncertainty, and guardrails in one decision rule.

• ROI framing: Convert uplift into dollars using defensible unit economics (margin-adjusted revenue, LTV, variable costs). Be explicit about which costs are fixed vs variable.
• Risk framing: Add downside terms (brand risk, fairness, operational load) as constraints or penalties, not footnotes.
• Constraint framing: If engineering capacity is the bottleneck, compare interventions by “profit per engineer-week,” not just uplift.

Common mistake: treating p<0.05 as “ship.” In decision-grade practice, p-values are rarely the best decision boundary. Another mistake is optimizing an intermediate KPI that is not causally tied to the outcome (a broken KPI tree). Your practical outcome from this section: a one-page “action-to-outcome” tree with a ship/no-ship threshold and guardrails that stakeholders agree to before seeing results.

Section 5.2: Interpreting confidence/credible intervals for non-technical teams

Intervals are where trust is won or lost. Non-technical stakeholders often hear a point estimate as a promise. Your job (Milestone 2) is to make uncertainty usable: “Here’s the plausible range of outcomes, and here’s what we’d do under each.”

Use consistent language and avoid technical detours. For a confidence interval, you can say: “Based on this experiment design and sample, the data are consistent with an uplift between A and B.” For a Bayesian credible interval, you can say: “Given our model and prior, there’s a 95% probability the uplift is between A and B.” In either case, immediately translate the interval into business impact: “That corresponds to +$120k to +$480k per quarter.”

Teach teams to focus on decision-relevant slices of the interval. Example: “Our 95% interval for conversion uplift is [-0.2%, +1.1%]. That crosses zero, so we can’t rule out mild harm. However, the probability conversion uplift exceeds +0.5% is 62%, and +0.5% is our breakeven threshold.” That phrasing supports a recommendation under uncertainty (Milestone 4) without implying false certainty.

• Separate statistical uncertainty from implementation uncertainty: The interval reflects sampling variability, not rollout bugs, seasonality drift, or metric instrumentation issues.
• Use absolute numbers: “+0.6% conversion” is abstract; “+600 additional purchases per 100k users” is concrete.
• Pre-commit to the estimand: Clarify ITT vs per-protocol, time window, and population. Otherwise stakeholders will shift the question after seeing results.

Common mistakes: overemphasizing “significant/not significant,” hiding wide intervals, or presenting too many intervals without a decision lens. Practical outcome: a standard results template where every estimate is paired with (1) interval, (2) translation to dollars/users, (3) comparison to threshold, and (4) a brief risk statement.

Section 5.3: Heterogeneous treatment effects and segmentation pitfalls

Stakeholders will ask “Does it work better for segment X?” That’s a valid causal question, but it’s also where teams accidentally overclaim. Heterogeneous treatment effects (HTE) can reveal where value concentrates, but segmentation multiplies noise and invites storytelling.

Start with a disciplined approach (Milestone 3). Pre-specify a small set of segments tied to mechanism hypotheses in your causal KPI tree: new vs returning users, high vs low intent, or regions with different operational constraints. If the mechanism is “reduces onboarding friction,” then “new users” is a plausible moderator; “favorite color theme” is not.

Use interaction estimates rather than running separate experiments per segment. Report: (1) overall average treatment effect (ATE), (2) interaction term(s), and (3) segment-level estimated effects with partial pooling when possible. Hierarchical modeling (or shrinkage methods) helps prevent extreme segment estimates from dominating decisions due to small sample sizes.

• Avoid post-treatment segmentation: Do not segment by variables influenced by treatment (e.g., “users who clicked the new banner”). That creates selection bias and can flip conclusions.
• Beware the “winner’s curse”: The segment that looks best is often the noisiest estimate. Expect regression to the mean on replication.
• Decide what you’ll do with HTE: If you can’t target or operationalize the segment, HTE is interesting but not actionable.

Communicate HTE as a prioritization tool, not a guarantee: “Evidence suggests higher uplift among new users, but uncertainty is large; we recommend a targeted follow-up test with new users only.” Practical outcome: a short “HTE appendix” that lists pre-registered segments, sample sizes, adjusted intervals, and the operational action each segment would enable.

Section 5.4: Multiple testing, metric fishing, and governance guardrails

Decision-grade measurement requires governance. Without it, organizations drift into metric fishing: running many cuts, many metrics, and many stopping points until something looks good. This creates false positives, erodes trust, and eventually makes experimentation politically unsafe.

Put guardrails in place before the test begins. First, classify metrics into (1) primary outcome (the decision metric), (2) guardrails (must-not-harm constraints), and (3) diagnostics (instrumentation and mechanism checks). Tie this back to the KPI tree: the primary should be closest to the true business outcome; guardrails should reflect ethical, operational, and customer constraints.

Second, define your stopping rules. If you peek daily and stop on a good day, your false positive rate inflates. Use one of: fixed-horizon tests; group-sequential designs; or Bayesian monitoring with a pre-defined decision boundary. What matters is not the method—it is the pre-commitment and documentation.

• Multiple comparisons: If you must test many metrics or segments, apply corrections (Holm/BH) or explicitly label analyses as exploratory.
• Version control for analysis: Lock the analysis notebook/script and the metric definitions at launch. Changes must be logged with rationale.
• Metric dictionary: Maintain a single source of truth for event definitions, attribution windows, and known data quality issues.

Common mistakes: adding a “secondary metric” after seeing primary results, redefining the population midstream, or quietly excluding outliers. Practical outcome: an experimentation checklist and lightweight review process (one page) that requires: estimand, primary/guardrail metrics, randomization unit, power, stopping rule, and analysis plan sign-off.

Section 5.5: Visuals that work: uplift plots, cumulative effects, and funnels

Most stakeholder confusion comes from charts that optimize for statistical completeness instead of decision clarity. Use a small set of visuals that answer executive questions quickly: “How big is the impact?”, “How sure are we?”, “Where in the funnel did it change?”, and “Is it stable over time?”

An uplift plot should show the treatment-control difference on an absolute scale, with intervals, and a reference line for the decision threshold. Avoid stacked percentage charts that hide the baseline. If you have multiple variants, show them side-by-side with consistent axes and clearly marked primary metric.

Cumulative effect plots are ideal for time dynamics. Plot cumulative incremental conversions/revenue over time with confidence bands. This helps diagnose novelty effects (early spike then fade), ramp effects (slow adoption), and seasonality. Pair it with a simple “days to break-even” annotation that ties back to ROI framing in Section 5.1.

Funnels are where mechanism meets outcome. Show a funnel decomposition: exposure → engagement → activation → retention → revenue. For each step, show absolute counts and conversion rates. Then highlight where the causal KPI tree predicted change. If the primary outcome moved but the funnel didn’t, investigate instrumentation, attribution, or interference.

• One chart, one message: Every visual needs a headline that states the claim (“+0.7% conversion, likely above breakeven”).
• Show denominators: Sample sizes by arm and by key segment prevent overconfidence in thin data.
• Respect uncertainty visually: Error bars or bands are not decoration; they are the core of decision-grade communication.

Common mistakes: truncating axes to exaggerate effects, mixing relative and absolute changes without labeling, or showing too many metrics in a single slide. Practical outcome: a reusable slide library with three standard charts (uplift with threshold, cumulative incremental impact, funnel step changes) that matches your org’s metric definitions.

Section 5.6: Executive storytelling: claim, evidence, assumptions, next action

Your readout is not a lab report; it is a decision document. The best structure is simple and repeatable: Claim → Evidence → Assumptions/Risks → Next action. This format lets you communicate limitations without losing trust because you are explicit about what is known, what is uncertain, and what you will do about it.

Claim: State the decision recommendation in one sentence (Milestone 4). Example: “Recommend shipping to 50% of traffic for two weeks while monitoring guardrails; expected profit uplift likely exceeds breakeven.” Avoid hedging here; put uncertainty in the evidence section.

Evidence: Present the primary effect size with interval and business translation (Milestone 2): “Conversion +0.6% [0.1%, 1.1%], +$260k to +$520k/quarter.” Include one mechanism chart (funnel) and one stability chart (cumulative effect). If HTE is relevant, summarize it cautiously: “New users show higher uplift; exploratory and needs confirmation” (Milestone 3).

Assumptions/Risks: List the top 3–5 items that could change the decision: metric validity, interference, novelty effects, data exclusions, multiple testing, or operational constraints (Section 5.4). Frame them as testable: “If novelty effect fades, cumulative plot should flatten by day 10; we will re-check.”

• Next action: Specify who does what by when: rollout plan, monitoring dashboard, follow-up experiment, or instrumentation fix.
• Q&A defense (Milestone 5): Prepare answers to: “What’s the counterfactual?”, “What did we exclude and why?”, “What’s the worst-case impact?”, “Which segments lose?”, “What would change your recommendation?”

Common mistakes: burying the recommendation, presenting every metric equally, or using uncertainty as an excuse to avoid action. Practical outcome: a stakeholder-ready readout template that consistently turns causal estimates into decisions, while documenting assumptions and governance so the organization can learn safely over time.

Section 6.1: Portfolio blueprint: one strong causal project vs many weak ones

If you build only one portfolio artifact, make it a single end-to-end case study with causal rigor. One strong project beats many weak ones because it demonstrates the full chain: decision context → causal question → estimand → design → diagnostics → uncertainty → limitations → recommendation. The hiring signal is your judgment under constraints, not the number of dashboards you can produce.

Start with a business decision you can narrate in one sentence (for example: “Should we change the onboarding flow to increase 30-day activation without increasing refunds?”). Then write the estimand precisely: the average treatment effect of the new flow versus control on 30-day activation among eligible new users, with a clear assignment mechanism. Add a DAG that includes likely confounders (marketing source, device type), mediators (time-to-first-action), and selection issues (users who drop before eligibility). Explicitly state what you will and will not control for and why.

• Design choice: Prefer an A/B test; if impossible, justify a quasi-experiment (DiD with parallel trends checks, RD with manipulation tests, IV with exclusion restrictions).
• Credibility checks: Balance tests for randomization, pre-trend plots for DiD, density tests for RD, and sensitivity analysis for unobserved confounding when relevant.
• Guardrails: Define at least two (e.g., refunds, support tickets) and explain stopping rules.

Write the case study as if it were a real internal readout: assumptions, what could go wrong, and how you would instrument the missing data. This is Milestone 1: assemble a portfolio case study with causal rigor. Common mistakes: reporting uplift without confidence intervals, controlling for post-treatment variables, and “finding significance” by trying many metrics without correction or a pre-registered plan.

Section 6.2: The experimentation playbook: intake, prioritization, and cadence

Teams fail at experimentation not because they lack statistics, but because they lack a system. Your playbook is that system: how ideas enter, how they are prioritized, how decisions are made, and how results are communicated. This directly supports Milestone 2 (reusable templates) and Milestone 3 (operating model proposal).

Define an intake process that forces clarity. Every request should include: decision to be made, user population, primary metric, expected direction, and constraints (engineering effort, launch date, legal risk). Do not accept “test button color” unless the requester can tie it to a behavioral mechanism and an estimand that matters. Next, add prioritization criteria beyond “leader wants it”: expected impact, confidence, effort, and risk. If your org is mature, add opportunity cost (what you are not testing) and learning value (will it reduce uncertainty for future bets?).

• Cadence: weekly triage, biweekly experiment review, monthly metric health review.
• Roles: product owns the decision; engineering owns implementation and logging; decision science owns design, analysis plan, and inference; data platform owns metric definitions and reliability.
• Artifacts: brief → analysis plan → launch checklist → readout → decision log.

The most practical element is a “definition of done.” An experiment is not done when the p-value is computed; it is done when you have a decision recommendation, a documented limitation, and the metric definitions are preserved for reuse. Common mistakes: shipping without an exposure event, changing the primary metric mid-flight, and relying on ad hoc dashboards rather than a consistent readout format.

Section 6.3: Data and instrumentation requirements for trustworthy measurement

Trustworthy causal inference is inseparable from instrumentation. Many “failed” experiments are actually data failures: missing exposure logs, inconsistent user identifiers, delayed events, or metric definitions that drift over time. As you transition from BA to decision science, your edge is knowing how operational processes create data artifacts—and how those artifacts bias estimates.

Start with a measurement map: for each metric, specify the event(s), the entity (user, account, session), the time window, and the inclusion/exclusion rules. Then define the assignment and exposure: assignment is who was randomized (or selected by policy), exposure is who actually saw the treatment. You need both to run intent-to-treat (ITT) and treatment-on-treated (TOT) analyses correctly, and to diagnose noncompliance.

• Minimum logging: assignment_id, variant, timestamp; exposure event with the same keys; outcome events with stable schemas.
• Identity resolution: document how anonymous users map to accounts and what happens when users have multiple devices.
• Metric layer: a single source of truth for metric definitions with versioning and tests.

Engineering judgment appears in tradeoffs: logging everything increases cost and privacy risk, but logging too little makes inference impossible. Write a launch checklist that includes event validation (counts by variant, missingness, latency), sample ratio mismatch checks, and guardrail monitoring. Common mistakes: calculating conversion without deduplicating users, attributing outcomes that occur before exposure, and ignoring interference (spillovers) when users share households, teams, or marketplaces.

Section 6.4: Governance: ethics, fairness, privacy, and stakeholder alignment

An experimentation program without governance becomes either reckless (harmful tests) or paralyzed (no one trusts results). Governance is the operating model layer that aligns incentives, ethics, and accountability. It should be lightweight enough to keep velocity, but firm enough to prevent predictable failures.

Establish decision rights and escalation paths. For low-risk UI tweaks, a standard review is enough. For anything affecting pricing, credit, healthcare, minors, or sensitive traits, require a formal ethics and privacy review before launch. Explicitly define “do-not-test” zones and how to handle informed consent where applicable. Fairness should be treated as an outcome and a constraint: measure heterogeneous treatment effects across key segments, and decide in advance what disparities are unacceptable versus expected due to baseline differences.

• Ethics checklist: potential harm, reversibility, vulnerable groups, transparency, and mitigation plan.
• Privacy checklist: data minimization, retention, access controls, and purpose limitation.
• Stakeholder alignment: pre-commit to success metrics, guardrails, and stopping rules to avoid post-hoc politics.

Common mistakes: using protected characteristics as targeting variables without justification, letting teams “shop” for metrics until they find a win, and ignoring long-term effects because the experiment window is short. Good governance includes a decision log: what was decided, why, and what uncertainty remains. That log becomes institutional memory and prevents repeating expensive mistakes.

Section 6.5: Interview kit: whiteboard DAGs, experiment design, and pitfalls

Interviewers are testing whether you can reason causally in real time, communicate clearly, and spot pitfalls before they ship. Prepare a compact kit you can reproduce on a whiteboard: a DAG workflow, an experiment design checklist, and a set of “classic traps” you proactively call out. This is Milestone 4: prepare for interviews with causal questions, tradeoffs, and critiques.

For a DAG prompt, practice a 60-second structure: define the decision and estimand; list key variables; draw arrows for causal relationships; identify backdoor paths and what you need to adjust for; flag mediators you should not control for. Then translate into an analysis plan: regression adjustment (pre-treatment covariates only), stratification, or CUPED-like variance reduction if appropriate, with clear assumptions.

• Experiment design: unit of randomization, eligibility, power/MDE, primary metric, guardrails, duration, and stopping rules.
• Pitfalls to mention: sample ratio mismatch, novelty effects, noncompliance, interference, peeking, multiple comparisons, and metric gaming.
• When no A/B test: propose DiD/RD/IV with the key validity checks and what would falsify your assumptions.

Bring one printed (or memorized) readout narrative: “Here is what we tested, what we learned, what we recommend, and what we still don’t know.” The strongest candidates do not overclaim; they quantify uncertainty and explain limitations without sounding evasive.

Section 6.6: Career positioning: translating BA experience into decision science impact

Your BA background is an advantage if you position it correctly: you already understand stakeholder incentives, operational constraints, and how metrics get misused. Decision science adds the causal discipline to make those metrics decision-grade. Milestone 5 is to ship a 30-60-90 day plan that proves you can land and deliver value quickly.

Translate your experience into outcomes: instead of “built dashboards,” say “created metric definitions and alerting that reduced decision latency,” or “standardized funnel measurement to prevent contradictory KPIs.” Then add the causal layer: “introduced analysis plans and guardrails that reduced false wins.” Hiring teams want to know you can operate across product, engineering, and leadership without losing statistical integrity.

• 30 days: learn the product, audit key metrics, review past experiments for repeatable issues (logging gaps, shifting definitions), propose a brief + readout template, and start a decision log.
• 60 days: run (or redesign) one high-leverage experiment end-to-end, establish SRM checks and launch checklists, and create a prioritized backlog with stakeholders.
• 90 days: propose the operating model: cadence, governance tiers, metric layer ownership, and a roadmap for quasi-experimental capability where randomization is limited.

Common mistakes in transition plans are being too tool-focused (“I’ll build a causal forest”) or too vague (“I’ll improve experimentation”). Your plan should name the few systems you will install—templates, checks, governance, cadence—and the first decision you will de-risk. That is what makes you a decision scientist, not just an analyst with new vocabulary.