Q-Learning & Policy Gradients Deep Dive for RL Mastery

Every RL project starts with an interface contract between the agent and the environment. At time step t, the agent receives an observation (often called a state) s_t, chooses an action a_t, and then the environment returns a reward r_t+1 and the next state s_t+1. This interaction produces a trajectory (or rollout): (s₀, a₀, r₁, s₁, a₁, r₂, ...).

When you “map a task to an MDP,” the first practical step is to write down three lists: (1) what the agent can observe (candidate state variables), (2) what actions are available (discrete buttons, continuous torques, API calls), and (3) what reward signal is returned. Do this before selecting an algorithm. Many failures that look like “the learning algorithm is unstable” are actually interface problems: rewards that arrive too late, action spaces that are poorly scaled, or observations that omit critical context.

A useful workflow is to prototype the environment with a random policy and log trajectories. Inspect them like you would inspect a dataset: reward distribution, episode lengths, frequency of terminal events, and whether actions have visible effects on state transitions. If the reward is almost always zero, learning will be slow regardless of method. If the reward spikes for unintended behaviors, you are setting up reward hacking (the agent finds loopholes that maximize the reward without solving the intended task).

• Practical outcome: You can produce and visualize trajectories, confirming that state transitions respond to actions and that the reward is informative.
• Common mistake: Treating observations as “the state” without checking whether they contain enough information to predict what happens next.

This trajectory-centric view also sets up later ideas: returns are computed along trajectories, Bellman targets are one-step slices of trajectories, and evaluation compares trajectory statistics across many runs.

An MDP formalizes the RL loop with five components: states S, actions A, transition dynamics P(s'|s,a), reward function R(s,a,s') (or R(s,a)), and discount factor γ. In an MDP, the key assumption is the Markov property: given the current state s and action a, the next state distribution does not depend on earlier history. This assumption is not a philosophical detail; it determines whether your learning targets are well-defined and whether your algorithm’s convergence claims apply.

In practice, your environment rarely hands you a perfect Markov state. Many tasks are partially observable: the agent sees an observation o_t that is a noisy or incomplete view of the true state. If you treat o_t as s_t, you may create apparent non-stationarity: identical observations lead to different outcomes depending on hidden variables (e.g., velocity unobserved, cooldown timers hidden, or other agents’ private states). This is a classic “it won’t learn” cause.

Engineering judgment here means deciding how to restore approximate Markov-ness. Common tactics include stacking recent frames, including previous action and reward, adding explicit memory (RNN policies), or redesigning the observation to include the missing variables. Before changing algorithms, run a diagnostic: can you build a simple predictor that estimates s_t+1 or r_t+1 from (o_t, a_t)? If prediction error is high due to missing context, your RL agent will struggle similarly.

• Practical outcome: You can specify (S, A, P, R, γ) for your task and list which parts are approximations.
• Common mistake: Debugging learning-rate schedules when the observation violates the Markov assumption and requires memory or richer state.

Clear MDP framing also helps you diagnose reward and dynamics issues: if P changes over time (drifting simulator, randomized latency), your environment is non-stationary; if R is misaligned, you can expect reward hacking even when learning “works.”

RL optimizes return: the cumulative reward over time. For a trajectory starting at time t, the discounted return is G_t = r_t+1 + γ r_t+2 + γ² r_t+3 + .... Discounting is not just “preference for immediate reward.” It is also an engineering tool that controls horizon length and stabilizes learning by ensuring the infinite sum is finite in continuing tasks.

The discount factor γ trades off long-term planning against variance and sensitivity to modeling errors. Higher γ (e.g., 0.99) makes the agent care about delayed outcomes but increases the effective horizon; value estimates become harder because far-future rewards are noisy and depend on many uncertain transitions. Lower γ (e.g., 0.9) shortens the horizon and often improves stability, but may produce myopic behavior (optimizing immediate reward at the cost of long-term success). A practical rule is to relate γ to a characteristic time scale: the effective horizon is roughly 1/(1-γ) steps. If meaningful consequences occur 200 steps later, γ=0.99 (horizon ~100) may already be borderline; you might need γ=0.995 or reward shaping that provides earlier signal.

Episodic tasks end in terminal states (game over, goal reached), so the return naturally stops. Continuing tasks (inventory control, thermostat regulation) do not end; you either use discounting or average-reward formulations. When implementing algorithms later, you must handle termination carefully: bootstrapped targets should not add value beyond terminal states, and evaluation should report both per-episode return and episode length where relevant.

• Practical outcome: You can compute returns from logged trajectories and explain how changing γ changes the planning horizon and learning variance.
• Common mistake: Using a high γ with sparse rewards and then misattributing slow learning to “bad exploration,” when the signal-to-noise ratio of returns is the bottleneck.

Discounting decisions are also a reward-design decision: if a reward is delayed, a high γ or better intermediate rewards are often required to make credit assignment feasible.

Value functions translate “maximize return” into quantities you can estimate and improve. For a policy π, the state-value is V^π(s) = E[G_t | s_t=s], and the action-value is Q^π(s,a) = E[G_t | s_t=s, a_t=a]. These definitions matter because most RL algorithms are ways of approximating one of these functions and using it to improve the policy.

The Bellman equations give a recursive structure. For example, V^π(s) = E[ r + γ V^π(s') ] under transitions induced by π. For control, the optimal action-value satisfies Q*(s,a) = E[ r + γ max_a' Q*(s',a') ]. In practice, this recursion is your learning target: a one-step bootstrapped estimate of long-term return.

Thinking in Bellman operators helps you reason about why algorithms fail. Bootstrapping creates a feedback loop: your target depends on your current estimate. With tabular representations and sufficient exploration, Q-learning converges; with function approximation, the “deadly triad” (bootstrapping + off-policy learning + function approximation) can cause divergence or oscillation. Even before deep networks, you can see instability if learning rates are too high or if your target changes too quickly.

Engineering takeaway: when you later implement Q-learning and DQN, treat the Bellman target as a signal that needs stabilization. That is why target networks (slow-moving copies for bootstrapping) and replay buffers (decorrelate samples) exist; they are not optional tricks, they address the moving-target problem implied by Bellman recursion.

• Practical outcome: You can write the Bellman expectation and optimality equations and identify what is being approximated in a learning update.
• Common mistake: Debugging network architecture when the real issue is an inconsistent target definition at terminals, or bootstrapping from invalid next-states.

Once you can articulate your targets with Bellman equations, you can also design sanity checks: compute one-step TD errors on a held-out buffer and confirm they decrease as learning progresses.

A policy maps states to actions. Deterministic policies choose a single action a=π(s); stochastic policies define a distribution π(a|s). Both are useful, but you should choose deliberately. Value-based methods often act greedily with respect to Q (deterministic at decision time), while policy-gradient methods commonly learn stochastic policies because differentiating through action sampling is natural and because stochasticity can be essential in environments with uncertainty or adversaries.

Exploration is the practical problem of collecting trajectories that reveal good actions before you already know them. Exploitation is using current knowledge to get reward. The tension is unavoidable: if you always exploit, you may never discover better actions; if you always explore, you may never consolidate gains. For tabular Q-learning, a classic approach is ε-greedy: with probability ε choose a random action, otherwise choose argmax_a Q(s,a). The critical engineering detail is the schedule: start with higher ε to discover behaviors, then decay to a smaller value to stabilize performance. Too-fast decay causes premature convergence to suboptimal policies; too-slow decay yields noisy learning curves and weak final performance.

When actions are continuous, exploration often comes from adding noise (Gaussian, Ornstein–Uhlenbeck) or learning a stochastic policy directly. The right choice depends on dynamics: if small action changes produce large state changes, your exploration noise must be scaled down; otherwise trajectories become chaotic and rewards uninformative.

Exploration also interacts with reward design. Sparse rewards make random exploration ineffective; you may need curriculum learning, shaped rewards, or better state representations. Misaligned rewards invite reward hacking: the agent exploits loopholes, which is “exploration success” but task failure. Treat strange high-reward behaviors as a signal to audit R, not as a sign the algorithm is clever.

• Practical outcome: You can choose a policy class (deterministic vs stochastic) and define an exploration strategy with an explicit schedule and rationale.
• Common mistake: Reporting evaluation results using an exploratory policy (e.g., ε-greedy with ε=0.1), which understates true performance and increases variance.

Later chapters will build on this: Q-learning needs explicit exploration; policy gradients “bake in” exploration through stochastic policies but require variance control.

RL results are noisy because trajectories are random: initial states vary, transitions may be stochastic, and exploration policies deliberately inject randomness. Without a careful evaluation protocol, you can convince yourself an idea works when it doesn’t, or miss real improvements. Treat evaluation as part of the system design.

Start by separating training from evaluation. During training, your agent explores. During evaluation, use a fixed, mostly greedy policy (e.g., ε=0 for value-based methods, or mean action / low-entropy sampling for stochastic policies) to measure capability, not exploration. Report metrics that match the task: average episodic return, success rate, constraint violations, episode length, and possibly cost metrics (energy, collisions). Always include uncertainty: mean and standard error over multiple episodes and multiple random seeds.

Seeds matter because RL can be sensitive to initialization, environment stochasticity, and data order. A practical minimum is 5–10 seeds for algorithm comparisons, and consistent environment versions. Log everything needed to reproduce a run: code commit, hyperparameters, reward coefficients, observation normalization, and hardware details if relevant. If you change the reward function, treat it as a new experiment; reward tweaks can dominate algorithmic differences.

Baselines prevent self-deception. Include at least one simple baseline (random policy, heuristic controller, or previous best method) and a sanity-check baseline (e.g., “no learning,” frozen network) to catch evaluation bugs where reward increases due to environment drift or logging errors.

Finally, use evaluation to diagnose failure modes. If training return rises but evaluation return does not, you may have overfitting to exploratory behavior or a bug in action selection. If performance collapses after improving, suspect non-stationary dynamics, unstable bootstrapping targets, or reward hacking that emerges later in training. If learning is inconsistent across seeds, investigate partial observability, overly aggressive learning rates, or high-variance returns.

• Practical outcome: You can set up metrics, seeds, and baselines that make learning curves interpretable and comparable.
• Common mistake: Comparing two methods on a single seed and declaring victory, despite high run-to-run variance.

With these evaluation habits in place, the next chapters’ algorithms—tabular Q-learning, DQN, REINFORCE, and actor-critic methods—become debuggable engineering systems rather than fragile demos.

Dynamic programming gives the conceptual target for RL: if you knew the environment dynamics (transition probabilities and expected rewards), you could compute optimal values by solving the Bellman optimality equations. For a fixed policy \(\pi\), the state-value function satisfies \(V^{\pi}(s)=\mathbb{E}[r+\gamma V^{\pi}(s')\mid s]\). For control, the optimal value satisfies \(V^*(s)=\max_a \mathbb{E}[r+\gamma V^*(s')\mid s,a]\). DP algorithms such as policy iteration and value iteration repeatedly apply these Bellman backups to push estimates toward a fixed point.

In model-free RL, you do not have the transition model, so you cannot take expectations directly. Temporal-difference learning replaces the DP expectation with a sampled backup. The core idea is a bootstrapped target: you update current estimates toward a target that includes your own next-step estimate. For a state-value function, TD(0) uses the target \(r+\gamma V(s')\), producing the error \(\delta=r+\gamma V(s')-V(s)\), then updates \(V(s)\leftarrow V(s)+\alpha\delta\).

Two engineering implications matter immediately. First, bootstrapping reduces variance compared to Monte Carlo returns (which wait until episode end), but introduces bias that depends on the quality of your current estimate. Second, because the target depends on the current parameters, large step-sizes or rapidly changing policies can cause oscillation—an issue you will later see in deep Q-learning as well. In tabular settings, you can often stabilize learning with careful exploration and step-size schedules, plus basic diagnostics like visit counts and TD error statistics.

From a workflow standpoint, adopt a loop that logs at least: episode return, moving-average return, number of steps per episode, and a notion of coverage (how many state–action pairs have been visited). These are the minimum signals to tell whether the Bellman backups are improving values or simply chasing noise.

Q-learning is TD learning applied to the action-value function \(Q(s,a)\), enabling direct greedy control without a separate model. The optimal action-value function satisfies the Bellman optimality equation:

\(Q^*(s,a)=\mathbb{E}[r+\gamma\max_{a'}Q^*(s',a')\mid s,a]\).

Q-learning approximates this fixed point with sampled updates. After observing a transition \((s,a,r,s')\), define the TD target \(y=r+\gamma\max_{a'}Q(s',a')\) and update:

\(Q(s,a)\leftarrow Q(s,a)+\alpha\big(y-Q(s,a)\big).\)

The intuition is “one-step lookahead”: you treat your best estimated next action as if you will take it, regardless of what action you actually take during exploration. This is the key off-policy property: the behavior policy that generates data (often epsilon-greedy) can differ from the target policy you are learning (the greedy policy with respect to \(Q\)).

Off-policy bootstrapping is powerful but can be brittle if exploration is poor or if the max operator systematically overestimates values (in tabular settings this is usually manageable; later, in function approximation, it motivates Double Q-learning). Practically, ensure you initialize Q-values sensibly. Optimistic initialization (e.g., starting Q to a high value) can encourage exploration in deterministic tasks, but can also mislead in stochastic tasks by causing long periods of chasing phantom high returns. A safer default is zero initialization plus explicit exploration scheduling.

Implementation details that matter: (1) represent \(Q\) as a 2D array indexed by discrete state and action; (2) ensure terminal transitions do not bootstrap (set \(\max_{a'}Q(s',a')=0\) when done); (3) keep separate random seeds for environment and agent when debugging. Finally, verify that your greedy action selection and your update indexing match—off-by-one errors in state encoding are among the most common reasons tabular Q-learning “does nothing.”

SARSA is the on-policy counterpart to Q-learning. It updates \(Q(s,a)\) toward the return you actually expect to get under your current behavior policy. Given \((s,a,r,s',a')\) where \(a'\) is chosen by the same policy used for acting, SARSA uses target \(y=r+\gamma Q(s',a')\) and update \(Q(s,a)\leftarrow Q(s,a)+\alpha(y-Q(s,a))\). The difference from Q-learning is subtle but important: SARSA bootstraps from the action you will really take, including exploratory moves, while Q-learning bootstraps from the greedy action regardless of exploration.

This distinction shows up in safety-sensitive or “cliff” environments. With epsilon-greedy behavior, Q-learning learns values assuming greedy behavior, but the agent still sometimes explores; near hazards, those exploratory steps can be costly. SARSA internalizes that risk and often learns a more conservative policy. When you compare SARSA vs Q-learning, treat it as a comparison of on-policy vs off-policy learning objectives, not merely “two update formulas.”

Expected SARSA bridges the two by replacing the sampled next action in SARSA with the expectation under the behavior policy: \(y=r+\gamma\sum_{a'}\pi(a'\mid s')Q(s',a')\). This often reduces variance while remaining on-policy. In engineering terms, expected SARSA can be a great default in small discrete action spaces because it smooths learning without losing the on-policy interpretation.

Control variants worth knowing: (1) Greedy in the limit with infinite exploration (GLIE): a schedule where exploration decays slowly enough that every state–action pair continues to be visited while the policy becomes greedy asymptotically; (2) Exploring starts (in episodic tasks): force random initial states/actions to improve coverage; (3) Q(\(\lambda\)) with eligibility traces: spreads credit across multiple steps, often speeding up learning in sparse reward settings. Even if you do not implement traces now, recognize the pattern: better credit assignment usually means either multi-step targets or richer exploration.

Exploration is not an afterthought; it is the data-generation mechanism for your learning problem. In tabular Q-learning, poor exploration usually manifests as Q-values that look reasonable in visited regions but are arbitrary elsewhere, causing sudden policy failures when the agent drifts into rarely visited states. Start with epsilon-greedy: with probability \(\epsilon\) choose a random action, otherwise choose \(\arg\max_a Q(s,a)\). It is simple, robust, and works well when actions are few and similarly “plausible.”

Scheduling \(\epsilon\) is where engineering judgment enters. A common practical schedule is linear decay from 1.0 to 0.1 over some fraction of training, then to a small floor like 0.01. For episodic tasks, you can decay per episode; for continuing tasks, decay per step. The diagnostic you want is visitation coverage: if many state–action pairs remain unvisited, decaying \(\epsilon\) too quickly will freeze learning. Conversely, if returns plateau while TD error remains high, \(\epsilon\) may be too large, preventing exploitation of learned structure.

Softmax (Boltzmann) exploration chooses actions with probability proportional to \(\exp(Q(s,a)/\tau)\), where \(\tau\) is a temperature. Softmax can be less “spiky” than epsilon-greedy because it prefers better actions more smoothly, but it is sensitive to the scale of Q-values; reward scaling or large \(\gamma\) can make Q magnitudes large and cause near-deterministic behavior unless \(\tau\) is tuned. If you use softmax, log action entropy to ensure it decays as intended.

Upper-confidence ideas (UCB-style) add an explicit bonus for uncertainty, e.g., choose \(\arg\max_a \big(Q(s,a)+c\sqrt{\log N(s)/N(s,a)}\big)\). In tabular settings, UCB can provide principled exploration by directing the agent to under-sampled actions. The common mistake is applying UCB naively in nonstationary learning (where Q-values are changing); the bonus helps coverage, but you still need stable learning rates and clear stopping criteria. A pragmatic approach is to use epsilon-greedy for baseline experiments, then try UCB-like bonuses if you see persistent under-exploration of rare-but-important actions.

Three knobs dominate tabular stability: the step-size \(\alpha\), discount \(\gamma\), and the effective scale of rewards. Step-size determines how aggressively you chase the TD target. Too large: oscillations, divergence-like behavior (values bounce), and high sensitivity to stochastic rewards. Too small: painfully slow learning and apparent “stuck” curves. A strong default for many small tasks is \(\alpha\in[0.05,0.5]\), but the right choice depends on reward variance and how often each state–action pair is visited.

A practical technique is per-(s,a) step-size decay, such as \(\alpha_{t}(s,a)=1/N_t(s,a)\) or \(\alpha_{t}(s,a)=\alpha_0/(1+kN_t(s,a))\). This often improves convergence in tabular domains because frequently visited pairs naturally get smaller updates. If you keep a constant \(\alpha\), monitor whether Q-values keep drifting late in training; drifting suggests you are not settling to a fixed point under your exploration regime.

The discount factor \(\gamma\) sets the planning horizon. \(\gamma\approx 0\) makes the agent myopic; \(\gamma\to 1\) makes long-term outcomes dominate and increases value magnitude, which can amplify instability and slow propagation of information in sparse reward tasks. In episodic tasks with bounded length, \(\gamma\in[0.95,0.99]\) is common; in short-horizon problems, \(0.9\) may be sufficient and easier to learn. Importantly, changing \(\gamma\) changes what “optimal” means, so treat it as a modeling decision, not only a tuning parameter.

Reward scaling is the quiet stabilizer. If rewards are extremely large (or inconsistent across tasks), Q-values become large, gradients in the tabular update become large, and exploration mechanisms like softmax become brittle. A practical guideline: normalize rewards so typical returns per episode are on the order of 1 to 100, then tune \(\alpha\). When debugging, also check sign conventions (accidentally negating a reward) and terminal rewards (double-counting at episode end). Many “hyperparameter issues” are actually reward definition issues.

Tabular RL is uniquely debuggable because you can inspect everything. Start with visitation. Track \(N(s)\) and \(N(s,a)\) and visualize them (as tables or heatmaps). If learning fails, the first question is often: did the agent even visit the states where reward is available? Sparse reward tasks frequently look like “Q-learning doesn’t work” when the real problem is that the exploration schedule never reaches reward.

Next, inspect the Q-table directly. For a small gridworld, print \(\max_a Q(s,a)\) as a value map and print the greedy policy arrows. If the policy is nonsensical, check state indexing, action semantics, and terminal handling. A classic bug is bootstrapping from terminal states (forgetting to zero the next-state value), which inflates values and can produce optimistic loops.

Learning curves should be paired with diagnostics. Plot: (1) episodic return with a moving average; (2) episode length; (3) average TD error magnitude; (4) fraction of greedy actions taken (or policy entropy). Interpretation examples: if return improves but episode length explodes, the agent may be reward-hacking a living reward; if TD error stays high while return is flat, \(\alpha\) may be too large or rewards too noisy; if return is noisy but TD error decreases smoothly, the environment may be stochastic and you need longer evaluation windows.

Use evaluation correctly: during evaluation episodes, set exploration to zero (or near-zero) and do not update Q. Mixing training and evaluation hides whether your learned greedy policy is actually good. Finally, run a “sanity environment” before complex tasks: a two-state MDP where the optimal policy is obvious. If Q-learning cannot solve that in a few hundred steps, you almost certainly have an implementation bug rather than a conceptual limitation.

In tabular Q-learning, the parameterization is “one parameter per (s,a).” Function approximation replaces that with a shared parameter vector θ and a model Q(s,a;θ). Two common forms are linear approximation and neural approximation. In linear Q-learning, you choose features ϕ(s,a) and set Q(s,a;θ)=θᵀϕ(s,a). The learning update becomes a gradient step on the TD error: θ ← θ + α·δ·∇θQ(s,a;θ), where δ = r + γ max_a' Q(s',a';θ) − Q(s,a;θ).

Linear features force you to think explicitly about representation. Good features are (a) bounded or normalized, (b) informative for predicting long-term return, and (c) aligned with generalization you actually want (similar states share features). A common mistake is using raw, unscaled inputs (pixel values, large-magnitude coordinates) and then blaming “RL instability” when the real issue is poor conditioning.

Neural Q-functions (DQN-style) replace hand-crafted features with learned representations. This increases capacity and reduces manual feature work, but also increases the chance of chasing noise and amplifying feedback loops. Practical outcome: before deep networks, implement a small linear baseline. It will expose whether the problem is fundamentally about exploration/reward design or about representational capacity. You can also measure approximation error by holding out transitions and computing prediction error of Q on those transitions, or by checking whether the greedy policy induced by Q is stable under small perturbations of the input.

• Rule of thumb: if a linear model with reasonable features cannot learn at all, a deep model often learns “something” but may be unreliable; fix the task signal and scaling first.

The “deadly triad” names three ingredients that, together, can cause divergence: bootstrapping (targets depend on current estimates), off-policy learning (learning about one policy while sampling from another), and function approximation (shared parameters generalize updates). Tabular Q-learning is bootstrapped and off-policy, yet typically stable under standard assumptions; the difference is that tabular updates do not generalize across unrelated states, so errors do not propagate as aggressively.

With approximation, one bad target can contaminate many state-action values because they share weights. Consider a simple example: two regions of state space share features (or a network representation) but have very different optimal values. A large TD error in one region updates parameters that also control values in the other region, pushing those values in the wrong direction. Bootstrapping then uses these corrupted estimates as future targets, and off-policy sampling can keep visiting the problematic region even if the current greedy policy would avoid it. The result can be value blow-up or oscillation.

Concrete signs you are in the deadly triad regime: (1) the average TD error decreases for a while, then spikes and never recovers; (2) Q-values drift to extreme magnitudes while rewards remain small; (3) performance collapses abruptly after improving. Engineering response: reduce one triad component. In practice, you rarely remove bootstrapping entirely, but you can soften it (n-step returns, target networks), reduce off-policy mismatch (more on-policy data, smaller replay “staleness”), or constrain approximation (regularization, smaller networks, bounded outputs).

Function approximation failure is often a data problem disguised as an optimization problem. Your Q-learner trains on a distribution of transitions induced by behavior (exploration policy, replay buffer contents), but it is evaluated under the greedy policy implied by the current Q-values. As Q changes, the induced policy changes, and the state-action distribution shifts. When the learner trains on yesterday’s distribution but is used on today’s policy, errors appear in regions with little coverage—and bootstrapping can make those errors self-reinforcing.

The max operator adds a second issue: overestimation bias. If each action value estimate has noise, taking max_a Q(s,a) tends to select actions whose estimates are accidentally high. That makes the target systematically optimistic, inflating values even if the true returns are modest. This bias is especially harmful under approximation because the same parameters produce correlated errors across actions and states.

Practical mitigation patterns include: (a) Double Q-learning, which decouples action selection and action evaluation (select argmax with one set of parameters, evaluate with another); (b) conservative target smoothing (e.g., entropy regularization in policy methods, or clipped double Q targets in some actor-critic variants); and (c) coverage checks: inspect how often each action is sampled per state region (or per cluster of observations). A common mistake is attributing instability to “learning rate too high” when the underlying issue is that the max operator is exploiting regions where the model is extrapolating.

Decision point for your project: if your task has large action spaces, continuous actions, or severe distribution shift (e.g., rare states with high reward), policy-based or actor-critic approaches may be more robust than pure max-based value learning.

Before adding replay buffers, target networks, or advanced algorithms, apply basic stabilization. Many “algorithmic” failures are actually scale failures. If observations vary over wildly different magnitudes, gradients become ill-conditioned: some features dominate, others are ignored, and small changes in weights produce large changes in Q. If rewards are unbounded or heavy-tailed, TD targets become unstable and the optimizer chases outliers.

Observation normalization: for vector inputs, maintain running mean and variance and normalize to roughly zero mean and unit variance. For images, standardize pixel ranges (e.g., [0,1]) and consider frame stacking or differencing carefully—overly correlated inputs can slow learning and amplify bootstrapping errors. Reward normalization is trickier because it changes the meaning of the return; a safer pattern is reward clipping to a bounded range (common in Atari-style tasks) or scaling rewards by a constant so that typical returns are in a manageable range (e.g., tens, not millions).

Gradient clipping (global norm or per-parameter) can prevent rare TD errors from causing catastrophic parameter jumps. This is not a cure for a wrong target, but it buys time for diagnostics and can prevent replay buffers from filling with transitions generated by a temporarily broken policy. Practical workflow: start with normalization + clipping, log value scales early, then decide whether you need deeper stabilization techniques.

• Common mistake: clipping rewards without realizing the task depends on reward magnitudes (e.g., proportional costs). If relative magnitudes encode priorities, prefer scaling over hard clipping.

Q-learning is a fixed-point iteration: it tries to find Q such that Q = T*Q, where T* is the optimal Bellman operator. In the tabular case, under standard assumptions, repeated application of the Bellman update converges toward that fixed point. With approximation, you are no longer applying T* exactly—you are applying it and then projecting back into the function class representable by your model. The combination “Bellman update + projection” may not be a contraction, so the iteration can cycle or diverge.

Target construction is therefore not a detail; it is the core stabilizer. A key pattern is the fixed target network: hold a copy of parameters θ⁻ constant for some period and build targets with it: y = r + γ max_a' Q(s',a';θ⁻). This reduces the moving-target problem: the model is not simultaneously changing both sides of the regression as quickly. Another pattern is n-step targets, which reduce reliance on bootstrapped estimates by incorporating more real rewards before bootstrapping, often improving bias-variance tradeoffs.

To reason about whether your targets are sane, check three things: (1) boundedness (are targets within an expected range given reward scale and γ?), (2) consistency (do targets change smoothly over training, or jump violently?), and (3) coverage (are targets being computed in state regions where the model is extrapolating due to lack of data?). If these fail, consider whether you should switch approaches: policy gradients optimize performance directly and avoid a hard max backup, which can be beneficial in continuous control or when approximation error dominates.

When Q-learning breaks with approximation, you need instrumentation that points to the failing link: targets, gradients, or data distribution. Start by logging TD error statistics: mean, median, and high percentiles of |δ|. A stable learner typically shows a noisy but bounded TD error distribution. If the 99th percentile grows steadily, you are likely seeing rare outliers (often from distribution shift or unnormalized inputs) driving instability.

Next, log value scale: histograms of Q(s,a) and targets y. Compare these to a back-of-the-envelope bound: if rewards are roughly in [−1,1] and γ=0.99, then typical optimal returns are on the order of 1/(1−γ)≈100. If your Q-values are in the tens of thousands, you are not “learning faster”—you are diverging or overestimating. Also monitor advantage gaps (difference between max and mean action value); abnormally large gaps can indicate the max operator is exploiting noise.

Finally, inspect gradient pathologies: gradient norm over time, per-layer gradient norms (for deep nets), and parameter update magnitudes relative to parameter norms. Exploding gradients often correlate with exploding targets, but they can also come from poor feature scaling. Vanishing gradients can appear when outputs saturate (e.g., bounded activations) or when rewards are too small after scaling/clipping.

Diagnosis workflow you can apply immediately: (1) freeze the policy (stop acting greedily) and evaluate Q on a fixed batch to separate learning instability from distribution shift; (2) temporarily remove the max (evaluate a fixed action) to see if overestimation is the trigger; (3) reduce bootstrapping strength (smaller γ or n-step) and check whether stability returns. The practical outcome is faster iteration: you stop guessing and start isolating the mechanism that makes your Q updates non-contractive in your chosen function class.

Section 4.1: DQN architecture and dataflow: acting vs learning

A DQN agent is best understood as two loops running at different “temperatures.” The acting loop interacts with the environment to generate transitions, while the learning loop samples past transitions to update the Q-network. Mixing these responsibilities is a common beginner mistake: if you update from the most recent trajectory only, your gradients see highly correlated data and the bootstrap targets shift too quickly.

Concretely, the acting loop uses an exploration policy such as epsilon-greedy over Q_online(s, a). Each step produces a transition tuple (s, a, r, s', done) that is appended to a replay buffer. The learning loop wakes up every k environment steps (often 1–4) and performs one or more gradient updates on mini-batches sampled from the buffer.

• Online network: parameters b8, used for action selection and trained by SGD.
• Loss: mean squared (or Huber) TD error: L = E[(y - Q(s,a;θ))^2].
• Target: y = r + γ(1-done) max_a' Q_target(s', a') (basic DQN).

Engineering judgment: decouple acting and learning with a buffer warmup (e.g., 5k–50k steps) before training begins, otherwise early updates overfit a tiny, biased dataset and can push Q-values into extreme ranges. Also, log Q-value magnitudes; exploding Q-values often indicate reward scaling issues, too large a learning rate, or a broken terminal handling (forgetting the (1-done) mask).

Practical outcome: you should be able to sketch your dataflow as: env  transitions  replay buffer  batch sampler  TD targets  loss  optimizer step. If any arrow is ambiguous in code, it is a future bug.

Section 4.2: Experience replay: decorrelation, coverage, and pitfalls

Experience replay makes deep Q-learning look less like online control and more like supervised learning on an evolving dataset. Its first job is decorrelation: sampling random mini-batches breaks up sequential dependencies so gradients are not dominated by a single trajectory. Its second job is coverage: the buffer retains rare but informative events (e.g., success states, failures, sparse reward transitions) long enough for the learner to revisit them.

A practical replay buffer has capacity (often 1e5–1e6 transitions), uniform sampling (initially), and stores enough to reconstruct targets: observation, action, reward, next observation, done. If observations are images, store them compactly (uint8 frames) and reconstruct stacks on sample; memory pressure is a real limiter, and “it fits in RAM” is not the same as “it samples fast enough.”

• Pitfall: distribution drift. Your training distribution is the buffer mixture, not the current policy distribution. If epsilon decays too quickly, the buffer becomes narrow and learning plateaus.
• Pitfall: terminal bugs. Not masking terminals or mishandling time-limit truncations changes the MDP and injects false bootstrapping targets.
• Pitfall: replay imbalance. In sparse reward tasks, uniform replay may rarely sample reward transitions; later sections address prioritized replay and n-step returns as remedies.

Engineering judgment: choose a sampling strategy consistent with your goal. If you want stable evaluation, keep acting exploratory but evaluate with a separate greedy (or low-epsilon) policy. Mixing evaluation episodes into the replay buffer can pollute the dataset (and is usually unnecessary). Finally, measure throughput: if your learner cannot keep up with environment steps, you effectively reduce updates per sample and change the algorithm.

Practical outcome: with replay, learning curves typically become smoother, and the agent can improve even when the acting policy has moved on. Without replay, DQN-like methods often oscillate or diverge under function approximation.

Section 4.3: Target networks and lagged bootstrapping

Bootstrapping is the defining feature of Q-learning—and also its destabilizer under function approximation. The target y depends on Q-values, so if you update the same network that generates the target, you create a moving objective that can chase its own errors. Target networks fix this by introducing a delayed copy Q_target with parameters b8b2 that change slowly.

In standard DQN, the TD target is computed with the target network: y = r + γ(1-done) max_a' Q_target(s', a'; θ¯), while the prediction uses the online network: Q_online(s,a;θ). You update b8 via gradient descent, and periodically update b8b2 b7 b8.

• Hard update: every C steps, copy parameters (b8b2 b7 b8). Simple and common.
• Soft update: b8b2 b7 c4b8 + (1-c4)b8b2 each step. Smoother, often used in actor-critic, can also work for DQN variants.

Common mistake: updating the target network too frequently makes it nearly identical to the online network, losing the stabilizing effect; updating too rarely makes targets stale and slows learning. The right cadence depends on environment stochasticity, learning rate, and reward scale—so treat it as a knob you tune by watching TD-error and return stability.

Practical workflow: implement a DQN training loop that (1) collects steps, (2) starts learning after warmup, (3) performs a fixed number of gradient steps per environment step, (4) updates the target network on schedule, and (5) evaluates periodically with exploration turned off. If your learning collapses, your first suspects should be terminal masking, target updates, and optimizer settings before you blame the algorithm.

Section 4.4: Double DQN, dueling networks, and value decomposition

Basic DQN tends to overestimate action values because the max operator selects actions based on noisy estimates, and the same network both selects and evaluates. Double DQN fixes this with a simple but powerful change: use the online network to choose the best next action, and use the target network to evaluate it.

The Double DQN target is: a* = argmax_a' Q_online(s', a'; θ), then y = r + γ(1-done) Q_target(s', a*; θ¯). This decoupling reduces overestimation bias and often improves stability with almost no added complexity. If you implement only one “beyond DQN” change, implement this.

Dueling networks address a different issue: in many states, the choice of action barely matters, but the state value matters a lot. Dueling architectures decompose Q into a state-value stream and an advantage stream: Q(s,a) = V(s) + (A(s,a) - mean_a A(s,a)). This helps the network learn which states are good even when action differences are subtle, improving sample efficiency in some domains.

• When Double DQN helps: environments with high variance returns, noisy value estimates, or where early overestimation destabilizes learning.
• When dueling helps: many similar-valued actions per state; large discrete action spaces where learning per-action values is slower.

Engineering judgment: do not stack enhancements blindly. Each additional mechanism changes the learning dynamics and adds hyperparameters. Start from a clean DQN baseline, then add Double DQN, then (optionally) dueling. Validate each change with an ablation: same code, same hyperparameters where possible, one feature toggled. If the “improvement” only appears for one random seed, it is not yet an improvement.

Practical outcome: you can reduce Q-value inflation and get more reliable learning curves, especially early in training, by implementing Double DQN targets and verifying that action selection and evaluation networks are truly separated in code.

Section 4.5: Prioritized replay and importance sampling corrections

Uniform replay treats all transitions as equally useful, but in practice some samples drive learning far more than others. Prioritized Experience Replay (PER) samples transitions with probability proportional to a priority, typically the magnitude of TD error: p_i d (|δ_i| + b5)^α. This focuses updates on surprising or underfit experiences and can greatly improve learning speed in sparse or deceptive reward settings.

But PER changes the training distribution, introducing bias: you are no longer optimizing the same objective as uniform replay. The standard correction is importance sampling (IS) weights: w_i = (1 / (N P(i)))^β, normalized by dividing by max w_i in the batch to keep gradients well-scaled. You then weight the TD loss per sample by w_i.

• b1 controls prioritization strength: b1=0 is uniform; higher values focus more aggressively on high-error samples.
• b2 controls bias correction: b2 often anneals from a smaller value to 1.0 over training to reduce bias late.
• Practical pitfall: priorities can become stale; always update priorities after computing new TD errors for sampled items.

Bias/variance trade-off: prioritization can reduce variance in “useful update directions” but increase variance in gradient magnitudes if a few samples dominate. Watch for instability: if learning becomes spiky, try lowering b1, increasing batch size, using Huber loss, or clipping TD errors used for priorities.

Engineering judgment: PER is worth it when your baseline is data-hungry or rarely revisits meaningful transitions. If your environment is dense-reward and easy, PER can be unnecessary complexity. Always test with and without PER as an ablation, and report mean and spread across multiple seeds; PER sometimes improves best-case performance but worsens worst-case stability.

Section 4.6: n-step returns and multi-step bootstrapping

One-step TD targets propagate reward information slowly: you only move value information back one step per update. n-step returns speed up credit assignment by backing up multiple rewards before bootstrapping. The n-step target is: y = a3_{k=0}^{n-1} b3^k r_{t+k} + b3^n (1-done_{t:t+n}) max_a Q_target(s_{t+n}, a) (or the Double DQN variant for the bootstrap term). This can improve sample efficiency, especially when rewards are delayed.

Implementation is less trivial than it looks because your replay buffer must store or reconstruct n-step transitions. A common practical design is an intermediate FIFO queue of the last n transitions collected by the acting loop. When the queue fills, you aggregate the n-step reward and push a single replay item containing (s_t, a_t, R^{(n)}, s_{t+n}, done_n). If an episode ends early, you flush the queue with shorter returns (properly truncated at terminal).

• When n-step helps: delayed rewards, sparse signals, or tasks where “getting started” requires long credit chains.
• When n-step can hurt: highly stochastic environments where multi-step returns have higher variance; large n can destabilize learning.

Engineering judgment: treat n as a bias/variance knob. Small n (3–5) often yields a good trade-off. Combine n-step with replay and target networks carefully: your bootstrap term still relies on the target network, and your terminal handling must ensure you never bootstrap beyond true episode termination.

Evaluation discipline: n-step returns can change learning speed, so compare methods by both (1) environment steps (sample efficiency) and (2) wall-clock time (throughput). Finally, run ablations and interpret stability across seeds: report mean, standard deviation, and ideally confidence intervals. If n-step improves one seed dramatically but increases variance overall, your conclusion should reflect that trade-off, not just the best curve.

Section 5.1: Stochastic policies and why gradients matter

A stochastic policy \(\pi_\theta(a\mid s)\) outputs a distribution over actions rather than a single action. Sampling from this distribution matters when the environment is partially observable, adversarial, multi-modal, or when exploration is essential and naive \(\epsilon\)-greedy behavior breaks down (e.g., continuous actions, safety constraints, or highly deceptive reward landscapes).

Policy gradients optimize expected return directly: \(J(\theta)=\mathbb{E}_{\tau\sim\pi_\theta}[\sum_t \gamma^t r_t]\), where \(\tau\) is a trajectory. Instead of learning \(Q\) and extracting a policy via \(\arg\max_a Q(s,a)\), we nudge \(\theta\) so that sampled actions leading to higher returns become more likely. This is especially attractive when the best behavior requires randomness (mixed strategies), when actions are continuous (Gaussian policies), or when value-based maximization is unstable due to approximation error.

Engineering judgment: choose policy gradients when you can afford on-policy data and need smooth control or principled stochasticity. Choose value-based methods when off-policy replay is a major advantage and the action space is small and discrete. A common mistake is blaming “policy gradients are noisy” without checking basics: reward scale too large, episode lengths too variable, or poorly normalized observations. These issues amplify gradient variance and can make even correct implementations appear broken.

Section 5.2: Likelihood-ratio trick and the policy gradient theorem

The key mathematical tool is the likelihood-ratio trick (also called the score-function estimator). It converts gradients through a sampling process into gradients of log-probabilities:

\[\nabla_\theta \mathbb{E}_{x\sim p_\theta}[f(x)] = \mathbb{E}_{x\sim p_\theta}[f(x)\nabla_\theta \log p_\theta(x)].\]

Apply this to trajectories sampled from \(\pi_\theta\). The policy gradient theorem yields a practical estimator that avoids differentiating environment dynamics:

\[\nabla_\theta J(\theta)=\mathbb{E}_{s_t\sim d^{\pi}, a_t\sim\pi_\theta}\big[\nabla_\theta\log\pi_\theta(a_t\mid s_t)\,Q^{\pi}(s_t,a_t)\big].\]

Two practical takeaways: (1) you only need \(\nabla_\theta \log \pi_\theta(a\mid s)\), which autodiff can compute for common distributions; (2) the “critic” target can be \(Q^{\pi}\), a return estimate, or an advantage estimate. Credit assignment enters through the multiplier: if you put a noisy estimate of \(Q\) there, your gradient becomes noisy. If you put a biased estimate there, you may stabilize training at the cost of correctness.

Common mistakes: using raw probabilities instead of log-probabilities (causes numerical underflow), forgetting to stop gradients through sampled actions for reparameterization vs. score-function paths, and mixing on-policy gradients with off-policy data without correcting via importance sampling. A simple implementation check is to verify that when rewards are all zero, the gradient norm trends toward zero (aside from entropy terms).

Section 5.3: REINFORCE algorithm and Monte Carlo returns

REINFORCE is the canonical Monte Carlo policy gradient method: it uses complete (or truncated) episode returns as the learning signal. For each timestep \(t\), compute a return \(G_t=\sum_{k\ge t}\gamma^{k-t} r_k\). Then ascend the gradient estimate:

\[\Delta\theta \propto \sum_t \nabla_\theta\log\pi_\theta(a_t\mid s_t)\,G_t.\]

Workflow you can implement reliably:

• Collect trajectories on-policy (no replay buffer at first). Store \(s_t, a_t, r_t, \log\pi_\theta(a_t\mid s_t)\).
• Compute \(G_t\) by reverse cumulative sum; normalize returns per batch (often helps) but monitor if it changes the learning objective too aggressively.
• Compute loss as \(L=-\sum_t \log\pi_\theta(a_t\mid s_t)\,G_t\) (negative for gradient descent frameworks).
• Backprop, clip gradients if needed, and keep learning rate conservative (policy gradients can diverge from overly large steps).

Why gradients can be noisy: Monte Carlo returns contain all future randomness—environment stochasticity, policy sampling, and delayed rewards. Long-horizon tasks make \(G_t\) high variance, and early actions get credit (or blame) for outcomes far in the future. Practical outcomes: REINFORCE can solve small problems and is an excellent debugging baseline, but it becomes sample-inefficient and unstable as horizons grow. A common mistake is to judge REINFORCE “not working” when the issue is simply that you need many more trajectories than value-based baselines, or that reward scaling makes updates too large.

Section 5.4: Baselines, variance reduction, and control variates

A baseline subtracts a state-dependent value from the return without changing the expected gradient. The classic result: for any function \(b(s_t)\),

\[\mathbb{E}[\nabla_\theta\log\pi_\theta(a_t\mid s_t)\,b(s_t)] = 0,\]

so you can replace \(G_t\) with \(G_t - b(s_t)\) and keep the gradient unbiased while often dramatically reducing variance. This is a control variate: it doesn’t change the mean, only the spread.

In practice, choose \(b(s)=V_\phi(s)\), a learned value function trained by regression to returns (or to bootstrapped targets later). This introduces the actor-critic pattern: the actor updates \(\theta\) and the critic updates \(\phi\). Engineering choices that matter:

• Critic target: Monte Carlo returns are unbiased but noisy; bootstrapped targets are lower variance but biased.
• Update ratio: if the critic lags, the baseline is poor and variance remains high; if the critic overfits, it can destabilize due to non-stationary targets.
• Detaching the baseline: do not backprop from actor loss into critic parameters through \(b(s)\) unless you intend coupled optimization; typically you stop-gradient on the advantage term in the actor loss.

Common mistakes: using an action-dependent baseline (can bias gradients if not handled correctly), training the critic on normalized returns but using unnormalized advantages (scale mismatch), and forgetting that baseline quality affects learning rate sensitivity. A practical diagnostic is to track advantage standard deviation—if it explodes, your critic or reward scaling likely needs attention.

Section 5.5: Advantage functions and generalized advantage estimation

The advantage function \(A^{\pi}(s,a)=Q^{\pi}(s,a)-V^{\pi}(s)\) expresses how much better an action is than the policy’s average behavior at that state. Using advantages improves credit assignment: the policy increases probability of actions that are better-than-expected, and decreases probability of worse-than-expected actions, relative to the baseline.

With a critic \(V_\phi\), a common estimator is the temporal-difference (TD) residual:

\[\delta_t = r_t + \gamma V_\phi(s_{t+1}) - V_\phi(s_t).\]

Generalized Advantage Estimation (GAE) forms an exponentially-weighted sum of TD residuals:

\[\hat A_t^{\text{GAE}(\gamma,\lambda)} = \sum_{l=0}^{\infty} (\gamma\lambda)^l\,\delta_{t+l}.\]

Here \(\lambda\in[0,1]\) controls the bias-variance tradeoff. \(\lambda=1\) approaches Monte Carlo (low bias, high variance). Smaller \(\lambda\) uses more bootstrapping (higher bias, lower variance), often improving stability and sample efficiency in deep RL.

Practical workflow:

• Compute \(V_\phi(s_t)\) for all states in a rollout (vectorized).
• Compute deltas \(\delta_t\), then compute advantages by reverse-time recursion: \(A_t=\delta_t+\gamma\lambda A_{t+1}\).
• Optionally normalize advantages per batch (almost always helpful for stable actor updates).
• Use \(A_t\) in actor loss: \(L_\pi = -\sum_t \log\pi_\theta(a_t\mid s_t)\,\text{stopgrad}(A_t)\).

Common mistakes: forgetting terminal handling (set \(V(s_{T})=0\) at episode end unless using truncated bootstrapping), using inconsistent \(\gamma\) between critic and advantage computation, and selecting \(\lambda\) without considering horizon—longer horizons often benefit from smaller \(\lambda\) to keep variance manageable.

Section 5.6: Entropy regularization and exploration in policy space

Exploration in policy gradient methods is not an afterthought like \(\epsilon\)-greedy; it’s part of the policy distribution itself. Entropy regularization encourages higher-entropy (more random) policies early in training, preventing premature collapse to a near-deterministic policy that stops exploring.

Add an entropy bonus to the objective:

\[J_{\text{total}}(\theta)=J(\theta)+\beta\,\mathbb{E}_t[\mathcal{H}(\pi_\theta(\cdot\mid s_t))].\]

In loss form (minimization), this becomes \(L = L_\pi - \beta\,\mathcal{H}\). For categorical policies, entropy is \(-\sum_a \pi(a\mid s)\log\pi(a\mid s)\). For Gaussian policies, it depends on log standard deviation parameters. Engineering judgment: start with a moderate \(\beta\), then anneal it or tune it per environment. If entropy stays high and returns do not improve, \(\beta\) may be too large; if entropy collapses immediately, \(\beta\) is too small or your advantages have extreme scale.

This connects to deciding when stochastic policies outperform value-based control. In continuous control, stochastic policies with entropy bonuses often explore more smoothly than value maximization. In tasks with multiple equally good behaviors, entropy helps avoid brittle commitment to one mode. Common mistakes include applying entropy to already-squashed distributions incorrectly (e.g., after tanh) and ignoring action bounds, which can create misleading entropy signals. A practical outcome is more reliable early learning and fewer runs that get stuck due to early unlucky sampling.

In actor-critic, you train two models (or two heads of one model). The actor represents a stochastic policy \(\pi_\theta(a\mid s)\). The critic estimates either \(V_\phi(s)\) or \(Q_\phi(s,a)\). The central idea is to replace the high-variance Monte Carlo return in REINFORCE with an advantage estimate:

\[\n\nabla_\theta J(\theta) \approx \mathbb{E}[\nabla_\theta \log \pi_\theta(a\mid s)\, \hat A(s,a)]\n\]

where \(\hat A(s,a)\) is often \(\hat A = \hat G_t - V_\phi(s_t)\) (return minus baseline) or a bootstrapped TD-based estimator. The critic is trained to minimize a regression loss such as \(\mathcal{L}_V = \mathbb{E}[(V_\phi(s_t) - \hat G_t)^2]\) or against a bootstrapped target.

Network design choice: shared trunk vs separate networks. A shared trunk with two heads (policy logits/parameters and value scalar) is parameter-efficient and often works well in vision-based tasks. Separate networks reduce gradient interference (policy gradients and value gradients pulling features in different directions) and can be more stable when rewards are sparse or the value function is hard to fit. A practical compromise is a shared trunk with separate optimizers and carefully weighted losses.

• Compatible objectives: the critic must match what the actor needs. If the actor uses \(\hat A = \delta_t\) (TD error), the critic must be trained with the same bootstrapping and discounting assumptions.
• Common mistake: mixing an advantage estimate computed with one \(\gamma, \lambda\) setup while training the value head with a different target; the actor then chases an inconsistent learning signal.
• Outcome: you can implement an actor-critic loop that collects trajectories, computes advantages/returns, updates the policy with an advantage-weighted log-prob loss, and updates the critic with a value regression loss.

Finally, be explicit about what is “on-policy.” Classic actor-critic assumes the data comes from the current policy. Reusing old data without correction breaks the gradient. PPO will later relax this with conservative updates and importance ratios, but you should still treat large policy drift as a primary risk.

A2C (Advantage Actor-Critic) and A3C (Asynchronous Advantage Actor-Critic) are not different objectives so much as different data collection and update strategies. Both typically use multi-step bootstrapped returns (e.g., \(n\)-step) and compute advantages from those returns.

A3C historically used multiple CPU workers running environments in parallel, each computing gradients on its own rollouts and applying updates asynchronously to shared parameters. The appeal was decorrelated experience and high throughput without a replay buffer. The cost is non-determinism and “stale gradients” (workers compute gradients on older parameters), which can either regularize or destabilize learning depending on the task.

A2C is the synchronous variant: collect rollouts from many environments, aggregate them into a batch, then do a synchronized update. This is much easier to reason about and reproduce. In modern deep RL stacks, A2C-style synchronous collection plus GPU-accelerated networks is the default.

• Engineering judgement: choose synchronous updates when you care about reproducibility, clean ablations, and stable learning curves. Choose asynchronous only if you truly need CPU scalability and can tolerate more variance in results.
• Common mistake: using too short a rollout horizon in environments with delayed rewards; the critic then bootstraps heavily from inaccurate values, creating biased advantages that point the actor in the wrong direction.
• Practical workflow: set a fixed rollout length (e.g., 128–2048 steps per environment), compute \(n\)-step returns or GAE (next section), then run a few epochs of optimization on that batch.

Even before PPO, you can improve baseline actor-critic stability by controlling update size: smaller learning rates, gradient clipping, and monitoring the KL divergence between successive policies. These ideas motivate PPO’s “don’t change the policy too much” principle.

PPO (Proximal Policy Optimization) is best understood as a practical trust-region method: it tries to improve the policy while preventing destructive updates that move too far from the behavior policy that generated the data. The core tool is the importance sampling ratio:

\[ r_t(\theta) = \frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{old}}(a_t\mid s_t)} \]

The clipped surrogate objective uses \(r_t\) but clips it to keep updates conservative:

\[ \mathcal{L}^{clip}(\theta)=\mathbb{E}[\min(r_t(\theta)\hat A_t, \text{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\hat A_t)] \]

Intuition: if an action had positive advantage, PPO allows increasing its probability, but not by more than about \((1+\epsilon)\) per update; similarly for negative advantage, it avoids collapsing probability too aggressively. This “soft constraint” often prevents the sudden performance crashes seen in vanilla policy gradients.

There are two common alternatives or additions:

• KL penalty: add \(\beta \cdot \text{KL}(\pi_{old}||\pi_\theta)\) to the loss. You can adapt \(\beta\) to target a desired KL per update.
• KL constraint / early stopping: monitor empirical KL during optimization epochs and stop early if KL exceeds a threshold. This is simple and effective in practice.

Common mistake: interpreting clipping as a guarantee of monotonic improvement. Clipping reduces risk but does not eliminate it; if advantages are wrong (bad critic, reward bugs, partial observability), PPO can still confidently optimize the wrong direction. Another frequent bug is computing \(\log \pi_{old}(a|s)\) after updating the policy; you must store old log-probabilities during rollout collection.

Practical outcome: you can implement PPO updates by (1) collecting rollout data under \(\pi_{old}\), (2) computing \(\hat A_t\) and value targets, (3) performing multiple SGD epochs on shuffled minibatches using the clipped objective plus a value loss and entropy bonus, while (4) monitoring KL and stopping early when necessary.

Stable actor-critic training is mostly about controlling scale and preventing any single component from dominating. Three practices have outsized impact.

1) Advantage estimation and normalization. Generalized Advantage Estimation (GAE) computes advantages with a bias-variance tradeoff controlled by \(\lambda\):

\[ \hat A_t^{GAE(\gamma,\lambda)} = \sum_{l=0}^{\infty} (\gamma\lambda)^l \delta_{t+l}, \quad \delta_t = r_t + \gamma V(s_{t+1}) - V(s_t) \]

Then normalize advantages per batch: \(\hat A \leftarrow (\hat A-\mu)/ (\sigma+1e{-}8)\). This prevents large-magnitude advantages from causing huge policy steps and makes learning rates more transferable across tasks.

2) Value loss weighting and clipping. The critic is a moving target; if it overfits or oscillates, the actor receives noisy advantages. Use a value loss coefficient (often 0.5) and consider value function clipping (common in PPO): clip the change in predicted value between old and new parameters to avoid runaway critic updates. Also track explained variance; a near-zero or negative explained variance is a warning that the critic is not learning useful structure.

3) Gradient and reward normalization. Apply global norm gradient clipping (e.g., 0.5–1.0) to prevent rare large updates. For environments with unbounded rewards, normalize rewards or returns (or use reward scaling) to keep value targets in a reasonable range. In continuous control, observation normalization (running mean/std) is often essential.

• Common mistake: increasing PPO epochs or minibatch passes until the surrogate objective looks optimized, while KL silently explodes. More epochs is not “more learning” if it violates the trust-region intuition; you often want fewer epochs when the batch is small or the policy is changing quickly.
• Practical outcome: you can keep training stable by treating KL, value loss, entropy, and gradient norms as first-class signals, not just episodic return.

In practice, stability comes from a consistent package: advantage normalization, entropy bonus to avoid premature determinism, KL monitoring with early stopping, and a critic trained neither too weak (noisy advantages) nor too strong (overfitting and harmful bootstrapping).

Policy optimization can “look good” in training curves while being worse in real performance. Proper evaluation separates learning signals (surrogate objective, value loss) from task outcomes (return, success). Use a dedicated evaluation loop: run the current policy without exploration noise (or with fixed stochasticity), do not update parameters, and report confidence intervals over multiple episodes and multiple random seeds.

On-policy metrics to log each iteration include: mean/median episodic return, episode length, success rate (for goal-conditioned tasks), entropy, approximate KL, clip fraction (fraction of samples where clipping is active), explained variance, and value error. Clip fraction is especially diagnostic: near zero may indicate updates are too small; very high may indicate \(\epsilon\) is too tight or advantages are too large.

For safe comparisons across algorithm variants, keep environment steps fixed (not “updates”), control seeds, and compare at equal wall-clock when relevant. Many RL regressions are “throughput changes” masquerading as algorithmic improvements.

Sometimes you need off-policy estimators to compare a new candidate policy using data collected by an older behavior policy, especially when evaluation is expensive or risky. Basic importance sampling can be high variance, but it is still useful as a sanity check: if the importance weights explode, your new policy is far from the behavior policy and any off-policy estimate is unreliable—also a sign your training updates may be too aggressive. Weighted importance sampling or doubly robust estimators can reduce variance, but they still depend on good overlap between policies.

• Common mistake: reporting only the best-seed curve or only training returns from the same trajectories used to update the policy. Always separate evaluation rollouts.
• Practical outcome: you can build an evaluation protocol that detects reward hacking, brittleness to initial conditions, and performance collapse early, before you spend days training the wrong variant.

Robustness checks are cheap and revealing: randomize initial states, vary observation noise, and test across environment parameter ranges. A policy that only succeeds under one narrow configuration is usually overfit or exploiting quirks in dynamics.

Stable algorithms still fail in unstable engineering environments. Treat your RL project like a production system: reproducible experiments, traceable artifacts, and regression protection.

Reproducibility recipe. Fix and log seeds (Python/NumPy/framework/environment), version your environment and wrappers, and log hyperparameters (\(\gamma, \lambda, \epsilon\), rollout length, number of envs, epochs, minibatch size, learning rates, entropy/value coefficients). Save checkpoints regularly (policy and optimizer state), and store normalization statistics (observation/reward running means). Without these, “resume training” will silently change behavior.

Monitoring. Beyond returns, monitor KL, entropy, value loss, explained variance, gradient norm, and action distribution statistics. Set simple alert thresholds: early stop PPO epochs when KL exceeds target; abort runs if value loss diverges; flag when entropy collapses too early (often indicates exploration failure or an overly strong advantage signal from a buggy reward).

Ablations. Actor-critic systems have many interacting stabilizers. When something improves, verify with ablations: remove advantage normalization, remove value clipping, change KL threshold, swap shared vs separate networks. This prevents “cargo cult” stacks where no one knows which component matters in your domain.

• Regression tests: keep a small set of deterministic “micro-environments” (or fixed start states) where a correct implementation should reach a minimum return within N steps. Run these tests in CI when you change code.
• Artifact discipline: log rollout statistics and save a few trajectories for inspection; many reward/dynamics issues are obvious when you watch state-action sequences.

Practical outcome: you can move from “it learns sometimes” to an RL pipeline where changes are measurable, failures are diagnosable, and improvements survive refactors. That deployment mindset is what makes PPO and actor-critic methods a reliable tool rather than a fragile demo.