Building Game AI with OpenAI Gym: RL Agents from Zero

Historically, “OpenAI Gym” was the standard RL environment API. Today, the actively maintained fork is Gymnasium. Many tutorials still say import gym, while modern code often uses import gymnasium as gym. Conceptually, they provide the same job: a catalog of environments with a standard interface so your agent code can swap games without rewriting the loop.

Think of an environment as a contract between “the world” and your agent. The contract is defined by a few methods and properties: env.reset() starts a new episode and returns an initial observation; env.step(action) advances the environment by one time step given an action; env.observation_space and env.action_space describe valid inputs/outputs; and optional render() displays a frame for humans.

Installation is boring but important: pin versions to avoid silent behavior changes. In a fresh virtual environment, install Gymnasium and common extras. For classic control, you typically need an extra dependency set (and system packages on some OSes). Your first verification step should not be “train a model,” but “run five episodes and print returns.” If that fails, fix it before proceeding.

• Practical outcome: you can create an environment by ID (e.g., gym.make("CartPole-v1")) and confirm that reset/step work without errors.
• Common mistake: mixing old Gym return signatures with Gymnasium. Many bugs come from unpacking the wrong number of values from reset() or step().

Finally, recognize that environments can be parameterized. Time limits, rendering modes, and reward variants can change the task. In research or team work, always write down the exact environment ID and version, plus any wrappers or configuration you applied. Your agent is only as comparable as the environment contract you ran it against.

Every Gym/Gymnasium environment declares two spaces: observation_space and action_space. These are not just metadata; they are how you prevent invalid actions, shape neural networks correctly, and design preprocessing. You should make it a habit to print them at the start of every run.

The two most common space types in game AI are Discrete(n) and Box(low, high, shape, dtype). A Discrete action space means the agent chooses an integer action in [0, n-1] (e.g., left/right). A Box action space means the agent outputs a continuous vector within bounds (e.g., steering angle and throttle). Observations follow the same pattern: toy text tasks often use Discrete observations (a state index), while classic control often uses a Box observation (a vector of floats like position and velocity).

This matters immediately for algorithm choice. Tabular Q-learning and SARSA (coming in a later chapter) require discrete state/action representations, or a discretization strategy. Deep methods like DQN typically assume Discrete actions but can handle Box observations by feeding them into a neural network. Policy gradient methods are often used when the action space is Box, because the policy can naturally parameterize continuous distributions.

• Workflow tip: sample from spaces to test shapes: action = env.action_space.sample(), obs = env.observation_space.sample().
• Common mistake: treating a Box observation as unbounded. Use the declared bounds to reason about normalization, clipping, and numeric stability.

Toy text environments are especially good for building intuition about state indexing and dynamics. Even when the observation is an integer, you still need to understand what it represents (a cell in a grid, a room in a map, etc.). Don’t guess—inspect the environment documentation, and if possible, print decoded states. This habit pays off later when you debug reward shaping or unexpected agent behavior.

The heart of RL engineering is the step loop. In Gymnasium, reset() returns (obs, info), and step(action) returns (obs, reward, terminated, truncated, info). This is a key difference from older Gym, which used a single done flag. You should treat terminated as “the task ended naturally” (success/failure terminal condition) and truncated as “the episode was cut off” (time limit or external constraint).

A correct run loop checks terminated or truncated to end an episode. This is not pedantry: mixing them can distort learning curves and evaluation. If your environment has a time limit wrapper, it will often set truncated=True at the step limit. For evaluation, you may want to report both: success rate (often tied to termination conditions) and average return (affected by truncation).

Start by running environments without any learning, just to ensure you understand their dynamics. Run a few episodes on a classic control task (like CartPole) and a toy text task, printing per-episode return and length. Add info printing sparingly; it’s useful for debugging but can become noisy.

• Engineering judgement: decide where to accumulate rewards. The standard is to sum rewards per episode, but for sparse rewards you may also track additional signals (e.g., number of steps survived, distance traveled).
• Common mistake: forgetting to handle truncated. This can cause episodes to “never end” in your code if you only check terminated.

Finally, rendering: many environments support render_mode="human" for interactive display or render_mode="rgb_array" for frames you can save. Rendering inside the training loop can drastically slow experiments; prefer rendering only during evaluation or when recording a small number of episodes for debugging.

Reproducibility is not optional in RL. Learning curves are noisy, and without careful control you can “prove” anything. The first tool is seeding: initialize all random number generators so that reruns produce the same sequence of stochastic choices—at least as much as the environment and hardware allow.

In Gymnasium, you typically seed at reset(seed=...) for environment dynamics. You should also seed your action sampling (if using numpy/python random) and your ML framework (e.g., PyTorch). A minimal reproducible template sets a single integer seed in config, then derives secondary seeds for train/eval splits if needed.

Determinism has limits. Some environments contain hidden nondeterminism; GPU operations can be nondeterministic depending on kernels; multi-threading can reorder operations. The practical goal is not mathematical determinism everywhere, but controlled variability: the ability to rerun a specific experiment and get the same result distribution, and the ability to average across multiple seeds for robust comparisons.

• Recommended practice: run at least 3–10 seeds for any claim about algorithm improvement, and report mean and standard deviation (or confidence intervals).
• Common mistake: changing code and reusing old logs without tracking the exact git commit and config; you lose provenance.

Build a habit of logging: environment ID, library versions, seed(s), episode limit, and any wrappers. When something looks “too good,” reproducibility metadata is how you determine whether it’s a real improvement or an accidental change in settings. This chapter’s experiment template will make these fields first-class, not an afterthought.

Wrappers are one of the most powerful features of the Gym ecosystem: they let you modify observations, rewards, actions, episode lengths, and logging without changing the underlying environment. In game AI projects, wrappers are where you put “environment engineering” so your agent code stays clean and reusable.

Common wrapper uses include: clipping or normalizing observations, stacking frames for visual tasks, discretizing continuous actions (with care), and shaping rewards. Even simple monitoring is typically implemented as a wrapper: record episode returns and lengths, write them to disk, and optionally capture videos. Gymnasium provides utilities like RecordVideo (and monitoring tools) that can automatically save episode footage to a directory—crucial for visual debugging when an agent exploits a loophole in your reward function.

When recording, decide on a policy: record only evaluation episodes (e.g., every N training iterations), or record the first K episodes for sanity checks. Recording every episode can create huge files and slow training. Use deterministic evaluation seeds so videos are comparable across runs.

• Practical workflow: run with render_mode="rgb_array" and a video recorder wrapper, then replay saved videos when investigating failures.
• Common mistake: applying reward shaping in a wrapper but forgetting to log both shaped and original rewards; later you can’t diagnose whether the agent truly solved the task.

Wrappers also help you standardize observation preprocessing. For example, even for vector observations you might want to cast dtypes consistently (float32), clip outliers, or concatenate additional signals. Keep wrappers small and composable, and document them in your experiment config so others can reproduce the exact environment pipeline.

Before implementing Q-learning, DQN, or policy gradients, you need a baseline. The simplest baseline is a random agent: sample actions from env.action_space and track episode returns. This baseline is deceptively valuable. It verifies your loop, your logging, and your metrics, and it provides a sanity check: if your learned agent performs worse than random, something is wrong (or the reward is misleading).

Score tracking should be treated as part of the experiment, not console spam. Store per-episode return, episode length, and termination/truncation counts. Save a summary (mean return over last 100 episodes, etc.) so you can compare runs. Even in Chapter 1, start practicing evaluation discipline: separate training episodes (where exploration happens) from evaluation episodes (fixed policy, no learning) where you measure performance.

A practical experiment folder structure keeps outputs organized and reproducible. A common pattern is:

• configs/: YAML/JSON configs (env id, seeds, hyperparameters).
• runs/<timestamp>_<env>_<algo>/: one folder per run.
• runs/.../metrics.csv or metrics.jsonl: append-only logs.
• runs/.../videos/: recorded episodes for debugging.
• runs/.../checkpoints/: saved agent state (even for simple baselines).

Checkpointing is not only for neural networks. For debugging, it’s useful to save enough information to “replay” an episode: the seed, the action sequence, and (optionally) the observed transitions. When a bug appears—an unexpected reward spike, a sudden performance collapse—you can reload the exact episode and step through it. This practice turns RL from guesswork into engineering.

By the end of this chapter, you should have a working template that: (1) installs and runs Gym/Gymnasium environments, (2) inspects spaces, (3) executes a correct step loop, (4) controls seeds, (5) logs metrics to disk, and (6) records and replays episodes. That foundation will let you focus on learning algorithms in later chapters instead of fighting tooling.

Section 2.1: MDP framing and value functions in practice

Every Gym environment can be viewed as an MDP defined by (S, A, P, R, \gamma): states, actions, transition dynamics, rewards, and discount factor. In practice you rarely know P explicitly; you sample it by calling env.step(action). The key engineering step is deciding what you will treat as “state” for tabular learning. If the observation is already a small integer (e.g., Discrete(n)), you can index directly. If the observation is a tuple (like Taxi’s row/col/passenger/destination) you can often map it to a single integer via the environment’s built-in encoding or a custom hashing function.

Gymnasium exposes the information you need to model the MDP:

• env.observation_space tells you whether the observation is discrete or needs encoding.
• env.action_space.n gives the number of discrete actions for tabular Q tables.
• reset(seed=...) and action_space.seed(...) support reproducible experiments.

Value functions translate “game scoring over time” into something we can optimize. The state-value function V(s) is the expected discounted return from state s, and the action-value function Q(s,a) is the expected discounted return from taking action a in s and following a policy thereafter. In games, sparse rewards are common (e.g., +1 for reaching the goal, 0 otherwise). Tabular methods cope well as long as the state space is manageable and you explore enough.

Practical judgement: prefer Q(s,a) for control (learning to act), because selecting actions via argmax_a Q(s,a) is straightforward. Keep \gamma consistent with the task: for episodic games with a time limit, values like 0.95–0.99 are common. Setting \gamma too low makes the agent shortsighted; too high can slow learning when rewards are very delayed.

Common mistake: treating high-dimensional observations (images, continuous vectors) as tabular states. If observation_space is not Discrete (or a small MultiDiscrete), tabular methods will explode in size and effectively never revisit the same state. In that case you need function approximation (Chapter 3), but for now choose environments designed for discrete tabular learning.

Section 2.2: Bellman optimality intuition for game scoring

The Bellman equations are the “accounting identity” of RL: the value of a situation equals immediate reward plus the discounted value of what comes next. For action-values, the optimal Bellman equation is:

Q*(s,a) = E[r + \gamma * max_{a'} Q*(s',a')]

In game terms, imagine you are choosing a move now (a) and you assume you will play optimally afterward. The max represents “best possible future play.” This is why Q-learning is often described as learning from imagined optimal futures, even if your current behavior is exploratory or messy.

This equation also clarifies what “learning” does in code: each step you get a sample (s,a,r,s') and you adjust your table entry Q[s,a] toward a target that mixes the observed reward and your current estimate of future best play. You are not computing exact expectations; you are doing stochastic approximation with a learning rate \alpha. Over many samples, the noisy targets average out.

Engineering judgement: the Bellman target can be unstable if you bootstrap from bad estimates too aggressively. This is why the learning rate matters and why you should watch learning curves over multiple random seeds. Also be careful with terminal transitions: if the episode ends at s', there is no “future,” so your target should drop the bootstrap term (treat max Q(s',·) as 0). Failing to handle terminal states is a classic bug that silently prevents convergence.

When you interpret rewards as “game score,” remember that the agent maximizes discounted return, not necessarily the final score you intuitively care about. If the environment gives a living penalty or time penalty, the agent may learn to finish quickly even with lower final reward. That’s not wrong—it’s exactly what the MDP specifies. If your learned behavior surprises you, first re-check the reward function and termination conditions rather than immediately tuning hyperparameters.

Section 2.3: Q-learning implementation details and pitfalls

Q-learning is the standard starting point for discrete control because it is simple and off-policy: you can behave with an exploratory policy while learning the greedy policy implied by your Q table. A minimal training loop needs: a Q table, an action-selection rule (ε-greedy), and the Q-learning update.

Core update for a non-terminal transition:

td_target = r + gamma * max_a' Q[s_next, a']
td_error = td_target - Q[s, a]
Q[s, a] += alpha * td_error

In Gymnasium, step returns (obs, reward, terminated, truncated, info). Treat the transition as terminal if terminated or truncated is true. That distinction matters: terminated means the environment’s terminal condition; truncated means a time limit. For learning, both usually mean “no bootstrap beyond this step” unless you explicitly want time-limit bootstrapping (an advanced topic).

• Q table shape: for Discrete(nS) and Discrete(nA), use Q = np.zeros((nS, nA), dtype=np.float32).
• Seeding: use env.reset(seed=seed) at the start of each run, and seed env.action_space too.
• Episode loop: reset, then repeat choose action → step → update → transition until done.

Pitfall: mixing up observation types. Some environments return obs as an integer state index; others return numpy arrays or tuples. If you do Q[obs] and obs is an array, you’ll get wrong indexing or errors. Make state handling explicit: if the observation space is not a plain Discrete, create a deterministic encoding function and unit-test it on a few samples.

Pitfall: evaluating during training with ε still active. Q-learning can look “bad” if you keep exploring in evaluation episodes. Separate concerns: during training you use ε-greedy; during evaluation set ε=0 and run greedy actions only. Also log both episodic return and episode length. In sparse-reward tasks, length can reveal that the agent is learning to avoid failure (or exploit time limits) even before rewards increase.

Section 2.4: On-policy SARSA and when it matters

SARSA (State–Action–Reward–State–Action) looks similar to Q-learning but updates toward the value of the action you actually take next, not the greedy best action. The update is:

td_target = r + gamma * Q[s_next, a_next]

where a_next is selected by your current behavior policy (often ε-greedy). This makes SARSA an on-policy algorithm: it learns the value of the policy it is executing. In game AI terms, SARSA learns to be good while still making occasional exploratory mistakes, and it internalizes the consequences of those mistakes.

When does that matter? In “cliff-like” environments where a single wrong move causes catastrophic loss, Q-learning can learn a risky policy because its target assumes perfect greedy behavior after the current step. SARSA tends to learn safer paths when ε is non-zero, because it expects that exploration might push it into danger. This difference is not philosophical; it shows up in learning curves and final behavior.

Implementation detail: SARSA requires choosing a_next before you compute the update, which slightly changes your step loop:

• Choose a from s (ε-greedy).
• Step to get s_next, r, done.
• Choose a_next from s_next (ε-greedy) unless done.
• Update Q[s,a] toward r + gamma * Q[s_next, a_next] (or just r if done).
• Set s=a_next and continue.

Common mistake: forgetting to handle terminal states and still sampling a_next, which can index garbage or incorrectly bootstrap. Another mistake: comparing SARSA and Q-learning with different ε schedules. If you want a fair comparison of learning behavior, run both with the same alpha, gamma, and identical exploration schedule, and average results over multiple seeds.

Practical outcome: SARSA is a great diagnostic tool. If Q-learning “learns” but produces brittle behavior, implement SARSA and see whether on-policy learning stabilizes training. If SARSA is stable but underperforms, it often points to an exploration schedule that is too aggressive late in training.

Section 2.5: Exploration strategies (ε-greedy, decay, softmax)

Exploration is the difference between an agent that memorizes a lucky trajectory and an agent that reliably wins. In tabular RL, ε-greedy is the default: with probability ε choose a random action; otherwise choose argmax_a Q[s,a]. The engineering question is not “should we explore?” but “how much, for how long, and how do we measure sample efficiency?”

A practical schedule starts with high exploration and decays it. Two common choices:

• Linear decay: ε decreases from eps_start to eps_end over N steps/episodes. Easy to reason about and reproduce.
• Exponential decay: ε = eps_end + (eps_start-eps_end)*exp(-t/τ). Decays quickly at first, then slowly.

Measure sample efficiency by plotting average return versus environment steps (not episodes). This avoids misleading comparisons when different runs produce different episode lengths. Also log ε over time so you can correlate “performance jumps” with exploration turning down.

Softmax (Boltzmann) exploration is another option: choose actions with probability proportional to exp(Q[s,a]/T) where T is a temperature. High T is exploratory; low T is near-greedy. Softmax can be smoother than ε-greedy in environments with many actions, but it is sensitive to Q scale: if rewards are large, exp(Q/T) can overflow or collapse to near-deterministic too early. If you use softmax, normalize or clip Q values, or tune temperature carefully.

Common mistakes:

• Decaying ε too fast: the agent stops visiting key states before their values are learned, causing premature convergence to a suboptimal policy.
• Leaving ε high forever: training return may look noisy and low even if the greedy policy is good; evaluation must use ε=0.
• No random tie-breaking: if multiple actions share the same Q value (common early), argmax may always pick the first action, reducing effective exploration. Randomly break ties when exploiting.

A robust workflow is: pick a baseline ε schedule, train for a fixed step budget, evaluate greedily every K steps, and then adjust decay based on whether improvement stalls early (too little exploration) or remains noisy late (too much exploration). This tuning discipline carries directly into DQN later, where exploration schedules are even more consequential.

Section 2.6: Evaluation protocols for tabular agents

Training curves are not evaluation. During training you inject randomness (ε-greedy) and continuously change the policy. Evaluation should answer a narrower question: “How well does the learned policy perform when executed greedily?” To do that, freeze learning (alpha=0 or no updates), set ε=0, and run a batch of episodes with controlled seeds.

A solid protocol for discrete tabular agents includes:

• Separate training and evaluation episodes: do not update Q during evaluation.
• Multiple seeds: run several independent trainings (e.g., 10 seeds) and report mean ± standard error of evaluation return.
• Step-budgeted reporting: compare agents after the same number of environment steps to make sample-efficiency claims.
• Baselines: include a random policy baseline and (if available) a simple heuristic baseline.

Exporting a learned policy is straightforward with tabular methods: you can store either the full Q table (for later fine-tuning) or the greedy policy pi[s] = argmax_a Q[s,a]. For portability, save as a NumPy file (.npy) or a compact JSON (state → action). When you reload, ensure the state encoding is identical; mismatched encodings are a frequent source of “it worked yesterday” failures.

For evaluation runs, consider using wrappers like RecordEpisodeStatistics to capture returns and lengths, and RecordVideo for a small number of qualitative rollouts. Keep video separate from metrics: recording can slow environments and change timing, which matters in some tasks.

Finally, treat truncation carefully. If episodes are truncated due to a time limit, high returns might reflect “survival” rather than goal-reaching. Report both return and success rate when the environment provides a success signal (sometimes in info), and consider adding task-specific metrics such as steps-to-goal. With these protocols, your tabular agents become reliable experimental baselines—and a trustworthy foundation for the deep RL methods that follow.

Section 3.1: From tabular Q to function approximation

In tabular Q-learning, you store Q[s, a] explicitly and update it with the Bellman target. With high-dimensional observations, you cannot index a table, so you approximate Q(s, a) with a function Q_θ(s, a) parameterized by neural network weights θ. Practically, your network will take an observation tensor and output a vector of size n_actions, where entry i is the estimated return for action i.

The core target remains: y = r + γ * max_a' Q(s', a') for non-terminal transitions. The difference is how you apply it: you compute a predicted value for the chosen action Qθ(s, a), then minimize a regression loss between that scalar and the target y. This is the DQN learning step. You will implement it with a deep learning framework (PyTorch is typical): forward pass, compute loss, backprop, optimizer step.

Engineering judgment: treat the network as a drop-in replacement for a table only at the interface level. Internally, the network introduces new failure modes (scale sensitivity, gradient explosion, feature dependence). Two practical habits help immediately: (1) normalize or at least clip observations when ranges are extreme; (2) clip rewards to a small range when using pixel-based or sparse-reward games, because large reward magnitudes can produce unstable Q scales.

Common mistake: training “online” on consecutive transitions without replay. With tabular methods this may still work; with neural approximators it often diverges because consecutive samples are highly correlated and targets move too quickly. You will fix this in Section 3.3, but keep it in mind: a naive neural Q-learning loop is usually unstable even in simple tasks.

Section 3.2: Network architecture choices for simple game states

For vector observations (like CartPole-v1), a small multilayer perceptron (MLP) is enough. A practical starting point is two hidden layers of 128 units with ReLU activations. Your output layer should be linear (no activation) with shape [batch, n_actions]. Avoid making the network too large at first: bigger networks can overfit the replay buffer early and produce sharper, less stable Q estimates.

For grid-like observations (e.g., a 2D board state), a shallow convolutional network (ConvNet) can help. Still, keep your first implementation simple: one or two convolution layers followed by a small fully connected head. The point of this chapter is DQN mechanics (replay, targets, evaluation), not squeezing performance through architecture search.

• Input preprocessing: cast observations to float32; if using pixels, scale to [0, 1] and consider grayscale and frame stacking via wrappers.
• Output meaning: each output unit corresponds to a discrete action; selecting an action is argmax(Q(s, ·)) during exploitation.
• Device handling: move tensors and model to GPU consistently, but don’t mix CPU and GPU tensors inside the training step.

Engineering judgment: initialization and activation choices matter more than you might expect. ReLU is standard; if you observe “dead” neurons (loss stops improving and Q-values become flat), try LeakyReLU or reduce learning rate. Also consider using LayerNorm for non-stationary observation scales in some game tasks, but introduce it only if you have evidence (unstable gradients or sensitivity to observation magnitude).

Common mistake: using a softmax on outputs because it “feels like a policy.” DQN outputs action-values, not probabilities. Softmax will distort the Bellman regression problem and typically breaks learning.

Section 3.3: Replay buffer design and sampling strategies

Experience replay is the practical upgrade that makes DQN trainable. Instead of updating on the most recent transition, you store transitions (s, a, r, s', done) in a buffer and sample random minibatches. This breaks temporal correlations and allows you to reuse older experiences, improving sample efficiency and stability.

A robust replay buffer implementation is mostly careful bookkeeping. Use a fixed-capacity ring buffer so memory does not grow unbounded. Store observations efficiently (NumPy arrays are common) and convert sampled batches to tensors only when training. Include done flags so terminal transitions zero out the bootstrap term: for terminals, y = r; otherwise y = r + γ * max Q(s', ·).

• Capacity: for simple vector tasks, 50k–200k transitions is fine; for harder tasks, 1M is common, but requires memory planning.
• Warm-up: don’t start gradient updates until you have a minimum number of transitions (e.g., 1k–10k). Early training on tiny buffers tends to overfit and produce extreme Q estimates.
• Sampling: uniform random sampling is the baseline; it often works surprisingly well.

Sampling strategy is an engineering choice. Uniform sampling is simplest and should be your default until you diagnose a problem. If learning is slow because rewards are sparse, prioritized replay can help by sampling “surprising” transitions more often, but it adds bias corrections (importance sampling weights) and more moving parts to debug. In a course setting, you should first get a clean uniform replay implementation that logs: buffer size, average TD error, and loss.

Common mistake: storing references to mutable arrays (especially when environments reuse buffers). Always store a copy of observations when pushing to the buffer, or you may train on overwritten states and see inexplicable instability.

Section 3.4: Target networks, update cadence, and loss functions

Even with replay, DQN can diverge because the target depends on the same network you are updating. The fix is a target network: maintain a second set of parameters θ^- (often called q_target) used only to compute the bootstrap term. Your online network θ (called q_online) is optimized; the target network is updated more slowly.

The target becomes: y = r + γ * (1 - done) * max_a' Q_{θ^-}(s', a'). This reduces the “chasing a moving target” problem. You then minimize the loss between Q_{θ}(s, a) and y. In practice, use Huber loss (a.k.a. smooth L1) rather than MSE: it is less sensitive to occasional large TD errors and often prevents unstable gradients early in training.

• Hard updates: every N environment steps (e.g., 1,000–10,000), copy parameters: θ^- ← θ.
• Soft updates: every step, Polyak average: θ^- ← τθ + (1-τ)θ^- with small τ (e.g., 0.005).
• Gradient handling: compute targets with no_grad() so you do not backprop through the target network.

Action-value evaluation matters here: log both max_a Q(s, a) and the average Q for sampled states. If Q-values drift upward without corresponding reward improvements, you may have overestimation (addressed next section) or unstable targets (target update too frequent, learning rate too high).

Common mistake: updating the target network too often (e.g., every step) while also using high learning rates. This collapses the benefit of having a separate target and can make training as unstable as the naive version.

Section 3.5: Common failure modes: exploding Qs, dead exploration

DQN failure is usually visible in logs before it is visible in returns. The two most common patterns are (1) Q-values explode while episode return stagnates or degrades, and (2) exploration collapses early, producing a policy that repeats a suboptimal behavior forever.

Exploding Qs often come from a combination of bootstrapping and function approximation: a slight overestimate at s' becomes a larger target at s, and the error compounds. To detect this, track statistics of predicted Q on a fixed evaluation batch of states sampled from the replay buffer. If the mean/max Q grows steadily while rewards do not, act immediately: lower learning rate, increase target update interval, use Huber loss, clip gradients (e.g., max norm 10), and consider reward clipping.

Overestimation checks: DQN uses max over noisy estimates, which biases values upward. A practical diagnostic is to compare Q estimates against empirical discounted returns from short rollouts (not perfect, but can reveal massive inflation). Another check is to log the gap between the best and second-best action-values; if the gap becomes huge early, the network may be confidently wrong. The standard algorithmic fix is Double DQN (select action with online net, evaluate with target net), but even if you don’t implement it yet, recognizing overestimation is important engineering skill.

Dead exploration usually comes from an epsilon schedule that decays too fast, or from training beginning before the replay buffer contains diverse behavior. If epsilon drops quickly, the policy becomes near-greedy on a partially trained network and stops discovering better trajectories. Fixes: increase epsilon_decay_steps, keep a higher minimum epsilon (e.g., 0.05–0.1 for simple tasks), and delay learning until after a warm-up collection phase.

Common mistake: evaluating performance using the same epsilon-greedy policy as training. Your benchmark should use a separate evaluation loop with a fixed low epsilon (often 0.0) and no learning, run across multiple seeds for reliability.

Section 3.6: Practical DQN hyperparameters and training curves

To train a DQN end-to-end and benchmark it, you need a disciplined configuration and a clear view of training curves. A minimal but practical setup for a vector-observation environment (like CartPole-v1) might look like: discount γ=0.99, replay capacity 100,000, batch size 64, learning rate 1e-3 or 5e-4, target hard update every 2,000 steps, gradient clipping at 10, and an epsilon schedule from 1.0 to 0.05 over 50,000–200,000 steps. Use an Adam optimizer and Huber loss.

• Training frequency: update every environment step or every 4 steps; if you update too frequently early, you may overfit correlated early data.
• Warm-up steps: 1,000–10,000 random steps before learning, depending on environment difficulty.
• Evaluation protocol: every N steps, run 5–20 evaluation episodes with epsilon=0.0, no replay writes, no gradient updates.

Read your curves like an engineer. Plot (a) episode return (train and eval), (b) TD loss, (c) mean/max Q on a fixed batch, and (d) epsilon. Healthy learning typically shows eval return rising while loss decreases or stabilizes; Q-values should increase only to a scale consistent with achievable discounted returns. For example, if the maximum episode length is 500 with reward 1 per step, the true optimal return is about 500; Q-values orders of magnitude beyond that are a red flag.

Benchmarking performance means comparing against baselines and controlling randomness. Always run multiple seeds (at least 3; ideally 5–10), log wall-clock time and environment steps, and report mean and variability of evaluation return. If you change one ingredient (e.g., target update cadence), keep everything else fixed and rerun: this is the simplest form of ablation study and is how you build confidence that an improvement is real.

Practical outcome: by the end of this chapter’s implementation, you should have a DQN agent that (1) uses a neural Q-network, (2) trains from a replay buffer, (3) stabilizes bootstrapped targets with a target network, and (4) produces interpretable training logs that let you diagnose overestimation and exploration issues quickly.

In continuous control, the action space is not a small set but an interval (or multiple intervals), such as steering in [-1, 1] and throttle in [0, 1]. A DQN-style approach would require discretizing each dimension. With two dimensions and just 11 bins each, you already have 121 discrete actions; with three dimensions it becomes 1331. This “curse of discretization” increases network output size, slows learning, and can prevent fine-grained behaviors that feel natural in games.

Policy-based methods avoid this by modeling a distribution over continuous actions and sampling from it. The policy can express “try slightly left” or “apply gentle throttle” without creating thousands of discrete buttons. This is also better aligned with exploration: instead of epsilon-greedy (which jumps to unrelated actions), stochastic policies explore locally around the current behavior, which is critical in physics-heavy environments.

• Use policy gradients when the action space is continuous, when smoothness matters, or when you need stochastic behavior as part of the solution (e.g., bluffing or mixed strategies).
• Prefer value-based methods when the action space is small and discrete, and you want high sample efficiency with replay buffers (e.g., classic Atari-style controls).

Engineering judgement: start by asking whether discretization changes the nature of the task. If discretizing makes the game feel “grid-like” (steering snaps, aiming is stepwise), policy gradients will typically be simpler and higher quality.

REINFORCE learns a stochastic policy c0b8(a|s) by maximizing expected return. The key identity is the log-derivative trick: b7(b8)=E[a3 R] implies b1b7/b1b8 = E[b1 log c0b8(a|s)/b1b8 * G], where G is the return from that timestep. In practice, you sample trajectories with the current policy, compute returns, and do gradient ascent on log-probability weighted by return.

Workflow for a minimal REINFORCE loop:

• Roll out one episode (or several) using the current stochastic policy.
• Store (state, action, log_prob, reward) at each step.
• Compute Monte Carlo returns G_t = a3_{kbe t} b3^{k-t} r_k.
• Loss = -mean( log_prob_t * G_t ) (negative sign because optimizers minimize).
• Backpropagate, step the optimizer, reset buffers, repeat.

Common mistakes: (1) forgetting to detach the return tensor (it should not backprop through the reward computation), (2) mixing up signs (maximize return but minimize loss), (3) using an incorrect log_prob (ensure it matches the distribution you sampled from), and (4) computing returns across episode boundaries when you batch episodes together.

Practical outcome: you can now train a policy in Gymnasium tasks where the action is sampled from a distribution. Even before adding a baseline, you should see learning on simple problems, but it may be noisy and unstable—variance reduction is the next step.

Pure REINFORCE has high variance because returns can vary wildly between episodes, especially in sparse-reward or long-horizon games. The classic fix is to subtract a baseline b(s) that does not depend on the action. The gradient remains unbiased, but variance drops: replace G_t with (G_t - b(s_t)). The best baseline is the state value function Vc6(s), learned by regression to returns.

Implementation pattern (actor-critic style, but still Monte Carlo):

• Actor: policy network outputs a distribution c0b8(a|s).
• Critic: value network outputs Vc6(s).
• Compute returns G_t from rewards.
• Advantage A_t = G_t - Vc6(s_t).
• Actor loss = -mean( log_prob_t * stopgrad(A_t) ).
• Critic loss = mean( (Vc6(s_t) - G_t)^2 ).

Two engineering details matter. First, stopgrad (detach) the advantage in the actor loss so the critic does not get updated through the policy gradient term. Second, keep critic learning stable: use a separate optimizer or at least separate loss terms and possibly different learning rates. If the critic is too weak, advantages are noisy; if it overfits, it can introduce oscillations.

Advantage normalization is a simple, high-impact stabilizer: within a batch, normalize A to zero mean and unit standard deviation before the actor update. This does not change the solution (it scales the gradient), but it makes learning rates easier to tune and reduces sensitivity to reward scale.

For continuous actions, the most common stochastic policy is a factorized Gaussian: c0b8(a|s)=N(bcb8(s), c3b8(s)). Your network outputs a mean vector bc and either a log-standard-deviation vector logc3 (state-dependent) or a global parameter logc3 that is learned. Sampling uses reparameterization in PyTorch distributions: create Normal(bc, c3), sample, and compute log_prob of the sampled action.

Gymnasium environments often bound actions (e.g., Box(low=-1, high=1)). A raw Gaussian is unbounded, so you need to handle bounds. Two practical approaches:

• Clamp after sampling: simple but the log_prob no longer matches the executed action, which can hurt learning.
• Tanh-squashed Gaussian: sample u~N(bc,c3), set a=tanh(u), then scale to env bounds. Compute a corrected log_prob with the tanh change-of-variables term. This is more correct and typically more stable.

For a first implementation, you can start with clamping to validate the loop, then switch to tanh-squashing once learning works. When computing log_prob for multidimensional actions, sum per-dimension log_probs to get a scalar per timestep. Also, remember to store the log_prob from the distribution used at sampling time; recomputing later with updated parameters breaks the on-policy gradient.

Practical outcome: you can control continuous-action tasks (e.g., Pendulum, MountainCarContinuous, LunarLanderContinuous) with smooth behavior and meaningful exploration.

On-policy policy gradients are sensitive to how much data you collect per update. Updating after every single episode can work, but gradients are noisy. A more reliable pattern is to collect a batch of transitions (e.g., 2k–10k steps) possibly across multiple episodes, then do one or a few optimization epochs on that batch. This gives you better advantage statistics for normalization and more stable critic targets.

Recommended practical workflow:

• Collect N steps with the current policy (reset env as needed) and record per-step data.
• Compute returns per episode boundary; do not bootstrap unless you intentionally implement TD(bb) or GAE.
• Compute values V(s) and advantages A = G - V(s).
• Normalize advantages within the batch.
• Take one optimizer step for actor and one for critic (or a few critic steps if it lags).

Learning rates: policy gradients often need smaller actor learning rates than you expect (e.g., 3e-4 to 1e-3), while critics may tolerate similar or slightly higher rates. If returns oscillate or collapse, reduce the actor LR first. If the value loss explodes, reduce critic LR and consider clipping returns or rewards.

Gradient clipping is a practical safety net. Clip global norm (e.g., 0.5 to 1.0) for both actor and critic to prevent rare large advantages from causing catastrophic parameter jumps. This is especially important when reward scale is large or when early policies occasionally stumble into very high-return episodes that skew gradients.

Comparison note: DQN benefits from replay buffers and off-policy reuse of data; REINFORCE does not. Accept that policy gradients trade sample efficiency for compatibility with continuous actions and simpler action modeling.

Policy gradient training can fail silently: the code runs, but learning stalls or becomes unstable. Add diagnostics early. Three signals are especially useful: policy entropy, KL drift, and reward/advantage scale.

Entropy measures how random the policy is. For a Gaussian policy, entropy is directly related to logc3. Track mean entropy per batch. If entropy collapses too early (very small c3), exploration ends and the agent may converge to a bad local optimum. If entropy stays extremely high, the policy may never commit. A common remedy is adding an entropy bonus to the actor loss: L_actor = -E[log_prob * A] - b2 * E[entropy]. Tune b2 small (e.g., 1e-3 to 1e-2).

KL drift approximates how far the new policy moved from the old one on the same states. Large KL spikes indicate too-large updates (often from high learning rates or unnormalized advantages). Even without implementing PPO, you can compute KL(c0_old||c0_new) on the batch as a “speedometer.” If KL is consistently high, reduce actor LR, increase batch size, or tighten gradient clipping.

Reward scaling matters because returns directly scale gradients. If rewards are huge, gradients explode; if tiny, learning is slow. Standardize rewards with wrappers, rescale by a constant, or rely on advantage normalization (still track raw episode return as the true metric). Also watch the critic: a rapidly growing value loss often means the reward scale is too large or returns are poorly computed across episode resets.

Practical outcome: you can tell whether a learning problem is exploration (entropy), step size (KL), or signal scaling (reward/advantages). This makes iteration systematic rather than guesswork.

Section 5.1: Observation preprocessing (stacking, normalization)

Observation preprocessing is the first place wrappers pay off. Many Gym environments expose observations that are technically correct but awkward for learning: pixel frames with large dynamic range, low-level RAM values, or vectors whose components have very different scales. Your job is to present the agent with a consistent, information-rich signal while staying honest about what the agent can observe.

Frame stacking is the classic example. In Atari-style games, a single frame does not reveal velocity; stacking the last k frames makes motion inferable. The wrapper should (1) keep a fixed-size deque of frames, (2) return a stacked tensor with predictable layout (e.g., (k, H, W) for channels-first), and (3) handle episode resets by filling the stack with the initial frame. Common mistakes include forgetting to reset the buffer (leading to leakage across episodes) or stacking after resizing inconsistently (causing subtle shape changes).

Normalization is equally important for vector observations. Neural networks train faster when inputs are roughly standardized. A practical approach is an online running mean/variance wrapper that normalizes observations to zero mean and unit variance. However, be careful: if you normalize using statistics that include evaluation episodes, you contaminate your metrics. Separate training and evaluation environments, each with its own normalization state (or freeze stats for evaluation). For image observations, normalization often means scaling uint8 pixels to [0,1] or [-1,1] and optionally converting to grayscale and resizing.

• Engineering judgement: Prefer preprocessing that is reversible in intent (scaling, clipping, stacking) over transformations that destroy semantics. If you crop, confirm you are not removing score, timers, or enemy indicators.
• Debug tip: Log a few preprocessed observations (min/max, histograms, and sample frames) at startup. Many training “mysteries” are dtype/scale bugs.

When done well, preprocessing makes your training loop simpler: the model sees a fixed shape and stable numeric ranges across runs, seeds, and environment versions.

Section 5.2: Action constraints, clipping, and discretization tradeoffs

Action space design determines what your agent is capable of expressing. In Gymnasium, actions might be discrete (e.g., Discrete(n)) or continuous (e.g., Box(low, high)). Wrappers are the right place to impose constraints or map between representations so the policy outputs remain well-behaved.

For continuous control, clipping is a practical necessity: policies can output values outside the legal bounds, especially early in training. An ActionClipWrapper that clamps actions to [low, high] prevents invalid actions and environment errors. But note the learning implication: if many actions get clipped, the policy receives distorted gradients (it thinks it took action a, but the environment executed clip(a)). A better pattern is to parametrize the policy output with a squashing function (e.g., tanh) and then scale to bounds, so actions are always valid by construction.

Discretization is tempting when you want to use DQN-like algorithms on a continuous problem: discretize each dimension into bins and treat the cartesian product as a discrete action. The tradeoff is combinatorial explosion. Two dimensions with 11 bins each already yields 121 actions; four dimensions yields 14,641 actions, often impractical. Discretization can work for simple games or when only a subset of actions matters, but you must measure whether performance saturates due to coarse control.

• Anti-pattern: Adding a “do nothing” action to fix instability while the true issue is observation scale or reward design. This often delays learning rather than solving it.
• Practical compromise: For continuous tasks, start with policy gradients (or actor-critic) and use action squashing; for discrete tasks, keep the action set minimal but expressive, and consider action masking when some moves are illegal.

Action wrappers are also a clean place to implement game-specific ergonomics: map a smaller “meta-action” space to complex combos, or enforce constraints like “cannot move left and right simultaneously.” Always document the mapping in your experiment config so results remain interpretable.

Section 5.3: Reward shaping patterns and anti-patterns

Reward design is where you translate “what you want” into a learning signal. In games, sparse rewards (win/lose) often make exploration too hard, but naive shaping can produce agents that maximize shaped reward while ignoring the real objective. The goal is to add guidance without changing what optimal behavior means.

A dependable pattern is potential-based shaping: add a term of the form F(s, s') = gamma * Phi(s') - Phi(s), where Phi is a potential function encoding progress (distance to goal, remaining enemies, etc.). This preserves the optimal policy under standard assumptions while providing dense feedback. In practice, you implement it in a reward wrapper that tracks the previous potential and adds the shaped difference each step.

Another safe pattern is reward decomposition for logging. Keep the environment’s original reward as the “objective reward,” and add auxiliary components (e.g., exploration bonus, time penalty) but log each component separately. This helps you diagnose when the agent is “gaming” the auxiliary term.

• Common anti-pattern: Shaping that rewards intermediate behaviors you merely associate with success (e.g., “move right” in a platformer). Agents will learn to move right even when it is unsafe, because you encoded a shortcut, not progress.
• Common anti-pattern: Large negative penalties that dominate the return scale. If dying is -1000 and winning is +1, the agent may learn to stall forever to avoid risk. Keep reward magnitudes comparable and consider a small per-step cost to discourage stalling.
• Workflow: Start with the base reward; add the smallest shaping that fixes exploration; run an ablation (with/without shaping); and confirm the shaped agent improves on the base objective, not just on shaped return.

Finally, remember that wrappers can modify reward but should not hide it. Preserve both: train on the shaped reward if you must, but always evaluate and model-select on the original task reward to avoid self-deception.

Section 5.4: Curriculum learning and domain randomization basics

Curriculum learning makes hard problems learnable by ordering experiences from easy to difficult. In games, this can mean shorter levels, slower opponents, simplified physics, or starting the agent closer to the goal. The key is to control difficulty with a schedule that is explicit, logged, and reversible—so you can reproduce results and avoid accidentally training on a different task than you evaluate.

A practical implementation is a curriculum wrapper that reads a difficulty parameter and modifies environment settings at reset(). For example, every N episodes increase enemy speed by 5%, or expand the range of starting positions. Another approach is performance-based progression: when average episode return over the last 100 episodes exceeds a threshold, advance to the next difficulty stage. Performance-based schedules can be more data-efficient, but they can also “lock” if metrics are noisy; add patience and smoothing.

Domain randomization is a related idea aimed at robustness: randomize textures, spawn locations, wind, friction, or sensor noise. This prevents the policy from overfitting to one narrow configuration and often improves transfer to new levels or seeds. The engineering rule is to randomize factors that are nuisance variables, not the core objective. If you randomize too aggressively early, learning may collapse because the agent cannot form stable associations.

• Practical outcome: Curriculum improves sample efficiency; domain randomization improves generalization. Combine them by starting with narrow randomization and widening it as training progresses.
• Common mistake: Evaluating only on the curriculum’s latest stage. Always evaluate on a fixed “target” distribution (fixed seeds or a held-out seed set) so progress is meaningful.

When done carefully, curricula and randomization become knobs you can tune systematically—rather than a pile of ad-hoc environment changes that make experiments impossible to compare.

Section 5.5: Experiment tracking: logs, configs, artifacts

RL experiments are noisy, compute-heavy, and sensitive to seeds. If you cannot reconstruct what happened, you cannot trust your conclusions. A professional training pipeline treats tracking as a first-class feature: every run writes structured logs, saves configs, stores artifacts, and records the code version.

Start with a single source of truth for configuration (YAML/JSON or argparse) and save it verbatim into the run directory. Include: environment id/version, wrapper stack and parameters, reward shaping coefficients, model architecture, optimizer settings, seed(s), total steps, evaluation cadence, and any curriculum schedule. If you use sweeps, log both the sampled hyperparameters and the sweep definition. This is what makes “controlled hyperparameter sweeps” actually controlled.

For logs, write both step-level training metrics and episode-level metrics. At minimum: episode return (objective and shaped), episode length, success rate (if defined), loss values, Q-value statistics or entropy (depending on algorithm), and wall-clock throughput (steps/sec). For evaluation, run deterministic episodes with fixed seeds and log mean/median return plus confidence intervals across seeds.

• Artifacts worth saving: periodic checkpoints, best-model checkpoint by evaluation score, wrapper-normalization statistics, and short evaluation videos. Videos are expensive but excellent for debugging reward hacking.
• Baselines: always log a random policy baseline and, when possible, a simple heuristic. It prevents you from celebrating “learning” that is actually just environment bias.

Whether you use TensorBoard, Weights & Biases, MLflow, or plain CSV, the principle is the same: produce a complete story of each run so you can compare seeds, run ablations, and explain results months later.

Section 5.6: Checkpointing, model selection, and early stopping

Checkpointing is not just “save weights sometimes.” In RL, a checkpoint must capture enough state to resume training without changing the trajectory. That typically includes: model parameters, target network parameters (for DQN-like methods), optimizer state, replay buffer contents (or a clear policy for rebuilding it), running normalization statistics, environment and RNG states (Python, NumPy, torch), and the current global step/episode counters.

A robust pattern is to save two checkpoint streams: latest (for resuming after interruption) and best (for model selection). “Best” should be defined using evaluation on the objective reward, not the shaped reward. If you select on shaped return, you may pick a model that exploits the shaping term but performs worse on the real task. Also, keep evaluation separate from training: use a distinct environment instance and disable exploration noise (e.g., greedy policy for DQN, mean action for Gaussian policies) to reduce variance.

Early stopping in RL requires caution because learning curves can be non-monotonic. A practical approach is patience-based stopping on a smoothed evaluation metric: stop if the moving average has not improved for K evaluations, but only after a minimum number of steps to avoid terminating during initial exploration. For hyperparameter sweeps, early stopping can save compute, but ensure you apply the same stopping rule across trials, or your comparison becomes biased.

• Common mistake: Resuming from a checkpoint but changing the wrapper stack or observation normalization. That silently changes the problem and invalidates continuity.
• Model selection tip: Save multiple top checkpoints (top-3) and report performance across several evaluation seeds. Single-seed “best” models are often luck.

With checkpointing, careful model selection, and sensible stopping rules, your training pipeline becomes scalable: you can run long jobs, recover from failures, and produce results that stand up to scrutiny.

Section 6.1: Train vs eval separation and avoiding leakage

First, draw a hard line between training and evaluation. Training is allowed to be noisy, exploratory, and adaptive (epsilon-greedy, entropy bonuses, learning rate schedules). Evaluation should be deterministic and comparable across runs: fixed policy (no exploration), fixed environment configuration, fixed preprocessing, and consistent termination rules.

In Gym/Gymnasium code, this usually means creating separate environment instances: env_train and env_eval. Even if they are the “same” environment, they must be reset with different seeds and must not share wrappers that mutate state in subtle ways. For example, a normalization wrapper that updates running mean/variance during evaluation leaks evaluation statistics into the policy. The correct approach is: update statistics during training, then freeze them for evaluation.

Another leakage vector is checkpoint selection. If you evaluate every N steps and then pick the best checkpoint based on eval scores, you have effectively tuned on the eval set. A practical fix is to use three splits: (1) training rollouts, (2) validation rollouts for early stopping/model selection, and (3) a final test evaluation that you run once at the end. In smaller projects you can approximate this by selecting checkpoints based on training proxies (loss curves, Q-values) and only using eval for reporting, but the safest pattern is train/validate/test.

• During eval: disable exploration (epsilon=0, sample=False), disable learning, and freeze normalizers.
• Use identical wrappers between train and eval except those that must differ (e.g., frame stacking is identical; reward clipping should match if it’s part of the deployed agent).
• Log environment version, wrapper stack, and config hashes so you can prove train/eval parity.

Engineering judgment: if you plan to ship an agent into a game build, your evaluation environment should mimic the runtime as closely as possible (observation resolution, action repeat, frame time). Differences here often explain “it worked in the notebook” failures.

Section 6.2: Metrics: average return, success rate, regret, stability

“Average episodic return” is the default RL metric, but games often need additional measures to reflect player-facing outcomes. Choose metrics that map to your design intent, and report more than one so you can detect trade-offs (e.g., a high score strategy that occasionally catastrophically fails).

Start with average return over evaluation episodes (mean and distribution). Then add success rate when the task has a clear binary outcome (level completed, boss defeated). Success rate is easier to interpret than return when reward shaping is heavy or when dense rewards do not align perfectly with “winning.”

Regret is useful when you have a known benchmark, curriculum, or baseline policy. Define regret as the gap between your agent and a reference (optimal if available, or best baseline) per episode or per timestep. In practice, you can compute regret relative to a strong heuristic bot or a scripted policy: regret = return_baseline - return_agent (or the reverse, depending on convention). This helps quantify “how far are we from acceptable play?”

Stability captures reliability across time and across seeds. Report variance (or standard deviation) of returns, the worst-percentile performance (e.g., 5th percentile), and failure modes. In a shipped game agent, the tail matters: one in fifty catastrophic episodes can still be unacceptable if it looks like the AI “breaks.”

• Mean return: overall performance level.
• Success rate: goal completion reliability.
• Regret: distance to a baseline/target behavior.
• Stability: variance, percentiles, and failure frequency.

Common mistake: averaging returns across episodes with different lengths without also reporting episode length. Some agents “game” the metric by ending episodes early (intentionally or accidentally). Track average episode length and, if relevant, time-to-success.

Section 6.3: Statistical best practices (seeds, CIs, significance)

RL results can vary dramatically with random seeds. A single run is a demo, not evidence. Your evaluation harness should run multiple training seeds and multiple evaluation seeds per checkpoint. A practical baseline is 5 training seeds; in heavier environments you may choose 3, but then be explicit about uncertainty.

Implement an evaluation harness that takes: (1) a trained policy checkpoint, (2) a list of eval seeds, and (3) a fixed number of episodes per seed. Aggregate metrics across all episodes and report confidence intervals (CIs). Bootstrapped CIs are simple and robust: resample episodes with replacement, compute the mean, repeat (e.g., 10,000 times), and take the 2.5/97.5 percentiles for a 95% CI.

When comparing agents (e.g., DQN vs DQN+Double, or with/without target network), use paired comparisons when possible: evaluate both agents on the same set of seeds and episode indices. This reduces noise and makes ablation conclusions sharper. If you report significance, prefer non-parametric tests (like Wilcoxon signed-rank) on per-seed means, but avoid overemphasizing p-values. In engineering terms, effect size and confidence bounds are more actionable than a binary “significant/not.”

• Fix and log RNG seeds for: environment, NumPy, PyTorch, action sampling, and replay sampling if applicable.
• Use deterministic inference settings (e.g., torch.no_grad(), model.eval()).
• Store raw per-episode returns so you can recompute metrics later.

Common mistake: changing code between runs without versioning. Your harness should record git commit, dependency versions, and environment IDs. Otherwise, seed control won’t save you from silent drift.

Section 6.4: Robustness tests: perturbations, shifts, stress cases

After you have a clean eval harness, expand it into a robustness suite. Robustness asks: does the policy still behave reasonably when the world is slightly different from training? In games, this is unavoidable: frame timing changes, textures differ, physics updates vary, and players create states your training distribution rarely saw.

Start with observation perturbations. Add controlled Gaussian noise to continuous observations, salt-and-pepper noise to pixel inputs, or random occlusions (drop a small patch). If you use preprocessing like grayscale, resizing, or normalization, apply perturbations after preprocessing to simulate sensor noise at the policy interface. Measure degradation curves: performance vs noise level.

Next, test environment shifts. Examples include changing enemy speed, gravity, action repeat, or reward scaling. In Gymnasium, you can implement shifts via wrappers or environment parameters. Keep shifts small and interpretable; you want to learn whether the agent relies on brittle timing hacks or memorized trajectories.

Finally, add stress cases: adversarial initial states, rare corner cases, and longer horizon episodes. For example, randomize spawn positions more widely, extend max episode steps, or insert “stuck” states to see whether the agent recovers. Log qualitative traces for failures (state, action, Q-values/probabilities) so debugging is possible.

• Perturbations: observation noise, missing frames, input latency.
• Shifts: physics constants, dynamics, reward scaling, level layouts.
• Stress cases: rare starts, long episodes, recovery scenarios.

Engineering judgment: don’t chase robustness blindly. Decide which shifts reflect real deployment variation, and prioritize those. A robustness test suite is most valuable when it mirrors how the game will actually change (different device performance, patched content, or player-driven state diversity).

Section 6.5: Deployment packaging: saving models, inference wrappers

Shipping an agent means turning “a training script that produced a checkpoint” into “a versioned component that can run fast and predictably.” Begin by saving not just weights, but the full inference contract: observation preprocessing parameters, action space mapping, and any wrappers required to interpret the environment.

For PyTorch-based agents, store: model state_dict, architecture config (layer sizes, frame stack count), and normalization statistics. Consider exporting to TorchScript or ONNX if you need stable, optimized inference in a production runtime. Always verify numerical parity between the exported model and the original within a small tolerance.

Wrap inference in a small class with a single method, e.g., act(obs) -> action. Inside it: apply preprocessing, move tensors to the correct device, run no_grad, and return an environment-valid action. This wrapper is the boundary you will test and benchmark. Include a batch mode (act_batch) if you plan to run many agents or parallel simulations.

Optimize speed by reducing per-step overhead: avoid Python-side list manipulations, pre-allocate tensors, and keep the model on the target device. For games, latency matters more than throughput; measure wall-clock time per decision under realistic conditions (same hardware, same frame budget). If you use frame stacking, manage a ring buffer to avoid copying full stacks each step.

• Save artifacts: checkpoint, config JSON/YAML, wrapper stack description, and training metadata.
• Provide a minimal demo runner that loads the agent and plays N episodes headlessly.
• Benchmark: milliseconds per step, memory usage, and initialization time.

Common mistake: deploying with training-time wrappers that alter reward or termination. Deployment should only include observation/action transformations needed to run the policy, not training conveniences like reward clipping unless your policy truly expects it.

Section 6.6: Communicating results: plots, reports, and checklists

A strong experiment is one you can explain quickly and reproduce later. Your final deliverable should include: (1) plots that answer the performance question, (2) a short report that explains what you did and why, and (3) a reproducibility checklist. This is the difference between a promising prototype and an engineering-ready result.

For plots, include learning curves with shaded confidence bands across training seeds. Show evaluation metrics over training steps (not just at the end) so readers can see stability and potential overfitting. Add bar charts or tables for robustness suites (performance under noise/shift levels). If you ran ablations, plot them side-by-side: “full agent” vs “without target network” vs “without replay,” etc. Ablations are not busywork—they are how you prove which components are actually buying performance.

Your report can be concise (1–2 pages) if it is structured: environment description, algorithm and key hyperparameters, compute budget, baselines compared, ablation summary, robustness findings, and deployment notes (inference speed, export format). Include failure analysis: one or two representative failure modes and what triggers them.

• Reproducibility checklist: environment ID/version, code commit, dependency lockfile, seeds used, hardware, training steps, evaluation protocol, and artifact locations.
• Baseline checklist: random policy, heuristic/scripted policy, and previous best agent (if any), all evaluated with identical harness.
• Ablation checklist: remove one component at a time; keep everything else fixed, including seeds.

Common mistake: only reporting the best run. Always report across seeds with uncertainty. Decision-makers care about expected performance and risk, not a single lucky outcome.