Reinforcement Learning for Beginners: Teach AI Better Actions

RL starts with a simple idea: an agent improves by acting and receiving feedback. If you’ve ever adjusted your driving based on a “too close” warning sound, you’ve experienced a reinforcement loop. The crucial difference from many other AI approaches is that the agent is not told the correct action upfront. Instead, it discovers which actions tend to lead to better outcomes.

“Trial and error” can sound random or wasteful, but in RL it becomes systematic. You define what the agent can do (actions), what it can sense (observations or states), and how success is measured (rewards). Then you run repeated interactions and record experience. The system learns patterns like: “In this situation, action A tends to pay off more than action B.”

Engineering judgment matters because the feedback signal shapes everything. Poorly chosen rewards produce agents that optimize the wrong thing. A common beginner mistake is to reward something easy to measure rather than what you actually want. For example, rewarding a robot for moving quickly can create reckless behavior unless you also penalize collisions or unsafe speed. Practical outcome: before writing any learning code, write down what feedback will be available, how frequently it arrives, and what “good” behavior looks like in measurable terms.

The agent–environment loop is the backbone of RL. The agent is the learner/decision-maker: a game bot, a trading program, or a robot controller. The environment is everything the agent interacts with: the game world, the market simulator, or the physical room. At each step, the agent chooses an action, and the environment responds with a new situation and a reward.

You’ll see both state and observation in RL. A state is the full information needed to predict what happens next (in an ideal model). An observation is what the agent actually gets to see, which may be partial or noisy. Beginners often assume the observation is always the true state; in real systems it may not be. Practically, you start with the best summary of the situation you can reliably compute: position on a grid, remaining battery, current speed, etc.

A reward is a numeric score the environment returns after an action (sometimes immediately, sometimes later). Rewards are not “labels.” They are feedback. For everyday examples: in a thermostat controller, actions are “heat on/off,” reward might be +1 when temperature is in the comfort band and -1 otherwise. In a recommendation system, action could be “show item X,” reward could be a click or dwell time. Milestone check: you should be able to point to any interactive task and identify actions, observations/state, and rewards without ambiguity.

RL interaction is often organized into episodes: a sequence of steps that starts in an initial situation and ends when a terminal condition is reached. In a maze, an episode ends when the agent reaches the goal or times out. In a game of chess, the episode ends at checkmate or draw. Each decision point is a step (also called a time step).

Why bother with episodes? Because goals are usually about total outcome, not one-step outcome. A short-term reward might tempt the agent into choices that look good now but are bad overall. Consider a robot that gets +1 for picking up an object but -10 for dropping it. If it picks up quickly without navigating safely, it may rack up early rewards and then suffer a large penalty. RL formalizes “better actions” as those that maximize long-term cumulative reward, not just immediate reward.

Practically, this is where you define the goal. If your goal is “reach the exit quickly,” you might give a small negative reward each step (to encourage shorter paths) and a large positive reward at the exit. If your goal is “survive as long as possible,” you might reward each step survived. Common mistake: mixing goals (speed, safety, style) without shaping rewards carefully, leading to unstable learning. Clear episodes and a clear objective function make the learning problem tractable.

A policy is the agent’s strategy: a rule that maps situations to actions. It can be as simple as a lookup table (“in state S, take action A”) or as complex as a neural network. In beginner RL, you often start with a small discrete state space and represent the policy indirectly through action values (Q-values). The core idea remains: the policy tells the agent what to do next.

Two practical forces shape any policy: exploitation and exploration. Exploitation means choosing the best-known action right now. Exploration means trying actions that might be worse in the short run but could reveal better options. This is not philosophical; it is operational. If you never explore, you can get stuck with a mediocre strategy. If you explore too much, you waste time and may never settle into good behavior.

A common starter approach is epsilon-greedy: with probability ε, pick a random action (explore), otherwise pick the action with highest estimated value (exploit). Engineering judgment: choose ε based on risk and time budget. In a toy grid world, you can explore aggressively. In a real robot, exploration can be dangerous, so you may constrain actions or explore in simulation. Milestone check: you should be able to justify when and how your agent will try new behaviors versus repeating known good ones.

One of RL’s central challenges is credit assignment: figuring out which earlier actions deserve credit or blame for a reward that appears later. If a reward arrives only at the end of an episode (win/lose), how does the agent learn which move 20 steps earlier mattered? This is why RL can be harder than supervised learning: the feedback is delayed and sparse.

Q-learning addresses credit assignment by updating value estimates using bootstrapping: it uses the current estimate of future value to improve the estimate of the present. Conceptually, it pushes reward information backward through the chain of decisions. In practice, you store values like Q(state, action) and update them after each step. Over repeated episodes, the estimates become more accurate.

Common mistakes come from reward design and logging. If your reward is too sparse (only at the end), learning can be slow. If your reward is noisy (random spikes), the agent may chase randomness. If you forget to log transitions (state, action, reward, next state), you can’t debug learning. Practical outcome: design rewards that align with the goal while providing enough signal to guide learning, and ensure your system can trace which experiences led to changes in behavior.

Let’s model a tiny RL problem: a 2x2 grid where the agent starts at the top-left and wants to reach the bottom-right goal. The states are the four grid squares: S00, S01, S10, S11 (goal). The actions are {Right, Down}. If an action would leave the grid, the agent stays in place. The reward is +10 for entering the goal state S11, and -1 for every other step to encourage shorter paths. An episode ends when S11 is reached.

We can build a Q-table with rows as states and columns as actions. Initialize all Q-values to 0. Choose learning rate α = 0.5 and discount γ = 0.9. Suppose the agent is in S00 and takes Right to S01, receiving reward -1. The Q-learning update is:

Q(S00, Right) ← Q(S00, Right) + α [ r + γ max_a Q(S01, a) − Q(S00, Right) ]

Numbers: Q(S00, Right)=0, r=-1, max_a Q(S01,a)=0 initially. So Q becomes 0 + 0.5[ -1 + 0.9*0 - 0 ] = -0.5.

Next, from S01 take Down to reach S11 and get +10. Update Q(S01, Down): 0 + 0.5[ 10 + 0.9*max_a Q(S11,a) - 0 ]. In terminal S11, treat max future value as 0, so Q becomes 5. Now the table encodes experience: from S01, Down looks good; from S00, Right looks slightly bad so far. After more episodes, Q(S00, Right) will improve because it leads to a state with a high-value action. This is the core workflow: define states/actions/rewards, collect transitions, update the Q-table step by step, and then derive a policy by taking the action with the highest Q in each state.

Section 2.1: Observations vs states (what the agent really “knows”)

In RL, it is tempting to say “the state is everything about the world.” In practice, the agent only has access to what you feed it—its observations. A state is the information you assume is sufficient to choose a good action. If you leave out something essential, the same “state” can require different best actions, and learning becomes noisy or impossible.

Start with Milestone 1: choose an environment you can fully describe. Then do Milestone 5 early: spot missing information and fix the state description. For our vacuum example, the raw world might include the agent’s room (Left/Right), whether each room is dirty, and the battery level. If you only include the agent’s room and ignore battery, you create contradictions: the best action in the Left room might be “clean” when battery is high, but “move right to charge” when battery is low. The agent will see identical inputs but different outcomes, which looks like randomness.

A practical rule: if the outcome of an action depends on some variable, that variable probably needs to be in the state. Another rule: prefer small, discrete states for beginner projects. For example, battery can be bucketed into {Low, High} instead of a continuous percentage at first. That gives you a manageable number of states and a Q-table you can inspect.

Common mistake: mixing up “what would be nice to know” with “what the agent can know.” If the agent cannot sense dirt in the other room, do not include it in the state unless you explicitly allow that sensor. If you want partial observability, you can still learn, but you’ll need memory or belief-state methods later. For now, design the state so it matches the observation and is sufficient for good decisions.

Section 2.2: Action spaces: small lists of choices vs many choices

An action space is the set of choices the agent is allowed to make. For a first RL project, keep it small and explicit (Milestone 2: write down states and actions without ambiguity). In the vacuum environment, you might define actions as: {MoveLeft, MoveRight, Clean, Charge}. Each action must have a clear meaning, preconditions, and consequences. For example, what happens if the agent chooses Charge in the Left room where no charger exists? Options include: “no-op” (nothing happens), “illegal action penalty,” or “action masked out” (not available). Pick one and document it. Ambiguity here produces confusing learning signals.

Small discrete action spaces are a great fit for Q-learning and Q-tables. You can store a value for each (state, action) pair and see what the agent prefers. With many choices—say a robot arm with continuous torques—tables become impossible, and you need different algorithms. Even with discrete actions, the count can explode quickly if you encode “move to any grid cell” rather than one-step moves. A key engineering judgment is choosing actions that are simple primitives but still expressive enough to reach the goal.

Exploration vs exploitation starts to show up immediately. If the action space is small, an b5-greedy policy is easy: usually pick the current best action (exploit), but with probability b5 pick a random action (explore). When actions have different risks (e.g., Clean costs energy), exploration can be expensive. Your design goal is to make exploration safe enough that the agent can learn, for example by bounding negative rewards or terminating episodes before catastrophic spirals.

Common mistake: defining actions that secretly include extra intelligence, like “go clean the nearest dirty room.” That may be useful later, but it hides decision-making inside the action and makes it harder to learn and interpret. Beginners learn faster by using simple, atomic actions and letting the policy emerge.

Section 2.3: Reward design basics: positive, negative, and shaping

Rewards are the training signal. Milestone 3 is to design rewards that match the real goal. If your real goal is “keep both rooms clean while avoiding running out of battery,” your reward must reflect that—not just “cleaning is good.” A simple base reward scheme might be: +10 for successfully cleaning a dirty room, -1 per time step (to encourage efficiency), and -20 if the battery hits zero (failure). This already creates a meaningful trade-off: cleaning is valuable, but wasting time or dying is costly.

Negative rewards (penalties) are often more important than positives, because they prevent loopholes. Without a step penalty, the agent might wander indefinitely after cleaning one room. Without an “out of battery” penalty, it might keep cleaning until it dies, if cleaning yields enough reward per step.

Reward shaping adds intermediate signals to guide learning. For example, you might add +2 for moving toward the charger when battery is Low, or +1 for reaching a fully-clean state (both rooms clean). Shaping can speed up learning, but it can also distort behavior if it becomes the true objective. A practical guideline: shaping rewards should correlate with real success and not be easily “gamed.” If you reward “being near the charger” too much, the agent may camp at the charger forever.

Common mistake: using rewards that measure what is easy to compute rather than what you actually want. Another mistake: mixing incompatible scales (e.g., +0.1 for cleaning, -100 for a small mistake) that make learning unstable or overly cautious. Start simple, test with a few hand-simulated episodes, and adjust. You should be able to explain in plain language what behavior the reward encourages.

Section 2.4: Transitions: what changes after an action

A transition is “what happens next” after the agent takes an action. This is where your environment becomes a system the agent can probe through trial and error—systematic trial and error, not random guessing. In our example, taking MoveRight changes the agent’s location; taking Clean changes dirt status; every action reduces battery by 1; Charge increases battery, but only in the right room. These rules define the dynamics the agent is trying to learn.

Write transitions as explicit, testable rules. If you can, implement a small step function: input (state, action) be output (next_state, reward, done). Even if you are not coding yet, describe it like you are. This practice forces you to remove ambiguity (Milestone 2) and to ensure the state contains what the transition depends on (Milestone 5).

Decide whether transitions are deterministic or stochastic. Deterministic means Clean always works if there is dirt. Stochastic might mean Clean succeeds 90% of the time. Stochastic transitions are realistic and still learnable, but they increase the amount of experience needed. For beginners, deterministic transitions reduce debugging time because you can reproduce behavior step by step.

Common mistake: silently allowing “impossible” transitions (e.g., moving left from the leftmost room) without defining the result. If you choose “no-op,” the agent might learn to exploit it if the reward makes it beneficial. If you choose a penalty, you teach boundary awareness. The key is consistency: the same (state, action) should produce a well-defined distribution over next states and rewards.

Section 2.5: Termination and success criteria

An episode is one run of interaction from a start condition to a terminal condition. Milestone 4 is deciding when an episode starts and ends. This is not cosmetic: termination shapes what the agent learns because it defines which outcomes count as “final” and how long-term consequences matter.

In the vacuum environment, possible start states include random dirt configurations and a starting battery level. Terminal conditions could include: (1) battery reaches zero (failure), (2) both rooms are clean and battery is at least Low (success), or (3) a maximum step limit is reached (timeout). The max step limit is a practical tool: it prevents infinite wandering and makes training stable, especially before rewards are tuned.

Define success in terms you care about, not just “got reward.” If the goal is sustained cleanliness, you might end the episode when both rooms are clean for the first time, but that teaches “clean once” rather than “maintain clean.” Alternatively, you can run fixed-length episodes and reward cleanliness at each step. That teaches maintenance behaviors but can be harder to learn initially. This is an engineering decision: pick the version that matches your objective and your learning method.

Common mistake: changing termination rules mid-experiment without tracking it. Another is having terminals that the agent can reach too easily (episodes end before meaningful learning) or too rarely (the agent gets little feedback). A good beginner setup provides frequent, interpretable endings: clear success, clear failure, and a timeout that encourages efficiency.

Section 2.6: A gentle MDP view: the five parts without heavy math

What you built in the previous milestones is essentially a Markov Decision Process (MDP) description—without needing formulas. An MDP has five parts: (1) States: what the agent knows and uses for decisions; (2) Actions: the choices available; (3) Rewards: the scoring rule; (4) Transitions: how the world changes after actions; (5) Termination (or terminal states): when an episode ends. If you can write these five parts clearly, you have turned a real situation into RL ingredients.

This framing also supports systematic “trial and error.” The agent is not trying random things blindly; it is estimating which actions lead to better long-term reward from each state. In a tiny MDP, you can store these estimates in a Q-table. Each entry Q(state, action) is the agent’s current guess of “how good it is” to take that action from that state, considering future rewards too.

Even before doing numeric Q-learning updates, you can sanity-check the table’s shape. If you have 8 states and 4 actions, you expect 32 Q-values. If you accidentally defined 200 states because you used a continuous battery percentage, you will feel it immediately: the table becomes sparse, learning slows, and debugging becomes difficult. This is why Milestone 1 (small, fully describable environment) and Milestone 5 (state completeness) matter.

Finally, the MDP view clarifies what you are not modeling. If you omit a variable, you are saying the agent cannot use it. If you simplify transitions, you are defining a simplified world. That is acceptable—often necessary—as long as it serves the practical outcome: a learnable problem where the learned behavior transfers to the real goal you care about.

Section 3.1: What “value” means: why some states feel promising

When we say a state has “value,” we mean it tends to lead to good outcomes if you behave well from there. In practice, value is a prediction. A state might feel promising because it’s close to a goal, because it offers many good action choices, or because it avoids dangerous outcomes. This is the mental model: the agent is trying to navigate toward high-value states and away from low-value ones.

In a small, tabular problem, we often store action-values, written as Q(s, a). A Q-value is the agent’s current estimate of “how good it is to take action a in state s,” considering not only the immediate reward but also what might happen next. A Q-table is just a grid of these numbers. Early on, most entries are zero or random, which is exactly why greedy choices can fail early: the “best” action is often best only because everything else is untested.

Example: imagine a tiny grid world with a start state S and two actions: Right and Up. The first time the agent tries Right, it accidentally hits a small reward (+1). If it now always exploits greedily, it may keep choosing Right forever—even if Up would lead, after two steps, to a much bigger reward (+10). Value is about the bigger picture, and early evidence is too thin to trust.

• Practical workflow: initialize Q-table; run episodes; update Q(s,a) after each step; periodically inspect which states/actions have high Q.
• Common mistake: interpreting early Q-values as “truth.” They’re provisional estimates that require exploration to become reliable.

As you read the rest of the chapter, keep this goal in mind: exploration is not “being random for fun,” it’s how you earn the right to exploit confidently later.

Section 3.2: Immediate reward vs long-term return

Agents do not optimize single-step rewards; they optimize return: the total reward accumulated over an episode (often discounted, which we’ll cover later). Tracking return makes “trial and error” systematic because it forces you to evaluate sequences of decisions, not isolated moves. This is Milestone 3 in practice: you should be able to compute the total reward across an episode and use it to compare behaviors.

Consider an episode with rewards over four steps: -1, -1, -1, +10. The immediate rewards look bad at first, but the episode return is +7, which is good. If the agent only chases immediate reward, it will avoid those initial -1 steps and never reach the +10 outcome. This tension shows up in real systems too: a robot may need to “waste” motion to reposition; a recommendation system may need to show slightly less certain content to learn user preferences.

To make this concrete in a Q-learning update, imagine you are in state s, take action a, receive reward r, and land in s'. Q-learning updates your estimate toward: r + (best future value from s'). Even without heavy math, the idea is straightforward: credit assignment spreads the eventual success backward to earlier actions. That is how the agent learns that those early -1 steps were “worth it.”

• Engineering judgment: define episodes so that return is meaningful (e.g., “until goal,” “until failure,” or a fixed horizon). If episodes end too early, long-term payoffs cannot be learned.
• Common mistake: logging only per-step reward and forgetting to compute per-episode return. Two policies can have similar step rewards but very different returns if one reaches the goal faster or more reliably.

When you later compare policies (Milestone 4), use returns across many episodes, not a single lucky run. RL is stochastic; you want a trend, not a story.

Section 3.3: Exploration strategies: random, epsilon-greedy, soft preferences

Exploration answers a practical question: “How do I try alternatives without throwing away everything I’ve learned?” The simplest approach is pure random actions. That guarantees coverage, but it can be wasteful because it ignores what you already know. Most beginner systems quickly move to epsilon-greedy (Milestone 2): with probability ε you explore (pick a random action), and with probability 1-ε you exploit (pick the current best action in your Q-table).

Here’s a small example. Suppose in state A your Q-table says: Q(A, Left)=2.0 and Q(A, Right)=1.5. With ε=0.2, 80% of the time you choose Left (exploit). The remaining 20% you choose randomly; if there are two actions, that means 10% Left and 10% Right. So overall you choose Left 90% and Right 10%. This is enough to keep testing Right occasionally, which prevents “early lock-in” where greedy choices fail due to limited data (Milestone 1).

A more nuanced family of methods uses soft preferences: actions with higher estimated value are chosen more often, but not deterministically. A common version is a softmax-like choice rule (sometimes called Boltzmann exploration). You don’t need the formula to use the intuition: increase “temperature” to explore more evenly; decrease it to behave more greedily. Soft preferences are helpful when you want exploration to focus on near-best actions rather than uniformly random ones.

• Practical guidance: start with epsilon-greedy because it is easy to implement and debug. Add soft preferences later if uniform random exploration is too wasteful.
• Common mistake: using ε=0 or ε too small at the start. If your Q-table begins as all zeros, greedy selection becomes arbitrary but consistent—your agent may repeat the same arbitrary choice and never learn alternatives.

In real training loops, log how often you explore versus exploit. If exploration nearly disappears early, you may be over-trusting immature Q-values.

Section 3.4: The role of randomness and why it’s useful

Randomness is not a bug in reinforcement learning—it’s a tool. It serves three practical purposes. First, it ensures coverage: the agent actually visits states and actions that would otherwise remain unknown. Second, it helps escape local optima: a habit that is “good enough” but blocks discovery of something better. Third, it provides statistical robustness: by sampling different trajectories, your Q-values become estimates based on multiple experiences rather than one path.

There is also a subtle point: environments are often stochastic. The same action can lead to different results due to noise, opponents, or changing conditions. If the agent never repeats an action in varied contexts, it can form overconfident beliefs from a single lucky outcome. Controlled randomness (like epsilon-greedy) forces repeated sampling, which reduces the chance that a fluke becomes permanent policy.

From an engineering standpoint, randomness must be managed. Use random seeds for reproducibility during debugging. When you think you “fixed” learning, rerun with multiple seeds to confirm it wasn’t a coincidence. If your results swing wildly across seeds, you may need more exploration, slower learning rates, or more episodes before evaluation.

• Common mistake: evaluating a policy while still exploring heavily. If ε is large during evaluation, you are measuring a noisy mixture of behaviors, not your learned policy.
• Practical outcome: separate training and evaluation: train with exploration on; evaluate with ε=0 (or near-zero) so you can compare policies fairly (Milestone 4).

Used well, randomness is how an agent becomes confident. Used carelessly, it is how you convince yourself learning is unstable when the real issue is that you’re measuring the wrong thing.

Section 3.5: Discounting (gamma) explained with everyday trade-offs

Discounting controls how much the agent cares about the future. The parameter γ (gamma) is between 0 and 1. If γ is near 0, the agent is short-sighted: it mostly values immediate rewards. If γ is near 1, it is far-sighted: it treats future rewards as almost as important as current ones. This is an everyday trade-off. Choosing between “eat a snack now” versus “wait for a full meal later” is a discounting decision; so is “take a quick shortcut with risk” versus “take a safer longer route.”

Discounting matters directly in your Q-learning update. Conceptually, you update Q(s,a) toward: immediate reward + γ × (best estimated value of the next state). If γ is small, the future term barely matters, and the agent may never learn multi-step strategies. If γ is large, the agent will tolerate short-term costs to reach later gains—but it may also propagate noisy future estimates backward, which can slow stabilization.

This connects to Milestone 5 (tuning exploration for stability) because γ and exploration interact. With high γ, the algorithm relies more on estimated future values; those estimates are uncertain early on, so aggressive exploration can create large swings in Q-values. You can counter this by reducing ε gradually (epsilon decay), training longer, or using smaller learning rates so the table updates more gently.

• Practical rule of thumb: for short tasks (few steps to goal), γ around 0.8–0.95 often works. For longer-horizon tasks, push γ higher—but expect slower, noisier learning.
• Common mistake: setting γ=1 without thinking. In continuing tasks or very long episodes, this can make learning unstable or overly sensitive to delayed randomness.

Discounting is not just a math trick; it is the knob that translates “long-term goals” into a learnable signal.

Section 3.6: Measuring progress: average reward, success rate, steps to goal

To know whether exploration is helping, you need metrics that reflect your goal. Three practical measures cover most beginner projects: average episode return, success rate, and steps to goal. Average return tells you whether the agent is collecting more reward over time (Milestone 3). Success rate is a clean signal when episodes have a clear win/fail outcome. Steps to goal captures efficiency: two agents might both succeed, but one learns to do it faster and with fewer penalties.

This is where you compare policies (Milestone 4) in a disciplined way. For example, Policy A might be highly exploratory (ε=0.3) and Policy B more exploitative (ε=0.05). During training, A may look worse because it “wastes” steps exploring, lowering immediate return. But when evaluated with ε=0 (pure exploitation), A might outperform B because it discovered a better route and filled in the Q-table more thoroughly. The right comparison is: train both under their exploration settings, then evaluate both under the same evaluation setting.

To improve stability (Milestone 5), track these metrics as moving averages (e.g., over the last 50 episodes). Single episodes are noisy. Also watch for warning signs: if average return increases but success rate falls, the agent might be gaming the reward function rather than solving the task. If success rate is high but steps to goal are flat, it may have learned a safe but inefficient strategy.

• Practical workflow: log per-episode return, success/fail, and step count; plot moving averages; evaluate periodically with exploration turned off; repeat across several random seeds.
• Common mistake: tuning ε based on one metric only. Balance reliability (success rate) and efficiency (steps) against reward totals to get a complete picture.

Progress in RL is measured, not guessed. Once you can track and compare these metrics, exploration stops feeling like a mystery and becomes an adjustable design choice.

Section 4.1: From values to action-values (why Q is useful)

In reinforcement learning, you often hear about “value.” A state value answers: “How good is it to be in this state?” That’s useful, but it hides an important detail: in most states you can choose multiple actions, and those actions can lead to very different outcomes.

An action-value (a Q-value) answers a more actionable question: “How good is it to take action a when I’m in state s?” That is exactly what an agent needs to decide what to do next. If you know the Q-values for the available actions, you can pick the best action immediately, rather than first estimating the state’s value and then reasoning about transitions.

This is why Q-learning is popular for beginners: it gives you a direct path from experience to behavior. Every experience tuple—(state, action, reward, next state)—can update a single cell in memory. Over time, those cells become a map of “what tends to work here.”

Practical outcome: you can implement decision-making without building a full model of the environment. Common mistake: treating Q-values as guaranteed outcomes. They are estimates based on limited experience, and they can be wrong early on. Engineering judgement is about keeping that in mind and balancing learning (updating estimates) with using what you have learned (acting on the best current estimate).

Section 4.2: The Q-table structure: rows as states, columns as actions

A Q-table is a matrix where each row is a state and each column is an action. The cell at row s, column a stores Q(s, a): your current guess of the long-term “goodness” of taking action a in state s.

Milestone 1 is being able to lay out this table for a tiny environment. Start by listing discrete states. For a toy navigation task you might define states as positions: S0, S1, S2, Terminal. Then list legal actions: Left, Right (or Up/Down/Left/Right in a grid). Build a table with one row per non-terminal state; terminal states often don’t need actions because the episode ends there.

Milestone 2 is reading Q-values correctly. A Q-value is not “how much reward you get immediately,” but an estimate of total future reward (discounted if you use a discount factor). If Q(S1, Right) is larger than Q(S1, Left), that means “from S1, going Right tends to lead to better outcomes over time.”

• Initialize Q-values to 0 for unknown tasks, or small optimistic values to encourage exploration.
• Only include legal actions for each state, or mask illegal actions so you never choose them.
• Keep state definitions consistent. If you accidentally treat the same situation as two different states, your table fragments learning.

Common mistakes include mixing up states and observations (especially if you later move to partially observable tasks), or forgetting that “state” must contain enough information to make a good decision. In a tiny example, you control this easily, which makes Q-tables a great learning tool.

Section 4.3: Learning rate (alpha): how fast to change your mind

When you update a Q-value, you are revising a belief. The learning rate, usually written as alpha (α), controls how strongly new experience overrides old estimates.

Conceptually, an update is: new estimate = (1 − α) × old estimate + α × new information. If α is 1.0, you completely replace your old guess with the new target from the most recent step. If α is 0.1, you move only 10% of the way toward the new target, which makes learning slower but more stable.

Engineering judgement: choose α based on how noisy and how non-stationary the environment is. If rewards and transitions are stable and deterministic, a higher α can work fine and learns quickly. If outcomes are noisy (e.g., stochastic rewards), a smaller α helps you average over many experiences rather than overreacting to one lucky or unlucky transition.

Milestone 3 starts here: you should be able to do the arithmetic of an update with a calculator and see how α changes the result. Example: if Q(s,a)=2.0 and your new target is 6.0, then with α=0.5 you update to 4.0; with α=0.1 you update to 2.4. Same evidence, different willingness to change your mind.

Common mistakes: setting α too high and watching values swing wildly, or setting α too low and concluding “it doesn’t learn.” Also, α is not a reward scale; if you change reward magnitudes by 10×, you may need to revisit α (and other hyperparameters) to keep learning stable.

Section 4.4: Bootstrapping idea: learning from a guess plus new evidence

Q-learning is built on a powerful shortcut called bootstrapping: you improve an estimate using another estimate. Instead of waiting until the end of an episode to know the total return, you update after each step using immediate reward plus your current guess about the future.

The standard Q-learning target is:

target = reward + gamma × max_a' Q(next_state, a')

Here, gamma (γ) is the discount factor (how much you care about future rewards). The term max Q(next_state, a') is your best current guess of how much reward you can still get from the next state if you act well from there onward.

This is “learning from a guess plus new evidence.” The new evidence is the reward you just observed. The guess is your current table entry for what comes next. With repeated experience, these guesses become better and the whole table self-consistently improves.

Practical workflow: after each transition (s, a, r, s'), compute the target, then update Q(s, a) toward that target using α. This makes learning online and incremental—important for agents that must learn while acting.

Common mistakes: forgetting the max over next actions (accidentally using the Q-value of the action you happened to take next), or bootstrapping from terminal states incorrectly. For a terminal next state, the future term is 0 because the episode ends: target = reward.

Section 4.5: Greedy policy derived from Q (argmax in plain terms)

Once you have Q-values, you can turn them into a behavior rule called a policy. The simplest policy is greedy: in each state, choose the action with the highest Q-value. In plain terms, this is “pick the action that looks best according to your table.”

Mathematically you may see:

policy(s) = argmax_a Q(s, a)

Argmax just means “the index (action) with the biggest number.” This is Milestone 4: improving a policy by choosing the best Q action. If your row for state S1 has Q-values [Left=1.2, Right=3.7], the greedy choice is Right.

However, greedy behavior can get stuck if the table is incomplete or wrong early on. That’s why exploration matters. A common practical approach is epsilon-greedy: with probability ε, pick a random action (explore); otherwise pick the greedy action (exploit). Even a small ε (like 0.1) can prevent the agent from prematurely committing to a suboptimal action just because it got an early lucky reward.

• If ε is too low too early, you may never discover better actions.
• If ε is too high for too long, learning looks noisy and performance may not improve.
• Tie-breaking matters: if two actions have equal Q, choose randomly among them to avoid bias.

Practical outcome: you can watch a policy emerge directly from the table. When the best action stabilizes in most rows, your agent’s behavior becomes consistent and goal-directed.

Section 4.6: Worked mini-example: filling a Q-table over a few episodes

Now we’ll fill a Q-table manually for a tiny environment to make the update process feel mechanical and predictable (Milestone 3) and to show how a better policy emerges (Milestone 4). Consider a 3-state “line world”:

• States: S0, S1, S2 (S2 is terminal).
• Actions: Left, Right (illegal moves keep you in the same state).
• Rewards: reaching S2 gives +1 and ends the episode; all other moves give 0.
• Hyperparameters: α = 0.5, γ = 0.9. Initialize all Q(s,a)=0.

Episode 1 (one possible path): Start at S0.

Step 1: in S0 take Right → next state S1, reward 0. Compute target = 0 + 0.9 × max(Q(S1,Left), Q(S1,Right)) = 0. Update Q(S0,Right): old 0 → new = 0 + 0.5 × (0 − 0) = 0. Nothing changes yet because the future estimate is still zero.

Step 2: in S1 take Right → next state S2 (terminal), reward +1. For terminal, max future value is 0, so target = 1. Update Q(S1,Right): old 0 → new = 0 + 0.5 × (1 − 0) = 0.5.

After Episode 1, your table has learned one useful fact: from S1, going Right seems good (0.5).

Episode 2: Start at S0 again.

Step 1: in S0 take Right → S1, reward 0. Now max Q in S1 is max(0, 0.5)=0.5. Target = 0 + 0.9 × 0.5 = 0.45. Update Q(S0,Right): old 0 → new = 0 + 0.5 × (0.45 − 0) = 0.225.

Step 2: in S1 take Right → S2, reward +1. Target = 1. Update Q(S1,Right): old 0.5 → new = 0.5 + 0.5 × (1 − 0.5) = 0.75.

Now notice the bootstrapping: S0’s value for going Right increased even though S0 itself never produced reward. It learned because S1 looks promising, and Q-learning propagates that promise backward through experience.

Reading the table as a policy: In S1, greedy action is Right (0.75 beats 0). In S0, greedy action is also Right (0.225 beats 0). With more episodes, Q(S0,Right) will continue moving toward 0.9 × 1 = 0.9 (because from S0 you need two steps to reach reward, so discount applies once at S0’s update).

Milestone 5 (when tables stop scaling): This example works because the state space is tiny and discrete. In real problems, states explode: position × velocity × inventory × time-of-day × user context, etc. A table needs one entry per (state, action). If you have 1,000,000 states and 20 actions, that’s 20 million numbers to store and update—often too big, and many states may never be visited. This is the key limitation: Q-tables don’t generalize. If you learn that “S1, Right is good,” it tells you nothing about “a similar but unseen state.”

Practical takeaway: use Q-tables to master the mechanics—state/action design, update arithmetic, and policy extraction. Then, when the environment grows, you’ll know exactly what you’re replacing when you move from a table to a function approximator (like a neural network) that can generalize across similar states.

Section 5.1: Q-learning goal: learn best actions without a model of the world

Q-learning is designed for a common situation: you can try actions and observe what happens, but you do not have (or do not trust) a complete model that predicts the next state and reward for every possible action. In other words, the agent learns directly from experience. This is why Q-learning is called model-free.

The “Q” stands for “quality.” A Q-value, written Q(s, a), is an estimate of the quality of taking action a in state s. High Q means “this action tends to lead to good total reward.” The word “total” matters: RL is usually not just about the immediate reward. Many tasks have delayed outcomes (you sacrifice now to gain later). Q-learning handles this by learning an estimate of the future-return you can expect after acting.

Practically, beginners often start with a tiny world where states and actions are countable. For example, a 3-room navigation task might have states {A, B, C} and actions {Left, Right}. Your Q-table would then have 3×2 entries. The agent runs episodes (a start state, a sequence of steps, then a terminal state), and after each step it slightly updates the relevant Q-table cell. Over many episodes, the table stabilizes into a policy: “in each state, pick the action with the highest Q.”

• Outcome you should expect: Q-learning can discover effective behavior even if your reward signal is imperfect, as long as it contains some learning signal.
• Common beginner misconception: “The table stores the right answer immediately.” It doesn’t. It stores estimates that are noisy early on and improve via repeated updates.

This section connects to the bigger RL workflow: you are still modeling the problem as states, actions, and rewards (the MDP idea), but you are not solving it with heavy math; you are letting the agent approximate action quality through trial and error.

Section 5.2: The update rule explained using a simple sentence template

Milestone 1 is being able to say the Q-learning update rule in words. Use this sentence template:

New Q(s, a) = Old Q(s, a) moved a bit toward “what I got now” plus “the best I can likely get later.”

In symbols, after you take action a in state s, observe reward r, and land in next state s':

Q(s,a) ← Q(s,a) + α [ r + γ max_a' Q(s',a') − Q(s,a) ]

Read it as three pieces:

• Old estimate: Q(s,a)
• Target (what we wish Q were): r + γ max_a' Q(s',a')
• Step size: α controls how strongly we move toward the target

Milestone 2 is to run updates on paper. Here is a tiny numeric example you can compute step-by-step. Suppose all Q-values start at 0, and you use α=0.5 and γ=0.9. You are in state A, take action Right, get reward r=+2, and move to state B. In state B, the best known action currently has Q(B,·)=0 (because everything is still 0). The target is 2 + 0.9*0 = 2. The update becomes:

Q(A,Right) ← 0 + 0.5*(2 − 0) = 1

Now do one more step: from B you take action Right, get reward r=+5, and reach terminal state (episode ends). For a terminal next state, treat max Q(terminal,·)=0. Target is 5. Update:

Q(B,Right) ← 0 + 0.5*(5 − 0) = 2.5

Notice what happened: the earlier action at A did not yet “know” that B could later yield +5. But after many episodes, the max Q term propagates value backward: once Q(B,Right) becomes large, future visits to A will update Q(A,Right) toward 2 + 0.9*(2.5), increasing A’s estimate. That is how delayed rewards become learnable through repeated experience.

Section 5.3: Off-policy intuition: learning best-next-action while exploring

Milestone 4 is understanding (without heavy theory) why Q-learning is often described as off-policy. The short intuition: Q-learning can behave one way to gather experience, while it learns the value of behaving optimally.

Look back at the update target: r + γ max_a' Q(s',a'). The max means “assume that from the next state onward, I will take the best action I know.” But during actual interaction, you might not take the best action—you might explore. So your behavior policy (how you act) can differ from the greedy policy (what the Q-table currently recommends). Q-learning still updates toward the greedy future, which is why it can learn an optimal policy even while acting non-optimally sometimes.

This matters in practice because exploration is not optional. If you always pick the current best-known action, the agent may never discover better actions. With Q-learning you can safely use an exploration strategy (like ε-greedy in the next section) and still push your Q-values toward “what would happen if I acted optimally from here.”

• Engineering judgment: early training should be more exploratory; late training should be more exploitative. Off-policy learning makes this transition easier.
• Practical interpretation: your Q-table is trying to represent “best possible future” even if some of your collected steps came from random moves.

One caveat for beginners: off-policy is not magic. If exploration is extremely poor (you never see key states), Q-values cannot become accurate. Off-policy helps you learn from whatever data you have, but it cannot invent experience you never collect.

Section 5.4: Hyperparameters that beginners can tune: alpha, gamma, epsilon

Milestone 3 is adding exploration and watching learning improve over time. In Q-learning, three beginner-friendly knobs strongly affect whether learning is stable and fast: α (alpha), γ (gamma), and ε (epsilon).

Alpha (α): learning rate. It sets how much each new experience overrides the old estimate. If α is too high (close to 1), Q-values may bounce around based on recent random outcomes. If α is too low (close to 0), learning can be painfully slow. A practical starting range is 0.1 to 0.5 for toy problems. If your environment is noisy (rewards vary), reduce α.

Gamma (γ): discount factor. It controls how much you care about future rewards compared to immediate rewards. If γ=0, the agent becomes short-sighted and only learns immediate payoff. If γ is near 1 (like 0.95 or 0.99), the agent strongly values long-term outcomes but may learn more slowly because distant consequences matter. For short episodes, γ can be high; for tasks where you truly want quick payoff, lower it.

Epsilon (ε): exploration rate. The common ε-greedy rule is: with probability ε choose a random action; otherwise choose the action with max Q(s, a). Early on, you might set ε=0.2 or 0.3 so the agent samples alternatives. Over time, you often decay ε (for example, multiply by 0.99 each episode until reaching a floor like 0.05). This creates a natural shift: explore to discover, then exploit to refine.

• Beginner workflow: (1) start with moderate ε, (2) confirm Q-values change in sensible directions, (3) decay ε, (4) adjust α if values are unstable, (5) adjust γ if the agent ignores long-term reward.
• Practical outcome: a Q-table where the best action in each state becomes consistent across episodes, and episode returns trend upward.

When you “run it on paper,” you can simulate ε-greedy by literally flipping a coin (or rolling a die) to decide whether you explore on a step, then applying the same Q-update. This makes the explore/exploit trade-off feel concrete rather than theoretical.

Section 5.5: Common issues: sparse rewards, loops, and reward hacking

Milestone 5 is diagnosing failures. Q-learning is simple, but simple does not mean foolproof. Three failure modes show up repeatedly in beginner projects.

1) Sparse rewards. If the agent gets reward only at the very end (e.g., +1 for reaching the goal, 0 otherwise), learning can be extremely slow because most steps look identical. Symptoms: Q-values stay near zero and the agent wanders. Fixes include shaping rewards (small positive signals for progress), shortening episodes, or ensuring exploration reaches the goal occasionally (higher ε early, or better start-state variety).

2) Loops and “safe wandering.” The agent may find a loop that avoids negative outcomes and never reaches the goal. This is especially common if you have no step penalty. Symptoms: episodes hit the time limit; Q-values for loop actions grow because they lead to states with similarly valued actions. Fixes: add a small negative reward per step (a “living cost”), add penalties for revisiting states, or enforce terminal conditions that prevent infinite wandering.

3) Reward hacking. The agent optimizes exactly what you measure, not what you meant. If your reward can be exploited (e.g., giving points for touching a checkpoint, and the agent learns to bounce on the checkpoint forever), Q-learning will happily do that. Fixes: redesign the reward to reflect the true goal, cap repeated reward from the same event, or add a stronger terminal reward that dominates small exploit rewards.

• Engineering judgment: when behavior looks “wrong,” don’t assume the algorithm is wrong first—inspect the reward and termination rules.
• Practical debugging tip: print (or log) tuples (s, a, r, s’) for a few episodes. Many issues become obvious when you see the actual rewards being generated.

Remember that Q-learning updates are local: they only change the visited (state, action) pair. If your agent never experiences a crucial transition, its Q-table cannot reflect it. Many “algorithm” bugs are actually “data coverage” bugs caused by insufficient exploration or overly sparse reward signals.

Section 5.6: Practical checklist: when learning looks “stuck” and what to try

When Q-learning looks stuck, you want a repeatable checklist rather than guesswork. Use the following sequence; it maps directly to the milestones you achieved in this chapter.

• Verify the update rule with one hand-computed step. Pick a single transition (s, a, r, s’). Compute the target r + γ max Q(s’,·) and confirm your code moves Q(s,a) toward it by α. This catches sign errors, wrong indexing, and terminal-state mistakes.
• Confirm episodes actually end. If episodes never terminate, learning can be dominated by loops. Add a max-steps cap and treat it as terminal; consider a step penalty.
• Check exploration is real. Print how often random actions are chosen under ε-greedy. If ε decays too quickly, the agent may lock into a mediocre policy. Try keeping ε higher for longer or setting a minimum ε (like 0.05–0.1).
• Inspect reward scale and signs. If rewards are tiny (e.g., 0.001) compared to noise, Q-values move slowly; if rewards are huge, Q-values can become extreme and unstable. Normalize or rescale rewards so typical episode returns are in a reasonable range (often single to tens for toy tasks).
• Tune α and γ with a purpose. If Q-values oscillate and policy keeps changing, lower α. If the agent ignores delayed payoff, raise γ. If it over-values distant rewards and becomes indecisive, lower γ slightly.
• Measure progress with two plots (or logs). Track (1) episode return over time and (2) the fraction of greedy actions chosen. Rising return with decreasing randomness is a healthy pattern.

Finally, interpret your Q-table directly. For each state, read off the best action (argmax over actions). If the recommended action is nonsense, locate which Q-values are inflated or never updated. This is the practical bridge between “numbers in a table” and “an agent that acts.” Once you can do this diagnosis, Q-learning stops being a formula and becomes a tool you can control.

Section 6.1: Scaling limits: why tables break with many states

Q-tables are perfect for learning because they are transparent: each state–action pair has a number you can read. The scaling problem is that the table grows as states × actions. That growth becomes impossible surprisingly fast. If your “state” includes multiple features—position, speed, inventory, remaining time, user segment, device type—then the number of distinct states explodes. Even when each feature has only a few possible values, the combinations multiply.

In real projects, the state is often continuous (temperature, distance, confidence scores) or very high-dimensional (images, text embeddings). A Q-table cannot store entries for infinitely many states, and even a large but finite table becomes sparse: you will visit only a tiny fraction of entries, so most Q-values stay unlearned. This creates brittle behavior: the agent performs well in familiar states and fails when anything changes.

Common mistake: discretizing everything aggressively to “make a table work.” Coarse discretization can hide important differences and create reward hacking. For example, if you bucket all customer wait times into “short/long,” the agent may treat 1 minute and 9 minutes as identical, which can produce unacceptable service outcomes.

• Milestone 1 (project brief): explicitly define what information is in the state and what is not. If you cannot write the state in a sentence, it is probably too large or too vague.
• Milestone 2 (baseline): before RL, implement a simple policy (e.g., greedy rule, heuristic thresholds). If a baseline works, you have a safety net and a comparison point.
• Practical outcome: you can recognize when a Q-table is a learning tool versus a deployable solution.

When you hit scaling limits, do not “fight the table.” Instead, change the representation: move from memorizing each state to learning a function that generalizes across similar states.

Section 6.2: The idea of approximation: using a model to estimate Q

Function approximation is the key conceptual upgrade: instead of storing Q(s,a) in a table, you learn a model that takes a state (and optionally an action) and outputs an estimated Q-value. You are replacing “lookup” with “prediction.” The advantage is generalization: if the model learns that certain patterns in the state lead to good outcomes, it can produce reasonable Q estimates for states it has never seen exactly before.

You can start simple. A linear model might estimate Q from a weighted sum of state features: if inventory is high and demand is low, discounting is good. A small decision tree might learn that certain regions of state space prefer certain actions. These approximators are easier to debug than deep networks and often strong enough for moderate problems.

Milestone 1 becomes more important here: your project brief must specify which state features are available at decision time and how they are computed. Feature leakage is a classic mistake—using information that is only known after the action (for example, including “next day churn” as a state feature). Leakage will make training look great and deployment fail.

Milestone 2 (baseline and metric) keeps you honest. Choose a success metric that reflects the goal and costs: average return per episode, constraint violations, or a weighted business metric. Also define a baseline policy like “do nothing,” “always choose action A,” or a hand-tuned heuristic. If your approximated Q policy cannot beat the baseline reliably, it is not ready.

• Engineering judgment: prefer the simplest approximator that meets your performance target and is stable under small changes.
• Common mistake: treating approximation as magic and ignoring data coverage; the model only generalizes within the patterns it has seen.

Approximation is not just about scaling. It also forces you to think carefully about what the agent can observe and how you will validate that the learned policy is robust.

Section 6.3: What deep reinforcement learning adds (high-level only)

Deep reinforcement learning (deep RL) is function approximation using neural networks. Its main benefit is representation learning: instead of hand-designing features, the network can learn useful internal features from raw inputs like images, audio, or large vectors of signals. This is why deep RL appears in game-playing agents, robotics, and complex simulation environments.

What deep RL adds in practice is a larger toolbox for stability and sample efficiency. Training can be unstable because the target you are trying to predict (future returns) depends on the policy you are changing. Popular methods introduce techniques like experience replay (reusing past transitions), target networks (slowing down moving targets), normalization, and carefully chosen exploration strategies. You do not need to implement all of these to understand the principle: deep RL is powerful but sensitive to setup details.

A practical way to avoid getting lost is to treat deep RL as a later milestone, not step one. First, prove the problem is well-posed with a small environment and a baseline. Second, define your episode boundaries and reward clearly (Milestone 1). Third, confirm you can measure success and compare it to a baseline (Milestone 2). Only then consider deep RL if simpler approximators cannot handle the state space or the input modality.

• When deep RL is justified: high-dimensional observations (camera frames), complex continuous control, or policies that must react to many interacting signals.
• Common mistake: using deep RL to solve a problem that is actually supervised learning, contextual bandits, or a fixed optimization problem.

Deep RL can unlock capability, but it also increases the need for careful evaluation and safety checks, because failure modes are harder to interpret than with a Q-table.

Section 6.4: Evaluation basics: train vs test, variance, and repeat runs

In toy problems, you can eyeball learning curves and watch the agent improve. In real projects, you need evaluation discipline. First, separate training from testing. Training is where the agent explores and updates its policy. Testing is where you freeze learning and measure behavior. Mixing them hides problems: a policy that looks good during training might rely on lucky exploration or transient dynamics.

Second, account for variance. RL results can change across random seeds, environment stochasticity, and initial conditions. A single run can mislead you. Repeat runs and report averages and variability (for example, mean return with a confidence interval). This is not bureaucracy; it is how you learn whether improvements are real.

Milestone 2 (baseline and success metric) becomes your anchor in evaluation. If your success metric is “average return,” also track safety and constraint metrics (collisions, budget violations, latency). Compare against a baseline policy under the same test conditions. If RL only wins by violating constraints, it did not actually win.

• Practical workflow: (1) build a minimal simulator or offline evaluation harness, (2) run the baseline, (3) run the RL policy with learning turned off, (4) repeat across seeds, (5) inspect failure cases.
• Common mistake: tuning on the test set—changing rewards, hyperparameters, or stopping criteria based on test results. Keep a validation loop separate from final reporting.

Good evaluation is what turns “trial and error” into a systematic engineering process. It also gives you the confidence to proceed to safer real-world tests.

Section 6.5: Safety and alignment basics: rewarding the right thing

RL agents do what you reward, not what you meant. This is the central safety lesson: mis-specified rewards create unintended behavior. If you reward “speed,” the agent may drive dangerously. If you reward “clicks,” it may learn manipulative recommendations. If you reward “short call time,” it may hang up on customers. These are not edge cases; they are predictable outcomes of optimization.

Milestone 1 (project brief) should include a reward specification section with: (a) what you are rewarding, (b) what you are penalizing, (c) what constraints must never be violated, and (d) what signals are proxies rather than true goals. Write down at least three “ways the agent could cheat” and how you will detect them.

Milestone 5 is your safety plan. Start with safe testing layers: unit tests for reward calculation, small-scale simulation tests, offline replay tests (if available), and staged rollouts with strict monitoring. Use “guardrails” where appropriate: action filters that block unsafe actions, rate limits, budget constraints, or a human-in-the-loop approval step for high-impact actions.

• Common mistake: only measuring the reward and ignoring side effects. Always log auxiliary metrics that reflect user harm, resource usage, or policy compliance.
• Practical outcome: you can design rewards that better align with real goals and build detection for reward hacking.

Safety is not an afterthought in RL. Because the agent learns through interaction, the cost of mistakes can be real. Good alignment work starts at the reward and continues through testing and deployment controls.

Section 6.6: Your next steps: tools to learn next and mini-project ideas

To keep momentum without getting overwhelmed, choose a “next step” that matches your current skill. If you are comfortable with Q-learning updates and interpreting a Q-table, your next goal is to implement a small end-to-end RL workflow: define an environment, write a project brief, pick a baseline, train, test, and report results with repeat runs.

Tools to learn next (pick one layer at a time): a lightweight RL environment library (so you can focus on the agent), plotting tools for learning curves, and simple approximators (linear models) before deep networks. Also learn experiment hygiene: configuration files, fixed seeds, and structured logging of rewards and constraints.

Mini-project ideas that naturally enforce the milestones:

• Inventory replenishment simulator: state = inventory and demand signal; actions = order amounts; reward = profit minus holding cost. Baseline = reorder point rule. Metric = average profit with stockout rate constraint.
• Website notification timing: state = time since last notification and user activity; actions = send or wait; reward = engagement minus annoyance penalty. Include a strict constraint on maximum notifications.
• Grid navigation with hazards: scale up from a toy grid by adding stochastic movement and unsafe zones. Reward shaping and safety constraints become visible quickly.

Milestone 4—knowing when to use RL—should guide your project selection. If the best action is obvious from a fixed rule, use the rule. If there is no sequential dependency (no delayed consequences), consider simpler methods like supervised learning or bandits. Use RL when actions influence future states in meaningful ways and you can define a reward and evaluation process you trust.

By the end of these projects, you will not only “run RL,” you will manage it: clearly scoped, measurable, and safer to iterate. That is the real transition from toy problems to real work.