Reinforcement Learning by Building Small Win-or-Lose

Section 1.1: Learning from rewards in daily life

The easiest way to understand reinforcement learning is to begin with ordinary life. Imagine learning to ride a bicycle. No one gives you a complete table of correct moves for every possible wobble. Instead, you try, adjust, fall, recover, and slowly discover which actions help you stay balanced. Or think about finding the quickest walking route across a campus. After a few attempts, you remember which turns save time and which paths are blocked or crowded. This is the everyday shape of reinforcement learning: trial and error guided by consequences.

What makes RL different from other kinds of machine learning is that the learner is not handed perfect examples of the right answer in advance. Instead, it must act first and evaluate later. A move is made, the world changes, and then a reward or penalty arrives. Sometimes the signal is immediate. If you touch a hot pan, the penalty is obvious and instant. Sometimes the signal is delayed. If you study consistently for a month, the reward may only appear on exam day. RL must handle both cases.

This practical viewpoint matters because beginners often imagine reinforcement learning as a mysterious optimization machine. It is more helpful to see it as structured experimentation. The system is constantly asking, “What happens if I do this here?” In a small game, the answers are visible. Move left into a wall: no progress. Move right toward the goal: better. Step into a trap: lose. The environment becomes a teacher through feedback.

There is also a useful engineering lesson here. Rewards do not need to be emotional or human-like. They are simply numbers or signals that rank outcomes. A coin collected in a grid world may be worth +1. Falling into water may be -5. Reaching the exit may be +10. The agent does not need to “understand” coins or water the way we do. It only needs a consistent signal that allows comparison between choices. Once that signal exists, repeated experience can turn random play into improved behavior.

A common mistake is to assume that one good outcome proves the agent has learned. It does not. Random behavior occasionally wins by luck. In RL, learning means improving the odds of future success across many attempts. That is why we care about repeated episodes, not just isolated wins. Over time, behavior should shift from accidental success toward reliable strategy.

Section 1.2: The agent and the environment

Every reinforcement learning problem begins by separating the decision-maker from the world it acts in. The decision-maker is called the agent. The world around it is called the environment. This distinction is simple but powerful. The agent chooses actions. The environment responds with a new situation and a reward. That loop repeats until the attempt ends.

In a maze game, the agent might be a small marker on paper or a character in code. The environment is the maze itself: the walls, open spaces, traps, coins, and exit. In a coin hunt, the agent decides where to move next, while the environment determines whether that move reaches a coin, hits a boundary, or wastes a turn. In a path game, the environment may contain safe paths, dead ends, and a finish square. Once you start thinking this way, many problems become easier to model.

Why is this separation important? Because good RL design depends on assigning responsibilities clearly. The agent should only control what it can decide. The environment should enforce the rules. If the agent can move up, down, left, or right, those are actions. If there is a wall at the top of the current square, the environment blocks that move or leaves the agent in place. This helps prevent a common beginner error: blending the logic of choice with the logic of world response until the learning loop becomes hard to reason about.

There is also an engineering judgment here about simplicity. In your first environments, keep the agent limited and the world explicit. Avoid hidden rules. If touching a red square means losing, define that clearly. If collecting all coins ends the game, state it upfront. Ambiguous environments make debugging painful because you cannot tell whether the agent is learning badly or whether the world itself is inconsistent.

When you later build a Q-table, this agent-environment view becomes even more useful. The table stores what the agent has learned about actions in particular situations. It does not store the whole world directly. That means your environment must present situations in a form the agent can recognize. Even in a paper game, this habit matters. Name the world. Name the player. List the legal actions. Define the results. Clear definitions lead to learnable systems.

Section 1.3: States, actions, and simple choices

Once you know who the agent is and what the environment is, the next step is to describe the agent’s current situation. In RL, that situation is called a state. A state is the information the agent uses to decide what to do next. In a simple grid game, the state may just be the current square location. In a coin hunt, the state might include both the location and whether a coin has already been collected. In a path game, the state could be the current node on a small map.

An action is a choice the agent can make from a state. Common actions in beginner environments are move up, move down, move left, move right, stay still, or pick up an item. The key is that the action set should be small and explicit. When you are learning RL, a tiny set of actions is a feature, not a limitation. It helps you inspect behavior closely.

Practical RL starts by asking two design questions. First, what information does the state need so the agent can make a good decision? Second, what actions should actually be allowed? If the state hides something important, the agent may appear confused even though the real issue is poor representation. If you allow too many actions, the search problem grows before you understand the basics.

For a first paper-based game, a 3x3 or 4x4 grid works well. Write the coordinates on each square. Pick a start square and a goal square. That coordinate is the state. The actions are the movement directions. Then play several rounds by hand. At each state, choose an action randomly for a while and record what happened. You will quickly see that some states have obvious better actions than others. That observation is the seed of a Q-table: storing which action seems best in each state.

A common mistake is to confuse states with outcomes. “Win” is not usually a state in the middle of the game; it is a terminal result that may occur after an action. Another mistake is to define states so loosely that the same label covers different situations. In RL, clear states make learning stable. Good action design makes comparison possible. Together, they turn vague decision-making into something a machine can improve.

Section 1.4: Rewards, penalties, and delayed results

Rewards are the feedback signals that drive reinforcement learning. If states describe where the agent is, rewards describe how well things are going. A reward can be positive, negative, or zero. Positive values encourage behavior. Negative values discourage it. Zero means no obvious gain or loss from that step. In tiny environments, rewards are often attached to clear events: reaching a goal, collecting a coin, wasting a move, or entering a trap.

This sounds simple, but reward design is where many RL projects succeed or fail. Suppose your maze gives +1 for every move and +5 for reaching the exit. The agent may learn to wander forever because moving itself is rewarded. If instead you give a small penalty for each step, such as -1, and a larger reward for the goal, the agent is pushed to find shorter paths. The lesson is practical: the reward function defines the behavior you are likely to get, not the behavior you vaguely hope for.

You should also distinguish between win, lose, and score signals. A win signal might be a strong reward for reaching the finish. A lose signal might be a large penalty for falling into a trap. A score signal might reward partial progress, like collecting coins along the way. These can coexist. In a coin hunt, the agent may gain small rewards for coins but a bigger reward for finishing the hunt. That creates a layered objective and makes the environment more realistic without becoming too complex.

Delayed rewards are especially important. Sometimes the best action now does not pay off immediately. Moving away from a nearby coin might be the right choice if it leads to the exit faster. Beginners often expect learning to come only from immediate feedback, but RL shines when outcomes unfold over time. The agent must connect present choices to future consequences. That is why repeated episodes matter and why a memory structure like a Q-table becomes useful.

When testing a new environment, inspect the rewards manually. Ask yourself: can the agent exploit a loophole? Could it earn points without actually solving the task? Are the penalties so large that exploration becomes impossible? Good engineering judgment means starting with simple, visible rewards and adjusting only after observing behavior. Reward signals are your training language. Write them carefully.

Section 1.5: Episodes, goals, and stopping points

Reinforcement learning is usually organized into repeated attempts called episodes. An episode begins at a start state, continues through a sequence of actions and rewards, and ends at a stopping point. That stopping point might be reaching the goal, losing the game, or hitting a maximum number of steps. Episodes are valuable because they turn an ongoing decision process into units you can measure, compare, and improve.

In a maze, one episode might start at the entrance and end when the agent exits, gets trapped, or exceeds twenty moves. In a path game, an episode could end when the player reaches the finish or walks into a losing square. In a coin hunt, it might stop when all coins are collected or when time runs out. These rules matter because they define success and failure clearly. Without stopping points, an agent may drift endlessly and learning becomes difficult to evaluate.

Goals should be concrete. “Do well” is not a usable RL objective. “Reach the exit in as few steps as possible” is usable. “Collect at least two coins before finishing” is usable. Clear goals let you design rewards and choose terminal conditions that match the task. They also help when comparing policies later. A random policy may eventually stumble into the goal. A greedy policy may follow the best-known action at each step. An explore-versus-exploit strategy may spend some time testing uncertain options to discover whether an even better route exists.

A frequent beginner mistake is ending episodes too late or too early. If an episode ends immediately after one minor mistake, the agent may never explore enough to learn. If it runs too long, the feedback signal becomes weak and noisy. Another mistake is to change the goal halfway through testing. In engineering terms, moving targets produce unclear learning signals.

For your first environments, choose short episodes with visible endings. That keeps the data understandable. You can count wins, losses, total score, and average steps. Those simple measures are often enough to reveal whether learning is happening. Episodes give RL its rhythm: start, act, observe, stop, repeat, improve.

Section 1.6: Mini project plan for a win-or-lose world

Now put the ideas together by designing a tiny paper-based decision game. Keep it small enough to simulate by hand in ten minutes. A good first project is a 4x4 grid with one start square, one goal square, one trap square, and one coin square. The agent starts in the same place every episode. The legal actions are up, down, left, and right. If the agent tries to move into a wall, it stays where it is. Reaching the goal is a win. Entering the trap is a loss. Collecting the coin gives a small bonus.

Write your environment rules first. This is part of good engineering discipline. Define the state representation, such as grid coordinates. Define the actions. Define the rewards, for example: step penalty -1, coin +3, goal +10, trap -10. Define the stopping conditions: end the episode on goal, trap, or after fifteen steps. Once the rules are written, do not change them during your first few trials. Stability helps you see how behavior changes.

Next, run several episodes using random choices. Record the sequence of states, actions, and rewards. This gives you a baseline. Random play is not useless; it shows the natural difficulty of the environment and exposes any design problems. Then make a simple greedy experiment. If one action from a state has led to better results in your notes, prefer it. You are informally approximating what a Q-table will later formalize: choosing actions based on learned estimates of value.

As you work, compare three behaviors: random action selection, greedy action selection using current best guesses, and a mixed approach that sometimes explores and sometimes exploits. This last idea is central to RL. If you only exploit what currently looks best, you may miss a better route. If you only explore, you never settle into a reliable strategy. Learning requires both.

Finally, review the project like an engineer. Did the rewards guide sensible behavior? Was the goal reachable? Did episodes end cleanly? Were there loopholes, such as bouncing safely forever? A tiny win-or-lose world is not just a toy. It is a controlled laboratory for understanding the full RL loop. Once this paper design is clear, turning it into code and then into a basic Q-table becomes much easier and much more meaningful.

Section 2.1: Creating a small maze or board world

The first design task in reinforcement learning is to create a world that is small, consistent, and easy to inspect. A grid world is the most common starting point because it is visual and structured. Imagine a board made of squares, such as 3x3, 4x4, or 5x5. Each square represents a possible location for the agent. The agent begins on one square and tries to reach another square while following rules. This is enough to model many useful ideas: navigation, obstacles, goals, penalties, and paths.

For a first environment, choose the smallest board that still allows decisions. A 4x4 grid is a good default. It gives enough room for choice without becoming messy. Mark one square as the starting position. Mark one or two squares as special locations such as treasure or trap. Leave the rest as ordinary tiles. At this stage, do not worry about graphics. A simple row-and-column representation is enough. In code, the world can be stored as a two-dimensional list, a matrix of symbols, or even just a set of coordinate rules.

Engineering judgment matters here. The board should be understandable at a glance. If the agent starts too close to the goal, the task becomes trivial. If the goal is buried behind too many obstacles, random play may almost never succeed, making debugging harder. A useful balance is to place the start and goal far enough apart to require several moves, but not so far that the game becomes mostly wandering. Keep the number of special tiles small so you can predict likely paths.

A common mistake is mixing world layout with learning logic too early. Keep them separate. The environment should answer questions like: where is the agent, what squares exist, and what happens when the agent tries to move? The learning algorithm comes later. This separation makes the system easier to test. You can run the world with a human-written sequence of actions and check whether it behaves correctly before you ask the agent to learn inside it.

The practical outcome of this step is a tiny board world with a clear map, a starting square, and space for wins and losses. Once that exists, every later concept in reinforcement learning has somewhere concrete to live.

Section 2.2: Defining legal moves and boundaries

After creating the board, define exactly what the agent is allowed to do. In a grid world, the standard actions are up, down, left, and right. These are the agent's possible moves. But an action list alone is not enough. You must also define what happens when the action points outside the board or into a blocked square. This is where the environment becomes precise rather than vague.

Legal move design has an important teaching purpose. It shows that actions depend on the current state. If the agent is in the top row, moving up may be illegal because there is no square above. If the agent is in the leftmost column, moving left may also be illegal. You can handle this in two common ways. One approach is to reject the move and leave the agent in the same place. Another approach is to prevent the action from being chosen at all. For beginners, allowing the action but keeping the agent in place is often easier to implement and explain.

Boundaries should be deterministic. The same action in the same state should always produce the same result unless you intentionally want randomness in the environment. For example, if the agent at position (0,0) chooses up, and your rule says boundary hits keep the agent in place, then the next state should remain (0,0) every time. This reliability is essential for debugging and later for learning a stable value table.

There is also an engineering trade-off in whether to punish illegal or useless moves. Some environments give a small negative reward for bumping into a wall, because it discourages wasted actions. Others simply allow the move to fail with no extra penalty. Both are valid, but be deliberate. If everything has zero cost except winning, the agent may wander for a long time without pressure to improve. If penalties are too harsh, the game may become dominated by fear rather than goal-seeking. Small, consistent signals work best.

A common mistake is forgetting to document the move rules clearly. If you later build a Q-table, you need to know exactly which actions exist and how the state changes after each one. Practical reinforcement learning depends on this clean contract between the agent and the environment. The output of this step is a complete action system: what moves exist, where they are allowed, and what happens at every edge of the map.

Section 2.3: Adding win squares, lose squares, and safe squares

A world becomes meaningful when different squares have different consequences. This is where we turn plain locations into reinforcement learning signals. The simplest pattern is to classify tiles into three types: win squares, lose squares, and safe squares. A win square ends the episode with a positive reward. A lose square ends the episode with a negative reward. A safe square allows the game to continue, usually with no reward or a small step cost.

This design turns winning and losing into numbers. That may sound cold, but it is exactly how an agent learns. The reward is not emotion. It is feedback. If treasure gives +10, a trap gives -10, and an ordinary move gives -1 or 0, then the agent can compare outcomes over many episodes. It starts to discover that some paths produce better total reward than others. The reward function is therefore the teaching signal of the world.

Good reward design is one of the most important pieces of engineering judgment in reinforcement learning. If the win reward is too small, the agent may not care much about reaching it. If the lose penalty is tiny, traps may not matter. If there is no cost for taking extra steps, the agent may eventually find the treasure but learn slow, sloppy paths. A common beginner setup is: treasure = +10, trap = -10, normal step = -1. That small step penalty encourages shorter routes without overwhelming the main objective.

Safe squares deserve attention too. They are not just empty spaces. They shape the decision problem. A longer safe corridor may protect the agent from risk but cost more steps. A shorter path might pass dangerously close to a trap. This is where strategy begins to emerge. Even in a tiny board, reward design can create real trade-offs.

A common mistake is hiding the goal in too many special cases. Keep the reward rules simple and readable. The environment should be easy to explain in one sentence: reach the treasure, avoid the trap, and try not to waste moves. When rewards are simple, later learning behavior is easier to interpret. The practical result is a world where the agent can now win, lose, or continue safely, and every outcome is measurable.

Section 2.4: Recording steps, turns, and outcomes

Once the world has actions and rewards, every move should be recorded as a state change. This is a core habit in reinforcement learning. At each turn, the environment should know at least five things: the current state, the chosen action, the next state, the reward received, and whether the episode is finished. That sequence is the story of learning. Without it, you cannot inspect behavior, debug the environment, or later update a Q-table properly.

Think of an episode as one full playthrough from start to finish. If the agent begins at the start square, takes six moves, reaches a trap, and the game ends, then those six transitions form one episode. Recording them lets you answer practical questions. Did the agent move the way you expected? Did the boundary logic work? Did the reward only appear when it should? Was the episode terminated correctly when the agent landed on a lose square?

A useful beginner logging format is simple text output such as: step number, old position, action, new position, reward, done. This may seem basic, but it is powerful. You can watch the agent move through the tiny world and immediately spot errors. For example, if the agent steps onto treasure but the episode does not end, your done flag is wrong. If moving right from column 3 sends the agent to column 4 in a 4x4 board, your boundary logic is broken.

Tracking turns also helps control experiments. Many environments set a maximum number of steps per episode, such as 20 or 30. This prevents endless wandering when the agent acts randomly. If the step limit is reached, the episode ends even without a win or loss. This is a practical safeguard and an important design choice. It tells the agent that time matters and keeps training runs manageable.

Common mistakes include recording too little information or mixing printed output with reward logic in confusing ways. Keep the environment step function clean: it should take an action and return the next state, reward, and terminal status. Then let a separate loop handle reporting and analysis. The practical outcome is an environment whose behavior can be audited move by move, which is essential before any serious learning begins.

Section 2.5: Why random play is a useful starting point

Before asking an agent to be clever, let it be random. This is one of the most useful habits in early reinforcement learning work. Random play means the agent chooses from its available actions without strategy. It does not know where the treasure is, does not try to avoid traps, and does not optimize anything. At first, that sounds pointless. In practice, it gives you a baseline and a test harness at the same time.

Random behavior answers an important question: does the environment work at all? If you run 100 random episodes and the treasure is never reached, perhaps the map is impossible or the rules are too restrictive. If the agent constantly gets stuck against boundaries, maybe the action handling is wrong. If reward totals look wildly off, you may have a bug in outcome scoring. Random play exposes these issues before learning logic makes them harder to diagnose.

It also teaches an essential idea about exploration. An agent cannot improve if it never experiences different states and outcomes. Random actions are the simplest form of exploration. They scatter the agent through the environment and reveal what can happen. Later, when you compare random choice with greedy choice and explore-versus-exploit strategies, this baseline becomes valuable. You will see clearly that learning is not magic. It is improved decision-making relative to uninformed behavior.

From an engineering perspective, random play is also useful for collecting sample trajectories. You can measure average episode length, average reward, win rate, and loss rate. These numbers create a starting benchmark. Suppose random play wins 12% of the time on your treasure board. Later, if a simple policy wins 70% of the time, you have concrete evidence of progress.

A common beginner mistake is dismissing random behavior as too dumb to matter. But in reinforcement learning, randomness is not just noise. It is a way to probe the world. Another mistake is evaluating performance after only a few random episodes. Because outcomes vary, use enough trials to see stable patterns. The practical result of this step is confidence that your tiny world behaves sensibly and a baseline against which smarter methods can be judged.

Section 2.6: Project 1 setup: a tiny treasure hunt game

Now we can assemble everything into a first complete project: a tiny treasure hunt game. Use a 4x4 grid. Let the agent start in the top-left corner. Place the treasure in the bottom-right corner. Add one trap somewhere along a tempting route, such as near the center. Every other tile is safe. The available actions are up, down, left, and right. Moves that cross the boundary leave the agent in the same position. This is enough structure to create meaningful trial and error.

Define the rewards clearly. Reaching the treasure gives +10 and ends the episode. Landing on the trap gives -10 and ends the episode. Every ordinary move gives -1, including failed moves into walls if you want to discourage waste. Set a maximum episode length, such as 20 steps, so the game cannot continue forever. If the step limit is reached without treasure or trap, the episode ends with whatever total reward has been accumulated so far.

The workflow for the project is straightforward. First, represent the state as the agent's grid position, such as (row, column). Second, implement a step function that accepts an action and returns the next state, reward, and done flag. Third, write an episode loop that resets the game, chooses random actions, applies them one by one, and logs the results. Fourth, run many episodes and summarize what happened: how often the treasure was found, how often the trap was hit, and how many steps episodes lasted on average.

This project is intentionally small, but it introduces all the core ideas you need for later chapters. States are locations. Actions are moves. Rewards convert winning and losing into signals. Each action causes a state change. Random play provides baseline behavior. Once this loop works, you will have a real environment ready for value-based learning. In the next stage, a Q-table can attach estimated usefulness to each state-action pair, allowing the agent to choose better moves over time instead of acting purely at random.

Be disciplined with testing. Try a few hand-crafted action sequences before running random episodes. Confirm that the treasure ends the game, the trap ends the game, and boundary collisions behave consistently. If the world is reliable now, learning later will be much smoother. The practical outcome is your first reinforcement learning playground: small enough to understand completely, but rich enough to demonstrate improvement through experience.

Section 3.1: Counting wins, losses, and average reward

Before we teach an agent to improve, we need to measure how random play performs. This step sounds simple, but it is one of the most important habits in engineering. If you do not establish a baseline, you cannot tell whether learning helped or whether the result just feels better. In a tiny win-or-lose environment, your baseline can include three basic numbers: wins, losses, and average reward per episode.

Suppose your agent starts in a small path world. One end is a trap, the other end is a goal. If it chooses actions randomly, let it play many episodes, not just one or two. Count how many times it reaches the goal, how many times it hits the trap or times out, and how much total reward it earns on average. If each step has a small penalty, average reward also captures efficiency. An agent that reaches the goal slowly may still have a worse average reward than one that reaches it quickly.

Practical workflow matters. Run a fixed number of episodes, such as 100 or 500. Keep the rules unchanged while measuring. Record:

• Win count
• Loss count
• Average reward
• Average number of steps

These numbers tell a fuller story than wins alone. Random behavior may occasionally win by luck, but it usually wastes movement. Average steps reveals that waste. This is especially useful in small projects, where a lucky streak can make random play look smarter than it is.

A common mistake is measuring too little data. If you test only 5 episodes, your results may swing wildly. Another mistake is changing the environment mid-test, such as moving the goal or altering rewards, then comparing the new results with old ones. That makes the baseline unreliable. Keep the world stable while you collect baseline numbers.

The practical outcome of this section is discipline. You learn to treat reinforcement learning as an experiment. Random play is not useless; it gives you the starting line. Once you know how often an untrained agent wins, loses, and what reward it tends to earn, you can later judge whether your learning method truly made better choices.

Section 3.2: Why memory helps an agent improve

An agent that acts randomly with no memory is doomed to repeat the same mistakes forever. If it falls into a trap after moving left from a particular state, but forgets that result immediately, then the next time it faces the same state it has no reason to behave differently. Learning begins the moment the agent keeps some record of what happened before.

In everyday language, memory means this: when a choice leads to something good, lean toward that choice next time; when a choice leads to something bad, avoid it. That is the entire spirit of reinforcement learning in simple form. We are not giving the agent a full map of the world. We are only letting it remember which actions have worked out better or worse in situations it has already seen.

This memory does not need to be complicated. In small environments, a number attached to each action can be enough. Higher number, better-looking choice. Lower number, worse-looking choice. Over repeated tries, those numbers become experience. The agent is still learning by trial and error, but now each error is useful because it leaves a trace.

Engineering judgment shows up in what kind of memory you choose to store. If the environment is tiny and fully visible, a simple table works well. If the environment were huge, this would not scale, but for our path games and toy mazes it is perfect. Small systems are where ideas become clear. You want memory that is easy to inspect and debug. A table lets you see exactly what the agent believes.

A common misunderstanding is thinking memory means the agent must remember the entire past. Not here. For this chapter, the agent only needs a compact summary: how good each action seems in each state. Another mistake is expecting improvement after one or two episodes. Memory helps gradually. Early on, the agent still explores, still gets things wrong, and still has incomplete information. Improvement comes from repeated feedback.

The practical outcome is powerful: once the agent remembers outcomes, random play becomes informed play. It is no longer choosing as if every state is brand new. It starts carrying lessons from previous episodes into future decisions, which is the first real step from blind action to better choices.

Section 3.3: Introducing the Q-table in plain language

A Q-table is a simple way to store an agent's experience. If the phrase sounds technical, reduce it to a plain picture: it is just a table of scores. For each state, the table stores a score for each possible action. That score answers a practical question: “If I take this action here, how good does it seem based on what I have learned so far?”

Imagine a tiny path game with five positions. In each position, the agent can move left or right. Your Q-table might have five rows, one for each position, and two columns, one for each action. At the beginning, all scores can start at zero because the agent has no experience yet. As episodes happen, some entries rise and others fall.

For example, if moving right from position 2 often leads toward the goal, the value for that cell should become more positive over time. If moving left from position 2 tends to lead into a trap or causes extra step penalties, that score should become lower. The table does not tell the agent everything about the world. It simply stores practical action preferences learned from results.

Why is this useful? Because it turns vague learning into something concrete. You can print the table after training and inspect it. You can ask: “Does the agent prefer moving toward the goal?” “Are bad actions getting low scores?” This visibility is valuable in beginner projects. It helps you understand not only whether the agent improved, but why.

Common mistakes appear quickly here. One is mixing up states and actions when building the table. Another is forgetting that the table stores estimates, not perfect truth. Early values are rough guesses. They improve with more experience. A third mistake is using a Q-table in an environment with too many states, where the table becomes massive. For our small win-or-lose environments, though, it is the right tool.

The practical outcome of learning the Q-table is confidence. You now have a physical representation of “what the agent thinks.” That makes reinforcement learning feel less mysterious. Instead of magic, you have a notebook of action scores that gets updated from trial and error.

Section 3.4: Updating action scores after each move

The Q-table becomes useful only when we update it from experience. After each move, the agent should look at what happened and adjust the score for the action it just took. In plain language, the update rule says: if this move led to a better outcome than expected, raise its score; if it led to a worse outcome, lower its score.

Here is the basic workflow. The agent starts in a state, chooses an action, lands in a new state, and receives a reward. Then it updates the table entry for the original state-action pair. In small projects, you can think of this as correcting a guess. The old score was your current belief. The reward and the next state's future possibilities give you better evidence. The updated score becomes a slightly improved belief.

A practical implementation usually includes three ideas: the current score, the immediate reward, and how promising the next state looks. If the next state has strong action scores, that should increase the value of the current move, because good future options matter. If the next state looks bad, the current move should not get much credit. This is how reinforcement learning connects short-term results with future consequences.

Engineering judgment matters in reward design. If you reward only the final goal and ignore all other behavior, learning may be slow because feedback is rare. A tiny step penalty can help the agent prefer shorter paths. But penalties that are too harsh can make every action look bad. In a toy path game, a simple setup often works well: positive reward for reaching the goal, negative reward for hitting the trap, and a small negative reward for each extra step.

Common mistakes include updating the wrong table cell, forgetting to use the next state when estimating future value, or making rewards inconsistent across episodes. Another common issue is expecting smooth progress every episode. Learning can be noisy. Some episodes look worse than earlier ones because the agent is still exploring.

The practical outcome is a repeatable learning loop. After every move, the agent slightly reshapes its own table of action scores. Those tiny updates accumulate. Over time, the agent starts favoring moves that lead to success and fewer wasted steps.

Section 3.5: Reading a table of better and worse choices

Once the Q-table has been updated over many episodes, you should learn to read it like a map of preferences. A high value in a cell means the agent currently believes that action is a strong choice in that state. A low or negative value means the action seems risky, wasteful, or harmful. Reading the table is how you move from “the agent improved” to “I understand how it improved.”

Take a simple example. In a path world, suppose middle states show higher scores for moving right than moving left, and the goal-side states strongly prefer the final move into the goal. That pattern makes sense. It suggests the agent has learned a directional strategy. If you see the opposite pattern, something may be wrong with your rewards, transitions, or update logic.

You should also compare the table with actual behavior. If one action has the highest value in a state, a greedy policy would choose it. That is where the idea of greedy choices enters. A greedy agent picks the action with the best current score. Random play ignores the table. In between those extremes is explore-versus-exploit behavior, where the agent usually follows the best-known action but sometimes tries another move to gather more information.

Engineering judgment is important when interpreting values. Do not obsess over exact numbers at first. Focus on ranking and patterns. Which action is preferred? Do states near the goal look more valuable? Are trap-leading actions lower? In small environments, these broad shapes matter more than tiny numeric differences.

Common mistakes include assuming the table must be perfect after limited training or reading one strange entry as proof the algorithm failed. Sometimes a single state was visited too rarely to get a strong estimate. More episodes often fix that. Another mistake is forgetting that exploration can still produce seemingly bad choices even when the table itself looks sensible.

The practical outcome here is interpretability. You can inspect a learned table and explain the agent's decision logic in plain language. That skill is valuable because reinforcement learning is not only about training agents; it is also about understanding whether they learned the right lessons.

Section 3.6: Project 2 setup: reach the goal in fewer steps

Now we turn the chapter ideas into a compact project. The goal of Project 2 is simple: build a tiny environment where the agent learns to reach a goal in fewer steps over repeated tries. A one-dimensional path is ideal. For example, place the agent at position 2 in a line of 5 positions. Position 4 is the goal, position 0 is a losing trap. The agent can move left or right.

Set rewards so the task teaches both success and efficiency. A practical scheme is: +10 for reaching the goal, -10 for entering the trap, and -1 for every normal step. That small step cost matters. Without it, the agent may learn that eventually winning is enough and may not care whether it takes 2 steps or 20. With a step penalty, shorter successful paths become more valuable.

Your workflow should be structured. First, measure random play for many episodes and log wins, losses, average reward, and average steps. Second, create a Q-table with one row per position and one column per action. Third, let the agent train over repeated episodes: observe state, choose action, get reward, update the Q-table, and continue until the episode ends. Fourth, compare the trained behavior with the random baseline.

This project is also the right place to compare random choices, greedy choices, and simple explore-versus-exploit behavior. During training, allow some randomness so the agent still explores. During evaluation, you can test a greedy policy that always picks the action with the highest learned score. That contrast shows whether the Q-table is truly guiding better decisions.

Common mistakes in this project include making the environment too large, using rewards that are too weak to signal anything useful, or evaluating only one final run. Keep it small and measurable. Print the table after training. Watch whether the number of steps decreases over time. That trend is the practical proof that learning is happening.

The main outcome of Project 2 is not just a working toy agent. It is a complete reinforcement learning loop you can understand end to end. You begin with random play, collect evidence, store action scores, update them from results, and finish with an agent that reaches the goal more efficiently than before. That is the core idea of reinforcement learning in action.

Section 4.1: The problem with always picking the current best move

A beginner often thinks, “Once the agent finds a good move, why not just keep doing it?” That sounds reasonable, but it causes a major learning problem. The current best move is only the best according to what the agent has seen so far. Early experience is incomplete, noisy, and sometimes misleading. If the agent locks itself into one action too soon, it stops gathering information and may never discover a better option.

Think of a coin hunt game with two hallways. In early rounds, the agent tries the left hallway and finds a coin quickly. It then decides the left hallway is excellent. If it always repeats that choice, it might never learn that the right hallway contains two coins or a more reliable reward pattern. In other words, a small amount of experience can create overconfidence. Reinforcement learning agents do not “know” that their knowledge is incomplete unless we design their action policy to leave room for discovery.

This problem becomes worse in win-or-lose environments because early results can be dramatic. One lucky win can make a mediocre action look amazing. One unlucky loss can make a strong action look terrible. If you always pick the current best move, your agent can become trapped by accidents from the first few rounds.

From an engineering point of view, this is a data collection issue. The agent is learning from its own behavior. If its behavior becomes too narrow, the data also becomes narrow. A narrow set of experiences produces a narrow Q-table. Then the agent appears confident, but its confidence is built on weak evidence.

• Too much early greed can freeze learning.
• Lucky rewards can create false favorites.
• Untried actions remain unknown, not bad.
• A Q-value is an estimate, not a guarantee.

A common mistake is to read a high Q-value as proof that an action is truly best. In small projects, remember that Q-values begin as rough guesses shaped by limited trials. The practical outcome is clear: your agent should not always repeat one move just because it currently has the highest value. Good learning requires space to test alternatives before committing too strongly.

Section 4.2: Exploration as safe curiosity

Exploration means trying actions that are not currently the top choice so the agent can gather more information. A helpful way to explain this is “safe curiosity.” The agent is not being random for no reason. It is checking whether the world contains better opportunities than the ones it already knows. In reinforcement learning, this curiosity is necessary because the agent usually starts with very little knowledge.

Suppose your agent is in a tiny maze and can move up, down, left, or right. At first, all directions are nearly unknown. If the agent only follows its first lucky result, it may miss the true route to the goal. Exploration lets it sample different actions, compare outcomes, and slowly build a more reliable map of which decisions lead to wins and which lead to losses.

Safe curiosity does not mean chaos. Good exploration is controlled. In a beginner project, you do not need a complex strategy. You just need a rule that allows occasional trial actions. This gives the agent a chance to correct its mistaken beliefs. If an action looked bad after one attempt, exploration can test it again. If an action looked great after one lucky round, exploration can compare it with alternatives.

Exploration also helps when the environment contains trade-offs. One move may give a small quick reward. Another may lead to a larger delayed reward. Without trying both enough times, the agent cannot learn that difference. This is especially important in path games, where a longer route may produce a more stable win rate than a short route filled with danger.

A practical workflow is simple: allow some exploration, record rewards, update the Q-table, then gradually let the better actions stand out. That process turns raw trial-and-error into useful learning. The engineering judgment is to explore enough to gather meaningful evidence, but not so much that the agent behaves like it knows nothing forever. Exploration is not the opposite of learning. It is one of the tools that makes learning possible.

Section 4.3: Exploitation as using what works

If exploration is curiosity, exploitation is confidence in action. Exploitation means the agent uses the action that currently looks best according to its Q-table or decision rule. This is how learning turns into performance. Without exploitation, an agent might discover many useful actions but fail to take advantage of them consistently.

In a practical project, exploitation is what makes the agent look smart. After enough rounds in a coin hunt or maze, the Q-table begins to contain patterns. Certain actions in certain states produce higher expected rewards. Exploitation means the agent starts choosing those actions on purpose. That is the point where trial and error becomes strategy.

However, exploitation should be understood correctly. It is not blind repetition. It is using the best evidence available right now. That phrase matters because the evidence can improve over time. A good reinforcement learning workflow alternates between learning and using what has been learned. The agent explores enough to gather data, then exploits enough to benefit from that data.

There is also an engineering advantage to exploitation: it helps you see whether the Q-table has become useful. If the agent exploits and its performance improves over many rounds, your learning system is probably capturing something real. If exploitation causes weak results, you may have problems with rewards, state design, or insufficient exploration.

• Exploration discovers possibilities.
• Exploitation converts knowledge into better choices.
• Too little exploitation makes learning look ineffective.
• Too much exploitation too early can trap the agent.

A common mistake is to frame exploration as good and exploitation as bad. That is not true. Exploitation is the reason the agent can win more often after learning. In tiny win-or-lose environments, you want the agent to increasingly use what works. The real skill is not choosing one side forever. The real skill is balancing them so the agent can both learn and perform.

Section 4.4: Epsilon-greedy choice without heavy math

The most beginner-friendly way to balance exploration and exploitation is epsilon-greedy choice. The name sounds technical, but the rule is simple. Most of the time, the agent picks the action with the highest current Q-value. A small part of the time, it picks a random action instead. That small chance is called epsilon.

For example, if epsilon is 0.2, then about 20% of the time the agent explores by choosing randomly, and about 80% of the time it exploits by choosing the current best action. If epsilon is 0.1, the agent explores less often. You do not need advanced math to use this. It is just a practical behavior rule.

Here is a plain workflow. First, look at the current state. Second, generate a random number. Third, if the number is below epsilon, explore by picking any allowed action. Otherwise, exploit by selecting the action with the highest Q-value in that state. After the action, observe the reward and update the Q-table as usual.

This approach works well because it is easy to code and easy to reason about. It prevents the agent from becoming too greedy too soon, but it still allows strong actions to be used most of the time. In small projects, that is often exactly what you need.

Engineering judgment matters when picking epsilon. If epsilon is too high, the agent keeps acting randomly even after it has learned useful behavior. If epsilon is too low, it may stop discovering better actions. A common beginner strategy is to start with a moderate epsilon such as 0.2 or 0.3, then slowly reduce it as training continues. That way, the agent is curious early and more focused later.

A common mistake is to think random means bad. In epsilon-greedy, randomness is a tool. Another mistake is to forget that tie-breaking matters. If two actions have equal Q-values, your code should handle that fairly, often by choosing randomly among the tied best actions. Epsilon-greedy is simple, practical, and strong enough to power many beginner reinforcement learning experiments.

Section 4.5: Watching learning change over many rounds

Exploration versus exploitation is easiest to understand when you watch behavior over many rounds instead of judging from one or two episodes. In early training, the agent often looks inconsistent. It wins sometimes, loses often, and makes moves that seem strange. That is normal. It is gathering experience. If your exploration rule is working, you should see a gradual shift: random-looking choices become less frequent, and useful actions appear more often.

One practical habit is to track simple measures across episodes. Count total wins, average reward, average steps to reach the goal, and how often each action is chosen in important states. These observations tell a story. If exploration is too high, action counts may stay spread out with no clear preference. If exploitation is too strong too early, one action may dominate before the agent has enough evidence.

In a tiny path game, for example, you might see this pattern. During the first 20 rounds, the agent tries both short and long routes with mixed success. By round 50, the safer route may start winning more often. By round 100, if rewards support it, the Q-table may strongly favor that route. This shift is the visible result of balancing trial and trust.

Another useful technique is to compare training behavior and testing behavior. During training, keep epsilon above zero so the agent still explores. During testing, set epsilon to zero and let the agent act greedily. This shows what the agent has actually learned when randomness is removed. Many beginners confuse training performance with learned policy performance. Separating them gives a clearer picture.

Common mistakes include changing epsilon too often, judging success from too few episodes, and ignoring reward trends. Reinforcement learning can look messy at first, so patience matters. The practical outcome is that over many rounds, a good exploration strategy should help the agent become less random, more informed, and more reliable at reaching wins.

Section 4.6: Project 3 setup: a risky path versus safe path game

To make this chapter concrete, set up a small game with two possible routes to the goal: a risky short path and a safe longer path. The agent starts at the same position each round and must choose actions that move it toward the goal. The risky path has fewer steps, but one tile can trigger a loss or a large negative reward. The safe path takes longer, but it usually avoids disaster. This setup is ideal for exploring explore-versus-exploit behavior because neither option is obviously best from the start.

Design the rewards so the trade-off is visible. For example, reaching the goal might give +10, stepping on a losing tile might give -10, and each normal move might give -1 to encourage shorter routes. With these numbers, the risky path can be excellent when it works, but painful when it fails. The safe path may have a lower immediate appeal, yet produce steadier long-term returns.

Now apply epsilon-greedy action choice. Early in training, use a moderate epsilon so the agent samples both routes. Watch whether the Q-values for the risky and safe decisions change over time. If the risky path wins often enough, the agent may prefer it. If the losses are too severe or too frequent, it may learn that the safe route is better overall. This is exactly the kind of behavior we want students to observe.

As an engineer, tune one variable at a time. First adjust epsilon. Then, if needed, adjust the penalty for danger or the step cost. Small changes can produce very different learned behavior. That is not a bug. It is a lesson about how reward design and exploration policy work together.

This project also teaches restraint. Do not assume the shortest route should win. Let the reward structure and repeated trials decide. By building this game, you will see why agents should not always repeat one move, how trying new actions can uncover better long-term outcomes, and how simple behavior rules can make a reinforcement learning system noticeably smarter.

Section 5.1: What training means in reinforcement learning

Training in reinforcement learning means letting the agent interact with the environment many times so it can improve its decisions from experience. In a small project, one interaction sequence is usually called an episode. The episode starts at some beginning state, continues as the agent takes actions, and ends when the game is won, lost, or reaches a step limit. During training, the agent is not expected to be good. In fact, early mistakes are useful because they reveal which actions lead to better or worse outcomes.

For a Q-table agent, training usually follows a practical loop. Start the episode, observe the current state, choose an action, apply that action to the environment, receive a reward, move to the next state, and update the Q-value. Repeat until the episode ends. Then start another episode. Over many episodes, the values in the table begin to reflect expected future reward. This is how trial and error becomes stored experience.

A key point is that training often includes exploration. If the agent always picks the current best-known action, it may never discover a better one. That is why many tiny RL projects use an explore-versus-exploit rule such as epsilon-greedy. With some probability, the agent tries a random action. With the remaining probability, it chooses the action with the highest current Q-value. During training, exploration helps the agent gather information. Without it, learning can get stuck in a poor habit very early.

Engineering judgment matters here. You should decide how many episodes to train, whether starting positions are fixed or random, how long an episode may last, and whether exploration decreases over time. In simple environments, it is common to begin with more exploration and reduce it gradually. That lets the agent discover possibilities early and settle into stronger choices later. A common beginner mistake is to stop training too soon because reward improved once or twice. Small environments can be noisy. You need enough episodes to see a stable pattern, not just a lucky run.

Good training also means keeping the environment rules consistent. If the rewards, map layout, or action meanings change during training, the agent is trying to learn a moving target. In tiny projects, clarity beats complexity. Keep the state representation simple, the actions limited, and the reward rule understandable. Training is not just “running the code many times.” It is the controlled process of giving an agent repeated experience in a well-defined world so the Q-table gradually becomes useful.

Section 5.2: What testing means and why it matters

Testing means evaluating the agent after training without letting the evaluation itself change the learned behavior. In plain language, training is practice and testing is performance. This separation is one of the most important habits in reinforcement learning because it prevents you from confusing “learning while being helped” with “already learned.” If your agent still explores randomly during evaluation, then your result does not show the true quality of the policy.

In a small RL project, testing usually means turning off or sharply reducing exploration. If you used epsilon-greedy during training, testing often sets epsilon to zero so the agent always chooses the highest-valued action from the Q-table. You then run multiple test episodes and observe outcomes such as win rate, average steps to goal, number of traps hit, or number of coins collected. The purpose is not to teach the agent anything new. The purpose is to ask: given what it currently knows, how well does it behave?

Testing matters because training rewards can be misleading. Imagine a maze agent that occasionally reaches the goal by luck while still wandering badly. During training, exploration may create some high-reward episodes, making progress look strong. But when tested without randomness, the same agent may repeatedly fail or take a very long route. Testing exposes that difference. It tells you whether the policy is reliable.

To test well, keep the conditions clear. Use a fixed number of test episodes. Record the same measurements each time. Avoid updating the Q-table during testing. If you want a more realistic evaluation, test from several starting positions rather than only one familiar start state. This helps you see whether the agent learned a general strategy or merely a narrow path from one location.

A common mistake is blending testing into training every few steps without noticing that the agent is still learning underneath. Another mistake is celebrating the single best episode instead of the average behavior. In engineering practice, consistent performance matters more than one impressive run. Good testing gives you an honest picture of the learned policy, helps you compare project versions fairly, and shows whether changes in reward design or environment rules actually improved the agent.

Section 5.3: Reward design and accidental shortcuts

Reward design is where many small RL projects succeed or fail. Rewards are not just points; they are instructions written in numbers. The agent does not understand your intention in a human way. It only learns which behaviors increase expected reward. If the reward rule is slightly misaligned with your real goal, the agent may discover an accidental shortcut that scores well but behaves badly.

Consider a coin hunt game. If you give a reward every time the agent steps onto a coin tile, but the coin never disappears, the agent may learn to move back and forth on the same tile forever. The code is working. The Q-learning update is working. The environment design is the problem. In another example, if a maze gives no penalty for extra steps and only rewards the goal, the agent may eventually reach the goal but learn a slow wandering route because wasting moves is not meaningfully discouraged. If a trap penalty is too small, the agent may treat traps as acceptable risks while rushing toward coins.

Good reward design starts by stating the desired behavior clearly. Do you want the agent to win as fast as possible, collect as many safe rewards as possible, avoid danger at all costs, or balance speed and caution? Then choose reward values that support that behavior. In tiny projects, a common pattern is: positive reward for success, negative reward for failure, and a small step penalty to reduce pointless wandering. If items can be collected, remove them after collection or mark them as already collected. Otherwise the reward system may invite exploitation.

When you spot weird behavior, ask a practical question: “Is the agent exploiting my scoring rule?” Often the answer is yes. Do not immediately blame the algorithm. First inspect the rewards and state transitions. Watch sample episodes. Look for loops, trap farming, wall-bumping, or repeated low-value behavior that still beats the intended path according to the reward totals.

The best improvement process is to change one reward rule at a time, retrain, and test again. That way you can connect cause and effect. In small RL environments, reward design is part of the project itself. Clear rewards produce clearer learning, while vague rewards produce confusing behavior that may look smart at first but falls apart under testing.

Section 5.4: Common beginner mistakes in small environments

Small environments are excellent for learning reinforcement learning, but they also hide mistakes because everything seems easy. One common beginner mistake is creating a state representation that leaves out important information. For example, in a coin game, if the state only stores the agent position and ignores whether a coin has already been collected, then two different situations may look identical to the Q-table even though the correct action should differ. The agent is then asked to learn from incomplete information.

Another mistake is mixing training and testing. Many learners run episodes, print rewards, and assume every improvement means the policy is strong. But if exploration is still active, those rewards reflect a mix of random actions and learned actions. You need separate test runs with stable conditions to know what the policy really does. A related mistake is updating the Q-table during testing, which quietly turns evaluation back into training.

Beginners also often choose reward values without thinking through side effects. A reward that is too large or a penalty that is too weak can distort behavior. In a tiny path game, a small positive reward for surviving each step might accidentally teach the agent to avoid the goal and stay alive longer. In a maze, no step penalty can encourage wandering. In a trap environment, inconsistent terminal rules can make outcomes hard to interpret.

There are also workflow mistakes. Changing several things at once makes debugging difficult. If you alter the map, reward values, exploration rate, and episode length together, you will not know which change caused improvement or failure. Better practice is to hold most settings steady and modify one factor at a time. Record what you changed and why.

Finally, many beginners trust totals without watching behavior. Numerical reward is useful, but visual inspection matters too. Run a few episodes step by step. Does the agent bounce between two cells? Does it ignore obvious rewards? Does it choose a long route for no clear reason? In small environments, direct observation is one of your strongest tools. These projects are tiny enough that you can actually watch the policy and develop intuition, which is part of becoming a careful RL builder.

Section 5.5: Measuring progress with simple charts and counts

You do not need advanced analytics to judge small RL projects. Simple charts and counts are often enough to reveal whether learning is happening and whether project changes are helping. The most useful mindset is to measure the outcomes that match your real goal. If your game is about reaching a target quickly, track both success rate and average steps. If your game includes traps, count trap hits. If it includes coins, count coins collected. One number rarely tells the full story.

A practical starting set of measurements is: average total reward per training block, win rate during testing, average episode length, and counts of key events such as coins collected or traps triggered. Plotting these across episodes or across batches of episodes makes trends easier to see. For example, if reward rises but test win rate stays flat, you may have a reward-design issue or too much dependence on exploration. If win rate improves while steps fall, that suggests the learned policy is becoming both successful and efficient.

In small projects, charts do not need to be fancy. A simple line plot every 50 or 100 episodes can be enough. Even a printed table of counts can work. The important part is consistency. Use the same test procedure each time so comparisons are fair. When you modify a reward rule or environment layout, record old and new results side by side.

Counts are especially useful for spotting accidental behavior. Suppose a coin collector shows high average reward. If a separate count reveals that the agent hits many traps, then the reward values may be letting coin gain hide dangerous behavior. Or suppose average episode length grows over time. That may mean the agent is learning to loop instead of finish. Numbers like these turn vague impressions into clear evidence.

Comparing multiple small RL projects also strengthens understanding. A maze may show falling episode length as learning improves. A coin project may show rising item counts. A trap-avoidance project may show declining failure counts. Looking across these patterns teaches you that “progress” depends on the environment goal. Measuring well helps you decide whether to keep training, change rewards, adjust rules, or redesign the project entirely.

Section 5.6: Project 4 setup: collect coins but avoid traps

For this chapter’s project, build a small grid world where the agent must collect coins while avoiding trap cells. This environment is useful because it combines positive rewards and negative rewards in one clear task. The goal is not just to move randomly until the episode ends. The agent must learn a trade-off: pursue useful rewards while staying away from costly mistakes. That makes it a strong practice project for separating training from testing and for spotting reward design problems early.

Start with a simple grid, such as 4x4 or 5x5. Place the agent at a fixed start position. Add a small number of coin cells and one or two trap cells. The actions can remain the familiar four moves: up, down, left, right. Decide whether hitting a wall keeps the agent in place or ends the move without position change. Keep this consistent. A practical reward design might be: +10 for collecting a coin, -10 for stepping on a trap, -1 per step to discourage wandering, and optional episode end after all coins are collected or after a trap is hit. If coins are collected, remove them or mark them so they cannot be collected repeatedly.

Now think carefully about the state. If coin collection changes the world, then the state should include more than the agent location. At minimum, it should somehow represent which coins remain. Otherwise the same location before and after collecting a coin would look identical to the Q-table, even though the best action might differ. In a tiny project, you can encode this with a simple flag pattern for remaining coins.

During training, allow exploration with an epsilon-greedy policy. Train for many episodes and record average reward, coins collected, trap hits, and steps per episode. During testing, turn exploration off and measure how reliably the agent collects coins safely. Watch for accidental shortcuts. If the trap penalty is too low, the agent may grab one coin and accept a trap. If the step penalty is too high, it may rush dangerously. If there is no step penalty, it may wander.

This project is also perfect for comparison. You can compare it with earlier mazes or path games and ask what changed. Did learning become slower because the agent must balance two goals? Did reward design matter more? Did testing reveal a gap between training reward and actual safe performance? Those are exactly the right questions. A well-built small project does not just produce a score. It teaches you how to reason about RL systems, improve rules for clearer learning, and judge whether an agent has truly learned the behavior you wanted.

Section 6.1: Choosing a final project idea you can finish

Your first complete reinforcement learning project should be intentionally small. Beginners often make the same mistake: they try to build a giant game, a complicated robot, or an environment with too many rules. That usually creates confusion before learning even begins. A better choice is a project with a tiny state space, a short list of actions, clear win and lose conditions, and rewards that are easy to reason about. The goal is not to impress people with scale. The goal is to demonstrate a complete workflow from setup to results.

A good final beginner project usually has three traits. First, it is easy to visualize. A 4x4 grid, a coin hunt, or a path-to-goal game works well because you can picture what the agent is doing. Second, it has a clear end. The episode should stop when the agent wins, loses, or reaches a step limit. Third, it should allow both bad decisions and good decisions, so learning can be observed. If every path is equally good, the agent has nothing meaningful to discover.

For this chapter, a project like Treasure or Trap is ideal. Imagine a small grid. The agent starts in the lower-left corner. One cell contains treasure worth +10. One cell contains a trap worth -10. Every move costs -1. The episode ends when the agent reaches treasure, reaches the trap, or uses too many steps. That setup gives immediate practical value. You can explain it to anyone in plain language: the computer learns to reach a good square and avoid a bad square while trying not to waste steps.

Use engineering judgment when choosing complexity. Ask yourself: Can I list all states? Can I list all actions? Can I explain the reward logic in two sentences? Can I tell whether the learned behavior is better than random? If the answer is yes, your project is the right size. If the answer is no, shrink it. Finishing a small project teaches more than abandoning a large one.

One more practical rule: choose a project where the environment is deterministic at first. That means the same action in the same state gives the same result. Deterministic environments are easier for beginners because they reduce noise and make patterns in the Q-table easier to understand. Once you can finish a deterministic project, you can later add randomness such as moving obstacles or changing rewards.

Common mistakes in project choice include too many states, hidden rules, unclear rewards, and no baseline for comparison. Avoid them by writing your idea in a few bullet points before coding. If you can describe the environment on paper, you are ready to build it.

Section 6.2: Setting states, actions, rewards, and endings

Once you choose the project, define the environment carefully. This is one of the most important practical skills in reinforcement learning. A weak environment definition creates weak learning, even if the training code is correct. Your job is to turn the game idea into exact pieces the agent can work with: states, actions, rewards, and endings. If these pieces are fuzzy, the agent will learn fuzzy behavior.

Start with states. In a small grid world, each square can be one state. A 4x4 grid gives 16 states. This is simple because the agent only needs to know its current position. You do not need extra information unless the rules require it. Beginners sometimes add unnecessary details, such as the full movement history or special memory flags, which makes the project harder to understand. In a beginner environment, the state should be the smallest amount of information needed to make a reasonable decision.

Next, define actions. In our final project, the actions are up, down, left, and right. That is enough. If the agent hits a wall or tries to move outside the grid, decide on one clear rule. You might keep the agent in the same state and still apply the step penalty. That teaches the agent that invalid moves waste time. Consistency matters more than cleverness here.

Rewards deserve extra care because they shape behavior directly. In the Treasure or Trap project, a common beginner reward design is: +10 for treasure, -10 for trap, and -1 for every normal move. These numbers are not magical, but they communicate a useful preference: reach the goal, avoid failure, and do it efficiently. If the step penalty is too small, the agent may wander. If the goal reward is too small, the agent may not care enough about success. Reward design is where engineering judgment and experimentation meet.

Now define episode endings. End the episode if the agent reaches treasure, reaches the trap, or exceeds a maximum number of steps. The maximum step count prevents endless wandering during early training. It also gives you a clean way to classify failure: the agent did not solve the environment in time.

• State: current grid position
• Actions: up, down, left, right
• Reward for treasure: +10
• Reward for trap: -10
• Reward per step: -1
• End conditions: win, lose, or hit step limit

A common mistake is to mix the goal with the reward in a confusing way. The goal is the big objective, such as reaching treasure. The reward is the feedback signal that helps the agent learn that objective. Keep those ideas aligned. When your environment definition is clear, your training process becomes much easier to trust and explain.

Section 6.3: Training the agent and checking improvement

With the environment defined, you can train the agent using a basic Q-table. This is the moment where trial and error becomes visible. The Q-table stores a score for each state-action pair. At the beginning, those values are usually zero or small random values. The agent does not yet know which actions are useful. Through repeated episodes, it updates the table using rewards it receives and the future value it expects.

A practical beginner workflow is simple. Reset the environment at the start of each episode. Let the agent choose actions using an explore-versus-exploit rule such as epsilon-greedy. With probability epsilon, choose a random action to explore. Otherwise choose the greedy action with the highest current Q-value. After each action, record the reward and update the Q-table. Repeat this for many episodes. Over time, gradually reduce epsilon so the agent explores a lot early on and behaves more greedily later.

This section is where you should compare behaviors. Random behavior gives you a baseline. A purely random agent may eventually find treasure sometimes, but it will often wander into the trap or waste steps. An exploratory learning agent behaves inconsistently at first because it is still gathering experience. A mostly greedy trained agent should begin to choose shorter, safer paths. This comparison helps you explain not just that learning happened, but how learning changed action selection.

Checking improvement does not need advanced charts. Use beginner-friendly measurements. Track average reward per 100 episodes. Track win rate, loss rate, and timeout rate. Track average number of steps before the episode ends. These are practical outcomes that anyone can understand. If average reward rises and the win rate improves, the agent is learning something useful.

Be careful with common mistakes. First, do not judge training from one lucky episode. Look for patterns over many episodes. Second, do not train and test with exploration turned fully on, or your results may look worse than the learned policy really is. When evaluating, lower epsilon or set it to zero so you can see the greedy policy clearly. Third, if nothing improves, inspect your reward design and step rules before blaming the algorithm.

In a tiny project, success often looks like this: early episodes are messy and inconsistent, but later episodes become shorter and more reliable. That is the beginner version of reinforcement learning working correctly. The computer is not “thinking” like a person. It is accumulating better action preferences through repeated feedback. The Q-table is simply the memory of that process.

Section 6.4: Explaining project results with confidence

Many beginners can train an agent but struggle to explain what happened. A strong portfolio project needs a clear story. The story should answer four questions: What was the environment? What did the agent need to learn? How did you train it? What changed after training? If you can answer those in simple language, you can present your results confidently even without advanced mathematics.

Start by describing the environment in plain terms. For example: “I built a 4x4 grid game where the agent tries to reach treasure, avoid a trap, and finish in as few moves as possible.” Then explain the learning setup: “The agent could move in four directions. It earned +10 for treasure, -10 for the trap, and -1 for each step. I trained it over many episodes with a Q-table and epsilon-greedy action selection.” This kind of explanation is strong because it is specific and understandable.

Next, talk about the results using direct observations. You might say: “At first the agent behaved almost randomly and often failed. After training, it reached the treasure more often and took shorter paths.” If you have numbers, use them. For example: “The win rate improved from roughly random performance to a much higher percentage after training.” Numbers make your explanation more credible, but plain language should still lead the way.

Engineering judgment also means being honest about limitations. Maybe the agent learned this one grid but would struggle if the map changed. Maybe the reward design made it prefer safety over speed. Maybe some states were visited less often than others. Mentioning limits does not weaken your work. It shows maturity. Real reinforcement learning projects always have assumptions and trade-offs.

A good result explanation often follows this structure:

• What the task was
• How the environment was defined
• How the agent learned
• What evidence shows improvement
• What the project still does not solve

This section matters because reinforcement learning is often misunderstood as magic. Your job is to remove that mystery. Explain that the agent improved because repeated rewards changed the Q-values, and those Q-values led to better action choices. That is a confident, accurate, beginner-friendly explanation of project results.

Section 6.5: Saving your work as a simple learning portfolio

Once your project works, do not leave it as a pile of code files. Turn it into a small learning portfolio item. This is where your technical work becomes something you can revisit, share, or build on later. A portfolio does not need fancy design. It needs structure, clarity, and enough detail that another beginner could understand what you made.

A practical portfolio entry for this chapter should include a short project title, a one-paragraph problem description, the environment rules, the learning method, the key results, and a brief reflection. For example, your title might be “Treasure or Trap: A Beginner Q-Learning Grid World.” The description can explain the goal in plain language. The rules section can list states, actions, rewards, and end conditions. The method section can mention the Q-table, training episodes, learning rate, discount factor, and epsilon schedule. Keep this concise but complete.

Include one visual or one simple table if possible. A small grid diagram, a sample Q-table snapshot, or a before-and-after win rate summary helps readers see evidence quickly. If you do not have plots, that is fine. Even a text summary of training outcomes is useful. The purpose is not beauty. The purpose is reproducibility and understanding.

Your reflection is especially valuable. Write a few sentences about what was harder than expected, what design decision mattered most, and what you would change next time. Maybe you discovered that reward design had a bigger effect than the update rule. Maybe you learned that too much exploration made evaluation confusing. These lessons show that you are thinking like a practitioner, not just copying steps.

A simple portfolio template might include:

• Project goal: teach an agent to reach treasure and avoid a trap
• Environment: 4x4 grid with terminal win and lose cells
• Method: Q-learning with epsilon-greedy exploration
• Evidence: win rate, average reward, and path length improved
• Lessons learned: reward design and evaluation settings matter

Saving work this way gives you something concrete to return to. Later, when you study deeper reinforcement learning, you will be able to compare new methods against this first complete project. That is why this chapter is called a portfolio chapter: it captures not just what you built, but how you learned.

Section 6.6: Where to go next after beginner reinforcement learning

After finishing a complete beginner project, the best next step is not to jump immediately into the most advanced deep reinforcement learning paper you can find. A better path is to extend what you already understand. Reinforcement learning becomes easier when new ideas grow from familiar examples. Your small win-or-lose project is the foundation for that growth.

One useful next step is to vary the environment while keeping the same Q-learning structure. Add walls that block movement. Change the start position each episode. Add multiple treasures with different values. Introduce a moving hazard. These changes teach an important lesson: even when the algorithm stays the same, environment design strongly affects what the agent learns. That is a core practical insight.

Another good step is to compare policy choices more intentionally. You already know random, greedy, and explore-versus-exploit behavior. Now you can test them side by side. How does a purely greedy agent behave if trained too early? What happens if epsilon stays high for too long? What if it falls too quickly? These are simple experiments, but they build real understanding.

From there, you can deepen the technical side. Learn the Q-learning update formula more formally. Study learning rate and discount factor in more detail. Compare Q-learning with SARSA to see how different update rules create different behavior. Later, when the state space becomes too large for a table, you can move toward function approximation and neural networks. That path leads naturally toward deep Q-networks and modern RL methods.

Just as important, keep developing your explanation skills. If you can describe reinforcement learning in everyday language, you will understand it more deeply yourself. Try explaining your project to three audiences: a friend with no technical background, a beginner programmer, and a more advanced learner. If your explanation works for all three, your foundation is becoming strong.

Your next-step plan can be simple:

• Build one variation of your current grid world
• Compare different exploration settings
• Record results in the same portfolio format
• Study one new RL algorithm or concept at a time
• Keep projects small enough to finish and explain

That is how beginner reinforcement learning turns into real skill. You start with tiny worlds, clear rewards, and visible wins or losses. Then you expand carefully. By finishing this chapter, you have done something important: you have built not just an agent, but a repeatable way to learn reinforcement learning through practice.