Build a Recommendation Bot with Reinforcement Learning Basics

When people say “the bot recommends,” they often imagine a hidden model that reveals the perfect item. In practice, a recommendation is a choice under uncertainty. The bot chooses one option from a set (a movie, a song, a product), knowing it might be wrong. That makes this a decision problem, not a prediction-only problem.

Reinforcement learning gives a clean vocabulary for this. The agent is your recommendation bot. An action is “recommend item X right now.” The environment is the user and the interface. After the action, the bot receives a reward based on what the user does (click, like, watch time, purchase), and then updates its future behavior. This is the “suggest, listen, improve” job description in engineering terms.

A concrete example: suppose you have 20 movies and you show one recommendation at a time. Each time you choose a movie, the user either clicks “Play” or ignores it. Your bot’s job is not to be correct in theory; it is to increase the chance of a good next outcome over repeated interactions. That is why RL is a natural fit: learning happens from experience, not from labels you already have.

Common mistake: treating recommendation as a one-time batch ranking problem and expecting learning to happen automatically. If you don’t close the loop—if you don’t capture feedback and update decisions—you have a static recommender, not a bot that improves.

Practical outcome for this course: we will start with the smallest possible “choice” setup (pick 1 item out of N) and a single feedback signal (a click or a simple satisfaction score). This keeps the learning loop visible and debuggable.

To build a recommendation bot, you must specify two interfaces: what the bot sees and what it says. In RL language, what the bot sees is the state (or context), and what it says is the action (the recommendation).

In real products, “state” can include user profile data, session context, time of day, device type, and recent interactions. In this course, we’ll keep state intentionally small to match the project scope. A practical starting state might be:

• Which items the user has already seen in this session
• The last item they clicked (or “none”)
• A simple preference tag if available (e.g., genre: action/comedy/drama)

The action is usually one of two shapes: recommend one item (simplest) or recommend a list of items (more realistic). Lists introduce extra complexity: position bias, diversity, and interactions between items. For a tiny bot you can finish, we will recommend a single item at a time. That gives us a clean action space: {item_1, item_2, …, item_N}.

Engineering judgement: the state must include only information you can reliably compute at decision time. A common mistake is to use “future” information accidentally (data leakage), like using the user’s rating that happens after you recommend. Another common mistake is building an overly rich state early on. If your state has 200 features and your bot behaves oddly, you won’t know why.

Practical outcome: we will define a minimal state object and a clear action API (e.g., choose_item(state) -> item_id). That boundary will make it easy to test, log, and iterate.

The bot improves only if feedback is converted into a usable reward. In recommendation systems, feedback is often noisy and indirect, so you must decide what signals you trust and how you translate them into numbers.

Common feedback signals include:

• Clicks / plays: fast, easy to log, but can be misleading (clickbait effects).
• Likes / thumbs up: stronger intent, but many users don’t provide it.
• Time spent: useful for media, but long time may mean “left it running.”
• Ratings: high quality, but sparse and delayed.

For a beginner RL recommendation bot, start with a simple reward: reward = 1 if the user clicks/plays, else 0. This creates a clear learning signal. Later, you can shape the reward: maybe +2 for “like,” +0.5 for a click, and a small negative reward (e.g., -0.2) for repeated ignores, which discourages spammy behavior.

Engineering judgement: reward design is where you encode “good means user value and business value.” User value might be satisfaction, discovery, or reduced effort. Business value might be retention, conversions, or revenue. A practical compromise is to define a primary reward (user action that indicates value) and track secondary metrics separately (revenue, retention), so you can detect misalignment early.

Common mistake: optimizing a single shallow signal (like clicks) without guardrails. The bot may learn to recommend sensational items that earn clicks but lower satisfaction. In this course, we will keep reward simple, but we will also add safety and quality rules later to prevent repetitive or annoying recommendations.

Practical outcome: you will implement a reward logger that records (state, action, reward) tuples, which is the basic training data RL needs.

Cold start is the day-one problem: the bot has little or no history, so it cannot “personalize” yet. This is not a rare edge case—it’s the normal starting condition for every new user, new item, or new product launch.

A practical cold start plan answers three questions:

• What will we recommend before learning? (popular items, editor picks, random exploration)
• How will we collect early feedback quickly? (simple UI choices, lightweight ratings)
• How will we avoid early annoyance? (diversity, no repeats, conservative exploration)

In RL terms, cold start is handled with exploration. If the bot always exploits a guess (e.g., always recommending the current “best” item), it may never learn about alternatives. A simple exploration strategy is epsilon-greedy: with probability ε, choose a random item (explore); otherwise choose the current best item (exploit). Early on, ε can be higher to learn faster, then gradually lower it to focus on quality.

Engineering judgement: exploration must be bounded. Randomly recommending anything can be harmful if some items are low quality or inappropriate. This is where you apply safety rules and content filters before the RL policy selects among remaining candidates.

Common mistake: confusing “cold start” with “the model is broken.” If you launch without an exploration plan and without a baseline policy, performance will look unstable. In this course, we’ll define a baseline recommender (e.g., simple popularity or uniform random among safe items), then let RL improve it over time.

Practical outcome: you will implement a day-one policy plus epsilon-greedy exploration so the bot can gather data while still providing reasonable recommendations.

Most beginner recommendation bots fail in predictable ways. Knowing these failure modes early will save you time and protect user trust.

• Spammy behavior: the bot chases a metric (often clicks) and becomes pushy—recommending the same “high click” content regardless of context.
• Repetitive loops: the bot finds one item that works and never leaves it, causing boredom and reduced discovery.
• Random feel: too much exploration (or a broken reward pipeline) makes recommendations seem unrelated to the user.

These failures usually come from one of three root causes: (1) reward is poorly defined or too noisy, (2) exploration is unmanaged, or (3) the action space allows low-quality or duplicate recommendations.

Practical guardrails you should plan from the start:

• No-repeat rules: don’t recommend the same item twice in a short window.
• Diversity constraints: rotate genres or categories so the bot doesn’t get stuck.
• Quality filters: remove items that are too new, too poorly rated, out of stock, or unsafe.
• Fallback policy: if the bot is uncertain (or data is missing), use a safe baseline.

Engineering judgement: guardrails are not “anti-AI.” They are product requirements. RL optimizes what you measure, not what you intend. Constraints express intent explicitly.

Practical outcome: later in the course, you’ll integrate simple rules around the RL policy so users don’t experience repetitive or annoying recommendations, even while the bot is still learning.

To ensure you finish a working system, we will keep the project scope intentionally small: a tiny recommendation bot that chooses one item from a small catalog and learns from user choices over time. You can later scale it to more items, richer state, and more advanced algorithms, but the first version must be end-to-end.

Here is the blueprint you will follow across the course:

• Define the catalog: a fixed set of items (e.g., 20 movies) with IDs and optional tags.
• Define state: minimal context (seen items, last click, optional preference tag).
• Define actions: recommend exactly one item per step.
• Define reward: start with click/play as 1, otherwise 0 (extend later).
• Implement learning: epsilon-greedy with value estimates per item (a simple bandit-style learner).
• Track metrics: click-through rate, average reward, simple satisfaction score if available, and regret (how much reward you missed compared to the best-known option).
• Add safety/quality rules: no repeats, basic diversity, and safe fallback recommendations.

Notice how this maps directly to RL basics: state → action → reward → update, repeated as a feedback loop. This is the plain-language core you should keep in mind: the bot tries something, observes the outcome, and changes its future behavior based on what worked.

Common mistake: attempting personalization, ranking lists, and deep models on day one. That makes debugging nearly impossible. If you can’t explain why the bot recommended an item, you can’t improve it safely. Our “tiny bot you can finish” approach ensures you can inspect logs, understand trade-offs between exploration and exploitation, and iterate with confidence.

Practical outcome: at the end of the next chapters, you will have a small but complete recommendation loop you can run, measure, and improve—an actual bot, not just a model.

Section 2.1: The agent and the environment

In RL, the agent is the decision-maker and the environment is everything the agent interacts with. For a recommendation bot, the agent is your code that chooses what to show next; the environment is the user plus the app context (time, device, session, what was shown earlier, and the user’s current mood—even if you can’t observe mood directly).

An everyday example helps: imagine you’re picking a restaurant for lunch. You (agent) choose a place (action), you experience the meal (environment response), and you remember whether it was good (reward signal). Next time, you use that experience to choose again. The key RL move is that learning happens through interaction, not from a static dataset labeled with “the correct restaurant.”

In recommendations, it’s tempting to treat the user as predictable. In practice, users change their minds, context shifts, and feedback is partial. That’s why the agent should be designed to handle uncertainty: it will make decisions with limited information, observe outcomes, and gradually adapt.

• Agent: the recommender logic that selects an item to display.
• Environment: the user + UI + inventory of items + constraints (e.g., content rules).
• Observation (state signal): what the agent can “see” (e.g., user segment, last item shown, session length).
• Feedback: what the environment returns (clicks, skips, ratings, complaints).

Engineering judgment: define the environment boundary so it matches what you can actually measure and influence. For example, you cannot control whether a user is tired, but you can observe time-of-day and recent behavior. Good RL setups start from reliable signals, then expand later.

Section 2.2: Actions: what the bot is allowed to do

An action is a choice the agent can make. For a recommendation bot, the most obvious action is “recommend item X.” But your action space must be realistic: the bot can only recommend items that exist, are eligible for the user, and are safe to show. Defining actions is not just math—it’s product policy encoded into the learning system.

Start small. A simple prototype might choose among 10 items (or 10 categories). In that setup, each step the agent chooses one of 10 actions. This is enough to demonstrate learning, explore vs. exploit strategies, and track outcomes. Later, you can expand to larger inventories by using candidate generation and then letting the RL agent choose among a shortlist.

• Item-level actions: choose one item from a small pool (good for learning the basics).
• Category-level actions: choose a topic (sports, cooking, music) and then pick an item inside that topic (good for scaling).
• UI actions: decide whether to show a new recommendation, repeat a prior favorite, or ask a preference question.

Common mistake: allowing actions that violate user experience rules, then “hoping rewards fix it.” If the bot is allowed to recommend the same item repeatedly, it may do so if it gets occasional clicks. Put guardrails into the action set: enforce diversity, frequency caps, and “no-repeat within N steps.” These are not afterthoughts; they are part of defining what the bot is allowed to do.

Practical outcome: by the end of this course, your bot’s action space should be explicit and testable. You should be able to log “chosen action” every time, and you should be able to explain why that action was eligible.

Section 2.3: Rewards: why numbers matter

A reward is a number that tells the agent how well it did. Rewards matter because RL algorithms do not understand “good recommendations” directly; they only optimize the numeric signal you provide. If you choose the wrong reward, you can build a bot that learns the wrong behavior very efficiently.

Design a reward you can actually measure. In recommendation systems, the easiest measurable signal is a click. A simple reward might be +1 for click, 0 for no click. That’s enough to build a working learning loop. But clicks can be misleading: users may click out of curiosity and regret it. If you have richer signals, you can shape rewards carefully.

• Binary click reward: +1 (click) / 0 (ignore). Simple and robust.
• Satisfaction reward: +2 if user rates “helpful,” -2 if “not interested.”
• Time-based proxy: +1 if dwell time > 10 seconds, else 0 (use carefully).
• Safety/quality penalties: -5 for “report,” -1 for repetitive item, -1 for too many recommendations in a row.

Engineering judgment: avoid rewards that are too delayed or too rare in an early prototype. If “purchase” happens once a week, the bot won’t learn quickly. Instead, combine short-term measurable feedback (click, save, dwell) with small penalties that discourage annoying behavior (spammy repetition, over-serving). This helps the bot learn not only “what gets interaction” but also “what stays pleasant.”

Common mistake: changing reward definitions midstream without versioning. If you redefine reward from “click” to “click + satisfaction,” your learning history is no longer comparable. In production, you would version reward functions and track metrics like click-through rate (CTR), average satisfaction, and regret (how much reward you lost versus the best action in hindsight, estimated from logs).

Section 2.4: Policies: rules for choosing actions

A policy is the agent’s rule for choosing actions. In beginner RL, the policy is often derived from estimated values: “I think item A is worth 0.6 expected reward, item B is worth 0.2, so pick A.” The tricky part is that the agent’s estimates start out wrong. If it always exploits its current best guess, it may never discover better options.

This is where exploration vs. exploitation becomes practical. A classic approach is epsilon-greedy: with probability ε, explore (pick a random eligible item); otherwise exploit (pick the item with the highest estimated value). For a small recommendation bot, epsilon-greedy is easy to implement and easy to explain to stakeholders.

• Exploit: choose the best-known item to maximize immediate reward.
• Explore: try less-known items to improve future decisions.
• Epsilon schedule: start with ε=0.2 and decay to ε=0.05 as confidence grows.

Write the learning loop in your head as: try → observe → update. The policy produces the “try” (choose item). The environment produces “observe” (click, skip, rating). The update step adjusts your value estimates so the policy improves. Even if you use a simple average reward per item, you still have a policy: “pick the item with highest average reward, except when exploring.”

Common mistake: exploring without constraints. Random exploration can accidentally produce repeated, low-quality, or sensitive recommendations. Combine epsilon-greedy with eligibility rules: exclude items shown too recently, exclude items failing safety checks, and ensure category diversity. This turns exploration into safe exploration, which is essential for user-facing systems.

Section 2.5: Episodes and steps (the timeline of learning)

RL happens over time, and you need vocabulary for that timeline. A step is one decision: the bot recommends something and observes feedback. An episode is a sequence of steps that naturally belong together. In recommendations, an episode is often a user session (open app → browse → leave) or a day of usage.

For the simplest recommendation bot, you can treat each step independently (a context-free bandit): show one item, observe reward, update that item’s estimated value. This already captures the try → observe → update loop and lets you track basic metrics like CTR and average reward.

As you make the problem more realistic, the state of the session matters. If a user just ignored three sports items, recommending a fourth sports item is likely a bad choice. That’s where you start introducing state (what has happened so far) and consider how actions affect future steps. Even then, the timeline is the same: step-by-step interaction and periodic evaluation.

• Step-level logging: user_id (or anonymized), timestamp, state features, chosen action, reward.
• Episode summaries: total clicks, total satisfaction, diversity count, number of “not interested.”
• Basic metrics: CTR, mean satisfaction, and estimated regret over a window.

Engineering judgment: decide what “done” means for an episode. If you define episodes as sessions, you can reset some state at session start (like recent items) while keeping learned item values across sessions. This avoids confusing short-term memory (session context) with long-term learning (item preferences).

Common mistake: optimizing a step metric while harming the episode. For example, clickbait can increase immediate clicks but lower session satisfaction and long-term retention. Even in a beginner bot, you can add small penalties for “quick back” or explicit negative feedback to reduce this risk.

Section 2.6: RL vs. other AI approaches (in simple terms)

It’s easy to confuse RL with supervised learning because both can produce “smart” predictions. The difference is in what data you have and what the system controls. In supervised learning, you train on labeled examples: “given features X, the correct label is Y.” In recommendations, that might look like predicting whether a user will click an item, trained on historical logs.

In reinforcement learning, the agent’s choices affect what data it gets next. The bot recommends an item; that action changes what the user sees and therefore changes the feedback. You cannot ask, “what would the user have done if we recommended a different item?” because you did not show it. This is why exploration matters: without trying alternatives, the system cannot learn about them.

• Supervised: learn from fixed datasets; labels exist for each example.
• RL/bandits: learn from interaction; feedback arrives after actions; must balance explore/exploit.
• Unsupervised: find patterns without explicit labels (e.g., clustering users) but does not directly optimize reward.

A practical way to phrase it: supervised learning predicts; RL decides. A supervised model might estimate click probability for each item; an RL policy uses those estimates (plus exploration and constraints) to choose what to show. In many real systems, you combine them: supervised models generate scores, while RL logic handles sequential decision-making, experimentation, and learning from live feedback.

Common mistake: calling any feedback-driven update “RL.” If you simply retrain a click model nightly, that’s supervised learning on logged data. If your bot changes its behavior online based on rewards and explicitly manages exploration vs. exploitation, you’re closer to RL (or contextual bandits). For this course, that distinction keeps your mental model clean and helps you design a bot that truly learns from user choices over time.

Section 3.1: Multi-armed bandits explained with a snack-choice analogy

Imagine a vending machine row with several snack buttons. Each time you stop by, you can press exactly one button. Some snacks are reliably good, some are hit-or-miss, and some are usually disappointing. You don’t know which is which at first. Your goal is to maximize your total enjoyment over many visits.

This is the multi-armed bandit problem. Each snack button is an “arm,” like the lever on a slot machine. Pulling an arm produces a reward drawn from an unknown distribution. The RL mapping is clean: the agent is your recommender, the action is selecting one item to show, and the reward is the user’s feedback (click = 1, no click = 0, or a more nuanced score). The feedback loop is repeated many times: choose → observe reward → update beliefs → choose again.

Bandits fit “pick one recommendation” problems because there is no need to plan multiple steps ahead. Your choice doesn’t change the world in a complicated way; it just gives you information about that item’s value for the current audience. That simplicity is also why bandits are a great first implementation: you can ship a learning system with minimal moving parts, then graduate later to contextual bandits or full RL when you truly need state and long-term planning.

A common mistake is to treat bandits as “magic personalization.” Standard bandits don’t use user context; they learn one global preference signal. That’s still useful for cold-start ranking, trending content, or learning which of a few candidates performs best overall. But be explicit about what you’re modeling so you don’t overclaim what the system can do.

Section 3.2: Arms as items: translating items into options

To turn a recommendation task into a bandit, you need to define the set of arms. In the simplest form, each arm corresponds to one item you might recommend: an article, a product, a video, or a learning exercise. At decision time, the bot chooses one arm and shows that item.

Start with a tiny test world: say 4 items. For example: Item A (popular), Item B (niche), Item C (new), Item D (risky/experimental). Keeping the set small lets you see learning dynamics clearly. Later you can scale to more items or use a candidate generation step that narrows thousands of items down to a small bandit set.

Engineering judgment: be careful with “what is an action?” If your UI shows a list of 5 items at once, the action might be a slate (a ranked set), not a single item. For this chapter we deliberately choose the easiest variant: recommend exactly one item. This helps you build the learning loop first.

Implement a baseline that does not learn so you have a fair comparison. Two practical baselines are: (1) uniform random (choose any item with equal probability), and (2) fixed rule (always recommend the same “best guess” item). Baselines often look bad, but they are essential: without them, you can’t tell whether your learning strategy genuinely improves outcomes or just looks busy.

Also add basic safety and quality rules up front, even in a toy system. Examples include: do not recommend the same item twice in a row; do not recommend items flagged as low-quality; and enforce simple frequency caps. These constraints can be applied before the bandit chooses (filter arms) or after (override unsafe picks). The key is to keep the learning logic clean while still protecting user experience.

Section 3.3: Tracking outcomes: counts, averages, and uncertainty

A bandit learns from feedback, so logging and statistics are not optional—they are the product. For each arm (item), track at least: n = how many times it was shown, and sum_rewards = total reward collected. The simplest value estimate is the sample average: avg_reward = sum_rewards / n. If rewards are clicks (0/1), avg_reward is the empirical click-through rate.

Uncertainty matters because early on you might have only shown an item once. Two items can have the same average, but one might have 1 impression and the other 100 impressions. The first is highly uncertain. Even if you don’t implement full Bayesian methods, you should think like an engineer about “how confident am I?” A practical proxy for uncertainty is simply the count n: low n implies high uncertainty.

To evaluate performance over time, track a few basic metrics: total clicks (or total reward), average reward per step, and a simple notion of regret. In a simulated environment where you know the true best arm, regret at time t can be defined as (best_expected_reward − reward_received). Summing regret shows how much reward you left on the table while learning. In real production you won’t know the true best, but regret is still useful in offline simulations and A/B experiments.

• Logging tip: store (timestamp, chosen_arm, reward, policy_version, eligible_arms_count). Without policy_version you can’t attribute changes to code vs. environment.
• Common mistake: updating averages incorrectly when n=0 or forgetting to increment counts on every impression, not every click.
• Outcome: you can now compare a non-learning baseline to learning strategies using consistent measurements.

These simple counters become your “state” for the learning algorithm: not user state, but internal belief state about each item’s quality.

Section 3.4: Epsilon-greedy: explore sometimes, exploit often

The core dilemma in bandits is exploration vs. exploitation. Exploitation means choosing the item with the highest current estimated reward. Exploration means trying something else to learn more, even if it might be worse right now. If you only exploit, you can get stuck on an early “lucky” item. If you only explore, you never capitalize on what you learn.

The simplest practical strategy is epsilon-greedy. With probability ε (epsilon), explore: choose a random eligible arm. With probability 1−ε, exploit: choose the arm with the highest avg_reward. Implementation is straightforward and robust enough for many small recommendation tasks.

Practical workflow for building it:

• Initialize each arm with n=0 and sum_rewards=0.
• On each step, filter arms using safety rules (e.g., no immediate repeats, remove blocked items).
• Sample a random number. If < ε, pick a random arm; else pick argmax(avg_reward) with tie-breaking (random among ties is fine).
• Show the item, observe reward, update n and sum_rewards, recompute avg_reward.

Engineering judgment: choose ε thoughtfully. A fixed ε like 0.1 is a common start. In many products, you’ll reduce ε over time (more exploration early, more exploitation later). But don’t decay ε to zero too fast; user preferences and item pools change. Keeping a small amount of ongoing exploration helps the bot adapt.

Common mistakes include exploring among ineligible items (violating safety rules) and using the same ε for every situation without monitoring. You should watch metrics like “repeat rate,” “unique items shown,” and average reward to ensure exploration is improving learning rather than just adding noise.

Section 3.5: Better exploration: optimistic start and UCB (conceptual)

Epsilon-greedy is easy, but it explores “blindly.” Two small upgrades can improve behavior without much complexity.

Optimistic initialization sets initial avg_reward for every item to a high value (or initializes sum_rewards and n with pseudo-counts). This causes early exploration naturally, because many arms appear promising until evidence pushes them down. In recommendation terms, you avoid under-serving new items just because they started with no data.

Upper Confidence Bound (UCB) adds a bonus for uncertainty. Conceptually, instead of choosing the highest avg_reward, you choose the highest:

score = avg_reward + bonus(n, total_steps)

The bonus is larger when n is small and shrinks as an item is tried more. This steers exploration toward items that are either performing well or not yet well-measured. The typical bonus grows with log(total_steps) and decreases with sqrt(n), which gives a principled “try uncertain things, but not forever” behavior.

Where this matters in practice: if you have one item with a slightly lower average but very few impressions, UCB will intentionally re-test it to reduce uncertainty. Epsilon-greedy might ignore it for long stretches if it’s not the current best, slowing learning.

Engineering judgment: UCB-style methods are sensitive to scaling of rewards (clicks vs. 1–5 ratings) and to how you count “steps” when eligibility filtering changes the available set. Keep the conceptual goal in mind—balance performance and information gain—then validate with simulation before using in user-facing settings.

Section 3.6: Simulating users to test learning safely

Before you let a learning recommender interact with real users, you should test it in a controlled “toy world.” Simulation is how you verify that your update logic works, your metrics make sense, and your safety rules actually prevent annoying patterns.

A simple simulator defines true click probabilities for each item, e.g., A=0.30, B=0.20, C=0.10, D=0.05. When the bot recommends item A, the simulated user clicks with probability 0.30 (sample a Bernoulli random variable). Now you can run thousands of steps quickly and compare strategies: random baseline vs. greedy vs. epsilon-greedy vs. optimistic vs. conceptual UCB.

What to track during simulation:

• Total reward and average reward over time (learning curves).
• Impressions per item (did the bot actually explore?).
• Cumulative regret relative to the known best item.
• Safety metrics: consecutive repeats, maximum frequency per item, proportion of blocked/overridden recommendations.

Common mistakes in simulation include accidentally leaking the “true probabilities” into the policy (making it unrealistically strong), failing to reset random seeds when comparing methods, and evaluating only final performance instead of the full learning curve. In real products, early performance matters because users experience the system while it learns.

Practical outcome: with a simulator you can iterate safely on decision rules, pick a reasonable ε (or bonus size), and ensure your baseline comparison is honest. Once you can demonstrate that learning beats the non-learning baseline in the toy world—and that safety rules keep behavior stable—you’re ready to move from “educational prototype” to a small controlled experiment with real feedback.

Section 4.1: Conversation flow as a decision loop

A recommendation bot is a conversation, but for RL it must also be a repeatable decision loop. The simplest stable loop is: (1) ask a short question to establish context, (2) suggest one item (or a small set), (3) ask for feedback, (4) update the policy, then (5) repeat. Each pass through this loop is one “step” where the agent takes an action and receives a reward.

Design the dialogue so the “state” is easy to compute. A practical state might include: the user’s declared goal (e.g., “something quick”), a category (music, article, product), and a small memory of recent suggestions (to avoid repeats). You do not need deep language understanding at this stage; you need consistent signals. Many teams fail here by asking open-ended questions that produce hard-to-interpret answers, which later forces guesswork when converting text to reward.

Keep the action space small and explicit. For a first bot, an action can simply be “recommend item i” from a curated list. If you want slight personalization, define actions as “recommend a genre” or “recommend from cluster k,” then pick a concrete item inside that cluster with a deterministic rule. This keeps RL responsible for the learning, not the item-selection plumbing.

• Ask: “What are you in the mood for—something short or something deep?”
• Suggest: “Try Item A (2 minutes). Want it?”
• Get feedback: buttons: “Yes”, “No”, “Show another”, “Save for later”
• Loop: update and propose the next suggestion

Engineering judgment: prefer explicit buttons (or numbered replies) over free text. This reduces ambiguity, makes reward mapping reliable, and gives you cleaner metrics. You can add richer natural language later, once the loop is proven.

Section 4.2: Data logging: what to save and why

RL without logs is guesswork. Your bot will occasionally behave oddly (repeat itself, “learn” the opposite preference, or get stuck exploiting too early). A simple interaction log is your black box recorder: it lets you reproduce what happened, compute metrics, and refine reward design. The log should be append-only, timestamped, and easy to query (CSV, JSONL, or a tiny database table).

At minimum, store one row per recommendation decision. Include identifiers and the exact context so you can replay learning offline. A practical schema:

• event_id, timestamp, user_id/session_id
• state: e.g., {goal: “short”, category: “articles”, recent_items: [...]}
• action: the recommended item_id (or action_id)
• policy_info: epsilon value, whether this was explore vs exploit, model version
• presented_items: if you showed a slate, store all candidates and their scores
• feedback: raw input (button clicked, text), plus parsed label
• reward: numeric reward used for learning (store final value)
• latency: response time, and whether the user abandoned

Common mistake: only logging “click/no click.” That hides critical issues like the bot not showing the same thing twice, the user not seeing the option, or rewards being computed differently across code paths. Also, store the reward you actually used, not just the raw feedback. Otherwise, when you change reward mapping later, you can’t tell which mapping produced which behavior.

Practical outcome: with this log, you can compute clicks, average reward, exploration rate, and simple regret proxies (e.g., how often the user rejected top-scored items). You can also diagnose whether the model is learning or merely reacting to random noise.

Section 4.3: Turning feedback into reward values

Users don’t speak in rewards; they give signals. Your job is to map those signals into numeric values that teach the bot what “good” means. The mapping should be (1) consistent, (2) aligned with user experience, and (3) robust to missing data. A classic beginner mistake is to treat every negative signal as equally bad, which can push the agent toward conservative, repetitive recommendations.

Start with a small reward table. For example:

• Click / “Yes”: +1.0
• Save / “Add to list”: +0.6 (interest, but not immediate)
• “Show another”: -0.2 (mild dissatisfaction, but still engaged)
• Explicit “No”: -0.6
• Report / “Not appropriate”: -2.0 (strong negative + safety trigger)

Why not use only +1 and 0? Because richer rewards help the bot distinguish “not now” from “never.” “Show another” often means the suggestion missed, but the user still wants help; punishing it too strongly can make the bot avoid exploring and instead cycle through the same “safe” item.

Use response time carefully. A fast “No” may indicate a poor match; a slow “No” might mean the user considered it. If you incorporate timing, do it gently (e.g., adjust reward by ±0.05) and log it, because timing is noisy and can correlate with distractions rather than preference.

Engineering judgment: keep reward scales stable over time. If you change reward definitions midstream, segment your analysis by “reward_version” in logs. Otherwise the bot’s learning curve becomes uninterpretable, and you may falsely conclude the model “got worse” when only the reward scale changed.

Section 4.4: Online updates: learning one step at a time

Online learning means the bot updates after each interaction, not after a nightly batch job. This is the heart of the RL loop: observe feedback, compute reward, update the policy, then make the next decision. For a “reinforcement learning basics” recommender, you can implement this with an epsilon-greedy multi-armed bandit per context bucket (or a simple contextual bandit).

A practical approach: maintain a table of action values Q(state, action). When a recommendation is made in a given state, you update that action’s value using an incremental mean:

Q ← Q + α (reward − Q)

Here α is a learning rate (e.g., 0.1). If you prefer exact averages, track counts N and use α = 1/N. The key is that you update only the action you took, using the reward you observed.

Decision rule (epsilon-greedy): with probability ε, explore (pick a random allowed item); otherwise exploit (pick the item with highest Q in that state). Start with ε around 0.2–0.3 for early learning, then slowly decay it (e.g., ε = max(0.05, 0.3 × 0.99^t)). Log whether you explored; otherwise you can’t interpret improvements.

Common mistakes:

• Updating the wrong key: e.g., mixing sessions or using an overly broad state so everything collapses into one bucket.
• Letting Q explode: inconsistent reward scales or double-applying rewards.
• Exploiting too early: ε too small causes the bot to “lock in” a mediocre item.

Practical outcome: after a handful of interactions per state, you should see the bot shift probability toward items that produce higher rewards, while still sampling alternatives to avoid blind spots.

Section 4.5: Handling ‘no response’ and unclear feedback

Real users are messy: they abandon, they get distracted, they reply with “maybe,” or they type something unrelated. If you treat every “no response” as a strong negative reward, your bot may learn that everything is bad—especially if users often leave mid-conversation. Instead, treat missing or unclear feedback as a distinct outcome with careful defaults.

First, define a timeout window (e.g., 60 seconds) after which an interaction is labeled no_response. Then decide how it affects learning. A safe baseline is a small negative reward (e.g., -0.05) or even 0.0, because silence does not reliably mean dislike. More important: log it and track its rate as a product metric.

For unclear text feedback, use a simple classifier or rule-based parsing, but include an “unknown” label. Examples: if the user says “what else?” map to “show_another”; if they say “stop recommending this,” map to “no” and trigger a repetition rule. If the parser is unsure, do not force a reward guess; store raw text and assign “unknown” with reward 0.0, then ask a clarifying question.

Add safety and quality rules that sit outside the learner:

• No-repeat window: don’t recommend the same item twice within the last K turns.
• Frequency cap: limit how often you ask for feedback to avoid fatigue (e.g., every other turn).
• Content filters: block disallowed items regardless of Q value.
• Diversity rule: if top-1 repeats too often, force exploration among top-N.

Engineering judgment: rules are not “cheating.” They protect user experience while your model is still learning, and they prevent the agent from exploiting loopholes in your reward (for example, repeatedly recommending a clickbait item that gets clicks but low satisfaction).

Section 4.6: A complete walkthrough of one learning cycle

Let’s walk through one full end-to-end cycle with concrete artifacts: state, action, log entry, reward, and update. Assume the user is browsing short articles. Your state representation is:

• goal = “short”
• category = “articles”
• recent_items = [102, 205]

Your action set is a curated list of candidate article_ids allowed by safety filters. The bot computes Q for each candidate in this state bucket. Suppose Q(“short/articles”, 310)=0.4, Q(..., 444)=0.1, Q(..., 555)=0.35. With ε=0.2, you roll exploitation this turn and pick item 310.

Conversation step: Bot: “Try Article 310: ‘Two-minute guide to habit loops.’ Want to read it?” User clicks “Yes.”

Logging: You append an event with timestamp, state, action=310, explore=false, ε=0.2, raw_feedback=“yes”, parsed_feedback=“yes”, reward=+1.0. This log entry is what you will later use to compute CTR and to debug unexpected behavior.

Reward and update: Using α=0.1 and current Q=0.4:

Q_new = 0.4 + 0.1 × (1.0 − 0.4) = 0.46

The bot has now slightly increased its belief that item 310 is good for this state. Next turn, your no-repeat rule prevents recommending 310 again immediately, so the bot either asks a follow-up question (“Want another short one?”) or suggests the next best item, possibly exploring if the ε roll triggers it.

Verify improvement over a demo session by tracking: (1) click rate over time, (2) average reward per turn, (3) exploration vs exploitation ratio, and (4) a simple regret proxy—how often users reject the top-ranked suggestion. In a short demo (20–50 turns), you may not see dramatic gains, but you should see a directional shift: fewer explicit “No” responses for frequently visited states and increasing Q for items that get positive feedback.

Common demo pitfall: changing multiple things at once (reward mapping, ε schedule, and state features). For a clean verification, change one variable at a time and compare logs. A reliable end-to-end loop—conversation design, logging, reward mapping, online updates, and robust handling of silence—is the foundation you’ll build on when you later add richer state features and better exploration strategies.

Section 5.1: What personalization means (and what it doesn’t)

In a reinforcement-learning-flavored recommender, “personalization” simply means the agent chooses actions differently depending on the user-related information in the state. If your state contains only a global counter (or nothing at all), your bot is effectively learning “what works on average.” Once you include a user profile feature—like new versus returning—you’ve taken the first step toward per-user behavior without maintaining a full identity graph.

Personalization does not mean you must predict everything about a person. It also doesn’t mean you need to store sensitive attributes. In fact, over-personalization is a common failure mode: the system learns a narrow view of the user and stops exploring, which can reduce discovery and make the bot feel repetitive. Another failure is confusing correlation with intent: if a user clicked “news” at 9am once, it doesn’t mean they always want news at 9am.

A practical baseline is to create a small “profile” object that is cheap to compute and safe to store:

• is_new_user: true until you have, for example, 5 interactions.
• recent_click_categories: a short rolling window (e.g., last 5 clicks) rather than a lifetime history.
• session_length_bucket: short vs. long session, which often correlates with exploratory vs. goal-directed behavior.

Engineering judgment: keep your first profile features stable and interpretable. If a feature changes every interaction, your learned values will be noisy. If a feature is too specific (like exact timestamps), you’ll split your data into tiny bins and never learn.

Section 5.2: Context: adding a few helpful details safely

Context features turn a generic recommender into a situational one. You’re not changing the core feedback loop—agent selects an action, user responds, agent updates—but you are enriching the “state” so the same action can be evaluated differently depending on the situation.

For this chapter, add three context signals that are easy to capture and explain:

• Time: bucket into simple bins like morning/afternoon/evening, or weekday/weekend.
• Category: what the user is browsing or what they last engaged with (e.g., “sports,” “coding,” “cooking”).
• Mood (optional): do not infer secretly; instead, allow the user to select a simple mode like “I want something quick” vs. “I want to explore.” If you can’t collect this explicitly, skip it.

A safe approach is to represent context as a small dictionary and then map it into a key you can use for learning. For example, your “state” for a contextual bandit can be the tuple (is_new_user, time_bucket, current_category, mood). This lets you maintain separate action-value estimates per context. The bot can learn that “returning users in the evening in cooking category” respond well to different items than “new users in the morning in coding.”

Common mistakes: (1) adding too many categories too soon, which makes learning sparse; start with a small taxonomy. (2) using raw time (like 13:07), which makes every state unique. (3) treating mood as a hidden label inferred from behavior; that crosses into fragile and potentially creepy territory. Practical outcome: with just a few bins, you’ll see higher click rates and less “why did you recommend that?” confusion.

Section 5.3: Contextual bandits in plain language

A contextual bandit is the simplest useful model for recommendations that react to user context. Think of it as “choose one recommendation now” rather than “plan a long sequence of moves.” In many recommenders, each interaction is mostly independent: the user sees an item, reacts, and you learn. That’s a bandit setting. When you add context, you get a contextual bandit: you still choose one action, but you choose it based on observed context.

In your code, the workflow looks like this:

• Build a context key ctx from profile + session context.
• For that ctx, maintain an estimate of value for each action (item or category to recommend).
• Select an action with epsilon-greedy: with probability epsilon explore; otherwise exploit the best-known action for that context.
• Observe reward (click, dwell, satisfaction score), then update the estimate for (ctx, action).

Because you’re avoiding heavy math, use incremental averages: store count[ctx, action] and value[ctx, action]. When reward r arrives, update with value += (r - value) / count. This is stable, easy to explain, and works well when rewards are bounded (e.g., 0/1 click or 1–5 rating scaled to 0–1).

Engineering judgment: keep the action space manageable. If actions are individual items and you have thousands of items, learning per context becomes slow. A pragmatic step is to let actions be “recommend a category” or “recommend from a curated pool,” then pick a specific item with a separate non-learning rule (e.g., newest in category). This keeps the RL part focused on the decision that matters: what type of content fits the current user context.

Section 5.4: Simple scoring rules for cold start and fairness

Cold start happens in two places: new users and new items. In a contextual bandit, it also happens for new contexts (a context key you haven’t seen before). If you do nothing, your bot will default to random exploration, which can feel low quality. The fix is to combine learned values with a few simple scoring rules that act as guardrails.

Start with a backoff ladder:

• If (ctx, action) has enough data (e.g., 20+ samples), use its learned value.
• Else back off to a broader context, like (is_new_user, time_bucket) without category/mood.
• Else back off to global averages.

Then add a cold-start prior: initialize all values to a reasonable baseline, such as the global click rate, rather than 0. This prevents the first few unlucky non-clicks from permanently burying an option. You can also enforce a minimum exploration budget per action so new items get a chance to be seen.

Fairness, in this simplified course setting, means avoiding systematic neglect. If your bot only optimizes short-term clicks, it may over-recommend “easy click” categories and starve others of impressions, making it impossible to learn their true value. A practical mitigation is an exposure floor: ensure each major category receives at least X% of recommendations over a day, or ensure each new item gets N impressions before it can be deprioritized. Another simple technique is to cap any single category’s share (e.g., no more than 50% of recommendations in a session). These are not perfect fairness solutions, but they are transparent, debuggable, and effective at preventing runaway feedback loops.

Section 5.5: Diversity controls: avoid recommending the same thing

Even a well-trained contextual bandit can become repetitive because exploitation keeps choosing the top option for a context. Users experience this as “the bot is stuck.” To fix it, add diversity controls that operate alongside learning. Think of them as quality rules: they don’t replace the agent; they shape the candidate actions the agent is allowed to pick.

Implement two layers of variety:

• Repeat suppression: keep a short “recently recommended” list and apply a penalty (or temporary ban) to actions seen in the last K turns.
• Category rotation: if the last two recommendations were the same category, require the next one to come from a different category unless the user explicitly asks for more.

One practical scoring pattern is: final_score = learned_value(ctx, action) - repetition_penalty(action) - category_streak_penalty(category). Then apply epsilon-greedy over final_score rather than raw learned value. This keeps exploration/exploitation intact while improving perceived quality.

Common mistake: using diversity as pure randomness. Randomness can feel like low quality. Better is controlled diversity: diversify among the top few actions, or diversify across categories while still using learned estimates. Practical outcome: you should see improved session satisfaction and fewer “I already saw this” complaints, while maintaining comparable click-through because the bot is still choosing from high-value options.

Section 5.6: Privacy basics: collect less, keep it simple

Personalization and context can drift into privacy risk if you collect too much or store it too long. The safest system is the one that never collects data it doesn’t need. For this course project, treat privacy as an engineering constraint: your recommendation quality must come from minimal, non-sensitive signals.

Use these practical rules:

• Data minimization: prefer coarse buckets (time-of-day) over exact timestamps; prefer broad categories over detailed content logs.
• Short retention: store only recent interaction windows (e.g., last 30 days) or session-only context when possible.
• Separate identity from behavior: if you need a returning-user flag, use a random user ID that is not an email or phone number.
• Make mood explicit or omit it: if the user doesn’t provide it, don’t guess it from sensitive signals.

Also keep your models simple enough to audit. With a contextual bandit table, you can inspect which contexts exist and what actions they favor. If a user asks “why did I get this recommendation?”, you can answer in plain language: “You’re a returning user browsing cooking in the evening, and this category performs well for similar sessions.”

Finally, be careful with logs. It’s easy to accidentally log raw context dictionaries, full item text, or unique identifiers. Log only what you need for metrics (clicks, satisfaction score, regret estimates, counts) and debugging, and remove or hash anything that can identify a person. Practical outcome: you get most of the value of personalization while keeping your system safer, easier to maintain, and more trustworthy.

Section 6.1: Metrics: success rate, average reward, satisfaction

Pick metrics that match your goal. The easiest trap in recommendation projects is to measure what is convenient (requests served, items shown) instead of what indicates user value. In RL, your reward is a training signal, but your metrics are how you judge whether the system is actually improving for the user and the product.

For a beginner-ready bot, use three simple metrics that cover different angles:

• Success rate: the fraction of recommendations that get a “positive” response. For example: click, “yes,” save, watch 10 seconds, or a thumbs-up. Define it clearly: success = click within 30 seconds, or success = user selects the recommended item as their final choice.
• Average reward: the mean of your reward values per interaction. If your reward is +1 for click and 0 otherwise, average reward equals click-through rate. If you have graded rewards (e.g., +2 for “love it,” +1 for click, -1 for “hide”), average reward captures both good and bad outcomes.
• Satisfaction score: a separate, human-facing signal that is harder to game. This can be an explicit rating (1–5) after a session, or a simple “Was this helpful? yes/no.” It is normal for satisfaction to be sparse; treat it as a high-quality audit metric.

Workflow tip: log every interaction as an event record: timestamp, user/session id (or anonymous), state features used, action chosen, exploration flag (did epsilon-greedy explore?), reward value, and any satisfaction label if available. Without a log, you cannot reproduce bugs or compare versions.

Common mistake: changing the reward definition midstream without versioning it. If you later compare “average reward” across days, you may be comparing different reward scales. Add a reward_version field to your logs, and keep a short markdown note explaining what changed and why.

Practical outcome: after this section, you should be able to print a daily or weekly report: interactions, success rate, average reward, satisfaction (if present), and a breakdown by item category. That report becomes your steering wheel.

Section 6.2: Regret as ‘missed opportunity’ (beginner-friendly)

Regret is a useful RL metric because it measures opportunity cost: how much reward you “left on the table” by not picking the best action. For beginners, think of regret as: “How much better could we have done if we had recommended the best option we reasonably could have known?”

In real recommendation systems, you rarely know the true best action for each moment. But in small projects you can estimate regret in simple ways:

• Bandit-style estimated regret: maintain an estimated value Q[item]. When you choose item A but item B has higher Q, estimated regret is max(Q) - Q[A]. This does not require ground truth; it measures whether your policy is selecting what it currently believes is best.
• Offline “counterfactual” regret (tiny version): if you sometimes show multiple items (e.g., list of 3) and the user picks one, you can treat the chosen one as “best among shown.” Regret for your top recommendation can be reward(chosen) - reward(top). It is imperfect because you did not show everything, but it gives a sanity check.
• Known-best sandbox: create a small simulated environment where certain items are truly better for certain user types. Here you can compute real regret and validate that epsilon-greedy learning reduces it over time.

Why regret matters: success rate alone can hide problems. Imagine your bot recommends a “safe” popular item that gets a modest click rate, while a slightly riskier choice would delight users more often. Regret highlights that missed upside, and it helps you tune exploration. If regret stays high, you might be over-exploiting too early or not learning from feedback effectively.

Common mistake: interpreting estimated regret as objective truth. It is only as good as your value estimates. Use it as a trend indicator: regret should generally decrease as the bot learns and as your guardrails prevent repeated low-quality choices.

Practical outcome: add a “regret over time” line to your report. Even a crude estimate can reveal whether policy changes are moving you toward better decisions.

Section 6.3: A/B testing basics for tiny projects

Testing changes safely is how you improve without breaking user experience. A/B testing sounds “big tech,” but you can do a tiny version with discipline: pick one change, split traffic, and compare metrics over the same period.

Start with simple experimental design:

• Define the hypothesis: “Reducing epsilon from 0.2 to 0.1 after 100 interactions will increase satisfaction without lowering success rate.”
• Choose a primary metric: pick one that matches the goal (e.g., satisfaction or average reward). Keep secondary metrics (success rate, regret) as safety checks.
• Random assignment: assign each user/session to A or B using a stable hash so they stay in the same group across visits. This avoids switching experiences mid-learning.
• Run long enough: for tiny projects, aim for a minimum number of interactions per group (e.g., 200–500) rather than a complicated power analysis. Note that noisy rewards need more samples.

RL adds a twist: the policy adapts during the experiment. To keep comparisons fair, you can freeze learning during the test (evaluate two fixed policies), or you can let both groups learn but from their own data. For beginner projects, freezing is simpler: train the bot for a while, snapshot parameters, then test policy A vs. policy B without updates.

Common mistakes include running multiple changes at once (cannot attribute results), peeking too often and stopping when the chart “looks good,” and comparing groups with different user mixes. Even with small data, you can protect yourself by logging group assignment, reporting confidence intervals if you can, and focusing on consistent direction across metrics.

Practical outcome: you will be able to ship improvements incrementally—changing exploration rate, reward shaping, or a guardrail—while measuring impact and avoiding surprise regressions.

Section 6.4: Safety guardrails: banned items, cooldowns, limits

A beginner-ready bot needs guardrails even if it is “just a demo.” RL will exploit whatever gets reward—even if that means repetitive, spammy, or inappropriate recommendations. Guardrails are rules that constrain actions so the learning happens inside acceptable boundaries.

Implement three practical guardrails:

• Banned items (blocklist): never recommend items that are unsafe, irrelevant, or disallowed (e.g., age-restricted content, broken links, items without metadata). Enforce this before scoring so the policy cannot pick them.
• Cooldowns (anti-repetition): prevent recommending the same item too frequently. A simple rule: do not show the same item within the last N interactions, or within a time window (e.g., 24 hours). You can implement a per-user recent-history set and filter candidates.
• Limits (caps and diversity): cap how often a category appears (e.g., no more than 2 “promo” items in 10 interactions), and ensure minimal diversity (e.g., at least 3 categories in the last 10). This protects the user experience and reduces the chance that one high-reward item dominates.

Where guardrails live in the pipeline matters. A robust pattern is: (1) build candidate list, (2) apply hard filters (bans, availability), (3) apply soft constraints (cooldowns, diversity re-ranking), (4) choose an action with your policy (epsilon-greedy or greedy), (5) log what was removed and why. Logging removals is crucial: if success rate drops, you need to know whether the policy degraded or whether the candidate pool shrank due to new rules.

Common mistake: hiding guardrail effects by not logging them. If 40% of items are filtered, your metrics and learning dynamics change. Treat guardrails as first-class features with versions and clear documentation.

Practical outcome: you will have a bot that is harder to “break,” less repetitive, and safer to show to real users—even while it is still learning.

Section 6.5: Debugging learning: when rewards mislead you

When an RL bot behaves oddly, the issue is often not the algorithm—it is the reward signal or the data pipeline. Debugging RL is mostly debugging feedback loops. Your goal is to confirm that (1) the bot sees the right state, (2) the action taken is what you think it is, and (3) the reward correctly reflects the user outcome.

Watch for these reward pitfalls:

• Proxy rewards: clicks can be a proxy for satisfaction, but clickbait can increase clicks while reducing satisfaction. If satisfaction drops while average reward rises, your reward is misaligned.
• Delayed rewards: if the “true” outcome happens later (e.g., user completes a lesson), but you reward immediate clicks, the bot learns to optimize the wrong part of the journey. Consider adding a delayed bonus when a session ends well, even if it is sparse.
• Missing negatives: if you only log positive events (clicks) and ignore skips or “hide” actions, the bot cannot learn what is bad. Add explicit negative or zero rewards for non-clicks, and log “impressions” (shown items) so absence of a click is still data.
• Instrumentation bugs: double-counted clicks, rewards attached to the wrong item id, or session resets that erase history. These can silently poison learning.

A practical debugging workflow:

• Replay a session: for one user/session, print the sequence of states, actions, filtered candidates, exploration flags, and rewards. Humans are good at spotting nonsense in a trace.
• Check distributions: plot reward frequency, per-item exposure, and per-item average reward. If one item gets 90% exposure, you may have a bug or a missing cooldown.
• Sanity baselines: compare against a simple baseline (random, most-popular, or category-round-robin). If your learner is worse than random, treat it as a signal that your feedback loop is broken.

Common mistake: tuning epsilon forever instead of fixing reward definitions and logging. Exploration cannot rescue a misleading reward; it only gathers more misleading data.

Practical outcome: you will be able to diagnose whether poor performance is due to (a) learning rate/epsilon choices, (b) guardrails constraining too much, or (c) reward/telemetry issues that must be fixed first.

Section 6.6: Shipping checklist and next learning paths

Shipping is not “deploy and hope.” Shipping is making the bot reproducible, observable, and safe enough that future improvements are easy. Use a lightweight checklist before you call the project done.

• Reproducible runs: a single command to start the bot, with pinned dependencies (requirements file), clear config (epsilon, learning rate, cooldown window), and sample data.
• Versioned policy: store a model/policy version id in logs. If you change reward shaping or guardrails, bump versions so you can compare.
• Logging + metrics report: event logs include state, action, reward, exploration flag, and guardrail filtering. Provide a small script/notebook that computes success rate, average reward, satisfaction, and estimated regret.
• Safety rules enabled by default: banned items list, cooldowns, and category limits turned on, with documented rationale and where to edit them.
• Fallback behavior: if the model fails or has too little data, fall back to a safe baseline (popular items, curated set, or random within constraints).
• Tiny experiment plan: a template for running A/B tests: what you will change, which metric is primary, how long you will run, and what would make you roll back.

After you ship the beginner version, the next learning paths are clear. You can extend from bandits to contextual bandits (richer state features), add better exploration strategies (softmax, UCB), and introduce longer-horizon RL if sequences matter (sessions with multiple steps). You can also improve product quality: better diversity re-ranking, per-user preferences, and more robust offline evaluation. The key is to keep the loop healthy: measurable goals, safe experiments, and guardrails that protect users while the bot learns.

Practical outcome: you finish this chapter with a bot you can demonstrate confidently—one that learns over time, reports whether it is improving, and avoids the most common ways recommendation systems annoy people.