Project-Based MLOps: Deploy a Model API, Monitor & Roll Back

Section 1.1: What MLOps means in hiring terms

In hiring, “MLOps” is less a job title and more a capability: you can take an ML idea from experiment to a reliable service. Recruiters and interviewers translate that into a few concrete questions: Can you reproduce training? Can you deploy behind an API? Can you observe behavior in production? Can you make safe changes and roll back when needed? Your portfolio project should answer those questions unambiguously.

Think of MLOps as “software engineering + statistical systems thinking.” You’re responsible not only for model code, but also for interfaces, dependencies, operational risk, and feedback loops. A model that scores 0.92 AUC in a notebook but can’t be rebuilt deterministically, can’t be deployed without manual steps, or can’t be monitored is not a production model—it’s a demo.

Common mistakes when transitioning into AI are (1) showing only notebooks, (2) relying on ad-hoc local files without artifact/version conventions, and (3) skipping acceptance criteria. The practical outcome of this chapter is that your project will read like an engineering deliverable: a structured repo, pinned environment, versioned artifacts, and a documented “golden path” that anyone can run. That combination signals seniority even if the model itself is simple.

• What hiring managers notice: repeatability, clear boundaries, automation hooks, and operational empathy.
• What they distrust: hidden state (notebooks), “works on my machine,” and metrics without business context.

You do not need a complex model to demonstrate MLOps skill. A baseline classifier with excellent packaging, monitoring, and rollback discipline is often more convincing than a fragile deep model with no operational story.

Section 1.2: Project scope, constraints, and acceptance criteria

Start by selecting a use case that is small enough to finish but rich enough to operationalize. Good candidates: churn prediction, fraud risk triage, ticket routing, or sentiment classification. Prefer problems with clear inputs/outputs and a dataset you can legally ship in a repo (or fetch deterministically). Your goal is not novelty; your goal is an end-to-end product slice.

Define a production-ready success metric. In production, you care about trade-offs and costs: false positives vs false negatives, latency budgets, throughput, and stability over time. For example, instead of “maximize accuracy,” define “achieve F1 ≥ 0.78 on the locked test set while keeping p95 inference latency ≤ 150ms on CPU.” This matters because you’ll later monitor for regressions and decide when to roll back.

Add constraints early. Constraints are not limitations; they are guardrails that make your system realistic. Choose: CPU-only serving, max container size, no external stateful services (for the course), predictable runtime config via environment variables, and a single command to reproduce training. Then write acceptance criteria that are checkable by CI or by a runbook.

• Scope: one model, one API service, one container, one monitoring/metrics layer.
• Constraints: deterministic training, pinned deps, versioned artifacts, and a documented release/rollback path.
• Acceptance criteria examples: “train script produces a model artifact with a version tag,” “/v1/predict returns schema-valid responses,” “smoke tests pass in CI,” “metrics endpoint exposes request count and latency histogram.”

A common mistake is picking a metric you can’t measure after deployment (e.g., using labels you won’t have online). If online labels are delayed, plan for proxy signals (input drift, confidence distribution shifts) and delayed performance evaluation. That decision should be explicit in your blueprint.

Section 1.3: Repository layout and contracts between components

Your repository is the product’s “map.” A good layout makes it hard to do the wrong thing (like mixing training and serving logic) and easy to automate tasks (CI, builds, releases). Aim for separation of concerns: data handling, training pipeline, inference service, and ops tooling.

Here is a practical layout you can implement immediately:

• src/ shared Python package (feature transforms, schemas, utilities)
• training/ train.py, evaluate.py, config files, metrics outputs
• serving/ FastAPI app, routers, model loader, request/response schemas
• data/ (optional) small sample data or scripts to fetch/build datasets deterministically
• artifacts/ local dev outputs (ignored in git), or a pointer to remote storage conventions
• ops/ Dockerfile, compose files, CI configs, deployment scripts
• docs/ runbooks and decision records (why you chose certain conventions)

The key idea is “contracts between components.” Training produces an artifact (model + metadata) with a known schema and location. Serving loads that artifact and exposes a stable API. Ops builds and runs the service with predictable configuration. If any component needs a “secret” assumption (like a hard-coded feature order), write it into the contract via explicit metadata and validation.

Common mistakes include importing training-only dependencies inside the API service, letting feature engineering differ between training and inference, and storing preprocessing steps only in a notebook cell. Your practical outcome here is a repo where the flow is obvious: fetch/build data → train → evaluate → package artifact → serve via API. That clarity is what enables CI checks and safe deployments later.

Section 1.4: Reproducible environments (venv/poetry/uv) and pinning

Reproducibility is the minimum bar for production-style ML. If you can’t recreate yesterday’s model, you can’t debug a regression, compare experiments, or trust your rollbacks. Environment management is where many projects quietly fail: “pip install -r requirements.txt” is not enough unless you pin and lock versions consistently.

You have a few solid options:

• venv + pip-tools: maintain requirements.in and compile a fully pinned requirements.txt with hashes.
• Poetry: pyproject.toml plus poetry.lock for deterministic installs.
• uv: fast installs; often paired with pyproject.toml and a lockfile workflow.

Pick one and commit to it for the course. The engineering judgment is to optimize for clarity and repeatability over novelty. A typical pattern is: define your dependencies in pyproject.toml, generate a lockfile, and ensure CI installs from the lockfile only. Also decide your Python version (e.g., 3.11) and enforce it in tooling and Docker, because model behavior and compiled wheels can differ across versions.

Pinning is not about freezing forever; it’s about controlling change. You should be able to intentionally update dependencies, run tests, and produce a new release candidate. Common mistakes: unpinned transitive dependencies (leading to “same code, different results”), mixing dev and prod dependencies, and relying on system packages that aren’t declared. The practical outcome: anyone can clone the repo, run one command to create the environment, and get the same versions you used to train and serve.

Finally, align your local environment with your container environment. If you develop on macOS but deploy on Linux, your lockfile and Docker build must be consistent enough that you don’t discover platform issues at release time.

Section 1.5: Data & model artifact naming, storage, and lineage

Once you move beyond a notebook, “the model” is not a single .pkl file. A production artifact should include: the trained weights, the preprocessing steps (or a pipeline object), the feature schema, and metadata describing how it was produced. Without lineage, you can’t answer basic questions like “Which dataset created this model?” or “What code version is running in production?”

Define conventions now. A simple, effective approach is semantic versioning for the API and immutable IDs for artifacts. For example:

• Dataset version: data/churn/v2026-03-26/ (or a hash-based ID if generated)
• Model version: model/churn/1.0.0/ with a build metadata tag (git SHA)
• Artifact bundle: model.joblib + preprocess.joblib + metadata.json

Your metadata.json should capture at least: training timestamp, git commit SHA, dependency lockfile fingerprint, dataset ID, label definition, feature list/order, training metric values, and threshold settings (if applicable). This is what makes rollbacks safe: you can redeploy a previous artifact and know exactly what you’re getting.

Storage can start local for development (an artifacts/ directory ignored by git), but design as if you’ll later move to object storage (S3/GCS/Azure Blob). That means artifacts should be immutable and addressable by version, not overwritten. Common mistakes: overwriting “latest,” forgetting to store the threshold used in evaluation, and silently changing feature engineering without bumping the model version. The practical outcome is a lineage trail that connects data → training run → artifact → deployed API version, which you’ll use later for monitoring and incident response.

Section 1.6: Runbooks, checklists, and definition of done

A runbook is the difference between a clever repo and an operable system. Your first runbook should describe the end-to-end “golden path”: the simplest happy-path procedure to go from a clean checkout to a running model API. It should be written for a teammate (or future you) who has no context and no patience for guesswork.

Include explicit commands and expected outputs. A strong golden path runbook usually contains:

• Setup: create environment from lockfile, verify Python version
• Data: fetch/build dataset deterministically, verify checksum or row counts
• Train: run training script with config, produce versioned artifact bundle
• Evaluate: compute metrics and compare to acceptance criteria
• Serve: start FastAPI locally, hit /health and /v1/predict with a sample request
• Package: build Docker image and run with environment variables

Pair the runbook with checklists and a “definition of done.” Definition of done is where you encode quality gates: lockfile committed, model artifact includes metadata, API schema validated, basic CI checks pass, and rollback plan exists (e.g., redeploy previous image tag + previous model artifact). This is also where you prepare for observability work later: decide what logs and metrics you must have before you consider the service “deployable.”

Common mistakes are leaving steps implicit (“install dependencies”), skipping validation of outputs, and writing runbooks that only work on the author’s machine. Your practical outcome: a documented, repeatable workflow that supports safe iteration—exactly what you need before you add CI/CD, monitoring, drift detection, and rollback mechanics in later chapters.

Start by treating data loading as part of the product, not a convenience. Your loader should be a stable function (or module) that reads from a known location, enforces schema, and produces a dataset object that downstream code can rely on. In a project repo, this usually lives under something like src/data/ and is invoked by a single training entry point (for example, python -m src.train).

The split strategy is where many “great in notebook, bad in prod” models are born. Choose a split that matches how the model will be used. If the model will score future events, prefer a time-based split (train on older data, validate on newer). If the model will score by user, use group-aware splits to avoid the same entity appearing in both train and validation. Random splits are only realistic when the deployment distribution truly mixes uniformly and i.i.d.—rare in operational systems.

• Determinism: set a global random seed and pass it into every split and learner component that uses randomness.
• Schema checks: verify required columns exist, dtypes are expected, and label values are valid before splitting.
• Holdout discipline: keep a test set out of the tuning loop; use validation for iteration, test for final reporting.

Common mistakes include leaking future information via time columns (splitting randomly while features implicitly encode the future), and “helpful” preprocessing done before splitting (like global normalization) that leaks validation statistics into training. A good rule: the raw data can be cleaned for obvious corruption (e.g., impossible values), but any transformation that learns parameters must be fit on the training split only.

Outcome: you can re-run training and obtain the same split sizes, the same row identities in each split, and an auditable record of how the split was produced. This realism pays off later when monitoring drift: your offline evaluation will actually resemble what production sees.

In production, the model is never just coefficients. It is “preprocessing + model,” and your biggest engineering risk is training-serving skew: training data gets transformed one way, but inference requests get transformed another. The remedy is to encode preprocessing as a formal pipeline with explicit fit and transform boundaries.

If you use scikit-learn, prefer a Pipeline (and ColumnTransformer for mixed types) so that encoders, imputers, scalers, and the estimator become a single object. The training pipeline fits transformers on training only, then uses the fitted transformers to transform validation/test. At inference time, you call pipeline.predict() and get consistent behavior without re-implementing preprocessing in the API layer.

• Boundary rule: anything that computes statistics (mean imputation, scaling, target encoding, vocabulary building) must be inside the pipeline and fit only on training.
• Leakage checks: scan features for post-outcome signals (e.g., “closed_date” predicting “churn”), and for IDs that behave like labels (high-cardinality identifiers can memorize).
• Stability checks: ensure transform output shape and feature order are stable across runs; lock down column lists and handle missing columns explicitly.

Engineering judgment: avoid “smart” preprocessing that is fragile in APIs. For example, heavy NLP pipelines or external lookups can introduce latency and failure modes; if you need them, treat them as first-class dependencies with caching and timeouts. Also prefer transformations that can tolerate missing values, because production payloads often have partial data.

Practical outcome: you have a single pipeline object that can be serialized, loaded by the inference service, and applied identically across training and serving. This reduces the surface area for bugs and makes rollbacks reliable because old artifacts carry their own preprocessing logic.

Training is not “run an algorithm.” Training is “optimize a model while producing evidence.” That evidence is your metrics, your baseline comparison, and your saved evaluation artifacts (predictions, confusion matrix, error analysis samples). Begin with a simple, transparent baseline and keep it in the repo. A baseline might be a majority-class classifier, a logistic regression with minimal features, or a naive “last value” forecaster—whatever matches your problem.

Define metrics that reflect deployment success. Accuracy alone is often misleading; you might need precision/recall at a specific threshold, ROC-AUC for ranking, or business-weighted costs. Pick one primary metric that will drive quality gates, plus secondary metrics to explain trade-offs.

• Reproducible training: set seeds; log hyperparameters; log data version identifiers (file hash, query timestamp, or dataset tag).
• Comparable evaluation: evaluate baseline and candidate on the same validation split with the same preprocessing rules.
• Artifacted outputs: save metrics to a machine-readable file (e.g., metrics.json) and optionally save plots for humans.

Common mistakes include tuning on the test set, reporting “best of many runs” without recording search space, and neglecting calibration. In deployed APIs, score calibration affects downstream decisions; if you serve probabilities, consider calibration checks or at least monitor probability distributions later.

Practical outcome: you can answer “Is this new model better than what we already have?” with a reproducible report. This is critical for CI/CD: the pipeline needs deterministic metrics to decide whether to accept or reject a candidate model.

Quality gates convert evaluation into an enforceable standard. Instead of relying on human judgment each time, you define thresholds and regression rules that can fail the build in CI. This is the MLOps equivalent of unit tests: a change that breaks quality should not silently ship.

Implement quality gates as code that reads metrics.json and compares it to a baseline reference. The baseline could be the previous release candidate’s metrics, a minimum absolute threshold, or both. Typical patterns include “AUC must be ≥ 0.82” and “AUC must not drop more than 0.01 versus the last approved artifact.”

• Absolute floors: prevent shipping obviously bad models when data shifts or bugs occur.
• Relative regression checks: protect against small degradations that accumulate over time.
• Data sanity gates: fail if label prevalence, missingness, or feature ranges are wildly different from expectations.

Engineering judgment: avoid gates that are too strict early in a project (they create constant failures and get bypassed), but also avoid gates that are so loose they are meaningless. Start with one primary metric gate plus two or three data sanity checks. When a gate fails, the fix should be actionable: either the model genuinely regressed, the data changed and you need a new baseline, or the pipeline has a bug.

Practical outcome: a pull request that changes features, preprocessing, or hyperparameters cannot be merged (or cannot produce a release candidate) unless it meets defined quality. This builds organizational trust: rollbacks become rare because regressions are caught before deployment.

Once you have a candidate pipeline, you must serialize it into an artifact that can be loaded by your inference service. The key requirement is that the artifact is self-contained enough to run reliably, and that its runtime dependencies are known. In Python ecosystems, common serialization choices include joblib/pickle for scikit-learn pipelines and framework-specific formats (e.g., ONNX, TorchScript) for cross-runtime portability.

For a project-based MLOps course, a practical default is to serialize the entire preprocessing+model pipeline using joblib.dump(). This is fast and preserves the pipeline object graph. The trade-off is that pickle-based formats are Python-version and library-version sensitive, so you must capture dependencies.

• Model file: store as model.joblib (or equivalent) inside a versioned artifact directory.
• Dependencies: pin library versions in requirements.txt or poetry.lock; record them again in the artifact manifest.
• Security note: treat pickle/joblib artifacts as untrusted input; only load artifacts you built in your pipeline.

Common mistakes include serializing only the estimator (forgetting preprocessing), relying on “latest” dependencies (breaking loads weeks later), and embedding environment-specific paths in the artifact. Keep file paths out of the model object; pass runtime configuration (like thresholds) separately via settings, not hard-coded training variables.

Practical outcome: the FastAPI service in later chapters can load a single file and immediately run inference with consistent preprocessing. You also reduce rollback risk: if you roll back to an old artifact, you roll back preprocessing behavior too.

A model artifact without metadata is a liability. You need to know what data, code, and settings produced it, and you need a human-readable narrative of what changed. Create a structured artifact directory that includes (1) the serialized model, (2) a manifest, (3) metrics, and (4) optional diagnostics (plots, sample errors).

The manifest is a small JSON/YAML file that makes the artifact traceable. Include: model name, semantic version (or a build number), training timestamp, git commit SHA, dataset identifier (hash or tag), feature list, preprocessing steps summary, dependency versions, and metric highlights. This becomes the handshake between training and deployment: the API can log the loaded model version, and monitoring can correlate incidents to a specific artifact.

• Version bumping: increment versions intentionally (e.g., 1.2.0 for feature changes, 1.2.1 for bug fixes) and keep the artifact path aligned (e.g., artifacts/model/1.2.0/).
• Release candidate registration: tag the artifact as rc (release candidate) before promoting it; store it in a predictable location or registry bucket.
• Changelog: write a short entry: what changed, why, expected impact, and any migration notes for inference.

Common mistakes include overwriting artifacts (“model.joblib” with no version), shipping a new model without updating documentation, and failing to record the git SHA—making debugging nearly impossible. Treat the artifact as a release: immutable, versioned, and auditable.

Practical outcome: you can point to a specific release candidate, know exactly how it was built, and promote or roll back with confidence. In the next chapter, that same version identifier will appear in your API endpoints and logs, enabling clean deployments and observability.

Section 3.1: FastAPI project structure for ML services

A practical ML API repository separates “serving code” from “training code” while still sharing types, feature logic, and model metadata. A common mistake is to dump everything into a single main.py, then struggle to test, version, or reuse components. Instead, aim for a small service package with explicit boundaries.

One workable structure looks like this: src/service/main.py for the FastAPI app factory, src/service/routes/v1.py for versioned endpoints, src/service/schemas.py for Pydantic models, and src/service/inference.py for model loading and prediction. Store the trained artifact under a predictable path (for example, artifacts/model.joblib) or fetch it at startup (covered later), but keep the “how to load” logic centralized.

• App factory pattern: expose create_app() to configure middleware, logging, and routes consistently in tests and production.
• Versioned routing: mount a router at /v1 so future breaking changes go to /v2 instead of silently changing behavior.
• Thin endpoints: endpoints should validate input, call inference code, and translate exceptions into HTTP errors—no business logic scattered across route handlers.

Engineering judgment: choose one inference endpoint that is stable (/v1/predict) and optionally a lightweight health endpoint (/health) used by orchestration systems. Keep health checks fast and independent of user input. If you add a readiness endpoint, decide whether it should fail when the model cannot load—this affects rollout behavior and how quickly failures surface.

Practical outcome: with this structure, you can evolve the model internals without rewriting API plumbing, and you can run contract tests against the exact same app instance that will run in production.

Section 3.2: Pydantic schemas, validation, and safe defaults

In ML services, most incidents start with bad inputs: missing fields, wrong types, out-of-range values, or “almost correct” payloads that slip through and produce garbage predictions. Pydantic schemas are your first line of defense. Define request and response models explicitly, and prefer strictness over permissiveness. If a client sends invalid data, fail fast with a clear 422 validation error rather than passing unexpected values into your feature pipeline.

Start by designing a PredictRequest with named fields (not anonymous lists), and add constraints where possible. For numeric fields, use bounds (e.g., non-negative). For categorical fields, consider enums or pattern constraints. Add a top-level request_id (optional) so clients can correlate logs and retries, and a model_version in responses so users can attribute behavior to a specific release.

• Safe defaults: defaults should be conservative and documented. Avoid defaulting a missing feature to zero unless that is truly semantically correct.
• Explicit error mapping: catch known inference exceptions (e.g., feature encoding errors) and return 400 with a human-readable message; treat unexpected exceptions as 500 with minimal disclosure.
• Schema evolution: adding optional fields is usually backward-compatible; changing meaning or removing fields is breaking and should trigger a new API version.

Common mistake: accepting “flexible” payloads (like Dict[str, Any]) and then doing ad-hoc parsing. This makes the service hard to debug and nearly impossible to contract-test. Another frequent pitfall is returning raw numpy types (e.g., np.float32) that fail JSON serialization. Ensure outputs are native Python types (float, int, str) and consider rounding rules for probabilities to reduce noise.

Practical outcome: your API becomes a stable contract. Clients know what to send, you know what you’ll receive, and invalid requests become predictable events rather than production mysteries.

Section 3.3: Model lifecycle: load, cache, and thread/process considerations

Inference performance is often dominated by model loading and feature preparation, not prediction itself. A classic production failure mode is loading the model on every request. This creates high latency, memory churn, and unpredictable throughput. Instead, load once at startup and reuse the loaded object across requests.

FastAPI supports startup events (or lifespan context) for warm starts. Use this to load the artifact, initialize any tokenizers/vectorizers, and run a small “smoke prediction” to confirm the pipeline is operational. Store the loaded model in an application state object (for example, app.state.model) or behind a module-level cache. Your goal is constant-time access per request.

• Thread/process model: if you run Uvicorn with multiple workers, each worker is a separate process and will load its own model copy. Plan memory accordingly.
• Thread safety: some model objects are not thread-safe if they mutate internal state during prediction. If in doubt, protect prediction with a lock or use process-based workers.
• Predictable latency: warm start avoids first-request spikes; consider exposing a readiness endpoint that only returns OK after the model is loaded.

Engineering judgment: decide where to do heavy work. If feature engineering is expensive, you may cache encoders or precomputed resources, but be careful caching per-request data (it can explode memory and leak PII). Also, avoid importing heavyweight libraries at module import time if you want faster container startup; load them in a controlled way during startup when you can log timing and failures.

Practical outcome: the service exhibits stable latency distributions and avoids “cold start roulette,” a key requirement before you layer on monitoring and autoscaling in later chapters.

Section 3.4: Testing strategy: unit, contract, and golden responses

ML APIs fail in two ways: the service breaks (HTTP errors, serialization issues) or the predictions silently change (model drift, training changes, preprocessing tweaks). Your testing strategy must cover both. The goal is not to prove the model is “good” in unit tests, but to make changes visible and safe to deploy.

Start with unit tests for pure functions: feature transforms, input normalization, and post-processing (e.g., thresholding). These should be fast, deterministic, and run on every commit. Next, add contract tests that spin up the FastAPI app (in-process using the TestClient) and assert: required fields are enforced, error responses have the expected shape, and the response schema contains prediction, probability (if applicable), and model_version.

• Golden response tests: choose a small fixed set of representative inputs and snapshot the expected outputs (or output ranges). This catches accidental changes in preprocessing or model selection.
• Stability vs. flexibility: for probabilistic models, assert within tolerances rather than exact floats, unless the model is fully deterministic and pinned.
• Negative tests: assert that bad inputs produce 422/400, and unexpected failures produce 500 with a traceable error ID in logs (not a full stack trace in the response).

Common mistake: writing only “happy path” tests. Another is allowing golden tests to become brittle by tying them to non-essential formatting (too many decimal places, ordering of JSON keys). Focus on what matters: schema compatibility and material prediction changes. When a golden test fails, treat it like a release decision point—either accept the change intentionally (update the golden files with review) or investigate a regression.

Practical outcome: you get CI-friendly checks that prevent breaking the API contract and provide early warning when inference behavior changes unexpectedly.

Section 3.5: OpenAPI, examples, and client usability

FastAPI’s built-in OpenAPI support is more than documentation; it is a communication tool that reduces integration friction. Treat your OpenAPI schema as the public face of your service. A well-documented inference API prevents support tickets and makes contract tests easier to reason about because the contract is explicit.

Provide clear endpoint descriptions, tag routes by version (v1), and include examples in your Pydantic models. Add field-level descriptions explaining units, expected ranges, and whether missing values are allowed. If your model expects a specific categorical encoding (e.g., country codes), document it. If you apply thresholding (e.g., classify as positive if probability ≥ 0.7), document that too, because it affects how clients interpret outputs.

• Request/response examples: include at least one realistic payload and a realistic response so users can copy-paste.
• Error response docs: document 422 validation errors, 400 domain errors (e.g., unsupported category), and 500 internal errors.
• Usage guide: add a short README snippet with curl and a minimal Python client call; show how to set headers and parse the response.

Engineering judgment: decide what to expose. Avoid leaking training features that are internal or sensitive. Prefer stable, user-meaningful inputs even if the internal model uses engineered features. If your internal preprocessing changes, you should not need to change the external schema unless the product requirements change.

Practical outcome: client teams can integrate without meetings. The service becomes self-serve, and your API versioning strategy becomes visible and enforceable through published docs.

Section 3.6: Config management and secrets hygiene

Production services run in multiple environments (local, CI, staging, prod). Hard-coding paths, ports, and secret values is a common reason deployments fail or accidentally leak credentials. Your FastAPI service should read configuration from environment variables, validate them, and provide sane defaults for local development.

Define a settings object (often using Pydantic settings) that loads values like: APP_ENV, LOG_LEVEL, MODEL_URI (local path or remote location), MODEL_VERSION, and optional credentials for artifact stores. Validate required settings at startup so misconfiguration fails early. Keep secrets out of logs, out of exception messages, and out of the OpenAPI schema.

• Do: inject configuration via env vars; use separate values per environment; rotate secrets without code changes.
• Don’t: commit API keys to the repo; bake secrets into Docker images; print env vars on startup “for debugging.”
• Predictable runtime: decide a single source of truth for the model artifact (image-bundled vs. pulled at startup) and document that operational choice.

Common mistake: mixing configuration styles (some env vars, some YAML files, some constants). Choose one approach and stick to it. Another pitfall is using “helpful” defaults for secrets (e.g., default password) that accidentally make it to production. Defaults should be safe for local development, and anything security-sensitive should be required and validated.

Practical outcome: you can ship the same container artifact across environments and change behavior only through configuration. This sets you up for CI checks and CD-style release steps later, because deployments become predictable and auditable.

Section 4.1: Dockerfile patterns for Python inference services

A strong Dockerfile for an inference API balances three concerns: security, predictability, and developer ergonomics. For Python/FastAPI services, the most common baseline is a slim Debian-based Python image (for compatibility) or distroless (for tighter security, but more friction). In early projects, choose predictability first: python:3.11-slim is usually a good compromise.

Use a multi-stage build. The first stage (“builder”) installs build tools and resolves Python dependencies. The final stage (“runtime”) copies only what you need to run: site-packages, application code, and model artifacts. This reduces attack surface and image size while keeping builds reliable. Also, run as a non-root user. It’s an easy win: even if your app has a vulnerability, the container will have fewer privileges.

Here’s a practical pattern (trimmed for readability):

FROM python:3.11-slim AS builder WORKDIR /app ENV PIP_DISABLE_PIP_VERSION_CHECK=1 PIP_NO_CACHE_DIR=1 COPY pyproject.toml poetry.lock ./ RUN pip install --upgrade pip && pip install poetry && poetry export -f requirements.txt -o requirements.txt --without-hashes RUN pip install --prefix=/install -r requirements.txt FROM python:3.11-slim AS runtime WORKDIR /app RUN useradd -m appuser COPY --from=builder /install /usr/local COPY app/ ./app/ COPY models/ ./models/ ENV PYTHONUNBUFFERED=1 USER appuser EXPOSE 8000 CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

Common mistakes: baking secrets into the image (never copy .env), running as root by default, and copying your entire repository (including tests and local caches) into production images. Be intentional about what goes in the runtime image: API code, model file(s), and only runtime dependencies.

Section 4.2: Image size, caching, and build reproducibility

Smaller images pull faster, scan faster, and have less “stuff you didn’t mean to ship.” But obsessing over size can backfire if it makes builds fragile. The practical target is: stable builds first, then reduce size with low-risk steps (multi-stage builds, removing build tools, avoiding extra OS packages).

Docker caching is your friend when used deliberately. Order your Dockerfile so the slowest, least-changing steps happen earliest and are cached most often. Dependency installation should be cacheable: copy dependency manifests (requirements.txt, or pyproject.toml/poetry.lock) before copying your changing application code. Then a code change doesn’t invalidate dependency layers.

For reproducibility, pin everything that matters: Python base image tag (prefer a specific version), Python dependencies (lock file), and—if you rely on OS packages—pin apt package versions where feasible. Also consider using BuildKit and recording image metadata (labels) such as git SHA and build timestamp. This helps you answer, “Which code is running in production?” without guessing.

• Use a .dockerignore: exclude .git, local venvs, notebooks, and data dumps to avoid bloating the build context.
• Prefer deterministic installs: lock files and “no implicit upgrades” reduce surprise regressions.
• Separate model artifacts from training outputs: copy only the model you intend to serve (e.g., models/model.joblib), not an entire experiments directory.

A subtle but important point: reproducible builds aren’t only about dependencies; they’re also about the input artifacts. If your image build downloads a model from a moving URL, you’ve lost traceability. Either copy a versioned model file into the image, or fetch it by immutable identifier (e.g., a content hash) during startup with strict verification.

Section 4.3: Runtime configuration, ports, and health endpoints

Containers should be configurable at runtime, not rebuilt for each environment. That means environment variables (or mounted config files) for things like: model version selection, log level, external service URLs, and feature flags. A clean rule is: the image contains code and default settings; the environment provides values that differ per deployment (dev/stage/prod).

Expose one port for the API (commonly 8000) and keep it consistent across environments. Consistency matters because it reduces operational “glue code.” Inside the container, bind to 0.0.0.0 so the port is reachable from outside the container. If you run multiple workers, be explicit (e.g., Gunicorn with Uvicorn workers) and test memory usage, since model loading can multiply per worker.

Health checks are not optional in real deployments. Implement two endpoints:

• /health (liveness): returns 200 if the process is running and event loop is responsive.
• /ready (readiness): returns 200 only if the model is loaded and dependencies are reachable (e.g., feature store or database, if used).

Then wire these into your container orchestration. Even in Docker-only environments you can add a HEALTHCHECK instruction so Docker can mark the container unhealthy and restart it. For smoke tests, call /ready and a small inference request against a fixed payload to catch serialization or model-loading regressions before you route real traffic.

Common mistakes: returning 200 from /ready before the model is loaded, making readiness depend on non-critical services (causing unnecessary outages), and forgetting to set timeouts (a stuck external call can hang readiness and break deployments). Design these endpoints with engineering judgment: be strict enough to prevent serving bad responses, but not so strict that transient external issues block your rollout.

Section 4.4: Docker Compose for local integration testing

Docker Compose gives you dev/prod parity without heavy infrastructure. The idea is to run your API the same way it will run in production—inside a container—while also spinning up the services it depends on. Even if your current project only has the model API, Compose becomes valuable as soon as you add a metrics stack (Prometheus), a dashboard (Grafana), or a reverse proxy.

A practical Compose file for this chapter includes at least: the inference service, and optionally a monitoring component later. You also want predictable configuration: environment variables in one place, ports mapped for local access, and a mounted volume only for development (so code reload works without rebuilding images).

Example outline:

services: api: build: . ports: - "8000:8000" environment: - LOG_LEVEL=INFO - MODEL_PATH=/app/models/model.joblib healthcheck: test: ["CMD", "curl", "-f", "http://localhost:8000/health"] interval: 10s timeout: 2s retries: 5

For integration testing, Compose enables a repeatable workflow: docker compose up --build, wait for “healthy,” run smoke tests, then tear down. This catches real-world issues that unit tests miss: missing system libraries, wrong working directory, file permission problems under a non-root user, and mismatched environment variables.

Common mistakes: using Compose as a “forever dev environment” without documenting commands, mounting the entire repository into the container in a way that masks what the image actually contains, and letting local volumes hide missing files that will be absent in production. Keep two modes: a dev override for hot reload, and a production-like mode that runs purely from the built image.

Section 4.5: CI pipeline: lint, tests, build, and artifact publishing

Manual builds are where “works locally” goes to hide. A CI pipeline makes the build and test process consistent and auditable. For a model API, the minimum CI stages are: lint/format checks, unit tests, container build, and artifact publishing (pushing the image to a registry). If you only do one thing, do this: require CI to pass before you can merge to your main branch.

Order matters. Run fast feedback steps first: lint and unit tests should fail quickly. Then build the Docker image, then run container-level smoke tests against the built image. That last step is critical: it validates the artifact you will deploy, not just the source code.

• Lint: enforce code quality (e.g., Ruff) and catch unused imports, security foot-guns, and inconsistent formatting.
• Tests: unit tests for preprocessing and request/response schemas; minimal integration tests for startup and one prediction.
• Build: docker build with BuildKit; label the image with git SHA.
• Publish: push to a registry (GHCR, ECR, GCR) only from trusted branches/tags.

Artifact publishing is where teams often get sloppy. Treat images as immutable: once pushed with a tag, do not rebuild and overwrite it. Instead, publish new tags. Also, don’t publish from pull requests originating from forks if secrets are required; structure CI so untrusted builds run tests but cannot push.

Finally, capture outputs: store test reports, and record the image digest produced by CI. The digest is the real identity of the artifact; tags are human-friendly pointers. In later monitoring and rollback work, knowing the digest you deployed will make debugging far faster.

Section 4.6: Release tagging and environment promotion strategy

Deployments become safer when you separate “build” from “promote.” Build once, then promote the same artifact through environments (dev → staging → prod). This avoids a classic failure mode: staging passed, but production runs a different image because it was rebuilt later with slightly different dependencies or base layers.

Start with a tagging strategy that supports traceability and rollbacks. A practical scheme uses multiple tags for the same image digest:

• Immutable tag: sha-<git_sha> (always unique, never overwritten)
• Release tag: semantic version like v1.3.0 for human-friendly releases
• Environment tags (optional): staging, prod pointing to the currently deployed digest (these are mutable pointers)

Environment promotion then becomes a controlled step: “move the staging tag to this digest” after smoke tests and basic checks pass; later “move the prod tag to that same digest.” If production misbehaves, rollback is simply repointing prod to the previous digest (or redeploying the previous immutable tag). This is the foundation for reliable rollbacks because it’s fast and doesn’t require rebuilding.

Where do smoke tests fit? Right before promotion. A typical flow is: CI builds and pushes sha-... on merge; CD (or a manual release workflow) deploys that digest to staging; automated smoke tests hit /ready and run a fixed inference; if successful, you promote the same digest to production. Engineering judgment shows up here: keep the gate small but meaningful. Overly complex gates slow down delivery without preventing the most common failures (bad startup, missing model file, schema mismatch).

A final common mistake is mixing model versioning with code versioning without a plan. Your image tag should identify the service build. Your API endpoints should be versioned (e.g., /v1/predict), and your model artifact should have its own version metadata. When you can answer “Which service build and which model produced this prediction?” you’re ready for the monitoring and rollback mechanics that come next.

Monitoring a normal API focuses on availability and performance: does the endpoint respond quickly and correctly? An ML API must answer those questions plus “is the model still valid?” Even if your FastAPI service returns HTTP 200, the underlying model can silently degrade due to changing input patterns (seasonality, new user segments), upstream pipeline bugs, or shifts in user behavior triggered by your own product changes.

Start by separating concerns into layers you can diagnose. Layer 1 is the platform: container restarts, CPU/memory usage, and dependency failures. Layer 2 is the web service: request volume, latency percentiles, and HTTP error codes. Layer 3 is inference: model load failures, preprocessing exceptions, time spent in feature construction, and prediction time. Layer 4 is model behavior: feature distributions and prediction distributions. Layer 5 is business outcomes: conversion, fraud capture rate, customer satisfaction—whatever your model influences.

Common mistake: teams only monitor /health checks. Health endpoints are necessary, but they don’t catch partial outages (e.g., 10% 500s), slow degradation (p95 latency creeping up), or correctness issues (model outputs out of expected range). Another mistake is to monitor “accuracy” in real time without labels; unless you have immediate ground truth, you’ll need proxies (explained later) and delayed evaluation jobs.

Practically, for an ML API you should monitor: (1) request/latency/error metrics for the service, (2) structured logs that capture enough context to debug failures, and (3) drift signals on inputs and predictions. Those three together let you answer: “Is this a system problem, a data problem, or a model problem?”

Use a consistent metrics vocabulary so alerts and dashboards remain interpretable. For request-driven services, RED is the classic taxonomy: Rate (throughput), Errors (failure ratio), Duration (latency). For resource-centric views (nodes, containers), USE is helpful: Utilization, Saturation, Errors. In practice you’ll combine both: RED on the FastAPI endpoint, USE on the container and host.

Implementing RED for your model endpoint typically means emitting counters and histograms. Counters: total requests, total errors by status code, total timeouts, total model-load failures. Histograms: request latency and model inference latency, with labels such as endpoint, model_version, and http_status. Track percentiles (p50/p95/p99) rather than averages; averages hide tail latency that users feel.

• Rate: requests/second by endpoint and model version
• Errors: 4xx vs 5xx; validation failures vs internal exceptions
• Duration: request latency p95; inference time p95; preprocessing time p95
• USE: CPU throttling, memory RSS, container restarts, thread pool saturation

Now add ML-specific metrics. You usually can’t log all features and compute full monitoring in-line, but you can emit lightweight “add-ons”: (1) input schema failures (missing fields, type mismatches), (2) out-of-range feature counts (e.g., negative ages), (3) prediction summary (mean, min/max, or bucket counts), and (4) abstentions or “low-confidence” flags if your model supports them. These are cheap signals that catch data pipeline bugs and sudden distribution changes.

Engineering judgment: keep metric labels bounded. A common mistake is putting unbounded values (user_id, raw query strings) into metric labels, which explodes cardinality and can take down your monitoring stack. Use coarse labels like model_version and endpoint, not per-request identifiers.

Metrics tell you that something is wrong; logs help you learn why. In production ML, logs are most valuable when they are structured (JSON), consistent, and traceable across systems. Instead of free-form strings, log events with stable keys like timestamp, level, message, endpoint, model_version, latency_ms, and error_type. This makes it possible to filter, aggregate, and build dashboards from logs.

Add correlation IDs to connect a user request across layers. A simple pattern: accept X-Request-ID if the client provides it; otherwise generate a UUID at the API boundary. Include that ID in every log line for that request, and return it in the response headers so support teams can ask users for it. If you later add distributed tracing (e.g., OpenTelemetry), this same mindset extends naturally to trace/span IDs.

Log at the right level of detail. For successful requests, log summaries: latency, model version, and perhaps coarse-grained feature checks (e.g., “schema_valid=true”). For failures, log stack traces and validation errors, but still keep structure. A common mistake is logging entire request payloads “for debugging,” which creates privacy risk and retention obligations.

Privacy and security are part of correctness. Apply a redaction policy: never log direct identifiers (emails, phone numbers), secrets, or raw free-text fields. If you need to debug data issues, log hashed or bucketed representations (e.g., age bucket, country code) and only log sampled payloads in secure, access-controlled environments. Also set retention periods: keep high-volume request logs short (days), keep aggregated metrics longer (weeks/months), and keep incident-specific artifacts under explicit access controls.

Practical outcome: with structured logs + correlation IDs, your incident response becomes faster. When an alert fires, you can pivot from “p95 latency spiked” to “all slow requests involve model_version=2 and feature_store timeout,” without guessing.

Drift is the core ML difference: the world changes. Data drift means your input feature distribution changes (e.g., more mobile users, different geographies). Concept drift means the relationship between inputs and the target changes (e.g., fraudsters adapt; customer preferences shift). Data drift is often detectable immediately from inputs; concept drift usually requires labels or strong proxies.

Start with two monitoring signals that are practical in most projects: input drift and prediction shift. Input drift compares current feature distributions to a baseline (often training or a recent “golden” window). Prediction shift compares the distribution of model outputs over time. Both are weakly diagnostic: they do not prove performance is worse, but they are early warnings that the model is operating in a new regime.

Choose a small set of features to monitor for drift: high-importance features, features known to break (categoricals with new values), and features tied to business cycles. Compute simple statistics in a batch job: missing rate, mean/std for numeric features, top-k category frequencies for categoricals, and a distance measure (PSI, Jensen–Shannon divergence, or KS test). The best choice is the one your team can explain and maintain; PSI is popular for its interpretability, but any consistent measure with calibrated thresholds can work.

When labels are delayed, use proxies. For example: for a churn model, monitor downstream cancellation requests as a delayed label; for a ranking model, monitor click-through rate segmented by predicted score buckets; for a fraud model, monitor chargeback rate with delay. Also monitor policy changes: if you change business rules that affect who gets scored, the input distribution will shift even if the world did not—your monitors should annotate such events to avoid false alarms.

Common mistakes: comparing to the wrong baseline (e.g., training data from a year ago), ignoring seasonality (weekend vs weekday), and failing to segment (drift may only occur in one region). Make drift monitors actionable by attaching “what changed” summaries: top drifting features, new category values, and which segments are affected.

Alerts are not monitoring; alerts are interruptions. Treat them as a product you design for the on-call engineer (even if that’s you). A good alert is actionable, time-bounded, and points to a clear next step. A bad alert fires constantly, teaches you to ignore it, and hides real incidents.

For service health, alert on RED signals with clear thresholds: sustained 5xx error rate above X%, p95 latency above Y ms, or request rate dropping unexpectedly (could indicate upstream outage). Prefer windowed conditions (e.g., “for 10 minutes”) to avoid flapping. For saturation, alert on repeated container restarts, memory nearing limits, or CPU throttling that correlates with latency spikes.

For error budgets and burn rates: if you define an SLO like “99.9% of requests succeed monthly,” you can alert on how fast you are consuming that budget. Burn-rate alerting catches both fast meltdowns (high error rate right now) and slow leaks (slightly elevated errors all day). Even without a formal SLO program, you can approximate this with two windows: a short window (5–15 minutes) and a long window (1–6 hours) with different thresholds.

For drift, avoid paging on weak signals by default. Drift alerts are often better as tickets or Slack notifications with severity levels. A practical pattern is: (1) “drift warning” when PSI/JS exceeds a small threshold for a day, (2) “drift critical” when multiple key features drift or prediction shift is large, and (3) page only when drift coincides with business KPI regression or elevated model errors.

• Noise control: use deduplication, grouping by model_version, and maintenance windows during deployments
• Annotations: mark deploys, feature pipeline changes, and backfills on charts
• Playbook: each alert links to steps: check dashboards, recent releases, logs by correlation ID, then rollback criteria

Your incident playbook should include rollback triggers. Example: “If 5xx > 2% for 10 minutes after deploy, roll back to previous model image; if p95 latency > 2x baseline and CPU is saturated, scale replicas or revert model that increased compute.” Make rollback a routine, not a failure—fast reversibility is a sign of mature MLOps.

A dashboard is successful when it supports decisions, not when it displays every metric you can collect. Build two views: an operator dashboard for engineers and an executive-ready summary for stakeholders. The operator view is for diagnosis; the executive view is for confidence and communication.

The operator dashboard should “tell a story” from top to bottom. Start with user impact: availability and error rate. Then performance: p95/p99 latency, broken down by endpoint and model version. Then capacity: CPU/memory, container restarts, queue/backpressure signals. Then inference internals: preprocessing time vs model time. Finally, ML signals: input drift and prediction shift panels, ideally with top features and a baseline window selector.

Include deployment annotations so you can visually correlate changes with releases. If latency increased exactly at a model version change, you can investigate feature engineering or model size. If drift increased after an upstream pipeline change, you know where to look. Segment charts by key dimensions like region, tenant, or product surface; many incidents only affect one segment.

The executive-ready summary should answer: Are we within SLO? Is the model stable? Are there known risks? Keep it simple: a small set of KPIs with trend arrows and brief notes. Example panels: monthly success rate vs target, p95 latency vs target, drift status (green/yellow/red), and “model performance (latest labeled batch)” when available. Add a short narrative field: “This week: deployed v3, latency improved 12%, drift warning in EU segment; mitigation in progress.”

Common mistake: building dashboards that require interpretation by the creator. Use clear titles, units, and thresholds. If a chart cannot answer a question, remove it. The practical outcome is confidence: when an alert fires, you know where to look; when a stakeholder asks “can we ship the new model?”, you can answer with evidence.

Safe releases start with choosing the right rollout strategy for your risk profile and infrastructure. Three common patterns are blue-green, canary, and shadow. Your course project can demonstrate at least one (canary) and describe the others clearly in documentation.

Blue-green means you run two full environments: “blue” (current) and “green” (new). You switch traffic all at once after verification. It’s simple to reason about and rollback is fast (switch back), but it costs more because you duplicate capacity. It also makes “gradual confidence building” harder: you learn only after the switch.

Canary means you route a small percentage of traffic to the new version (e.g., 1–10%), monitor key metrics, then gradually increase. Canary is ideal for model services because it gives early warning on latency, error rate, and prediction distribution shifts. For this project, define explicit decision criteria such as: p95 latency must not increase by more than 20%, 5xx error rate must remain below 0.5%, and prediction score distribution (e.g., mean/KS statistic) must remain within a tolerance.

Shadow means the new version receives a copy of real requests but does not affect user responses. It’s excellent for comparing predictions and latency without risk, but it requires request duplication and careful handling of PII/logging. Shadow is often the best first step for high-stakes models when you need offline confidence before exposure.

• Common mistake: doing canary without a “stop condition.” If you can’t say what would cause a rollback, you’re not running a canary—you’re hoping.
• Practical outcome: document a rollout plan in your repo (e.g., docs/release_playbook.md) including step sizes (1% → 10% → 50% → 100%), time windows, and thresholds.

In interviews, being able to explain why you chose canary for a model API—and what you monitored—signals real operational maturity.

A rollback is not a feeling; it’s a mechanism. For a database-less inference service (stateless FastAPI + model artifact), you should be able to roll back by redeploying a prior container image and configuration. This is why image tagging and config discipline matter.

Images: tag Docker images with immutable identifiers (e.g., Git SHA) and optionally a semantic version for humans (e.g., 1.2.0). Your CD step should deploy by digest/SHA, not by “latest.” Rollback then becomes: redeploy the last known good SHA.

Configs: keep runtime configuration (thresholds, feature flags, model version selection) separate from the image. Use environment variables or a config file mounted at runtime. Store default configs in the repo and document required variables. For canary, a simple feature flag can route traffic or choose a model artifact. Ensure configs are versioned so that you can revert both code and behavior.

Model artifacts: treat model files as versioned artifacts (e.g., in an artifacts bucket or a registry-like folder), with checksums. If you bake the model into the image, rollback is image-only. If you load models from storage, rollback might also require switching the artifact pointer. Either approach is valid—just make it deterministic.

• Rollback drill: intentionally deploy a “bad” version (e.g., force a higher latency by adding a sleep, or return an incorrect schema), let monitors detect it, then execute rollback and confirm recovery.
• Recovery metrics: measure time-to-detect (TTD) and time-to-recover (TTR). Track these in your case study; they are strong evidence of ops thinking.

Common mistake: assuming rollback is safe without rehearsing. Many teams discover during incident response that old images were deleted, configs drifted, or deployment scripts were not reproducible. Your project should prove the opposite.

After deploying a canary, you need verification layers that answer different questions. Think in three tiers: smoke tests, synthetic tests, and real-traffic monitoring. Together, they form your post-deploy “gate” before promotion.

Smoke tests are immediate, cheap checks: does the container start, does /health return 200, does /predict accept a known payload and return the expected schema. Automate these in CI and re-run them after deployment. Keep smoke tests strict about contract (fields, types) but not about exact prediction values unless deterministic.

Synthetic tests simulate usage patterns: send a small suite of representative inputs, measure latency, and validate invariants (no NaNs, probabilities sum to 1, outputs within bounds). Synthetic tests are where you can check model-specific logic: feature preprocessing, thresholding, or categorical handling. Run them continuously on a schedule so you can detect “it broke at 2am” even without new deploys (e.g., dependency or environment changes).

Real traffic monitoring is where canary decisions are made. Watch service SLO-style metrics (p95 latency, 5xx rate, timeouts) and ML signals (input feature distribution, prediction distribution, and—if you have labels—online performance). In the canary window, compare baseline vs new version: don’t just look at absolute numbers; look at deltas and confidence intervals when possible.

• Automated checks + decision criteria: define a promotion checklist that can be executed as a pipeline job: “If error rate delta < 0.2% and latency delta < 20% and drift score < threshold for 30 minutes, then promote to 100%; else hold or rollback.”
• Common mistake: using one metric (often accuracy) to decide. Many failures are operational (timeouts, memory) rather than predictive.

Even if your project runs locally or on a simple VM, you can still implement the logic: scripts that query metrics endpoints, compare to thresholds, and print a “PROMOTE/HOLD/ROLLBACK” recommendation.

Shipping safely also means knowing when to change the model and when to leave it alone. A mature model iteration loop defines triggers (signals that suggest retraining), gates (checks before deploying a new model), and governance (who approves, what’s recorded, and how you avoid silent regressions).

Retraining criteria should combine data and performance signals. Data drift alone is not always a reason to retrain; it might be seasonal or harmless. Practical triggers include: sustained drift on key features (e.g., PSI/KS beyond threshold for N days), a statistically significant drop in label-based metrics (e.g., AUC down by 2 points), or business KPIs moving in the wrong direction (e.g., precision at a fixed recall falls below target). For this course, write a simple “retraining trigger plan” document that names metrics, thresholds, and evaluation windows.

Governance can be lightweight but explicit: record the training dataset snapshot, code SHA, hyperparameters, evaluation report, and approval note. In a portfolio project, a structured reports/ folder plus a markdown “model card” is enough to demonstrate discipline.

Post-deploy evaluation closes the loop: define how you will collect labels (if available), how long you wait before trusting them, and how you compare model versions. If labels are not available, document proxy metrics and caution about their limits. Consider a two-stage approach: (1) canary checks for operational safety and distribution sanity, (2) longer-running evaluation for predictive quality before declaring the model “stable.”

• Common mistake: retraining on a schedule without evidence. That can create churn and repeated regressions.
• Practical outcome: add a pipeline stub (even manual) that says: “If trigger X fires, run training, generate report, run offline validation, then create a release candidate image.”

The goal is not to build an enterprise governance system; it’s to show you understand why models require lifecycle management beyond code deployment.

This project becomes valuable for career transitions when you translate tasks into the language hiring teams use: reliability, automation, observability, and controlled releases. Your narrative should connect the artifacts you built to common MLOps responsibilities.

Map your work explicitly:

• “Built CI/CD” → implemented CI checks (lint/tests/build), produced versioned images, and defined a repeatable release pipeline with promotion/rollback steps.
• “Improved reliability” → added canary rollout with thresholds, rollback drill, and measured time-to-detect/time-to-recover.
• “Observability” → instrumented structured logs and metrics (latency/error/request volume) and used them to drive release decisions.
• “Model monitoring” → tracked drift/performance signals and defined a retraining trigger plan with governance artifacts.

In interviews, tell a tight story: (1) baseline deployment, (2) introduced canary to reduce risk, (3) ran a controlled failure to validate rollback, (4) added post-deploy evaluation signals, (5) documented the operating model. Emphasize tradeoffs: e.g., “I chose canary over blue-green because I wanted incremental exposure and metric-based promotion; I kept the service stateless to make rollback a single command.”

Common mistake: presenting only the “happy path.” Hiring managers want to see that you can anticipate failure modes. Your rollback drill, thresholds, and runbook are concrete proof.

Practical outcome: write a 1–2 minute spoken summary and a 5–7 bullet resume entry set. Make each bullet start with an action verb and include a measurable result (even if it’s a lab metric like “recovered in <2 minutes”).

Portfolio packaging is where good engineering becomes legible. Your repo should let a reviewer answer: What does this do? How do I run it? How do I know it’s working? How do releases happen? The deliverables are a high-signal README, a simple architecture diagram, and a demo script you can execute reliably.

README structure: start with a one-paragraph overview and a diagram, then include (1) quickstart commands (Docker build/run, endpoints), (2) configuration variables, (3) observability (where to see logs/metrics), (4) release process (canary + promotion/rollback), and (5) limitations and next steps. Keep it runnable: copy/paste commands should work.

Architecture diagram: include boxes for client → FastAPI service → model artifact storage (if used) → metrics/logging sink. Annotate where canary routing happens (even if conceptual). A simple SVG or PNG checked into docs/ is sufficient, but ensure it matches reality.

Demo checklist: create a script that you can follow under pressure: build image, start baseline, send a request, show metrics/logs, deploy canary, run automated checks, decide promote/rollback, execute rollback, and show recovery metrics. Write it as a sequence of terminal commands plus expected outputs. This is also your “live interview” safety net.

• Common mistake: shipping a repo that only the author can run. If setup takes more than 10 minutes, most reviewers stop.
• Practical outcome: add make targets (e.g., make run, make test, make smoke, make canary, make rollback) to standardize your demo.

When these pieces are in place, your project reads like a production system: not because it’s huge, but because it’s controlled, observable, and recoverable.