NVIDIA GenAI Cert Workshop: Fine-Tune, Serve & Optimize LLMs

Start exam prep by treating the blueprint like a project plan. Most NVIDIA GenAI certification exams are organized around domains that correspond to the lifecycle: data and safety, model adaptation (fine-tuning), evaluation, deployment/serving, and performance optimization. Your scoring strategy should mirror how you would ship an LLM feature: secure the “must not fail” fundamentals first (data handling, reproducibility, GPU constraints), then stack on advanced techniques (LoRA/QLoRA, batching/KV cache, quantization, profiling).

Practical approach: build a one-page domain-to-artifact mapping. For example, map “data pipelines” to a versioned dataset directory, a schema definition (JSONL fields, labels, sources), and a validation script. Map “fine-tuning” to a training config file, an experiment tracking run, and a model card. Map “serving” to an inference container image, an endpoint contract (request/response schema), and load tests. Map “optimization” to profiler traces and before/after latency and cost numbers.

• Time allocation: prioritize domains with broad leverage. GPU memory math and serving constraints show up everywhere, not just in one section of the exam.
• Eliminate unforced errors: know key definitions (tokens, context window, KV cache, quantization types), and practice translating them into design choices.
• Show your work: in scenario questions, the best answers justify tradeoffs (quality vs latency vs cost) rather than claiming a single “best model.”

Common mistake: studying tools in isolation. The exam generally expects you to connect tools to outcomes (e.g., “use experiment tracking to ensure your fine-tuning run can be reproduced and audited,” not “I know what MLflow is”). Treat every concept as a lever that changes reliability, safety, cost, or measurable quality.

LLMs are transformer-based sequence models: they predict the next token given prior tokens. For engineering work, you don’t need to re-derive attention, but you must reason about how tokens and context length drive both quality and compute. A token is not a word; it is a subword unit produced by a tokenizer. This matters because costs and limits are measured in tokens, and “short prompts” in English can still be token-heavy (code and non-English text often tokenize into more tokens).

The context window is the maximum number of tokens the model can attend to at once (prompt + conversation history + retrieved documents + generated output). The hard limit is architectural (e.g., positional embedding/rope scaling choices) and the practical limit is resource-driven: long contexts inflate compute and VRAM via attention and KV cache. In production systems, context management is not optional—you must decide what to keep, summarize, or retrieve.

Architecture-related constraints show up in fine-tuning too. If you fine-tune a base model on instruction data, you are shaping how it follows directions across contexts, but you are not magically increasing its context window. If your task requires referencing long documents, you likely need RAG or long-context models rather than “more fine-tuning.”

• Tokens as a budget: define maximum input tokens and maximum output tokens for each endpoint. This becomes part of your API contract and protects latency.
• Stop conditions: configure stop sequences and max_new_tokens to prevent runaway generation.
• Safety boundaries: context window is also an attack surface—prompt injection and data exfiltration often ride along in long retrieved passages.

Common mistake: evaluating quality with a tiny prompt that fits comfortably in context, then deploying with real prompts that include long histories, tool outputs, and retrieved text. The model behaves differently when it is near its context limit (truncation, missing key instructions). Plan for realistic contexts from day one, including token counting in your data pipeline and evaluation harness.

GPU basics are a certification staple because they determine what is feasible. For LLM work, the two recurring bottlenecks are VRAM (memory capacity/bandwidth) and compute throughput (tensor cores, clock rates). You should be able to estimate whether a model will fit in memory for inference and for fine-tuning, and understand how batch size and sequence length change that fit.

Rule-of-thumb VRAM components for inference: (1) model weights, (2) KV cache, (3) activations/overheads. Weights dominate at small batch sizes; KV cache dominates at long sequences and high concurrency. A quick estimate: FP16 weights take ~2 bytes/parameter (so a 7B model is ~14 GB just for weights), while 8-bit quantization roughly halves that. KV cache grows with layers × hidden size × sequence length × batch size; that’s why doubling context length can break an otherwise stable deployment.

Batching improves throughput by amortizing overhead, but it increases peak memory. For serving, you often choose between static batching (predictable but can add latency) and dynamic batching (better utilization but needs careful queueing). For fine-tuning, micro-batches and gradient accumulation are your friends: keep per-step VRAM small while reaching an effective batch size that stabilizes training.

• Sequence length matters twice: it affects attention compute and KV cache memory. Don’t pick “max context” as a default; choose what your use case needs.
• Know your GPU: H100/A100 class GPUs behave differently than consumer cards; memory bandwidth and tensor core formats (FP16/BF16/FP8) influence optimization options.
• Profile early: if latency is high, determine whether you are compute-bound or memory-bandwidth-bound before changing parameters blindly.

Common mistake: planning only for single-request inference. Real endpoints handle concurrent users, streaming, retries, and worst-case prompts. Your compute plan should include concurrency targets and a token-per-second goal, then back into GPU count, quantization choice, and maximum request size.

A reproducible GenAI project is a set of artifacts connected by metadata: datasets, configs, training runs, evaluation reports, and deployable model binaries. Tooling is not “extra”; it is how you avoid repeating expensive mistakes and how you prove that a fine-tune actually helped. The exam expects you to recognize standard roles: experiment tracking (metrics and configs), model registries (versioned deployable outputs), and dataset/version control (repeatable inputs).

At minimum, structure your repo with clear boundaries: data/ (raw vs processed), src/ (pipelines, training, serving), configs/ (YAML/JSON for runs), reports/ (evaluation outputs), and models/ or a remote registry pointer. When you prepare instruction datasets, store (a) the source, (b) the transformation steps, (c) the final JSONL/Parquet, and (d) a manifest file with counts, token statistics, and filters applied (PII removal, deduplication, toxicity checks).

Experiment tracking should log more than loss. Log dataset version hashes, tokenizer/model revision, context length, LoRA ranks, learning rate schedule, and evaluation suite results. A model registry entry should include a model card: intended use, limitations, safety notes, and how it was evaluated. These details turn into fast answers on scenario-based questions and, more importantly, prevent “mystery regressions” when a teammate retrains with slightly different settings.

• Datasets are products: validate schema, run sanity checks (empty fields, label leakage), and compute token histograms to avoid outliers.
• Registries enable rollback: serving should reference a model version, not “latest.”
• Evaluation is a first-class artifact: store prompts, model outputs, and scoring results to support regression testing.

Common mistake: tracking only the best run. You also need to track failed runs and resource usage; they teach you where VRAM limits and instability thresholds are.

Certification questions often ask which lever to use. The correct choice depends on what you are changing: behavior, knowledge, or access to private data. Use prompting when you can express the task clearly and the model already has the capability. Use RAG when the model needs up-to-date or private knowledge and you can retrieve it reliably. Use fine-tuning when you need consistent behavior at scale (style, formatting, tool-call discipline), domain-specific instruction following, or reduced prompt length/cost because the behavior is “baked in.”

A practical decision framework is a three-column table: Quality, Cost/Latency, Risk. Prompting is lowest effort but can be brittle; long prompts increase cost and can exceed context limits. RAG adds retrieval complexity and introduces new failure modes (bad retrieval, prompt injection in documents), but it is usually the right answer for factual grounding. Fine-tuning requires curated data, tracking, and evaluation, but it can dramatically improve consistency and reduce prompt tokens.

Combine methods intentionally. A common production pattern is: system prompt sets policy, RAG supplies facts, and a small fine-tune enforces output format and tool usage. Parameter-efficient fine-tuning (LoRA/QLoRA) is especially useful when you want targeted behavioral change without the cost of full fine-tuning. However, do not fine-tune to “memorize” volatile facts; that is what retrieval is for.

• If failures are formatting/tooling: consider fine-tuning on structured I/O examples and add regression tests.
• If failures are factuality: add RAG and evaluate grounding (citation accuracy, answerability checks).
• If failures are rare edge cases: start with prompting + targeted few-shot examples before committing to training.

Common mistake: using fine-tuning as the first response to poor results. If the baseline model is mismatched (too small, wrong context length, wrong instruction tuning), no amount of LoRA will rescue it. Baseline selection is part of the decision: pick a model that meets constraints on quality, cost, and latency before you adapt it.

Reproducibility is how you turn experiments into an engineering system. In GenAI workflows, irreproducibility comes from many sources: non-deterministic kernels, changing dataset snapshots, unpinned dependencies, and silent config drift. For exam readiness—and for real deployments—adopt a discipline: every run is defined by a config file, a dataset version, a code commit, and an environment description.

Start with deterministic habits: set random seeds for Python/NumPy/PyTorch, record CUDA/cuDNN determinism flags when feasible, and log the exact model revision and tokenizer. Accept that perfect determinism is not always achievable on GPU, but aim for practical reproducibility: if you rerun, metrics should be within a narrow band and the qualitative behavior should not flip.

Package your work so it can be rerun by someone else. Use environment locks (e.g., pinned requirements, container images), and standardize entry points such as make train, make eval, and make serve. A runbook is the “operator manual” for your model: how to train, how to evaluate, how to deploy, what to monitor, and how to roll back. Include resource expectations (expected VRAM, expected tokens/sec), known failure modes (OOM at certain sequence lengths), and safety checks (PII filters, refusal policies).

• Config-first: no “magic constants” in notebooks; everything important is in versioned configs.
• Artifact discipline: save processed datasets, evaluation prompts, and model weights with immutable identifiers.
• Operational readiness: define alert thresholds (latency, error rate, GPU utilization) before serving to users.

Common mistake: treating reproducibility as an afterthought. In fine-tuning and optimization, you will make many small changes; without disciplined tracking and packaging, you cannot tell whether a performance improvement came from batching, quantization, or simply a different dataset slice.

Choose a data format that matches how the model will be used in production. A mismatch here is a common reason fine-tunes “look good” offline but fail in an endpoint. For classic supervised fine-tuning (SFT), each example is typically an instruction (or prompt), optional input context, and a single output. This format works well for single-turn tasks like classification, extraction, and short-form generation.

Most real deployments are multi-turn chat. In that case, store an ordered list of messages with explicit roles (system, user, assistant) and ensure you apply the same chat template used at inference. The system message is not optional: it is where you encode policy constraints, tone, and tool rules. If your training set omits system messages but production always includes them, you are effectively training a different distribution than you serve.

• SFT schema: {id, instruction, input, output, metadata}. Keep metadata for source, language, license, and quality flags.
• Chat schema: {id, messages:[{role, content}], metadata}. Preserve assistant refusals and safety behavior as first-class examples.
• Tool-call schema: include structured tool requests and tool results. A practical pattern is: user request → assistant tool call (JSON) → tool response (JSON/text) → assistant final answer. Validate that tool-call JSON is syntactically valid and stable.

Define acceptance tests aligned to the schema. If the endpoint must return JSON, require JSON in the reference outputs and run a parser in CI. If the assistant must call a tool for certain intents, create rubric rules that reward tool usage and penalize hallucinated tool outputs. Engineering judgment matters: don’t over-constrain creativity for open-ended tasks, but do harden anything that needs to be machine-consumable.

Finally, keep raw and rendered forms separate. Store the canonical example in a neutral schema, then render it into the model-specific template during training. This makes it easier to migrate between model families and reduces accidental “template drift” across experiments.

Fine-tuning data is “labels plus intent.” Decide early how you will produce labels and how you will measure their reliability. Human labeling is the gold standard for nuanced tasks (tone, policy compliance, domain reasoning), but it is expensive and inconsistent without a rubric. Write labeling guidelines that mirror your acceptance tests: format requirements, refusal conditions, and what counts as an error versus a preference.

Weak supervision is often the fastest path to scale. You can derive labels from existing signals: rules, regex, heuristics, database fields, or legacy systems. For example, if tickets already have resolution codes, you can train a classifier. The risk is label noise and hidden bias; mitigate it by sampling and auditing slices where heuristics are likely wrong (edge cases, new product lines, multilingual inputs).

Synthetic data is powerful when used to fill coverage gaps, not to replace reality. Use it to generate rare scenarios, tool-call traces, and “hard negatives” that teach the model what not to do. A practical loop is: (1) define failure modes from evals, (2) generate targeted synthetic examples, (3) add strict validation, and (4) re-run regression tests. Treat synthetic data as a controlled experiment—tag it in metadata so you can ablate it later.

• Human-first slices: high-risk intents (medical, legal, finance), refusal behavior, safety-sensitive outputs.
• Weak supervision slices: high-volume extraction/classification with clear mapping to fields.
• Synthetic slices: rare edge cases, tool-use trajectories, adversarial prompts aligned to policy.

Common mistakes include mixing label standards (different annotators, shifting rubrics), copying model outputs without review (model-collapse risk), and generating synthetic data with the same model you will fine-tune (amplifies its biases). Practical outcome: a labeled dataset where each example has a provenance tag, a confidence score (even coarse), and enough rubric clarity that you can reproduce labels later.

Quality control is not a single “cleaning” step; it is a set of gates that prevent bad data from entering training. Start with noise checks: empty outputs, encoding issues, corrupted markup, mis-ordered chat turns, and examples where the assistant output is actually the user content repeated. Run schema validation and minimum/maximum length checks, and keep failure logs so you can fix upstream ingestion rather than patching downstream.

Next, deduplicate aggressively. Near-duplicate prompts and answers cause overfitting, inflate evaluation metrics, and waste tokens. Use a two-stage approach: exact hashing for identical records, then approximate methods (MinHash/SimHash embeddings) for near duplicates. For chat logs, deduplicate at the conversation level and the turn level; it is common to see boilerplate policy text repeated thousands of times, which will dominate gradients if not controlled.

Safety filtering needs to be policy-driven. Define what you will exclude (e.g., explicit self-harm instructions, illegal activity instructions) and what you will keep with labeling (e.g., user mentions of self-harm that should trigger a safe response). Over-filtering can remove the very examples that teach safe refusal behavior. A practical pattern is to create three buckets: (1) remove, (2) keep-and-redact, (3) keep-and-train-with-safe-response.

Bias and coverage checks ensure you are not optimizing for a narrow slice. Measure coverage across languages, topics, user segments, and difficulty levels. If you are building an enterprise assistant, verify representation of product areas and error categories. Use targeted sampling to inspect model-critical slices: long context, ambiguous requests, adversarial prompts, and tool-required intents. Practical outcome: a dataset with quality metrics (dup rate, toxicity rate, PII rate, length distribution) you can track across versions.

Splitting is where many fine-tuning efforts accidentally invalidate their own evaluations. Your goal is to estimate how the model will behave on future inputs, not how well it memorized your dataset. Start by defining what “unit” you split on: user, account, document, conversation, or prompt template. If you split at the row level but the same document appears across rows, you will leak content into validation and test.

Use three splits with distinct roles. Train is for learning; validation is for early stopping, hyperparameter selection, and prompt/template decisions; test is for a final, rarely-touched report. For instruction tuning, keep a small “acceptance test” suite separate from all three—this is your regression set that should not be optimized against too frequently, or it becomes another training signal.

Contamination detection should be explicit. Run approximate matching between splits using hashing and embedding similarity to catch paraphrases. For generation tasks, also check answer leakage: if the same reference output appears in train and test with minor prompt changes, your metrics will be inflated. If you use synthetic data, ensure synthetic prompts derived from test cases are excluded from training; keep a strict boundary around evaluation artifacts.

• Group-aware splitting: split by document ID, customer ID, or source system to prevent cross-talk.
• Time-based splitting: for logs, train on earlier data and test on later data to simulate deployment drift.
• Template-aware splitting: keep prompt templates consistent across splits or deliberately hold out templates to test generalization.

Practical outcome: evaluation numbers that you can defend. When asked “does this improvement generalize?”, you can point to a split strategy, contamination checks, and a reproducible manifest of which IDs landed where.

Tokenization turns your carefully curated examples into the sequences the model actually trains on. If you ignore this step, you may silently drop crucial instructions, corrupt tool-call JSON, or bias learning toward short examples. Start by measuring token length distributions per dataset slice (train/val/test, per task). Then decide your max sequence length based on model context window and budget.

Truncation is the most common hidden failure. If the system message or rubric-critical instruction is truncated, the model will learn inconsistent behavior. Prefer strategies that preserve the beginning of the prompt (system + user intent) and the end of the assistant output when needed, but be deliberate: for long documents, you may need chunking or retrieval augmentation rather than truncation.

Padding affects efficiency and sometimes quality. Excess padding wastes compute; dynamic padding by batch reduces waste but complicates reproducibility if not controlled. Ensure attention masks are correct, especially for packed sequences.

Packing (concatenating multiple short examples into one long sequence) can dramatically improve throughput for SFT. However, packing requires correct boundary tokens and loss masks so the model does not “learn” across example boundaries. For chat data, packing is trickier; many teams pack only within the same conversation or avoid packing altogether when turn boundaries matter.

• Validate that tool-call JSON remains intact post-tokenization; run a decode-and-parse check on a sample.
• Track the percentage of examples that truncate; if it is high, you likely need data redesign, not just a higher max length.
• Keep tokenization configuration versioned (tokenizer name, vocab version, special tokens, chat template) to ensure repeatability.

Practical outcome: a training-ready dataset where sequence construction is deterministic, efficient, and aligned to your acceptance tests (format validity, refusal behavior, and tool correctness).

Governance is what lets you ship and sleep. For exam readiness and real-world deployments, you need to prove what data you used, why you were allowed to use it, and how you protected users. Start with a dataset card that describes purpose, sources, time range, languages, intended use, and known limitations. Include quantitative stats (counts, token totals, duplication rate, PII rate) and qualitative notes (labeling rubric, annotator training, known bias risks).

Lineage tracking is your audit trail. Every record should be traceable from raw source → normalized form → filtered/redacted form → final training example, with scripts and versions recorded. Store immutable raw snapshots when permitted, and store hashed identifiers when you cannot retain raw content. This is essential for incident response: if a user reports their data appeared in output, you need to know whether it was in training, and under what policy.

Licensing is not optional. Tag each source with license terms and usage constraints (commercial use, redistribution, derivative works). If licenses conflict, isolate those examples so they can be excluded from commercial models. For internal enterprise data, confirm you have training rights and that retention policies allow model training.

PII handling should be explicit and tested. Define PII types (emails, phone numbers, addresses, account numbers), choose redaction or pseudonymization methods, and verify that redaction does not break task structure (e.g., keep consistent placeholders like <EMAIL>). Retention policies should specify how long raw logs, intermediate artifacts, and curated datasets persist, and how deletion requests propagate through your pipeline.

Practical outcome: a dataset package you can hand to security, legal, or an auditor: documented purpose, clear permissions, reproducible transformations, and retention controls—without scrambling to reconstruct decisions after training.

Most instruction tuning starts as supervised fine-tuning (SFT): next-token prediction on prompt→response pairs. Even though the loss function is the same cross-entropy used in pretraining, the masking and format matter. In instruction tuning you typically compute loss only on assistant tokens (not on the user prompt), otherwise the model wastes capacity learning to “parrot” the prompt rather than produce better answers. A common mistake is silently training on the entire concatenated sequence, then wondering why helpfulness doesn’t improve.

Loss curves for SFT can look deceptively good. A steadily decreasing training loss may simply indicate memorization of responses, especially if responses are short or templated. You need at least three signals during the baseline run: training loss, validation loss, and one task-facing metric (for example exact match, ROUGE, pass@k on a tiny code set, or a rubric-based score). Log them per step and per epoch, and always keep a fixed “canary” evaluation set that you never train on.

Engineering judgement: don’t over-optimize the objective early. A minimal baseline SFT should validate your pipeline and formatting. Use a small number of steps, confirm that loss decreases, confirm that the model’s generations change in the expected direction, and confirm there’s no data leakage between train and validation. Then lock the dataset snapshot and move to PEFT for iteration.

• Baseline run checklist: correct chat template, loss masking on assistant tokens, deterministic shuffle + split, fixed max sequence length, and logging of loss + one task metric.
• Common pitfalls: mixed formatting across samples, incorrect end-of-turn tokens, truncation that removes the answer, or including system prompts in the loss region.

Practical outcome: by the end of your baseline SFT, you should have a run artifact (config + metrics + checkpoint) that becomes your “control” for all future comparisons. This is the anchor for regression testing once you start hyperparameter tuning or adapter experiments.

Parameter-efficient fine-tuning (PEFT) lets you adapt a model while updating a small number of weights. The most common approach is LoRA: you freeze the base model weights and learn low-rank updates on selected linear layers. This reduces GPU memory for optimizer states and speeds experimentation. QLoRA goes further by quantizing the base model (often 4-bit) while training LoRA adapters on top, making it feasible to fine-tune larger models on a single GPU.

LoRA has three key knobs you must choose deliberately: rank (r), alpha (scaling), and target modules. Rank controls capacity. Too low and the model won’t learn; too high and you lose the cost advantage and may overfit. A practical starting point: r=8 or r=16 for smaller models; r=16–64 for larger models or more complex domains. Alpha is often set proportional to r (e.g., 16/32/64) to stabilize update magnitudes. Target modules determine where adaptation happens: attention projections (q_proj, k_proj, v_proj, o_proj) are common; adding MLP projections can help for style or knowledge-heavy tasks but increases compute.

When results are unstable, first narrow scope rather than increasing rank. For example, apply LoRA to q and v projections only, then expand to all attention projections if needed. This keeps the update constrained and can reduce overfitting. Another common mistake is applying LoRA everywhere “just in case,” then being surprised by slower training and harder debugging.

Merging adapters matters for serving. During training, you keep the base model + adapter weights separate. For deployment, you can either load adapters at runtime (flexible, supports multiple domains) or merge them into the base weights (simpler inference graph, fewer moving parts). Merging is usually appropriate when you have one “blessed” adapter and want a single model artifact for production. However, once merged, you must version the merged model separately and keep the adapter checkpoint for traceability.

• Practical PEFT workflow: baseline SFT → LoRA run with fixed dataset → evaluate → optionally QLoRA for bigger base models → select best adapter → merge (or keep modular) → version and register.
• Operational tip: save adapter checkpoints frequently; adapter files are small, so you can checkpoint more often than full-model SFT.

Practical outcome: PEFT gives you fast iteration cycles and lower VRAM usage while still producing measurable quality gains. It also encourages clean model versioning: base model version + adapter version + merge hash.

Training operations often fail not because your approach is wrong, but because memory budgeting is. Fine-tuning requires activations, gradients, and optimizer states. Even with PEFT, activations can dominate VRAM at longer sequence lengths. Two standard techniques—mixed precision and gradient checkpointing—let you push batch size or context length without changing the model.

Mixed precision (FP16 or BF16) reduces memory and increases throughput. BF16 is typically more numerically stable on modern NVIDIA GPUs, especially for transformer training, because it has a wider exponent range than FP16. If your hardware supports BF16, it is often the safest default. FP16 can work well, but watch for NaNs and gradient overflows; loss scaling helps, yet it adds complexity.

Gradient checkpointing trades compute for memory by re-computing parts of the forward pass during backprop instead of storing all activations. This is useful when sequence length is large or when you need a larger micro-batch for stable optimization. The cost is slower steps, so you should use it only when memory is the bottleneck. A common engineering judgement call: if your GPU utilization is low and you’re memory-bound, checkpointing is often “free enough.” If you’re already compute-bound, it may reduce throughput too much.

• Rule of thumb: first enable BF16; then reduce sequence length or micro-batch; then turn on gradient checkpointing; finally consider QLoRA or a smaller base model.
• Watchouts: checkpointing can interact with certain attention implementations; validate correctness by comparing a few steps without checkpointing (when possible).

Practical outcome: with these optimizations, you can run a baseline fine-tune and PEFT experiments on limited VRAM while keeping training stable. This also sets you up for later serving optimizations, because memory discipline during training usually forces you to understand sequence length, padding, and batching behaviors early.

Hyperparameters determine whether your fine-tune is stable and whether it generalizes beyond the training set. Start with a small playbook and change one variable at a time. For SFT and PEFT, the highest-leverage knobs are learning rate (LR), warmup, effective batch size, and number of epochs/steps.

Learning rate: PEFT usually tolerates higher LRs than full fine-tuning because you update far fewer parameters. However, too high still causes divergence or “style drift.” Practical starting points: for LoRA, try 1e-4 to 2e-4; for QLoRA, 1e-4 is a common safe baseline; for full-model SFT, often 1e-5 to 2e-5 depending on model size and dataset. Use validation loss and task metrics to decide, not training loss alone.

Warmup and schedules: warmup prevents early instability when optimizer statistics are uncalibrated. A common setting is 1–5% of total steps. Cosine decay or linear decay both work; the important part is to avoid a constant high LR for long runs, which can harm generalization. If you see good early improvements followed by degradation, a decaying schedule and earlier stopping often fix it.

Batch sizing: distinguish micro-batch size (per GPU) from effective batch size (after gradient accumulation). If you’re constrained by VRAM, keep micro-batch small and use accumulation to reach an effective batch that yields stable gradients. For instruction data with variable lengths, monitor tokens per batch rather than examples per batch; token-based batching often improves utilization and stability.

• Stability signs: smooth loss decrease, no spikes to NaN/Inf, validation metrics improve or plateau slowly.
• Generalization signs: validation loss stops improving while training loss continues downward; task metrics fall even as loss improves.

Practical outcome: you can tune toward a “stable, boring” training curve that produces consistent gains. This is the foundation for trustworthy comparisons and for selecting a checkpoint to serve.

Fine-tuning without experiment tracking is indistinguishable from guesswork. Your goal is to make runs comparable: same data snapshot, same evaluation set, same prompt templates, and a recorded configuration. Track at minimum: base model ID and hash, dataset version, tokenizer/version, training code commit, hyperparameters, random seed, and the exact chat template used.

Log metrics at two levels. First, optimization metrics: training/validation loss, learning rate, gradient norm, tokens/sec, GPU memory, and throughput. These explain whether the run is healthy. Second, product metrics: a small suite of fixed prompts scored with task metrics and, when appropriate, an LLM-as-judge rubric. Keep judge prompts and scoring instructions versioned; otherwise, “evaluation” becomes another moving target. Pair this with lightweight regression tests: a handful of must-not-fail examples (policy compliance, refusal behavior, formatting constraints) that you run on every candidate checkpoint.

Reproducibility requires deterministic data handling. Save the resolved training file (after filtering and formatting), log the exact train/val split indices, and record max sequence length and truncation strategy. A subtle but common mistake is changing preprocessing between runs—e.g., switching from left truncation to right truncation—then attributing differences to hyperparameters.

• Versioning practice: treat checkpoints as immutable artifacts; tag “best on val” and “best on task metric” separately.
• Adapter management: store adapter configs (rank, alpha, targets) alongside weights; merging creates a new model version that must be traceable back to base + adapter.

Practical outcome: you can answer “what changed?” for any regression, and you can promote a checkpoint to serving with confidence that it is better than the baseline on agreed-upon metrics.

Training failures usually fall into a few repeatable categories. Divergence shows up as loss spikes, NaNs/Infs, or sudden degradation in outputs. The usual causes are excessive learning rate, unstable precision settings, or problematic batches (extreme lengths, corrupted tokens). Mitigations: lower LR, increase warmup, switch to BF16, enable gradient clipping, and inspect the exact batch that triggered the spike. If divergence happens at the same step across reruns, suspect a specific data example.

Overfitting is more subtle: training loss improves while validation loss or task metrics stagnate or worsen. This is common when datasets are small, homogeneous, or templated. Mitigations include fewer epochs, stronger decay schedules, smaller LoRA rank/target scope, and more diverse data. Also verify that your evaluation set is representative; otherwise you may “optimize away” from real-world requirements.

Data-induced collapse happens when the dataset teaches the model bad habits: excessive refusals, verbosity inflation, unsafe content patterns, or brittle formatting. This often comes from noisy instruction datasets, inconsistent system prompts, or mixing conflicting styles. The model may become less helpful even though loss improves because it is learning to imitate low-quality responses. The fix is not a new optimizer—it is data curation: filter low-quality samples, standardize templates, enforce policy labels, and add counterexamples that teach the desired behavior.

• Troubleshooting order: validate data formatting → reproduce with fixed seed → check LR/warmup and precision → inspect offending batches → simplify (shorter context, fewer targets) → re-run baseline.
• Regression discipline: never declare victory without running the same evaluation suite used for the baseline, including safety and formatting checks.

Practical outcome: you can respond to failures with a structured playbook. This is the difference between “fine-tuning sometimes works” and an operational training pipeline that produces improvements you can serve, optimize, and maintain over time.

Start your evaluation suite with objective task metrics. Even when your application is “chat,” you can usually define at least part of the output as a measurable artifact: a label, a JSON field, a set of entities, or a short factual answer. A golden set should include representative samples, hard cases, and “gotcha” formats (extra whitespace, alternate phrasing, missing fields). Keep it small enough to run frequently (dozens to a few hundred items) and stable enough to compare across model versions.

Exact match is ideal for deterministic outputs such as classification labels, tool calls, or canonical answers (e.g., “yes/no,” a function name, a single ID). However, exact match can be brittle: different punctuation or casing can unfairly penalize a correct answer. Use normalization (lowercasing, trimming, standardizing units) and enforce structured outputs where possible (JSON schema validation is often better than text matching).

F1 score is useful when answers are sets or spans: entity extraction, keyword lists, or multi-label tagging. Compute precision/recall on token sets or normalized entities. This gives partial credit and highlights whether your model is over-predicting (high recall, low precision) or too conservative (high precision, low recall).

BLEU/ROUGE can help for summarization or paraphrase-like tasks, but treat them as weak signals. They reward lexical overlap and can miss semantic correctness. If you must use them, pair them with human/LLM rubric scoring that checks factuality and coverage.

Custom scores are often the most valuable. Examples: JSON parse rate, schema compliance rate, tool-call success rate, “no hallucinated fields” rate, or latency-weighted quality (quality score minus penalty if output exceeds token budget). In exam and production scenarios, these metrics map directly to user experience and operational cost.

Workflow tip: compute a single dashboard table per run (model version × dataset version × metric set). Store the raw model outputs too—metrics tell you that something regressed, but examples show you why.

Many important qualities are hard to capture with exact-match metrics: helpfulness, completeness, tone, reasoning clarity, and whether an answer follows instructions. LLM-as-judge evaluation turns these into repeatable rubric scores. The key is to treat the judge as a measurement instrument that must be calibrated, not as an oracle.

Begin with a rubric that is short, unambiguous, and aligned to your product. For example: (1) follows instructions, (2) correct and grounded, (3) concise, (4) safe. Provide the judge with the user prompt, the model response, and (when applicable) a reference answer or retrieved context. Ask for a structured output: a numeric score per dimension and a short justification. This makes aggregation and debugging possible.

Calibration is essential. First, create a small “calibration set” of 20–50 examples that you (or a human reviewer) score manually. Run the judge and compare. If the judge is too lenient or too strict, adjust the rubric language, scoring anchors (what qualifies as a 1 vs 5), and constraints (e.g., “If any factual claim contradicts the reference, correctness must be ≤2”). Repeat until judge scores correlate with your intended standards.

• Control bias: randomize model IDs (A/B) so the judge doesn’t learn a pattern.
• Use pairwise ranking: comparing two outputs is often more stable than absolute scoring.
• Judge diversity: if feasible, use two judge models and require agreement or average.
• Temperature=0: judges should be deterministic to support regression testing.

Common mistake: using the same model family as both candidate and judge without controls. This can inflate scores due to shared biases. Another mistake is letting the judge see chain-of-thought or hidden reasoning; keep evaluation focused on observable outputs and verifiable facts.

Practical outcome: you gain an automated way to score qualitative dimensions and track improvements after fine-tuning, prompt updates, or RAG changes—without needing to re-run expensive human evaluations for every iteration.

Alignment checks are not optional in real deployments and are increasingly part of certification expectations. You need a repeatable way to verify that your model refuses disallowed requests, handles ambiguous safety situations appropriately, and doesn’t become easier to jailbreak after fine-tuning. Treat safety evaluation as its own golden set with explicit expected behaviors.

Build a policy test suite that includes: (1) clearly disallowed requests (e.g., instructions for wrongdoing), (2) borderline cases (e.g., historical discussion vs procedural guidance), (3) benign look-alikes (safe content that shares keywords with unsafe topics), and (4) jailbreak patterns (role-play, “ignore previous,” obfuscated text). For each item, define the expected outcome: refuse, comply, or comply with constraints (e.g., provide high-level safety info only).

Measure at least three metrics: refusal accuracy (refuse when required), false refusal rate (refuse when allowed), and jailbreak susceptibility (compliance rate under adversarial prompts). If you use tool calling, also check that the model does not call tools to obtain restricted info. In practice, many failures are “partial”: the model refuses but still leaks actionable steps—your rubric must catch that.

Automate offline evaluation by running the suite at every model change (fine-tune checkpoint, prompt template update, decoding parameter change). Store outputs and label failures. This becomes a regression gate: if jailbreak susceptibility increases, the build should fail.

Engineering judgement: fine-tuning on narrow instruction data can accidentally reduce safety behavior if the dataset rewards “always answer.” Counter this by mixing in policy examples, using system prompts that assert constraints, and validating with the same safety suite used before and after training. The practical outcome is confidence that improvements in helpfulness did not come at the cost of unsafe compliance.

Hallucinations often come from a mismatch between what the model was trained on and what the user asks. If your task depends on changing, domain-specific, or proprietary facts, Retrieval-Augmented Generation (RAG) is usually the first lever—not fine-tuning. RAG supplies relevant context at inference time, reducing the need for the model to “guess.”

A minimal RAG pipeline has four steps: ingest documents, chunk them, embed chunks, and index embeddings for retrieval. Chunking is the first major design decision. Chunks that are too large dilute relevance; chunks that are too small lose context. A practical starting point is 300–800 tokens with 10–20% overlap, then adjust based on retrieval performance. Preserve structure: split on headings, code blocks, and tables when possible, because semantic boundaries matter.

Embeddings convert text into vectors where semantic similarity corresponds to geometric closeness. Choose an embedding model that matches your language and domain. Normalize text consistently (case, whitespace, boilerplate removal) and consider adding metadata fields (source, timestamp, product version) that you can filter during retrieval.

Vector indexes (approximate nearest neighbor search) enable fast top-k retrieval. The key exam-style concept is understanding the trade-off: higher recall and accuracy vs latency and memory. Configure your index for your scale and update pattern. If documents change frequently, you need a predictable re-indexing strategy and a way to version the corpus so evaluation runs are reproducible.

Common mistakes: embedding whole PDFs without cleaning (headers/footers pollute similarity), ignoring chunk metadata (making citations impossible), and skipping an evaluation loop (assuming retrieval is “good enough”). The practical outcome of a minimal RAG pipeline is that you can answer factual questions with cited support and measure whether hallucinations decrease.

RAG only helps if retrieval finds the right evidence. You therefore need retrieval metrics and a tuning loop. Start by creating a retrieval golden set: for each question, label one or more “relevant” chunks (or documents) that contain the answer. Then compute recall@k: does the correct chunk appear in the top k results (k=5 or 10 are common)? High recall@k is a prerequisite for grounded answering.

If recall@k is low, fix it before touching generation. Typical levers include chunking changes, embedding model choice, and query formulation. Also watch for “semantic drift” where embeddings retrieve related but not answer-bearing chunks; this often happens with generic queries or overly broad chunks.

Reranking improves precision after initial vector retrieval. The standard approach is: retrieve top 20–50 via embeddings (fast), then rerank with a cross-encoder or LLM-based scoring (slower but more accurate) to produce the final top 5–10. In practice, reranking often yields a bigger gain than swapping embedding models, especially for nuanced questions where exact phrasing matters.

Query rewriting is a pragmatic tool when user queries are short, ambiguous, or conversational. Use a lightweight prompt (or a small model) to rewrite the user message into a search-optimized query, possibly expanding abbreviations and adding key entities. Keep rewriting deterministic, and log both original and rewritten queries to debug failure modes. A good rule: rewriting should clarify intent, not inject new facts.

• Track recall@k and MRR (mean reciprocal rank) on the retrieval golden set.
• Measure end-to-end “answer accuracy with RAG” separately from retrieval metrics.
• Version your corpus and index settings so regressions can be reproduced.

Practical outcome: you can explain whether a wrong answer was caused by retrieval failure (no relevant chunk returned) or generation failure (relevant chunk returned but not used). This separation is vital for deciding whether to tune the retriever, adjust the prompt, or fine-tune the generator.

Grounding is the discipline of ensuring that answers are supported by provided sources. In RAG systems, grounding is your primary defense against hallucination. To evaluate grounding, you need checks that go beyond “sounds right” and instead test whether claims are attributable to retrieved context.

Implement citation requirements in your output format: for each factual claim, include a citation to a chunk ID (or document URL + section). This can be enforced via structured output (e.g., JSON with fields answer and citations) and validated automatically. Your evaluation suite should measure citation coverage (how often citations are present when required) and citation validity (whether cited chunks were actually retrieved and exist in the index version).

Next, measure faithfulness: do cited sources actually support the claim? A practical offline test is: for each answer sentence, ask an LLM judge to label it as “supported,” “contradicted,” or “not in context,” given the retrieved chunks. Keep the judge deterministic and calibrate it with a small human-labeled set. Combine this into a faithfulness score (e.g., percent of sentences supported) and treat any “contradicted” label as a critical failure.

Also evaluate attribution behavior: the model should admit uncertainty when sources are missing (“Not found in provided documents”) rather than filling gaps. This is where hallucination risk assessment becomes concrete: you can quantify how often the model answers without evidence, even when retrieval fails.

Common mistakes: allowing the model to cite documents it did not retrieve, or retrieving evidence but not forcing the model to use it. The practical outcome is an auditable system: when an answer is wrong, you can trace whether the error came from the corpus, retrieval, or generation—and you can set regression gates that prevent “improvements” that secretly reduce grounding quality.

Decision guidance for exam scenarios: choose fine-tuning when you need consistent style, tool-call formats, or domain-specific reasoning patterns; choose RAG when facts change or must be sourced; choose a hybrid when you need both (e.g., fine-tune for schema/tool reliability, then RAG for up-to-date factual grounding). Your evaluation and grounding checks provide the evidence to justify that choice.

Choose a serving pattern based on latency targets, workload shape, and user experience. A synchronous request/response API is simplest: the client sends a prompt and blocks until completion. This works well for low-latency tasks (classification, short extraction) and internal services where you can enforce strict limits on prompt and output length. The common mistake is applying sync serving to long generations; one slow request can tie up connections, hit load balancer timeouts, and amplify tail latency.

Streaming is often the best default for interactive chat and long outputs. Instead of waiting for the full completion, the server emits tokens (or chunks) as they are decoded. Streaming improves perceived latency and reduces the risk of end-to-end timeouts, but it changes your API contract: clients must handle partial output, disconnects, and idempotency. In practice, you implement streaming via Server-Sent Events (SSE) or WebSockets, and you must test how proxies and gateways buffer data (a frequent source of “streaming that isn’t really streaming”).

Asynchronous patterns become essential when work can exceed typical HTTP limits, or when you need stronger fairness and scheduling. A job-queue approach (submit → get job_id → poll/subscribe) lets you enforce concurrency limits per tenant, prioritize premium traffic, and retry work without the client holding an open socket. For LLM batch processing (document summarization, nightly extraction), async queues also enable bulk batching and better GPU utilization.

• Sync: simplest; enforce tight limits; best for short tasks.
• Streaming: best UX for chat; must handle disconnects and partial responses.
• Async job queue: best for long or bulk work; enables scheduling and fairness.

Whichever pattern you choose, stand up an inference endpoint that supports batching. Batching can be “server-side dynamic” (aggregate multiple requests into a single forward pass) or “client-side micro-batches” for offline jobs. Pair batching with streaming carefully: many systems stream tokens per request while still batching decode steps internally.

Understanding the request lifecycle helps you optimize latency and diagnose bottlenecks. A typical LLM request flows through: (1) validation and normalization, (2) tokenization, (3) prefill (processing the full prompt to produce initial key/value tensors), (4) decode (iterative token generation), and (5) detokenization/formatting. Prefill cost grows with prompt length; decode cost grows with output length. Many teams optimize decode while ignoring prefill, then wonder why long-context prompts are slow.

Batching works differently across stages. Prefill batching benefits from grouping prompts of similar length; otherwise shorter prompts waste compute due to padding. Decode batching is sensitive to “stragglers”: if one request asks for 2,000 output tokens and others ask for 50, your batch can drag. A practical technique is to enforce max output tokens per tier, and route long generations to separate pools or async queues.

KV caching is the core performance lever for multi-turn chat. If you keep the key/value cache for the conversation context, follow-up turns avoid recomputing prior tokens. This reduces prefill time dramatically, but increases GPU memory pressure. Engineering judgment is required: aggressive KV caching boosts throughput until you run out of memory and trigger OOM failures. Common mitigations include limiting maximum context length, truncating older turns, using paged/blocked attention, and evicting caches with an LRU policy.

Package and register a deployable model artifact with the exact tokenizer and generation configuration used during evaluation. Serving bugs frequently come from mismatched tokenizers, missing special tokens, or different default sampling parameters (temperature/top_p). Your artifact should include: base model reference (or weights), adapter/LoRA weights, tokenizer files, a pinned inference config (max context, max output, stop sequences), and metadata (training dataset version, evaluation report links). Registering this artifact in a model registry makes rollout and rollback repeatable, and it supports regression testing across versions.

Reliability is not “five nines”; it’s predictable behavior under stress and failure. Start with timeouts at every layer: client timeout, gateway timeout, server request timeout, and model execution timeout. Without hard caps, one pathological prompt can monopolize a worker and cascade into a backlog. In LLM serving, you also want token-based limits (max input tokens, max generated tokens) because wall-clock time varies with hardware load and batching.

Retries must be used carefully. Retrying a failed LLM request can double cost and load. A practical policy is: retry only on clearly transient errors (network hiccups, 429 with a backoff hint), cap retries to 1–2, and add jitter. For streaming, implement resume semantics only if your protocol supports it; otherwise treat disconnects as failures and return an explicit status to the client.

Circuit breakers protect the system when downstream components degrade (GPU nodes unhealthy, model server OOMs). If the error rate or latency crosses a threshold, trip the breaker: fail fast rather than queueing indefinitely. Pair this with backpressure: return 429/503 early when queues exceed a safe depth. The common mistake is “infinite queueing,” which looks like availability but becomes minutes-long latency.

Fallbacks are your safety net. Examples include routing to a smaller, cheaper model when the primary pool is saturated, switching from streaming to async job mode for long tasks, or returning a cached response for repeated prompts (common in tooling or template-based workloads). Document these behaviors in your API contract so clients know what to expect.

Finally, build reliability into the artifact and endpoint design. Track model version and config in every response. When incidents happen, you need to correlate spikes in timeouts with a particular deployment, prompt pattern, or tenant. Reliability isn’t just infra; it’s traceability.

Serving turns your model into an attack surface. Start with strong authentication (authN) and authorization (authZ). Use signed tokens (OAuth2/OIDC or mTLS for service-to-service), validate audience/issuer, and enforce tenant scoping in every request. Authorization should be policy-driven: which tenants can use which model versions, maximum context/output per tier, and access to tools (retrieval, code execution, external APIs).

Secrets management is non-negotiable. Never bake API keys into containers or model artifacts. Store secrets in a vault/KMS, rotate them, and grant the smallest possible permissions to the serving runtime. A frequent production failure is leaking a third-party tool key through logs or returning it in model output due to tool misconfiguration—treat tool outputs as sensitive, and redact before logging.

Prompt injection defenses require layered controls. Input validation is your first guardrail: reject oversized payloads, enforce allowed content types, and normalize encodings. Then apply policy enforcement: content filters for disallowed categories, tool-use allowlists, and constraints on system prompts. In tool-augmented systems, the most important rule is to separate instructions from data: retrieved documents, user uploads, and web pages are untrusted data and must not be allowed to override system policy.

• Validate: size limits, schema checks, structured parameters.
• Constrain: explicit tool schemas, allowlisted domains/actions, maximum tool calls.
• Sanitize: strip or escape untrusted markup; redact secrets and PII in logs.
• Monitor: detect jailbreak patterns and unusually high tool-call rates.

Guardrails are not just moderation. They are engineering controls that make your endpoint safe and predictable. Treat policy as code: version it, test it, and deploy it alongside model versions so behavior changes are intentional and auditable.

GPU inference is expensive, and LLM cost is dominated by tokens. Cost control begins with quotas and budgets: per-tenant token quotas (input + output), per-minute rate limits, and maximum concurrent requests. Implement these limits at the edge (API gateway) and enforce them again in the model server to prevent bypass. Make quota errors actionable: return remaining quota and a reset time so clients can adapt.

Tiering aligns cost with value. A practical approach is to define tiers like free/dev, standard, and premium, each with different max context, max output, concurrency, and priority. Premium traffic can preempt standard queues, but you must ensure standard still meets its SLOs. The mistake is over-promising: if you can’t guarantee premium latency, you’ll pay for idle capacity or disappoint users.

Model routing is the most effective cost lever after batching. Route simple tasks to smaller or quantized models, and reserve large models for complex prompts. You can implement routing via: (1) static rules (task type → model), (2) confidence-based escalation (try small model, escalate if low confidence), or (3) learned routers. Add guardrails to prevent “router thrash” where requests bounce between models and double your token usage.

Batching and KV cache are also cost controls because they improve tokens-per-second per GPU. Measure cost as $ per 1M tokens and effective tokens/sec/GPU under realistic load, not single-request benchmarks. The goal is a stable operating point where you meet SLOs with predictable spend, and where rate limits and quotas prevent surprise bills.

Serving systems fail most often during change. Deployment hygiene makes change routine. Build CI/CD that produces immutable artifacts: a container image for the server, and a versioned model artifact registered in your registry. Pin everything: base model commit/hash, adapter version, tokenizer files, CUDA/container base image, and inference libraries. “Latest” is not a versioning strategy; it’s an outage strategy.

Use canary deployments for new model versions and new serving configs (batch sizes, quantization settings, decoding parameters). Route a small percentage of traffic to the canary, compare metrics (latency p50/p95/p99, error rate, OOMs, token throughput) and quality signals (LLM-as-judge score, user satisfaction proxies) before ramping. A common mistake is canarying only infrastructure metrics; a model can be fast and wrong.

Rollbacks must be fast and boring. Keep the last-known-good model version warm (or at least quickly loadable). Version pins enable rollback without rebuilding. If your serving stack supports it, use blue/green deployments to switch traffic instantly. If not, ensure your orchestrator can scale down bad replicas quickly and that your registry retains prior artifacts.

Implement observability as part of deployment hygiene: logs, metrics, traces, and SLOs. Log request metadata safely (tenant, model version, token counts, latency, error codes) without storing sensitive prompts unless explicitly allowed. Metrics should include queue depth, GPU utilization, cache hit rate, tokens/sec, and rejection counts from guardrails and rate limits. Traces should follow a request through gateway → validator → model server → tools. Define SLOs (e.g., p95 latency under X seconds for Y-token requests, availability, and maximum 5xx rate), and tie alerts to SLO burn so you page on real user impact.

Done well, deployment hygiene turns serving into a repeatable system: new fine-tunes become new registered artifacts, rollouts are gated by canaries and SLOs, and operators can explain any regression using versioned configs and end-to-end traces.

Section 6.1: Performance profiling: latency breakdown and telemetry

Optimization begins with a correct latency breakdown. Treat every request as a pipeline with phases: request parsing, tokenization, queueing, prefill (prompt processing), decode (token generation loop), detokenization, and response serialization. A common mistake is to profile only “model time” and miss queueing delay or tokenization overhead, which can dominate at low batch sizes or short outputs.

Instrument your server so every request emits structured telemetry: timestamps for each stage, prompt length, output length, batch size, and whether KV cache reuse occurred. On the GPU side, capture kernel timelines and memory activity (e.g., Nsight Systems for end-to-end traces and Nsight Compute for kernel-level hotspots). On the CPU side, include thread pools and event-loop latency, especially if you use Python-based servers where the GIL or excessive logging can distort results.

• Key metrics: p50/p95 end-to-end latency, tokens/sec (per GPU and per request), time-to-first-token (TTFT), GPU utilization, SM occupancy, HBM bandwidth, CPU utilization, and queue depth.
• Golden habit: correlate telemetry. A p95 spike that aligns with rising queue depth implies concurrency pressure; a spike aligned with HBM saturation implies memory-bound decoding; a flat GPU util with high latency often implies CPU or I/O bottlenecks.

Engineering judgment: only optimize what moves your target metric. If your SLA is TTFT, focus on prefill, tokenization, and queueing. If your SLA is tokens/sec at steady state, focus on decode throughput and batching. Another frequent error is benchmarking with a single prompt. Use a representative distribution of prompt lengths and output lengths; otherwise you will overfit your optimization to a best-case scenario and fail in production (and on exam scenarios that implicitly test generalization).

Section 6.2: Throughput levers: batching, speculative decoding, concurrency

Throughput is the art of keeping the GPU busy without destroying latency. The primary lever is batching: combine multiple requests so matrix multiplications run at higher arithmetic intensity. But batching has a cost—waiting to form a batch increases queueing delay. The practical solution is dynamic batching: aggregate requests for a short window (e.g., a few milliseconds) with a max batch size and a max queue delay, then execute.

Concurrency is the companion lever. You want enough in-flight requests to fill the GPU, but not so many that you thrash memory, blow up queueing delay, or cause tail latency from context-length extremes. Implement admission control: cap concurrent sequences, set per-tenant limits, and reject or shed load gracefully rather than letting latency become unbounded.

• Batching tactics: separate prefill and decode scheduling; batch prefill aggressively (large GEMMs), batch decode carefully (incremental steps) to protect TTFT.
• Speculative decoding: use a small “draft” model to propose tokens and a larger model to verify them, reducing large-model decode steps when acceptance is high. This can dramatically improve tokens/sec for long generations, but it can hurt on tasks with low acceptance (domain mismatch, highly constrained outputs).
• Parallelism choices: scale up (single GPU efficiency) before scaling out (multiple GPUs). If you must shard, pick tensor parallel for compute-heavy layers, pipeline parallel for very large models, and be mindful of interconnect bandwidth and added latency.

Common mistakes include benchmarking with batch=1 and concluding the model is “slow,” or pushing batch size to the maximum and then being surprised by time-to-first-token regressions. For certification readiness, be able to explain the tradeoff: batching increases throughput but can increase latency; the correct configuration depends on your SLA and workload mix.

Section 6.3: Memory levers: KV cache strategies and context management

Most real inference failures are memory failures. The KV cache (key/value tensors stored for each token per layer) grows with batch size, number of concurrent sequences, and context length. When you see sudden OOMs during decode, it is often not “model weights,” but KV cache expansion plus fragmentation from variable-length requests.

Start by estimating KV memory: roughly proportional to layers × heads × head_dim × tokens × dtype, doubled for K and V, and multiplied by the number of concurrent sequences. Use this estimate to set safe limits for max context and max concurrent sequences. Then choose a strategy that matches your product needs:

• Cache reuse: for chat, reuse KV for the conversation prefix so follow-up turns pay only incremental cost. Ensure correct cache invalidation when system prompts or tool context changes.
• Paged/blocked attention: allocate KV in blocks to reduce fragmentation and improve utilization under variable sequence lengths.
• Context management: enforce max context, apply truncation policies, and consider summarization or retrieval to keep prompts short while preserving relevance.

Engineering judgment: if TTFT is high, the culprit is often prefill on long prompts. That is a product design issue as much as a systems issue—encourage shorter prompts via UI, apply retrieval to narrow context, and keep system prompts lean. A frequent mistake is “just increase max tokens” to fix answer quality; that can silently destroy throughput and memory headroom, causing instability under load. In an exam-style scenario, you should be able to propose a safe context policy (limits, truncation rules, and monitoring) and justify it with KV cache mechanics.

Section 6.4: Quantization options: int8/int4, calibration, and pitfalls

Quantization reduces weight (and sometimes activation) precision to improve speed and reduce memory footprint. The goal is simple: fit larger models on the same GPU or increase throughput per dollar. The risk is also simple: degrade quality, especially for long-context reasoning, multilingual text, or tasks sensitive to small probability shifts.

Int8 is the standard “first step” because it often preserves quality with minimal tuning. Int4 can unlock major memory savings, but it is more sensitive to calibration quality and outliers. Calibration means selecting representative inputs to determine scaling factors; if your calibration set does not match production prompts, you may see regressions that are hard to diagnose.

• When to use int8: you want a safer speedup, reduced memory pressure, and minimal quality impact; ideal for general chat and many instruction-following tasks.
• When to use int4: you are memory-bound (KV + weights) or deploying on smaller GPUs; you accept stricter validation and occasional task-specific regressions.
• Pitfalls: calibrating on too-short prompts, ignoring domain-specific vocabulary, and failing to re-run regression tests after changing kernels or drivers.

Practical workflow: quantize, then measure (1) tokens/sec and TTFT under load, (2) peak memory and fragmentation, and (3) quality against your evaluation suite. Treat quality as a gate, not a nice-to-have. For certification readiness, be able to explain how you would select a calibration dataset, what metrics you would compare pre/post, and what you’d do if quality drops (e.g., switch to int8, adjust calibration, selectively keep certain layers higher precision, or use a different quantization scheme supported by your serving stack).

Section 6.5: Benchmarking: datasets, prompts, and reproducible load tests

Optimization without reproducible benchmarking is guesswork. Your benchmark suite should include (a) microbenchmarks for raw tokens/sec and TTFT, and (b) scenario benchmarks that mimic real traffic: mixed prompt lengths, mixed output lengths, and concurrency. Keep the suite small enough to run frequently, but broad enough to catch regressions.

Build a benchmark harness with explicit versioning: model hash, tokenizer version, server commit, GPU type, driver/CUDA version, and configuration (batching window, max batch, concurrency cap, context limit, quantization mode). Store raw results and summarize with p50/p95 latency, error rates, tokens/sec, and GPU memory. This is the “experiment tracking” mindset applied to inference.

• Prompt sets: include short Q&A, long-context summarization, tool-like structured outputs, and safety-sensitive prompts if relevant.
• Load tests: use fixed random seeds where possible, warm up the server, and run long enough to observe steady-state and tail behavior.
• Quality checks: pair performance runs with lightweight regression checks (task metrics or LLM-as-judge) so you don’t “optimize” into worse answers.

Common mistakes: benchmarking on an idle machine and deploying into a shared environment; forgetting warm-up (first request pays compilation/cache penalties); changing multiple knobs at once; and reporting only average latency. On the exam, expect scenarios where you must choose the right metric (TTFT vs tokens/sec), interpret tail latency, and propose a reproducible plan to validate improvements.

Section 6.6: Exam readiness: common traps, review map, and last-mile drills

Certification readiness is about execution under constraints: limited time, partial information, and the need to pick the highest-impact next step. A useful final exercise is an end-to-end capstone checklist that mirrors real exam tasks: profile a serving endpoint, identify the bottleneck, apply one throughput lever and one memory lever, then validate speed and quality with a reproducible benchmark run.

• Capstone checklist: (1) capture latency breakdown and GPU/CPU telemetry, (2) set a baseline p50/p95 and tokens/sec, (3) adjust batching/concurrency with a stated SLA target, (4) validate KV cache strategy and context limits, (5) try int8 (then int4 only if justified) and re-run quality gates, (6) produce a short report tying changes to metrics.
• Common traps: optimizing without baselines, ignoring tail latency, confusing throughput with latency, and “fixing” quality by increasing max tokens instead of improving prompts/context policy.
• Review map: profiling → throughput knobs → memory/KV cache → quantization → benchmarking discipline → deployment stability (timeouts, retries, backpressure).

For a timed mock exam drill, practice a strict loop: spend a few minutes reading the scenario and writing your metric goal, spend the next block collecting telemetry and forming a hypothesis, then make only one change at a time and re-measure. End with a short study plan focused on your weak links: if you struggle to interpret traces, rehearse Nsight timelines; if you struggle with batching tradeoffs, practice configuring dynamic batching and explaining TTFT impacts; if quantization surprises you, practice calibration selection and regression analysis. The practical outcome is confidence: you can justify your decisions with evidence, and you can reach a working answer quickly—exactly what the exam evaluates.