ML System
Design Practice

Define the decision and objective. Ask whether this is search or feed advertising, CPC or CPA billing, and whether value means qualified clicks or conversions. For this exercise, use CPC. CTR is a model output, not the sole product KPI. Consider revenue per thousand valid impressions, with conversion quality, hides/reports, latency, and retention as guardrails.

State scale assumptions. Assume a peak of 10k requests/s, 200 candidates per request, and a 100 ms p95 ranking-service budget. That is about 2M candidate scores/s, motivating batched scoring, cached features, and candidate pruning. These are interview assumptions.

Use impressions as examples. Join impression_id to user/ad/context features, position, candidate set, policy/model versions, timestamps, and click events. Deduplicate clicks and handle invalid traffic. Define a click-attribution window and only label a non-click after that window matures.

Use information available at decision time. Features include prior behavior, creative text/image representations, device, time, and pre-impression aggregates. Use point-in-time joins and chronological train/validation/test splits. Handle conversion delays separately. Record negative-sampling rates; unshown ads do not have observed click labels.

Start with an interpretable baseline. Use smoothed CTR by placement/category, then logistic regression or GBDT to verify data quality and incremental value. For large sparse feature spaces, consider embeddings and explicit feature interactions, such as a DLRM-style model. Encode text and images offline where feasible.

Specify the learning objective. Binary cross-entropy estimates p(click | user, ad, context, shown). Calibrate on a separate validation set with natural traffic prevalence. A multitask click/conversion/hide model requires explicit loss weights and attribution windows. Click-conditional CVR is not the same as impression-level conversion probability.

Build offline and online data paths. Impression/click streams feed durable storage, deduplication, watermarking, label maturation, and training validation. Recent counters update an online feature store. Daily retraining is a starting assumption; adjust it based on drift and measured value.

Make releases reversible. Version the model, feature schema, and calibrator together. Progress through replay, shadow traffic, a canary, and A/B testing. On feature timeouts, use a lightweight model trained to handle missingness. Eligibility, budgets, and frequency caps remain enforced. Monitor feature age, missingness, p95/p99 latency, and predicted versus observed CTR.

Separate ranking from probability quality. ROC-AUC measures discrimination; interpret PR-AUC alongside prevalence. Use log loss, Brier score, and reliability diagrams for probability quality. Evaluate on natural traffic and slice by placement, new ads, device, geography, and activity level.

Test product impact online. Use stable user assignment, check SRM and logging completeness, and predefine the primary metric, MDE, sample size, and duration. Shared campaign budgets may create interference, requiring campaign- or geo-level designs. Revenue gains must pass user and advertiser guardrails, with fast rollback available.

AUC rises but revenue falls: inspect calibration, slice shifts, candidate composition, latency, and budget consumption before increasing model size. New ads receive no exposure: add content features and constrained exploration rather than relying only on ID embeddings.

Main tradeoffs. Clickbait can increase CTR while hurting conversions and trust. Fresh features improve responsiveness but add dependencies and latency. Use minimal necessary features, layered fallbacks, natural-distribution calibration, and exploration data to limit feedback loops. Profile feature fetching and embedding tables as traffic grows.

Scope the surface first. Homepage, product-detail, and cart recommendations express different intents. Assume a personalized homepage shelf. Ask about sponsored content, anonymous traffic, and freshness requirements. A primary metric might be matured valid orders per user, with cancellations/returns, satisfaction, repeat visits, and latency as guardrails.

State scale assumptions. Assume 10M products, 5k peak requests/s, and 150 ms p95. Merge roughly 1k candidates, pre-rank to 200, and select 20. Tune these counts using recall–latency curves rather than treating them as fixed rules.

Distinguish behavioral signals. Log request_id, candidate and displayed items, positions, clicks, carts, orders, returns, timestamps, and model versions. Displayed-but-ignored items are weak negatives; unshown items are unknown. Purchases are sparser than clicks, and returns mature later, requiring separate label windows.

Make examples reproducible. Use pre-request history, session intent, category, price, text/image features, and inventory snapshots. Do not use future prices or post-purchase attributes. Split chronologically and retain cold-user/item slices. Normalize product variants and duplicate listings.

Give retrieval and ranking distinct jobs. Start with popular items, category candidates, and item-to-item co-purchase signals. Add a two-tower user/session and item model using a contrastive or sampled-softmax objective. In-batch negatives may be false negatives; account for popularity-driven sampling and product variants.

Use richer interactions in ranking. A GBDT or deep ranker predicts clicks, carts, purchases, and returns. Combine these through an explicit utility function and hard constraints. Retrieval dot products are not purchase probabilities. Image/text representations can help new items, but first measure a strong metadata baseline.

Version embeddings and ANN indexes together. Generate item vectors offline, maintain a main index plus incremental updates, and compact periodically. User/item towers and the index must be compatible; switch atomically and retain the previous version. Features, models, and indexes all need rollback paths.

Separate freshness requirements. Descriptions change slowly, session intent updates via streams, and price/inventory comes from an authoritative service with explicit TTLs. Filter during retrieval and recheck before output. Do not encode live stock as a permanent embedding property. Fall back to eligible popular or recently viewed items on ANN failure.

Diagnose by stage. Measure candidate Recall@K against known relevant items, then ranker NDCG@K, purchase log loss, and calibration. Check catalog coverage, diversity, and cold-start slices. ANN recall measures agreement with exact nearest-neighbor search; it is different from relevance recall.

Randomize by user. Predefine power and duration, check SRM, and measure orders per user and incremental value while allowing cancellation/return labels to mature. Whole-user metrics help distinguish additional purchases from moving purchases between surfaces.

New users: use context, session behavior, popularity, and optional preferences. New products: use metadata, image/text representations, and constrained exploration. Overspecialization: add category/brand diversity and suppress duplicates or no-longer-needed durable goods.

Main tradeoffs. ANN speed competes with long-tail recall; larger rankers compete with latency; real-time behavior adds both freshness and noise. Determine whether useful items were missed by retrieval or misordered after retrieval before investing. Exploration and temporal evaluation help limit popularity feedback loops.

Define the answer contract. Ask about formats, languages, tables, update frequency, cross-document reasoning, and permissions. Answers must address the question and cite evidence; missing evidence should trigger clarification or abstention. Optimize supported task completion, with authorization violations, false claims, latency, and cost as guardrails.

Separate latency measures. Assume 1M chunks, 100 queries/s, a 2 s p95 time to first token, and 8 s to completion. Bound answer length and tool calls. Streaming improves responsiveness but does not prove lower total generation latency.

Make documents traceable assets. Store doc_id, version, section/page, source offsets, updated_at, tenant_id, and ACL for each chunk. Respect headings and paragraphs; preserve table headers and units. Tune chunk size and overlap through evaluation. Propagate deletions and permission revocations to indexes and caches.

Build a challenging gold set. Include real questions, relevant spans, required facts, acceptable answers, unanswerable cases, and access-denied cases. Split by document family and time to avoid near-duplicate leakage. Synthetic questions can expand coverage, but need human auditing and independent evaluation.

Prove retrieval first. Start with BM25, a fixed prompt, and citations. Dense embeddings add semantic matching; BM25 preserves exact identifiers and terms. Fuse ranks with Reciprocal Rank Fusion, then rerank a bounded candidate set with a cross-encoder. Do not directly add incompatible score scales.

Generate from authorized, relevant evidence. Require source IDs and acknowledgment of missing evidence, and map citations back to source spans. Hold the generator fixed for retrieval ablations. Fine-tune retrievers/rerankers only when representative labels and error analysis justify it; RAG does not require LLM fine-tuning.

Separate indexing from serving. Content hashes support deduplication and incremental updates. Keep reversible index snapshots and consistent embedding versions. Authenticate and enforce tenant/ACL restrictions during search, then recheck revocations before reranking/output. Pure post-filtering can severely reduce recall.

Bound failures. Cache keys include permission scope, query, and document/index versions; revocations invalidate caches. On retrieval/model timeout, return authorized sources or explain that the answer could not be completed. Treat documents as data, not authorization to call tools or change instructions. Trace retrieval, prompt/model versions, citations, and stage latency.

Make evaluation attributable. Retrieval metrics include evidence Recall@K, MRR/nDCG, empty-result rate, and recall after ACL restrictions. Generation metrics include correctness, claim-level groundedness, citation precision/coverage, and abstention precision/recall. A citation being present does not mean it supports a claim. Isolate retrieval and generator changes in ablations.

Measure completion online. Test task success, reformulations, escalation, cost, TTFT, and full-response time. Calibrate an LLM judge against human labels with an explicit rubric, blinded comparisons, and language/length bias checks. Human-review consequential errors.

Correct document, wrong answer: inspect chunk boundaries, table units, conflicting versions, and context use. Missing evidence: inspect ingestion, ACL filters, query rewriting, and separate BM25/dense recall.

Main tradeoffs. Larger K can increase recall but also noise, tokens, and latency. Reranking trades cost for precision. Live price or inventory requires authoritative APIs rather than stale documents. Test prompt injection, outdated evidence, and unsupported synthesis explicitly; low temperature is not a guarantee against hallucination.

Clarify whether the feed contains friends, public content, or both, and whether chronological ordering is a baseline. Optimize valuable engagement or survey-calibrated satisfaction, with hide/report, safety, and retention guardrails. Assume 20 items and a 200 ms p95 budget for practice, with an explicit freshness requirement.

Log viewable impressions, not only served items, plus position, dwell, shares, hides, and reports. Dwell can indicate confusion rather than satisfaction. Use request-time features and chronological splits; unseen items lower in the feed are not strong negatives. Retain policy propensities and controlled exploration data.

Start with recency decay and relationship strength, then a GBDT or multitask deep ranker for valuable actions and negative feedback. Mix friend, interest, and trending candidates. Apply author/topic diversity and safety constraints after ranking. Objective weights need an interpretable utility definition and online validation.

Stream recent interactions and learn long-term interests offline. Avoid unconditional fan-out for very high-fanout authors; combine precomputed candidates with request-time retrieval. Cache candidates and batch scoring. Invalidate deleted content promptly, and fall back to an eligible chronological feed on timeout.

Offline, evaluate per-task log loss, NDCG, slice calibration, and creator coverage. Counterfactual estimates require propensity support and do not automatically eliminate bias. Online, check SRM and measure satisfaction, retention, and negative actions. Social spillovers may require cluster designs or explicit interference analysis.

High dwell can reward sensational content, and popular creators can become increasingly dominant. Use constrained exploration, concentration monitoring, diversity, and quality guardrails. If offline gains hurt retention, slice by source, creator, new users, and content length to find misaligned incentives.

Clarify whole-image versus selected-object input, exact-SKU requirements, and supported categories. Optimize successful queries and relevant interactions, with no-result rate and latency guardrails. For practice, scope to product images and 300 ms p95; let users select an object in multi-object scenes.

Collect query–item relevance, exact/category/style labels, impression clicks, and image–text pairs. Split by near-duplicate image and product family. Include hard negatives that look similar but differ in material or function, and audit missing or incorrect descriptions.

Start with a pretrained visual encoder and ANN. Compare image–text aligned encoders, domain contrastive fine-tuning, and crop-level representations. Rerank using color, material, text, and category. Graph-based representations such as PinSage-style embeddings can add candidates, but graph proximity is not identical to visual similarity.

Encode catalog items offline and queries online, versioning encoders and indexes together. Bound GPU batching by request deadlines. Cache common image hashes; handle corrupt/low-resolution images explicitly or offer text search. Incrementally index new products and remove invalid items.

Evaluate exact matching and semantic relevance separately with Recall@K, mAP/NDCG, plus ANN recall and category/lighting/background slices. Audit relevant representation differences such as skin tone for applicable image categories. Online, test successful searches and product interactions with cost, tail-latency, and no-result guardrails.

The model may exploit backgrounds or brand watermarks; test crops, background augmentation, and hard cases. Similar appearance can hide size/function differences, so allow text refinements and attribute constraints. Better crops and labels may outperform a larger encoder; verify through ablations.

Clarify pretraining versus fine-tuning, dense versus MoE, target loss, deadline, hardware, and interconnect. Optimize time/cost to quality, useful tokens/s, and recovery time rather than GPU utilization alone. Define global batch size and token accounting.

Deduplicate and filter data, track provenance/versions, and check held-out contamination. Derive next-token labels by shifting sequences and mask padding. Distributed sampling must avoid unintended duplication or omissions. Save shuffle seeds, data cursors, and consumed-token counts.

Establish a small-model reference at a reliable precision, then compare mixed precision. DDP replicates state; FSDP/ZeRO shard it at communication cost. Tensor parallelism favors fast interconnects; pipeline parallelism introduces bubbles. Global batch equals microbatch × accumulation steps × data-parallel world size.

Profile compute, collectives, data loading, and I/O. Activation checkpointing trades recomputation for memory. Save model, optimizer, scheduler, RNG, data cursor, and shard metadata with a completeness manifest. A checkpoint is useful only after a tested restore. Restart synchronous groups consistently after failures to avoid rank-step divergence.

Compare loss trajectories, gradients, and downstream quality with matched data and effective batches, then measure tokens/s/GPU, scaling efficiency, MFU, and cost/token. Validate the platform with cluster canaries, controlled workloads, and failure injection. A training platform does not need a click-rate A/B test; deployed model behavior gets its own product experiment.

For OOM, separate parameter, optimizer, and activation memory. For NaNs, inspect data, overflow, loss scaling, and learning rate. Poor scaling may come from collectives, stragglers, or I/O. Sharding adds communication; frequent checkpoints add I/O. Choose checkpoint intervals from failure frequency, save cost, and acceptable recomputation.

Define minimum quality, p95/p99 latency, peak traffic, and hardware. Optimize cost per successful task, decomposing token, idle GPU, cache, network, and retry/review costs. Cheap tokens can hide expensive failed tasks.

Log request shape, token counts, queue time, TTFT, completion time, route, hardware, and success labels, minimizing sensitive content. Replay sets must include long contexts, rare languages, peaks, and hard tasks rather than only easily cached requests.

Baseline a fixed model and standard serving. Profile before changing exact/prefix caching, small-model routing, quantization/distillation, or unnecessary output length. Train the router for quality and cost, escalating uncertain cases. Semantic caching needs applicability and authorization checks.

Version routing policies, models, quantization, and caches; canary one change at a time. Bound dynamic-batching queues by deadlines and scale using queue depth and arrivals. Define degraded-service quality limits instead of queueing indefinitely. Invalidate caches by data and permission version.

Evaluate matched tasks for quality, success, cost, and load behavior; construct a quality–cost–latency Pareto frontier. Online, compare cost per successful task and tail latency under stable assignment, sliced by cache hits, request length, and language, including idle infrastructure cost.

Large batches improve throughput but add waiting; small models can increase retries; quantization can damage specific languages or tool formats. Report total successful-task cost, not just fewer GPUs. If the actual scope is training, shift to data efficiency, checkpointing, utilization, and time-to-quality.

Clarify automatic versus creator-assisted tagging, multiple objects, and exact SKU versus category labels. Optimize correctly accepted tags, with mismatch and latency guardrails; incorrect links can be worse than missing tags.

Collect boxes, categories, attributes, SKUs, and corrections. Unannotated objects are not necessarily background. Split by product family/image group, including occlusions, rare brands, backgrounds, and viewpoints. Reserve unseen SKUs for open-set evaluation.

Start with a pretrained detector plus image-embedding retrieval, then fine-tune detection and image–text representations. Verify SKU matches using crops, multi-view catalog images, OCR, and attributes. Threshold confidence to choose exact match, category output, or abstention.

Process uploads asynchronously where possible and precompute catalog embeddings. Track model, index, and taxonomy versions; invalidate unavailable products. Route low-confidence cases to human/user confirmation and audit corrections before training on them.

Evaluate detection mAP, attribute F1, SKU Recall@K, and precision at coverage separately. Online, measure correct acceptance, corrections, and downstream interactions. Random human audits address the fact that an uncorrected tag is not necessarily correct.

Occlusion, imitations, and size variants can make exact matching unreliable; use alternatives and attribute verification. Higher thresholds reduce errors but reduce coverage. Choose the operating point based on mismatch cost and review capacity.

Clarify uploaded video, live streams, or pre-recommendation filtering. Define policy taxonomy, severity, and actions. Track violating-exposure prevalence with false-positive, successful-appeal, and review-SLA guardrails; live streams need shorter deadlines.

Use versioned-policy multilabel annotations, time spans, languages, and appeals. Preserve uncertainty in disputed cases and restrict sensitive data. Split by creator/near-duplicate video and time. Random traffic audits estimate prevalence; reports alone are selection-biased.

Start with hashes/rules and unimodal classifiers, then evaluate multimodal fusion. Prioritize recall for severe rare categories. Calibrate and threshold by policy, routing uncertain or consequential cases to reviewers as an explicit system component.

Use a cascade of cheap filters, expensive models, and human review, budgeting frame sampling and ASR. Version models and policies separately and replay policy changes. Prioritize review queues by risk/SLA, retain reasons and appeals, and define degraded behavior instead of blindly allowing or removing everything.

Measure per-policy PR-AUC, recall at fixed FPR, calibration, language/group slices, and reviewer agreement. Start with shadowing and audits, then constrained canaries. Do not casually relax severe-content policy for an A/B test. Use controlled rollouts to track prevalence, appeals, and resolution time.

Sarcasm, quotation, dialects, and background audio can cause errors; use context, multimodal evidence, and expert labels. Refresh hard cases for adversarial transformations and policy drift. Higher recall increases review load and false positives, so set thresholds jointly with severity and capacity.

Distinguish homepage from “up next.” Optimize satisfying viewing sessions, with not-interested actions, abandonment, safety, and retention as guardrails. Define how duration and completion contribute to value rather than using total minutes alone.

Log viewable impressions, plays, duration, seeks, skips, surveys, and video length. Autoplay is not active preference. Split chronologically, exclude future history, and slice by duration, new creators, and language. Survey responses may be selection-biased.

Start with popularity and co-watch. Use two-tower or sequence representations for retrieval and multitask viewing/satisfaction ranking. Content embeddings support new videos and session context captures current intent. Rerank for repeat creators, previously viewed items, and exploration.

Refresh video embeddings and models offline while streaming recent viewing. Deploy compatible index versions, support deletions and eligibility, and maintain cold-start candidates. On timeout, serve eligible subscription/popular content and log fallback exposure for evaluation.

Evaluate Recall@K, NDCG, watch-prediction error, satisfaction calibration, and new-video coverage. Online, measure satisfying sessions, retention, abandonment, and creator concentration, including fatigue across sessions.

Watch time alone can favor long or sensational videos; use duration slices, multiple objectives, and quality constraints. New videos need content signals and constrained exploration; metadata can be manipulated. Compared with commerce, session sequence and viewing depth deserve more attention.

Clarify commute, exercise, or discovery use, sequence continuity, and offline needs. Optimize satisfying sessions, saves, and return visits, with early skips, repetition fatigue, and playability as guardrails.

Log source, exposure, listening duration, skip timing, saves, repeats, and explicit dislikes. Skips are not equally negative. Evaluate temporal and artist/version slices; audio and regional availability are item features.

Start with favorite artists and popularity, then collaborative retrieval, audio embeddings, and session sequence ranking. Rerank for transitions, artist repetition, and discovery; repeated songs should not be universally banned.

Precompute track embeddings/candidates and stream session state. Prefetch queues to reduce playback gaps. Filter unavailable tracks promptly and use playable cached queues during outages. Preserve queue consistency across releases.

Offline, measure next-track Recall@K, skip prediction, save calibration, and discovery coverage. Online, test satisfying sessions, saves, early skips, and repeat visits by new/old tracks, context, and user activity.

Acoustic similarity may not fit the listener's mood. Too much exploration disrupts sessions; too little becomes repetitive. Use controlled exploration, context, and explicit dislikes without equating popularity with preference.

Clarify individual profiles versus shared households. Optimize successful discovery, time to a satisfying choice, and satisfaction, with early abandonment, failed search, and retention guardrails rather than starts alone.

Log row/title impressions and positions, starts, viewing, completion, continuation, and ratings. Completion differs by duration; separate profile mixing and unshown items. Split by time and series family.

Start with popularity, categories, and continue-watching. Retrieve through collaborative/content embeddings and rank for satisfying viewing. A second layer chooses row themes/order, limits cross-row duplication, and balances familiarity and discovery.

Precompute candidates/content features and add recent session context online. Filter by regional availability, profile settings, and title status. Fall back to eligible popular and continue-watching rows, preserving visual stability.

Evaluate title NDCG plus slate coverage/duplication, then online successful choice, satisfying viewing, early abandonment, and repeat visits. Choose profile- or household-level randomization according to interference risk.

Popular titles or series can dominate every row. Better individual titles do not guarantee a better page. Apply slate diversity and cross-row deduplication; shared profiles may need less aggressive personalization and stronger session context.

Define types, urgency, quiet hours, and cross-device duplicates. Route essential alerts separately. Optimize incremental valuable activity for optional messages, with opt-outs, interruption feedback, and retention guardrails.

Log candidates, send decisions, delivery/visibility, opens, dismissals, delays, and opt-outs. Unsent messages lack open labels. Randomized send/no-send holdouts are needed for causal uplift; handle delivery delays and selection bias.

Start with preferences, caps, and timing rules, then a response ranker. Train uplift models once adequate randomized data exists. Utility combines incremental activity and interruption cost under constraints, rather than sending everything with a high predicted open rate.

Deduplicate events, enforce TTLs and local time, and share state across devices. Use idempotent sends, recheck preferences at dispatch, and avoid replaying expired notifications. Fallback follows explicitly allowed rules.

Evaluate response calibration and uplift using randomized labels. Online, measure incremental activity, opt-outs, retention, and sends per user with stable holdouts, preserving essential notifications and user preferences.

Frequent messages can increase short-term returns while causing opt-outs. Add fatigue features and caps. Notifications may substitute for one another, so evaluate daily slates and total interruption cost, not only independent message scores.

Clarify languages, typo tolerance, trends, and personalization. Assume a 50 ms server p95 budget for practice. Optimize reduced typing followed by successful search, with misleading/unsafe suggestions and tail latency as guardrails.

Log prefixes, suggestion impressions, selections, final queries, and search outcomes, accounting for position/acceptance bias. Keep private or rare queries out of public candidates. Split temporally and test unseen prefixes.

Start with a trie/FST, frequency, and recency decay. Add locale, session, and query-quality ranking. Use bounded typo/semantic expansion, with safety eligibility separate from relevance scores.

Cache common prefixes, incrementally refresh trends, and atomically replace indexes. Use debouncing and request sequence IDs to discard stale responses. Fall back to cached suggestions; handle empty prefixes and IME composition separately.

Offline, measure MRR, Recall@K, useful-query coverage, and unsafe-candidate rate. Online, measure saved keystrokes, successful searches, reformulations, and latency rather than suggestion acceptance alone.

Popular queries may be noisy or manipulated, and short prefixes can reveal inappropriate information. Use privacy-conscious aggregation, candidate thresholds, and trend reliability checks. More personalization reduces cacheability and increases privacy complexity.

Clarify the surface, invitation frequency, and privacy settings. Optimize accepted, valuable connections, with ignores, blocks, reports, and recipient burden as guardrails. Enforce blocked/invisible relationships as hard exclusions.

Record exposures, invitations, acceptance, ignores, and downstream interaction. Acceptance is delayed, and no invitation is not rejection. Reconstruct graphs as of event time to prevent future-edge leakage; propagate privacy changes.

Start with common neighbors or Adamic–Adar, then graph embeddings/GNN retrieval and invite/conditional-accept ranking. Two-sided utility should not overwhelm popular recipients. Cold-start candidates can use voluntarily supplied, permitted context.

Precompute graph candidates and update new edges incrementally, bounding expansion around high-degree nodes. Recheck blocks, privacy, caps, and existing connections online, invalidating caches on revocation. Fall back to eligible candidates or an empty result.

Measure candidate Recall@K, acceptance calibration, and degree/cold-start slices. Online, measure useful connections and negative signals. Graph interference may require cluster randomization or spillover analysis beyond ordinary user A/B tests.

High-degree people can dominate exposure and receive excessive invitations; enforce recipient-capacity and diversity constraints. Mutual contacts can reveal sensitive relationships, so explanations must use permitted signals. Reduce recommendations when precision is low.

Separate spam, phishing, and malware actions. Track inbox spam prevalence, with legitimate-mail false positives, latency, and recoverability as guardrails. Personal preferences should not override explicit safety constraints.

Use spam/not-spam corrections, audits, and reputation signals; inaction is not a reliable label. Split by time and campaign to avoid duplicate leakage. Track label provenance and protect message content.

Start with rules plus TF-IDF logistic regression or Naive Bayes, then evaluate text encoders and header/link features. Set risk-aware thresholds with high cost for legitimate-mail false positives; calibrate on natural traffic.

Use fast models in delivery and heavier follow-up analysis where appropriate. Version models, rules, and reputation separately. Shadow and canary changes, preserving inbox recovery. Define quarantine behavior for service failures.

Measure PR-AUC, recall at very low FPR, and language calibration, using random audits for real error rates. Controlled rollouts track inbox spam, legitimate-mail recovery, and complaints rather than report counts alone.

Obfuscation, sender rotation, and poisoning cause drift; maintain new-campaign holdouts and audited labels. Stricter thresholds reduce spam but may hide important mail, making quarantine and recovery central tradeoffs.

Clarify authorization deadlines, amount distribution, verification actions, and review capacity. Optimize total decision cost including fraud, false declines, and friction, with legitimate approval, p99 latency, and appeal guardrails.

Join transaction IDs, devices, merchants, historical velocity, verification, and matured dispute/review labels. Define fraud carefully; not every dispute is fraud. Split by time and attack group, exclude post-decision information, and treat blocked outcomes as unobserved rather than automatically fraudulent.

Start with rules and GBDT; anomaly detection is an additional signal. Sequence/graph models need demonstrated incremental value. Calibrate risk and select actions using amount, false-decline cost, and review capacity rather than identical costs for every transaction.

Define event-time windows, deduplication, and out-of-order handling in streamed features. Align online features and training joins. Shadow/canary releases and use explicit risk-tier fallbacks on deadline failures. Retain reasons, versions, and auditability.

Evaluate PR-AUC, recall at fixed false-decline rate, amount-weighted loss, and calibration on matured labels. Within permitted bounded risk, compare legitimate approvals, losses, and review load online. Wait for outcome maturity and retain manual audits.

Attack drift, stale features, and poisoning can degrade performance. Monitor freshness, slice risk, and matured losses. Reducing false declines may increase fraud, so tune challenge paths and review capacity as well as thresholds.

Clarify global versus local trends, update frequency, and personalization. Optimize timely relevant event discovery, with manipulation, duplicate topics, misleading labels, and detection-delay guardrails.

Collect event time, language, region, author, text, and interactions. Deduplicate reshares and distinguish unique authors. Label topic merges and trend quality, retain seasonal baselines, and remove deleted or non-public data.

Start with regional/language window counts and growth rates, then add seasonal residuals, unique authors, reliability, and semantic clustering. Smooth tiny baselines and require minimum support so 1→10 events do not automatically dominate.

Use event-time windows, watermarks, and late updates. Partition aggregation/clustering and handle hot-key skew. Publish timestamped versioned lists; on stream failure, serve a clearly aged prior list or pause the affected region.

Replay events to measure precision@K, event coverage, cluster purity, and detection delay. Online, assess relevance, useful interactions, manipulation exposure, and duplicate topics; clicks do not establish reliability.

Bot bursts, ambiguous names, and cross-language duplicates distort trends. Use author diversity, reliability checks, entity/time disambiguation, and human fallback. Shorter windows react faster but need persistence thresholds to avoid instability.

Clarify advisory versus automatic pricing, revenue versus occupancy goals, horizon, and host constraints. Optimize matured net revenue per available night with cancellation, occupancy, adoption, and volatility guardrails.

Collect listing/date, availability, exposure, offered price, bookings, cancellations, and market context. Blocked/unlisted nights are not zero-demand observations. Booked-only data is selected, and events confound both prices and demand.

Start with seasonal comparable-listing medians. Model p(book | price, context), or a multi-day booking hazard, and optimize constrained expected net revenue including fees/cancellations. Observational association is not price elasticity; identification assumptions or controlled variation are required.

Generate horizon-based recommendations daily with event-driven updates. Version data, models, and decisions. Show explanations/uncertainty and host bounds. Use conservative comparable prices when data is sparse and recheck limits before publication.

Evaluate demand calibration, replayed revenue, and uncertainty coverage, while acknowledging backtests cannot prove counterfactual pricing gains. Consider market/listing clusters or switchbacks online, accounting for competition and booking horizons, and wait for cancellations/stays to mature.

New listings and events add uncertainty, while volatile prices hurt trust. Use intervals, rate-of-change bounds, and overrides. Short-term high prices may hurt occupancy or long-term value; evaluate the defined horizon rather than only the next booking.

Define origin/destination, prediction timestamp, arrival semantics, and refresh frequency. Optimize user-relevant ETA error with tail underestimation, jitter, and latency guardrails; waiting and travel models can differ.

Use GPS, map matching, route, time, traffic, and arrival events. Labels are remaining time from the prediction timestamp. Canceled trips are censored, not ordinary completions. Split by time/city and exclude future traffic.

Start with segment travel times plus turning/waiting costs, then a GBDT residual model before graph/sequence approaches. Quantile losses estimate p50/p90 and asymmetric uncertainty; check for quantile crossing.

Handle delayed GPS and outliers in traffic streams, caching segment features. Retrain periodically and monitor holidays/incidents. Fall back to routing estimates on timeout and smooth display updates without hiding real delays.

Measure MAE, p90 absolute error, quantile calibration, and city/route/weather slices; MAPE is unstable for very short trips. Online, track realized error, cancellations, and trust, considering selection effects when displayed ETAs change behavior.

GPS drift, closures, rare incidents, and rerouting cause errors. Use freshness checks, anomaly handling, intervals, and route refreshes. Frequent updates add cost and jitter, so choose frequency using experience metrics.

Clarify sensors, operational design domain, weather/roads, output types, and end-to-end deadlines. Scope this to perception rather than a complete autonomous-driving system. Success combines scenario risk, miss rates, and worst-case latency.

Collect synchronized multi-view/temporal data, calibration parameters, 3D boxes/lanes/occupancy labels, and hard scenarios. Split by drive, geography, and time to avoid adjacent-frame leakage. Track label uncertainty and synthetic-data domain shift.

Start with a validated detector and tracker, then compare temporal fusion, multitask spatial representations, and applicable sensor fusion. Output state and uncertainty; low confidence must not be interpreted downstream as “no obstacle.”

Training may be distributed, but edge deployment requires thermal, memory, and deterministic-deadline profiling. Version models/calibration, shadow first, and expand under a validation plan. Sensor failures or out-of-domain conditions trigger predefined system-level degraded behavior.

Evaluate 3D detection, tracking, occupancy, and uncertainty by distance/occlusion, emphasizing consequential scenarios such as pedestrians and construction. Validate through replay, simulation, closed-course testing, and controlled shadowing. Aggregate mAP does not establish road safety.

Glare, unusual obstacles, calibration drift, and domain shifts are central failures. Maintain scenario suites, active hard-case selection, and sensor-health checks. Larger models can undermine deadline reliability; evaluate improvements with system-level risk.

Bound the action scope. Ask whether the assistant recommends or buys, which merchant APIs exist, authentication, currency, and delivery region. For “a lightweight laptop under $500,” clarify intended use and required specifications. Optimize supported task completion, with incorrect prices, unavailable items, misleading sponsorship, tool errors, and cost as guardrails.

Connect dialogue, catalog, and tools. Log turn/request IDs, consented preferences, candidate/display sets, product versions, price timestamps, tool parameters/results, citations, and sponsorship flags. Labels include relevance, constraint satisfaction, factual accuracy, tool success, completion, and cancellations/returns. Split by time, product family, and user where appropriate.

Start with a workflow before an agent. Baseline constraint extraction, catalog search, and template comparison. Add hybrid/multimodal retrieval, ranking, and a bounded tool controller. Plan dynamically only when clarification or tool follow-up is needed. Enforce hard specifications, budget, and availability before ranking by relevance and utility; sponsored items must also meet eligibility/relevance thresholds.

Use explicit tool contracts. search_products returns identifiers and traceable metadata; get_price_stock(product_id, region) returns currency, price, stock, and timestamp. Enforce timeouts, schema validation, retry bounds, and circuit breakers. If buying is supported, show the final price/details for confirmation before executing with an idempotency key. Merchant text cannot authorize side effects.

Evaluate components and the whole task. Offline tests cover constraints, product Recall/NDCG, price freshness, claim groundedness, tool success, and sponsorship labeling. Include tool failures, conflicting specs, no valid products, and prompt injection. Online, track task success, satisfaction, cancellations/returns, cost, and latency, separating sponsored traffic and using blinded human review.

Price changes or stock-outs: revalidate and expose timestamps. Tool failure: do not invent values; show verified information and the gap. No eligible products: explain conflicting constraints and ask which can change. More autonomous agents add tool calls, cost, and uncertainty; use them only when measurable task gains justify the complexity.