閱讀方式：先看架構，再沿著曝光 → 標籤 → 特徵 → 模型 → 評估 → 上線閱讀。最後用 45 分鐘口說流程與反例檢驗設計。範例中的數字都是假設；程式碼入口已核對，沒有宣稱已訓練、部署或量測成效。
How to read: start with architecture, then follow impressions → labels → features → models → evaluation → serving. Finish with the 45-minute rehearsal and counterexamples. Numerical examples are assumptions; code entry points were inspected, with no claim of training, deployment, or measured results.
先定義預測目標與決策邊界Define the prediction target and decision boundary
設計社群動態中的廣告點擊預測。第一版負責為合格候選輸出可信的 pCTR；候選檢索、競價、pacing 與使用者體驗限制是相鄰元件，必須說明契約。
Design click prediction for ads in a social feed. The first version outputs trustworthy pCTR for eligible candidates. Retrieval, auction, pacing, and user-experience constraints are adjacent components with explicit contracts.
貫穿全篇的假設Running assumptions
使用者打開 feed，系統需為一組合格廣告預測「實際曝光後 W 內的有效落地頁點擊」。範例假設每秒 1,000 個請求、每次精排 200 個廣告、精排服務 p99 預算 80 ms。所有流量、窗口、模型尺寸與 SLO 都是設計練習的假設，應向面試官確認；不是 Meta 公開規格。
A user opens the feed; predict a valid landing-page click within W after an actual impression for each eligible ad. Illustrative assumptions: 1,000 requests/s, 200 candidates per ranking request, and an 80 ms p99 ranking-service budget. Traffic, windows, model sizes, and SLOs are exercise assumptions to clarify, not published Meta specifications.
pCTR = P(valid click within W | user u, ad a, context c, impression)
Decision unit: one rendered impression
Serving output: pCTR + model_version + feature_version + freshness status| 釐清問題Clarification | 本篇選擇Working choice | 對設計的影響Design implication |
|---|---|---|
| 什麼算 click？What counts as a click? | 有效落地頁點擊；重複／無效事件有明確規則。A valid landing-page click, with explicit duplicate/invalid-event rules. | 展開圖片、誤觸、購買不能混為同一個 label。Image expansion, accidental taps, and purchases are distinct targets. |
| 在哪裡預測？Where is the prediction used? | 候選精排，條件包含已知版位。Candidate ranking conditioned on a known placement. | 多個 slot 要逐 slot 評估或另建曝光模型；不能偷用未知最終位置。For multiple slots, score per slot or model exposure separately; do not leak the unknown final position. |
| 如何定義成功？What is success? | 離線先看機率品質，線上預先選定主要價值指標。Probability quality offline, a prespecified value metric online. | CTR、收入、轉換與長期滿意度不同；設負回饋、延遲與成本護欄。CTR, revenue, conversion, and long-term satisfaction differ; guard negative feedback, latency, and cost. |
| 需要哪些資料？What data is available? | 經允許使用的互動、creative、版位及版本化紀錄。Permitted interactions, creatives, placements, and versioned logs. | 明確定義資料保留、刪除傳遞、線上可用性與缺值語意。Specify retention, deletion propagation, serving availability, and missing-value semantics. |
以 CPC 為例，bid × pCTR 可作預期點擊出價價值的一部分，但不是完整拍賣公式。投放還受品質、預算、pacing 與 eligibility 約束。先保持預測服務輸出可解釋的機率，再由下游決定效用；不要把任意商業權重混進 label 後仍稱輸出為 pCTR。
For CPC ads, bid × pCTR can contribute to expected bid value, but is not a complete auction formula. Delivery also depends on quality, budgets, pacing, and eligibility. Keep the prediction service's output interpretable as a probability and let downstream logic define utility. Arbitrary business-weighted labels do not automatically preserve pCTR semantics.
90 秒開場如何回答？How do you open in 90 seconds?
我先確認 click、曝光與時間窗，將核心任務定義為合格候選的條件點擊機率。我會先畫線上 scoring 與離線學習閉環，再定義曝光／點擊 join、時間一致特徵和成熟標籤。建立 LR baseline，依交互與行為歷史假說增加模型能力；用未來時段的 log loss、校準和 slices 驗證。最後討論 A/B、延遲與資料故障時的止損。
I will clarify clicks, impressions, and the prediction window, then define the core task as a conditional click probability for eligible candidates. I will draw serving and learning paths, define impression-click joins, point-in-time features, and mature labels, and establish an LR baseline. I will add capacity for explicit interaction or history hypotheses, validate on future periods with log loss, calibration, and slices, then cover A/B tests, latency, and safeguards against data failures.
線上決策與離線學習閉環Serving decisions and the learning loop
圖上每一條箭頭都要說得出 payload、時間與錯誤行為。訓練不是 request 的同步相依；模型品質卻依賴 request 日誌的正確性。
For every arrow, explain the payload, timing, and failure behavior. Training is not a synchronous request dependency, but model quality depends on correct request logging.
| 邊界Boundary | 資料契約Data contract | 驗證與降級Validation / fallback |
|---|---|---|
| Retrieval → rankerRetrieval → ranker | request ID、candidate ad IDs、placement、eligibility version。Request ID, candidate ad IDs, placement, eligibility version. | 候選可能被預算／政策排除；未呈現者不能當作非點擊。Budgets/policy may remove candidates; unshown candidates are not observed non-clicks. |
| Features → modelFeatures → model | Dense／sparse／sequence tensors、missing mask、schema、資料年齡。Dense/sparse/sequence tensors, missing masks, schema, feature ages. | Schema 不符拒用該 bundle；允許的缺值走訓練過的 fallback。Reject incompatible bundles; permitted missingness uses a trained fallback. |
| Ranker → logsRanker → logs | 預測機率、模型版本、experiment arm；另由 render event 確認曝光。Predictions, model version, experiment arm; a render event separately confirms exposure. | 分清 scored、selected、rendered 三種事件；監控各段流失。Separate scored, selected, and rendered events; monitor attrition. |
| Trainer → releaseTrainer → release | 模型、encoder、vocabulary／hash、normalizer、calibrator、評估報告。Model, encoder, vocabulary/hash, normalizer, calibrator, evaluation report. | 相容性檢查、shadow、canary、可回滾版本；避免混搭。Compatibility gates, shadow, canary, rollback versions; avoid mixed artifacts. |
ScoreRequest {
request_id, user_key, placement, event_time,
candidate_ad_ids[], deadline_ms, experiment_arm
}
ScoreResponse {
scores: [{ad_id, pctr, status}],
bundle_version, feature_schema, feature_age_ms
}
# status examples: OK / MISSING_OPTIONAL / FALLBACK / UNSCORABLE
# Only rendered impressions become click-label opportunities.容量先做算術：1,000 requests/s × 200 ads = 200,000 candidate scores/s。不能據此宣稱需要幾張 GPU；先量 feature lookup、序列長度、batch、模型 FLOPs、memory bandwidth 與尾延遲。保留 request-level batching，避免 200 次小 RPC；critical path 用 trace 量測，階段 p99 不可直接相加。
Start with workload arithmetic: 1,000 requests/s × 200 ads = 200,000 candidate scores/s. This does not determine GPU count. Measure feature lookup, sequence lengths, batching, FLOPs, memory bandwidth, and tail latency. Batch candidates per request instead of making 200 tiny RPCs. Trace the critical path; stage p99 values are not additive.
資料準備：曝光、延遲回饋與時間切分Data preparation: exposure, delayed feedback, and temporal splits
一筆訓練資料是一次有效曝光，以及那次決策當下已知的特徵。先確保標籤與時間正確，才討論模型。
One example is one valid impression with features available at that decision. Establish label and time correctness before choosing a model.
impression_id, request_id, user_key, ad_id, placement
impression_event_time, impression_ingest_time
feature_snapshot_ref, feature_schema, model_version
click_event_time, click_ingest_time, attribution_rule_version
label_window_end, label_status, label_version
sampling_probability, logging_policy_version, experiment_arm在曝光 ID 上做有時間限制的 keyed join；先去重，再依固定歸因規則建立 label。label=0 需要觀測窗結束且相關資料源健康、處理進度足夠；不是「目前沒有 join 到 click」。事件遺失無法靠等久一點修好，應隔離受影響時段並回補。晚到修正需更新同一個 example version，避免把同一曝光當兩個互相矛盾的訓練樣本。
Use a bounded keyed join on impression identity, deduplicate, and apply a fixed attribution rule. A zero label requires a completed window, healthy sources, and sufficient processing progress; it is not merely the absence of a click so far. Waiting does not repair lost events: quarantine and backfill affected periods. Late corrections update the same example version rather than creating contradictory duplicate examples.
| 資料切分Split | 示範設計Illustrative design | 防洩漏規則Leakage control |
|---|---|---|
| TrainTrain | 先前連續數日；重播固定訓練截止點。Earlier contiguous days, replayed at a fixed training cutoff. | 只納入截止時已成熟的 labels；歷史特徵不能讀未來 click。Use labels mature by the cutoff; history features cannot read future clicks. |
| ValidationValidation | 後續時段做 HPO、early stopping。A later period for HPO and early stopping. | Vocabulary、scaler、feature selection 只 fit train。Fit vocabulary, scalers, and feature selection on train only. |
| CalibrationCalibration | 另一段代表目標分布的成熟資料。Another mature period representative of target traffic. | 只 fit calibrator；不能用 final test 挑校準器。Fit the calibrator only; do not choose it using final-test labels. |
| Final testFinal test | 最後的未碰觸未來時段，重複 rolling-origin backtests。An untouched future period; repeat rolling-origin backtests. | 模擬當時發布／取特徵條件。另報新 user／ad slice。Reproduce release and feature-availability conditions; report new-user/ad slices. |
為什麼只有 event_time < impression_time 還不夠？Why is event_time < impression_time insufficient?
某個行為雖發生得早，卻在曝光後才被系統收到。若 offline aggregate 使用它，而 online 當時不可能知道，就有 availability leakage。理想用 serving snapshot；若重建特徵，同時檢查 event time 和資料可用時間，並以 replay 比對線上輸入。
An event may occur earlier but arrive after the impression. Including it offline leaks information unavailable to serving. Prefer logged serving snapshots; when reconstructing features, check both event time and availability time, and compare replayed tensors with online inputs.
要不要完全按 user 切分？Should users be completely disjoint across splits?
主要評估要模擬未來服務，故以時間切分，同一使用者可以再次出現；這本身不是洩漏。另用 disjoint users／ads 評估冷啟動，並說明它回答的是不同問題。所有標籤衍生統計仍須嚴格 point-in-time。
Use temporal splits to represent future serving; recurring users are not inherently leakage. Add disjoint-user/ad evaluations for cold start and explain that they answer a different question. Label-derived statistics must remain strictly point-in-time.
抽樣、曝光偏差與機率校準Sampling, exposure bias, and probability calibration
類別不平衡、計算成本與曝光偏差是三個問題。negative sampling 可以省算力，但不會自動修好選擇偏差。
Class imbalance, compute cost, and exposure bias are distinct problems. Negative sampling saves compute; it does not automatically fix selection bias.
q = P(y=1 | x, retained)
s1 = P(retained | y=1), s0 = P(retained | y=0)
q/(1-q) = (s1/s0) * p/(1-p)
p = s0*q / [s1*(1-q) + s0*q]
logit(p) = logit(q) + log(s0/s1)
Example: q=0.10, s1=1, s0=0.10 → p≈0.010989抽樣如何改變 pCTR？How does sampling change pCTR?
修正後的機率 pCorrected probability p:
q 必須介於 0 與 1；保留率必須大於 0 且不超過 1。q must be between 0 and 1; retention rates must be greater than 0 and at most 1.
互動計算只示範 Bayes 修正；假設保留率只依 label、q 在抽樣分布已校準，且未另外用 inverse weighting 修正。
This calculator demonstrates Bayes correction only: retention depends on the label, q is calibrated on the sampled distribution, and inverse weighting has not already corrected the model.
這是 Bayes rule 的推導，前提是保留率只依 label（或已知每個 x 的保留率），而 q 確實代表抽樣分布。另一條路是使用 1/s_y 的 inverse-probability weighted BCE 估原始風險；兩條路不能不加分析地疊用，否則重複修正。Hard-negative mining 依 x 選樣，不能套一個全域負例率就宣稱已還原。
This follows from Bayes' rule when retention depends only on the label, or when retention conditional on x is known, and q represents the sampled distribution. Alternatively, inverse-probability weighted BCE using 1/s_y estimates original risk. Do not blindly combine weighting with posterior correction: that can double-correct. Hard-negative mining depends on x and cannot generally be undone with one global negative-retention rate.
| 問題Problem | 處理方案Approach | 不可忽略的假設Required caveat |
|---|---|---|
| 負例很多Many negatives | 先試 BCE；若需降成本，再記錄保留率並抽樣。Start with BCE; if cost warrants sampling, record retention probabilities. | Accuracy 不適合主要評估；加 class weight 會改變最優 posterior。Accuracy is a poor primary metric; class weights alter the optimal posterior. |
| 舊策略只曝光部分廣告Old policy exposes only some ads | 在安全的合格集合探索，記錄 action／slate propensity。Explore within eligible sets and log action/slate propensities. | 未曝光者沒有 click outcome；IPS 需要 support 與正確 propensity。Unshown ads lack click outcomes; IPS needs support and correct propensities. |
| 位置與 examinationPosition and examination | 依 slot 定義條件目標；用隨機化資料檢驗位置效應。Define the target conditional on a slot; use randomized data to study position effects. | position feature 不能消除所有 confounding；觀察 CTR 不等於 intrinsic relevance。A position feature does not remove all confounding; observed CTR is not intrinsic relevance. |
| 校準漂移Calibration drift | 獨立資料 fit sigmoid／isotonic calibrator，再分群驗證。Fit a sigmoid/isotonic calibrator on separate data, then validate by slice. | 低樣本 slice 需 shrinkage；校準器不能補救缺失標籤或錯誤 feature。Small slices need shrinkage; calibration cannot repair missing labels or broken features. |
為什麼不直接用 SMOTE 或 focal loss？Why not immediately use SMOTE or focal loss?
任意插值類別 ID 或 embeddings 不一定代表有效曝光。Focal loss 重新加權難例，可能改善某些目標，但不能假設輸出仍是原始 CTR。先以正確標籤、BCE、抽樣修正建立 baseline；任何新 loss 都需比較校準、排序、slice 與成本。
Interpolating categorical IDs or embeddings need not create valid impressions. Focal loss reweights hard examples and may help some objectives, but its output is not automatically the original CTR. Establish correct labels, BCE, and sampling correction first; evaluate any alternative loss on calibration, discrimination, slices, and cost.
特徵工程：訊號、時序與可用性Feature engineering: signal, time, and availability
每個 feature 都要有定義、來源、可用時間、缺值語意與實驗證據。增加一個特徵，同時增加訓練和 serving 的相依。
Every feature needs a definition, source, availability time, missing-value semantics, and experimental evidence. A feature creates dependencies in both training and serving.
| 特徵群Feature family | 例子與假說Examples / hypothesis | 處理與驗證Processing / validation |
|---|---|---|
| UserUser | 長期類別偏好、最近 session 行為、廣告疲勞。Long-term category affinity, recent session behavior, ad fatigue. | 區分新 user 與資料缺失；多窗口、frequency caps 相關計數。Separate new users from missing data; use multiple windows and exposure counts. |
| Ad／creativeAd / creative | campaign／advertiser／category IDs，文字與圖片表示。Campaign/advertiser/category IDs and text/image representations. | Creative 版本更新要重算表示；新 ID 可用內容特徵與上層 prior。Recompute on creative changes; new IDs use content and hierarchical priors. |
| ContextContext | 版位、裝置、時間、當前 session。Placement, device, time, current session. | 只使用 scoring 當下已知資訊；不同 surface 分群評估。Use only decision-time information; evaluate surfaces separately. |
| AggregatesAggregates | ad／advertiser 的 1h、1d、7d 曝光與 CTR。Ad/advertiser impressions and CTR over 1h, 1d, and 7d. | 範例窗口需驗證；計數／分母／資料年齡一起輸入，避免低樣本過度自信。Validate these illustrative windows; include counts, denominators, and age to handle low support. |
| InteractionsInteractions | user-interest × ad-category、device × creative-format。User-interest × ad-category, device × creative-format. | LR 手動 crosses 對照 FM／cross network；固定 feature set 比較。Compare manual LR crosses with FM/cross networks on fixed feature sets. |
| SequenceSequence | 曝光前的 item、action、time-gap、內容向量。Pre-impression item, action, time gap, content vectors. | 固定 padding／mask／ordering；避免把當前曝光的 click 放進歷史。Define padding, masks, ordering; exclude the target impression's click from history. |
smoothed_CTR = (clicks + alpha) / (impressions + alpha + beta)
prior_mean = alpha / (alpha + beta)
All counts: event_time < impression_time AND available_by(impression_time)
Numeric processing: train-fit clipping / log1p / normalization + missing indicator
Sparse processing: vocabulary + OOV OR versioned hashing + collision analysis平滑 prior 只用 train 資料估計，可按 advertiser／category 做有足夠支持的階層共享。缺失不是零：沒有歷史與確實觀測到零次互動需分開表示。Hashing 可限制 vocabulary，但碰撞會混合不同 ID；固定 seed 與 bucket 規則，量測頻率分布與碰撞敏感度。不要每次 retrain 改 ID 映射卻沿用舊 embedding。
Estimate smoothing priors from training data only, with hierarchical sharing across sufficiently supported advertisers/categories. Missing is not zero: no history differs from an observed zero count. Hashing bounds vocabulary but collisions mix identities; version the seed/bucket rules and measure frequency and collision sensitivity. Never change ID mappings while blindly reusing old embeddings.
新的廣告完全沒有 click，如何預測？How do you score an ad with no clicks?
以 creative 內容、類別、advertiser prior 與 context 做初始估計，ID 路徑用 OOV／新 ID 初始化。看曝光量分層的校準與 learning curve，評估是否需要有限探索。不能把低樣本高 CTR 直接視為高品質，也不能把未知廣告永久壓低而無法學習。
Start with creative content, category, advertiser priors, and context, using OOV or new-ID initialization for the ID path. Evaluate calibration and learning curves by exposure count, and consider bounded exploration. A high CTR from a tiny sample is not reliable quality; permanently suppressing unknown ads also prevents learning.
模型開發：從可信 baseline 到交互模型Model development: from a credible baseline to interactions
先提出可被推翻的假說，再選模型。一次改動 feature、sampling 和模型，很難知道改善來自哪裡。
State a falsifiable hypothesis before choosing a model. Changing features, sampling, and architecture together makes gains hard to attribute.
p_i = sigmoid(z_i)
BCE = -(1/N) * sum[y_i*log(p_i) + (1-y_i)*log(1-p_i)]
LR: z = b + w^T x
DeepFM: z = linear(x) + FM_pair_interactions(x) + MLP(embeddings(x))
DCNv2 cross layer: x_next = x0 * (W*x + b) + x # * is elementwise
Use numerically stable BCE-with-logits; specify sampling weights separately.| 候選模型Candidate | 歸納偏好與用途Inductive bias / purpose | 代價與失敗條件Cost / failure condition |
|---|---|---|
| LR／FTRLLR / FTRL | 稀疏特徵與人工 crosses；FTRL 是 optimizer，不是另一種網路。Sparse features and manual crosses; FTRL is an optimizer, not a separate network. | 交互需特徵設計；低成本 baseline 可揭露資料問題。Interactions need feature design; a cheap baseline exposes data issues. |
| GBDT → LRGBDT → LR | 樹葉指示特徵接 LR；可作 tabular 非線性 baseline。Tree-leaf indicator features feed LR as a nonlinear tabular baseline. | 樹更新會改 leaf feature space；發布與 warm start 要保持相容。Tree updates change leaf-feature space; releases and warm starts need compatibility. |
| DeepFMDeepFM | 共享 embeddings，學二階 FM 與高階 MLP 交互。Shared embeddings support second-order FM and higher-order MLP interactions. | 不會自動處理時間順序；參數與調參預算要公平。Does not inherently model temporal order; compare parameter and tuning budgets fairly. |
| DCNv2DCNv2 | 顯式 cross layers；low-rank 選項可控制成本。Explicit cross layers; low-rank options control cost. | 須量 expressiveness／rank／latency 的曲線，不保證每個資料集勝出。Measure expressiveness/rank/latency curves; no universal win is implied. |
| DLRMDLRM | Dense bottom MLP、sparse embeddings、interaction、top MLP。Dense bottom MLP, sparse embeddings, interactions, top MLP. | 適合練 embedding memory／communication；不是完整投放系統。Useful for embedding memory/communication practice, not a complete delivery system. |
初始選擇：LR＋必要 crosses，之後固定資料、特徵與 split 比較 DeepFM 或 DCNv2。記錄 seeds、HPO 次數、training wall time、模型／optimizer 記憶體和 serving 成本。先看 loss 曲線和切片：train 好、validation 差可能是稀疏 ID 過擬合或 drift；兩者都差則查標籤、特徵、容量與 optimizer。
Start with LR and necessary crosses, then compare DeepFM or DCNv2 using fixed data, features, and splits. Record seeds, HPO trials, training wall time, model/optimizer memory, and serving cost. Inspect loss curves and slices: strong training but weak validation suggests sparse-ID overfitting or drift; poor performance on both calls for label, feature, capacity, and optimizer checks.
多目標延伸：不要混淆條件機率Multi-objective extension: preserve conditioning
若產品要看 conversion，可另建 pCVR 與相關 head，但需定義是 P(conversion | click, impression) 還是 P(conversion | impression)。只有 conversion 必經定義中的 click 且窗口一致時，才能按機率鏈式法則寫 pCTCVR = pCTR × pCVR。View-through conversions 需要不同事件建模；不同 label maturity 也會影響訓練。
If conversion matters, add a corresponding head with an explicit target: P(conversion | click, impression) or P(conversion | impression). The chain-rule form pCTCVR = pCTR × pCVR applies only when conversion requires the defined click and windows align. View-through conversions require different event modeling, and label maturity differs across objectives.
為什麼不用更大的 Transformer 一步到位？Why not start with a larger Transformer?
如果沒有可靠 sequence、時間資訊或充足資料，容量可能只記住熱門 ID。先量 LR／interaction baseline 的瓶頸，提出序列順序或長期意圖假說，再比較 shuffled-history、pooled-history 與 ordered-history 對照。收益需在公平調參及延遲預算下成立。
Without reliable sequences, timing, and enough data, capacity may memorize popular IDs. Diagnose the LR/interaction baseline, then test an order or long-term-intent hypothesis using shuffled, pooled, and ordered histories. Gains must survive fair tuning and latency constraints.
行為序列與可共用的使用者表示Behavior sequences and reusable user representations
同一使用者對不同廣告可能有不同相關歷史。長期 user representation 與當前 candidate 的 matching 應有清楚分工。
Different ads can make different parts of a user's history relevant. Separate long-term user representation from matching to the current candidate.
DIN 用候選廣告條件化的 local activation 聚合歷史，提供一個可讀的起點。候選感知不等於時間順序建模：若只把歷史當 set，交換事件順序不一定改變結果。要主張 recency／order 有用，加入時間差、position 或相應序列結構，並做控制實驗。
DIN uses candidate-conditioned local activation to aggregate history and is a readable starting point. Candidate awareness is not temporal-order modeling: treating history as a set need not change under permutation. Claims about recency or order require time gaps, positions, or a suitable sequence structure, plus controlled experiments.
h_j = embed(item_j, action_j, time_gap_j, content_j)
r_a = sum_j g(h_j, e_ad) * h_j # DIN-style candidate-conditioned pooling
z = f(user_cached, recent_readout_a, ad_features, context)
pCTR = sigmoid(z)
# g need not be softmax-normalized in DIN; specify the actual implementation.
# Padding tokens are masked; no event after the decision enters the history.| 假說Hypothesis | 對照實驗Controlled comparison | 除了 AUC 還看什麼Beyond AUC |
|---|---|---|
| 近期事件有額外訊號Recent events add signal | 固定長期向量，比較有／無近期支路。Hold long-term vectors fixed; toggle the recent-event branch. | 資料年齡分桶、熱門事件期、cache miss 成本。Age buckets, trend periods, cache-miss cost. |
| 順序有用Order matters | 相同事件集合／長度，原順序對 shuffled；保留相同時間特徵定義。Same events/length: ordered versus shuffled, with identical time-feature definitions. | 確認 permutation 未同時破壞其他訊號；比較公平容量 baseline。Check that permutation does not confound other signals; match baseline capacity. |
| 長歷史有用Longer histories help | 相同資料 cutoff 下測多個長度與 event mixtures。Compare lengths and event mixtures at identical cutoffs. | Memory、p99、training cost、低活躍與冷啟動 slices。Memory, p99, training cost, low-activity and cold-start slices. |
| 內容向量能泛化Content vectors generalize | ID-only、content-only、combined；按 ad age 分群。ID-only, content-only, combined; slice by ad age. | 新 creative 的效益、encoder 更新成本、缺圖／缺文健壯性。New-creative gains, encoder refresh cost, missing-modality robustness. |
快取需包含 encoder version、schema、產生時間與最後事件 cutoff。線上 ranker 必須與向量空間相容；舊向量超過允許 age 就重算或走已評估的降級。候選相依 attention 不能跨廣告快取成同一個結果；可以共用的是 user-only encoding、部分 keys／values 或非同步長期表示。
Cache encoder version, schema, generation time, and event cutoff. The ranker must be compatible with the embedding space; embeddings beyond allowed age require refresh or an evaluated fallback. Candidate-dependent attention outputs cannot be reused as one universal result. User-only encoding, some keys/values, and asynchronous long-term representations may be shared.
訓練流程、分散式成本與安全更新Training, distributed cost, and safe updates
先建立可重現的批次流程，再用 freshness 實驗決定是否值得近線或即時更新。快不等於可信。
Build a reproducible batch pipeline first, then use freshness experiments to justify nearline or online updates. Faster updates are not automatically trustworthy.
validate_sources_and_label_health()
examples = build_mature_point_in_time_examples(cutoff)
train, val, calibration, test = temporal_split(examples)
preprocessor.fit(train)
model = train_with_logged_config(train, val, sampling_policy)
calibrator = fit_on_representative_holdout(model, calibration)
report = evaluate_frozen_bundle(model, preprocessor, calibrator, test)
if passes_quality_and_serving_gates(report):
register_bundle(data_manifest, code_revision, all_artifacts)
shadow_then_canary()| 設計面向Decision | 起始方案Initial choice | 何時改變When to revisit |
|---|---|---|
| 更新頻率Refresh cadence | 固定批次重訓；獨立控制 feature freshness。Scheduled batch retraining; control feature freshness separately. | 重播不同 model age／feature age 的品質曲線，避免混淆收益。Replay model-age and feature-age curves separately to identify gains. |
| Sparse parametersSparse parameters | 按 cardinality 估 embedding、gradient、optimizer states。Budget embeddings, gradients, and optimizer states by cardinality. | Table／row／column sharding，處理 skew、hot IDs 與 all-to-all。Consider table/row/column sharding, skew, hot IDs, and all-to-all traffic. |
| Dense parametersDense parameters | 資料平行 baseline；實測 batch、精度與 throughput。Data-parallel baseline; measure batch size, precision, throughput. | 通訊或 activation memory 成瓶頸再調整；低精度需驗品質。Adapt when communication/activations bottleneck; validate lower precision. |
| Warm startWarm start | 保留相容 ID 映射與 schema，記錄 optimizer 狀態。Preserve compatible ID mappings/schema and track optimizer state. | 新 feature／encoder／loss 改變時重新驗證，必要時重建模型。Revalidate new features, encoders, or losses; rebuild when needed. |
| CheckpointCheckpoint | 保存模型、optimizer、資料進度、RNG 與 embedding metadata。Save model, optimizer, data progress, RNG, and embedding metadata. | 演練中斷恢復，避免漏資料、重複更新或版本失配。Rehearse recovery to detect skipped data, duplicate updates, or mismatches. |
Illustrative raw embedding memory:
100,000,000 IDs * 64 dimensions * 4 bytes = 25.6 GB
This excludes gradients, optimizer states, metadata, replicas and activations.
Dense compute, embedding capacity and communication require separate budgets.Online learning 增加資料速度，也縮短錯誤標籤污染模型的時間。設資料健康 gate、限制發布頻率、保留 last-known-good bundle，且先將訓練與 serving 更新解耦。Replay 消費要定義 offset 與去重策略；恢復時的重複樣本會改變權重，即使 pipeline 沒有報錯。
Online learning increases freshness and the speed at which bad labels contaminate a model. Gate on data health, bound release frequency, preserve a last-known-good bundle, and decouple training from serving updates. Define offset and deduplication policies for replay: duplicate examples after recovery change effective weights even if the pipeline reports success.
GPU utilization 很低，是否應換更大模型？Low GPU utilization: should the model get larger?
先 profile input decode、embedding lookup、通信、CPU／GPU copy、batch waiting 和 kernels。Utilization 低可能是 I/O 或 all-to-all bound，不是模型太小。分開報 examples/s、request latency 與品質；用實測瓶頸選改善方式。
Profile input decoding, embedding lookup, communication, CPU/GPU transfers, batch waiting, and kernels. Low utilization can be I/O or all-to-all bound, not evidence of insufficient model size. Report examples/s, request latency, and quality separately, and optimize the measured bottleneck.
離線評估：辨識力、校準與可信比較Offline evaluation: discrimination, calibration, credible comparisons
同一個 AUC 可以對應不同的機率尺度；一個整體校準比例也可能掩蓋互相抵消的分群錯誤。
The same AUC can coexist with different probability scales; one aggregate calibration ratio can hide offsetting slice errors.
LogLoss = mean[-y*log(p) - (1-y)*log(1-p)]
NE_eval = LogLoss / H(mean(y_eval))
H(r) = -r*log(r) - (1-r)*log(1-r)
Calibration ratio = sum(p) / sum(y) # ideal: 1; undefined if no clicks
Brier = mean((p-y)^2)
NE can exceed 1. For all-positive/all-negative sets, H=0: do not report NE.本篇公式採 evaluation empirical CTR 作 entropy 分母，明確標為 NE_eval。也可用 train CTR 定義 reference predictor 並報相對 log loss，但必須寫清楚分母與期間，不能把不同定義的 NE 混比。跨時段 base rate 改變時，同時看 raw loss、base rate 與分群結果；normalized 指標不能保證不同流量可直接比較。
The formula here uses evaluation-set empirical CTR in the entropy denominator and labels it NE_eval. Another convention uses a train-CTR reference predictor and reports relative log loss. State the denominator and period; do not compare different NE conventions as if identical. Across base-rate shifts, inspect raw loss, base rate, and slices too; normalization does not make all traffic directly comparable.
| 面向Dimension | 要報告的內容Report | 常見誤判Pitfall |
|---|---|---|
| DiscriminationDiscrimination | ROC-AUC；必要時 PR-AUC，列出 prevalence。ROC-AUC; optionally PR-AUC, with prevalence. | 排序不變的單調轉換可能不改 AUC，卻大幅改變預期價值。A monotone transform can preserve AUC while changing expected values substantially. |
| CalibrationCalibration | Reliability plot、預測／實際 click、Brier／loss、各 bin 樣本量。Reliability plots, predicted/observed clicks, Brier/loss, counts per bin. | ECE 依 bin 選擇；global ratio=1 不保證 conditional calibration。ECE depends on bins; global ratio=1 does not imply conditional calibration. |
| SlicesSlices | 新 user／ad、ad age、曝光量、surface、device、feature age。New users/ads, ad age, exposure count, surface, device, feature age. | 低樣本區間需 uncertainty；多重切片探索要另做確認。Small slices need uncertainty; exploratory slice findings need confirmation. |
| UncertaintyUncertainty | 同一 test 上 paired comparison；user cluster／time blocks 依相關結構選。Paired comparisons on the same test; use user clusters/time blocks as appropriate. | 曝光彼此相關，不能直接假設每列獨立；seed variance 與資料 variance 不同。Impressions are correlated; seed variance and data-sampling variance differ. |
| Serving parityServing parity | 用相同輸入重播預處理、prediction、calibration 與缺值分支。Replay preprocessing, predictions, calibration, and missing-feature branches. | Notebook 評估的模型與實際 bundle 不同，離線提升可能消失。Notebook and deployed bundles can differ, erasing offline gains. |
AUC 提升 0.001，要發布嗎？AUC improves by 0.001: should you ship?
這只是一個訊號。先看 paired uncertainty、rolling windows、log loss、校準與重要 slices，再量模型成本與服務差異。若一致改善且符合護欄，進入小流量受控實驗；不能從離線 AUC 直接推論收入收益。
That is one signal. Check paired uncertainty, rolling windows, log loss, calibration, key slices, and serving cost/parity. Consistent gains within guardrails justify a controlled small-traffic experiment; offline AUC alone does not establish revenue lift.
離線反事實評估的邊界Limits of off-policy evaluation
IPS／SNIPS／doubly robust 方法需要明確 logging policy、overlap 和 outcome 定義。舊策略從未曝光的廣告沒有可靠 support；極小 propensity 產生高 variance，clipping 又引入 bias。預算與競價的動態效果也不會因一個 estimator 自動重現。把它作篩選與診斷工具，不能取代所有線上實驗。
IPS, SNIPS, and doubly robust methods require a specified logging policy, overlap, and outcome. Ads never exposed by the old policy lack support; tiny propensities create high variance, while clipping introduces bias. Dynamic budget and auction effects are not automatically reproduced. Use off-policy estimates for screening and diagnosis, not as a universal replacement for online experiments.
線上實驗：成功指標與因果邊界Online experiments: outcomes and causal boundaries
模型品質改善是起點。真正要回答的是：在固定的投放與量測條件下，新的預測是否改善預先定義的產品結果？
Model quality is a starting point. Does the new prediction improve a prespecified product outcome under controlled delivery and measurement conditions?
| 步驟Step | 具體方案Concrete proposal | 失敗時怎麼辦Failure response |
|---|---|---|
| HypothesisHypothesis | 新特徵／模型改善價值指標，且不增加負回饋、latency、timeout。A feature/model improves a value metric without worsening negative feedback, latency, or timeouts. | 一次只改可歸因的 bundle；改 auction 規則需另定義 treatment。Change an attributable bundle; auction-rule changes define a different treatment. |
| AssignmentAssignment | 從 user-level 穩定分流起步，預先定義 eligible population。Begin with stable user-level assignment and a predefined eligible population. | 檢查 cross-device、session 重入與共同 advertiser budget 的干擾。Inspect cross-device exposure, re-entry, and shared advertiser-budget interference. |
| Power / windowPower / window | 用分流單位的變異估 MDE 與樣本量；包含週期與 label 成熟時間。Estimate MDE/sample size at the assignment unit; include cycles and label maturity. | 事前選 fixed horizon 或有效 sequential 方法，避免每日偷看後任意停止。Prespecify a fixed horizon or valid sequential method; avoid arbitrary stopping after peeking. |
| ValidityValidity | 檢查 SRM、日誌、render coverage、assignment 版本與排除規則。Check SRM, logs, render coverage, assignment versions, and exclusions. | 解決資料品質問題再解讀 lift；不能悄悄丟掉異常 arm。Resolve data-quality problems before interpreting lift; do not silently drop an arm. |
| DecisionDecision | 報效果量、confidence interval、guardrails、slices 與實際成本。Report effect sizes, confidence intervals, guardrails, slices, and real cost. | CTR 上升但 ad hides／轉換品質惡化，需重新檢查目標。Higher CTR with more ad hides or worse conversion quality requires revisiting the objective. |
示範主指標可選 revenue／eligible user，但這是產品選擇，並非通用答案。CTR 是 ratio，需保留 clicks 和 impressions 的變化，避免把分母減少誤判為改善。若 treatment 影響哪些使用者被觸發，只分析事後被曝光者會引入選擇偏差；從預先界定母體做 intention-to-treat 分析，再做有理由的診斷。
Revenue per eligible user is one illustrative primary metric, not a universal choice. CTR is a ratio: inspect clicks and impressions separately so denominator changes are not mistaken for benefit. If treatment changes who is triggered, conditioning on post-treatment exposure introduces selection bias. Start with intention-to-treat analysis of a predefined population, then add justified diagnostics.
廣告主預算同時服務 control 與 treatment，怎麼辦？What if advertiser budgets serve both arms?
Treatment 可能提早耗盡某些預算而改變 control 的可用候選，破壞無干擾假設。先說明想估的效應，視市場結構考慮 budget isolation、cluster randomization 或有 washout 的 switchback，並評估功效與外推限制。沒有一種分流方式適用所有市場。
Treatment can exhaust budgets earlier and change control's eligible candidates, violating no-interference assumptions. Define the estimand, then consider budget isolation, cluster randomization, or switchbacks with washout as the marketplace permits. Evaluate power and generalization limits; no assignment scheme fits every market.
Serving：延遲、共用計算與版本發布Serving: latency, shared computation, and releases
模型前後還有取特徵、cache、queueing、校準與投放規則。以 end-to-end request 為單位 profile，別只報單次 forward 的平均時間。
A forward pass sits inside feature fetches, caches, queues, calibration, and delivery rules. Profile end-to-end requests instead of reporting average forward time alone.
| 優化Optimization | 方法Method | 驗證風險Validation risk |
|---|---|---|
| BatchingBatching | 同一 request 的候選批次計算；跨 request microbatch 有 deadline。Batch candidates within a request; deadline-bound cross-request microbatches. | Throughput 提升可能增加等待；固定 arrival process 測尾延遲。Throughput gains may add queueing; measure tails under a specified arrival process. |
| User compute sharingUser compute sharing | 重用 user-only encoding，再做 candidate-conditioned interaction。Reuse user-only encoding before candidate-conditioned interactions. | 檢查候選依賴邊界與 cached embedding 相容性。Verify candidate-dependency boundaries and cached-embedding compatibility. |
| QuantizationQuantization | 依硬體和敏感度選 precision，保留較敏感模組精度。Choose precision by hardware and sensitivity; preserve sensitive modules. | 比較校準、極端機率、cold-start 與線上 kernels 的一致性。Check calibration, extreme probabilities, cold start, and actual serving kernels. |
| Feature fetchFeature fetch | 批次讀取、bounded parallelism、freshness-aware cache。Batched reads, bounded parallelism, freshness-aware caches. | TTL 不是品質保證；資料 age 與 missingness 必須量測。TTL does not guarantee quality; measure age and missingness. |
| FallbackFallback | 允許缺 optional feature；失敗時走已測試的輕模型或保守策略。Allow optional-feature loss; use an evaluated lighter model or conservative policy. | 不得用隨意常數掩蓋故障；保留 calibration 與版本紀錄。Do not hide failure with arbitrary constants; retain calibration/version logs. |
bundle = {
model, feature_schema, vocabulary_or_hash_config, normalizer,
user_encoder_version, creative_encoder_version, calibrator,
allowed_freshness, fallback_policy, evaluation_manifest
}
features = fetch_batch(request, deadline=request.deadline)
validate_schema_and_embedding_versions(features, bundle)
route = choose_evaluated_route(features.status, request.deadline)
scores = route.score_candidates(features)
return calibrated_scores_with_versions(scores, route.bundle)發布時先跑 tensor／prediction parity 和固定負載測試，再 shadow 觀察實際缺值與尾延遲，最後 canary 評估完整決策。Shadow 不影響曝光，所以不能直接估新 policy 的真實 CTR。Rollback 要回整個相容 bundle，必要時連 user／creative embeddings 一起回退或路由到相容版本。
Before release, run tensor/prediction parity and fixed-load tests, shadow real missingness and tail latency, then canary the actual decisions. Shadowing does not change exposure and cannot directly estimate the new policy's realized CTR. Roll back a compatible bundle, including embeddings or routing to compatible encoder versions where required.
服務 HTTP 200，但 pCTR 全部接近零？HTTP 200, but almost all pCTR values approach zero?
先比 model／feature／calibrator 版本、特徵 coverage 與分布，檢查是不是 vocabulary 失配、zero-filled tensors、precision 問題或錯誤校準。若模型剛更新，回 last-known-good bundle；若 input 壞掉，切健康 fallback 並阻擋用受污染資料繼續訓練。
Compare model/feature/calibrator versions, coverage, and distributions. Check vocabulary mismatches, zero-filled tensors, precision, and calibration. Roll back a newly released bundle; for corrupted inputs, route to a healthy fallback and block training on contaminated data.
反例演練：觀察、診斷與止損Counterexamples: observe, diagnose, mitigate
切換六個固定案例，先自己回答再看建議。這些案例是原創教學 fixtures，沒有呼叫模型或廣告服務。
Switch among six fixed cases and propose a response before reading the suggested decision. These are original teaching fixtures, not live model or ad-service calls.
健康基準Healthy baseline
曝光與 click 都準時到達，feature schema 相容；新模型離線 loss 略降，p99 符合預算。
Impressions/clicks arrive on time, schemas match, and a candidate slightly improves offline loss within the p99 budget.
檢查校準、slices 與重播一致性，通過後進入預先定義的 canary／A/B。
Check calibration, slices, and replay parity, then proceed to the predefined canary/A/B plan.
成功條件是受控線上結果，不是訓練 loss 下降。
Success means controlled online outcomes, not merely lower training loss.
窗內晚到點擊Late in-window click
10:00 曝光；五分鐘 target。Click 在 10:04 發生，10:08 到達；10:05 的資料仍未完整。
Impression at 10:00, five-minute target. Click occurs at 10:04 and arrives at 10:08; data is incomplete at 10:05.
保留 pending 或依成熟政策處理；以 event time 標成正例。若已定案，依版本修正／回補政策處理同一曝光。
Keep pending or follow the maturity policy; label positive by event time. If finalized, correct/backfill the same exposure under the versioning policy.
10:06 才發生的點擊是窗外事件，與窗內晚到不同。
A click occurring at 10:06 is outside the target, unlike an in-window late arrival.
Click 日誌中斷Click-log outage
曝光量正常，click ingest 突降為零；來源 lag 飆升，新 batch 的 CTR 異常低。
Impressions remain normal, click ingestion falls to zero, source lag spikes, and the new batch CTR collapses.
隔離 labels、凍結更新與發布，保留健康 serving bundle。恢復後核對 completeness 再回補。
Quarantine labels, freeze training/release, and preserve the healthy serving bundle. Backfill after recovery and completeness checks.
等待成熟窗口不能補回遺失事件；不要自動把它解釋成使用者興趣改變。
A maturity window cannot restore lost events; do not automatically interpret this as changing user interest.
快取向量不相容Incompatible cached vectors
Ranker 預期 user encoder v8，cache 還有 v7；向量維度相同，因此 API 沒報錯。
The ranker expects user encoder v8, but the cache contains v7. Dimensions match, so the API reports no error.
檢查語意版本，路由到相容 bundle、重算或用已測 fallback；分群監控 feature age 和輸出。
Validate semantic versions, route to a compatible bundle, recompute or use evaluated fallback; monitor age and predictions by slice.
相同 tensor shape 不代表 embedding space 相容。
Equal tensor shapes do not imply compatible embedding spaces.
全新廣告New ad
新 ad ID 未見過，creative 圖文可用；同 advertiser 有歷史但只有少量新廣告曝光。
An unseen ad ID has text/image content; the advertiser has history but the new ad has few impressions.
用 OOV／新 ID 路徑、content 與 advertiser prior；量 cold-start 校準，若產品允許則有限探索。
Use the OOV/new-ID path, content, and advertiser prior; evaluate cold-start calibration and bounded exploration if permitted.
不要把一次點擊／一次曝光的 100% CTR 當可靠機率。
One click from one impression does not justify a reliable 100% probability.
AUC 好、收益差AUC up, value down
AUC 小幅提升，但新模型普遍高估 pCTR，還增加 deadline misses；線上價值指標下降。
AUC improves slightly, but pCTR is overestimated and deadline misses increase; online value falls.
分別檢查 calibrator、抽樣修正、版本與 latency；固定候選做 replay，再以隔離實驗確認原因。
Inspect calibration, sampling correction, versions, and latency separately; replay fixed candidates and isolate causes experimentally.
排序辨識力、機率尺度與成功服務率是不同面向。
Discrimination, probability scale, and successful serving are distinct dimensions.
| 告警訊號Signal | 先排除什麼First checks | 可接受的止損Mitigation |
|---|---|---|
| CTR 突降CTR drops | click source lag／loss、流量組成、曝光 coverage。Click-source lag/loss, traffic composition, exposure coverage. | 區分真實行為與記錄故障；隔離錯誤 labels，避免自動追著噪音訓練。Separate behavior from logging failure; quarantine labels instead of training on noise. |
| Cold-start 退步Cold start regresses | OOV rate、creative encoder、content branch 缺值。OOV rate, creative encoder, missing content-branch inputs. | 還原相容處理；檢查新廣告 exposure／校準與探索支持。Restore compatible processing; inspect exposure/calibration and exploration support. |
| p99 上升p99 rises | 長歷史、hot keys、queueing、cache misses、autoscaling lag。Long histories, hot keys, queueing, cache misses, scaling lag. | Bounded sequence／batch、deadline、已評估的 fallback；追蹤品質損失。Bound sequences/batches, enforce deadlines, use evaluated fallback; measure quality loss. |
| AUC 升，收入降AUC up, revenue down | 校準、競價輸入、timeout、候選分布與共享 budget。Calibration, auction inputs, timeouts, candidates, shared budgets. | 固定元件 replay 與分層實驗；不把收入變化都歸因模型架構。Replay fixed components and isolate layers experimentally; do not blame architecture alone. |
監控分四層：資料健康（coverage、lag、dedup）、特徵／模型（schema、分布、age、prediction）、決策（eligibility、auction、budget）、產品結果（成熟 CTR／conversion、負回饋）。每個告警有 owner、severity、freeze／fallback／rollback 動作與恢復條件，避免只畫 dashboard。
Monitor four layers: data health (coverage, lag, deduplication), features/models (schema, distributions, age, predictions), decisions (eligibility, auctions, budgets), and outcomes (mature CTR/conversions, negative feedback). Give alerts an owner, severity, freeze/fallback/rollback action, and recovery condition rather than stopping at a dashboard.
Repo 導讀與可重現實作路線Repository guide and reproducible implementation path
先做可信的 tabular CTR 實驗，再補事件 join 與 serving fixtures。以下是實作計畫與已核對入口，沒有宣稱完成訓練、部署或量測。
Build a credible tabular CTR experiment, then add event-join and serving fixtures. These are implementation plans and inspected entry points, not claims of completed training, deployment, or benchmarks.
| 資源Resource | 閱讀順序Reading path | 練習與限制Exercise / limits |
|---|---|---|
| FuxiCTRFuxiCTR | example1_build_dataset_to_parquet.py → example2_DeepFM_with_parquet_input.py → model config。example1_build_dataset_to_parquet.py → example2_DeepFM_with_parquet_input.py → model config. | 比較 LR／DeepFM／DCNv2；保存 feature map、設定和 split。框架不是完整廣告平台。Compare LR/DeepFM/DCNv2; save feature maps, configs, splits. The framework is not a full ad platform. |
| DeepCTR-TorchDeepCTR-Torch | run_classification_criteo.py → inputs.py → models/din.py。run_classification_criteo.py → inputs.py → models/din.py. | 理解 sparse／dense／sequence tensor。範例先對全資料 fit encoder／scaler，再 random split；需改成 train-only fitting。Inspect sparse/dense/sequence tensors. The demo fits encoders/scalers before random splitting; change to train-only fitting. |
| DLRM／TorchRecDLRM / TorchRec | dlrm_s_pytorch.py → torchrec_dlrm/dlrm_main.py → TorchRec tutorial。dlrm_s_pytorch.py → torchrec_dlrm/dlrm_main.py → TorchRec tutorial. | 追 embeddings、interaction、sharding 與 train loop；記憶體／通信 profile。Trace embeddings, interactions, sharding, and training; profile memory and communication. |
| MonolithMonolith | 論文 Figure 1、online joiner／parameter sync → repo demo。Paper Figure 1, online joiner/parameter sync → repository demo. | 讀動態 embeddings 與即時更新；repo 於 2025-10-13 封存，依賴需重新核對。Study dynamic embeddings and live updates; archived on 2025-10-13, so revalidate dependencies. |
Criteo 可以驗證什麼？What can Criteo validate?
1TB click logs 含 24 天按時間排序的資料、13 個數值與 26 個匿名類別欄位；正負例已用不同率抽樣，欄位語意未公開。適合 CTR benchmark；不能任意宣稱某欄是 user ID 來建立真實行為序列。未知原始抽樣率時，只能評估釋出分布上的機率品質，不能聲稱已校準到原流量。
The 1TB click logs contain 24 chronologically ordered days, 13 numeric and 26 anonymous categorical fields. Positive and negative examples were sampled at different rates; semantics are undisclosed. This supports CTR benchmarks, not invented user-ID semantics or authentic history reconstruction. Without original retention rates, evaluate probabilities on the released distribution rather than claiming calibration to original traffic.
train, val, cal, test = split_by_available_time(data)
vocab = fit_vocabulary(train, reserve_oov=True)
scaler = fit_numeric_transform(train)
X_train = transform(train, vocab, scaler)
X_val = transform(val, vocab, scaler)
X_cal = transform(cal, vocab, scaler)
X_test = transform(test, vocab, scaler)
for config in prespecified_trial_budget:
fit_and_select_on_validation(config, X_train, X_val)
selected = freeze_selected_model()
calibrator = fit_calibrator(selected, X_cal)
report_once(selected, calibrator, X_test)
# Save data split, seeds, code revision, preprocessing and model artifacts.| 里程碑Milestone | 產物Deliverable | 驗收條件Acceptance |
|---|---|---|
| A · DataA · Data | 依時間取可負擔 subset、split manifest、missing／OOV report。An affordable temporal subset, split manifest, missing/OOV report. | 不跨 test fit 前處理；抽樣不偷改 label 比例；標示資料能力限制。No test-fitted preprocessing; no silent prevalence changes; state dataset limits. |
| B · ModelingB · Modeling | LR＋一個 interaction model，同 HPO 預算、rolling backtest。LR plus one interaction model, equal HPO budgets, rolling backtests. | Log loss／AUC／calibration／uncertainty，另報成本；結果可重現。Reproducible log loss/AUC/calibration/uncertainty plus cost. |
| C · Event fixturesC · Event fixtures | 合成 impression／click stream、可調時鐘與晚到／重複／遺失事件。Synthetic impression/click streams, controllable clocks, late/duplicate/missing events. | 驗 join／label maturity，不把合成結果當真實模型收益。Validate joins/maturity; do not present synthetic results as real model lift. |
| D · ServingD · Serving | 批次 scoring API、版本 bundle、fallback 與 load generator。Batched scoring API, versioned bundle, fallback, load generator. | 報硬體、候選數、arrival rate、warm／cold cache、p95／p99。Report hardware, candidate counts, arrival rate, cache state, p95/p99. |
| E · SequenceE · Sequence | 另找有真實 identity／timestamp／exposure 的資料，做序列 ablation。Use separate data with authentic identities/timestamps/exposures for sequence ablations. | Next-item recommendation 與 impression CTR 是不同目標，需明確界定。Next-item recommendation and impression CTR are different targets; define them explicitly. |
Repo 的 demo 只是入口。面試更有價值的展示是：你發現了哪個資料／評估陷阱、如何修正、哪個假說沒有成立，以及品質改善花了多少算力。不要捏造 online lift；沒有流量時，以離線結果與 mock／load test 的範圍如實呈現。
A repository demo is a starting point. A stronger interview artifact explains a data/evaluation trap you found, its correction, a failed hypothesis, and the compute cost of any gain. Do not invent online lift; without production traffic, report the limits of offline results and mock/load tests.
45 分鐘回答流程與深入追問A 45-minute answer flow and deeper follow-ups
沿六步回答，每一步都說出一個具體選擇、一個取捨和一個驗證方法。先閉卷口說，再展開中英答案。
Use six stages, each with a concrete choice, tradeoff, and validation method. Rehearse without the text before opening either language.
00–05 · 釐清產品目標00–05 · Clarify the product goal
我先把任務限於社群 feed 的精排 CTR，預測有效曝光後 W 內的落地頁點擊。確認版位、候選量、延遲、click 定義和可用特徵。模型輸出條件機率，下游再結合競價與品質限制；線上主要價值指標和負回饋、延遲護欄需先對齊。
I will scope this to CTR prediction for social-feed ranking: a landing-page click within W after a valid impression. I will clarify placement, candidate volume, latency, click semantics, and available features. The model produces a conditional probability; downstream auction and quality rules consume it. We should agree on a primary value metric and negative-feedback/latency guardrails first.
05–13 · 資料與標籤05–13 · Data and labels
一筆樣本是一筆 rendered impression，以 impression ID join 去重後的 click。使用 event time 定義窗口，arrival time 控制資料可用性；成熟且來源健康後才定案負例。特徵用 serving snapshot 或雙時間條件重建，按時間切 train／validation／calibration／test。若抽負例，保存保留率並修正 posterior 或 loss。未曝光候選不直接標 0。
One example is a rendered impression joined to deduplicated clicks by impression ID. Event time defines the target window; arrival time governs availability. Finalize negatives only after maturity with healthy sources. Use serving snapshots or time-aware reconstruction, split chronologically into train/validation/calibration/test, and log retention rates if sampling negatives. Correct either the posterior or loss appropriately. Unshown candidates are not zero-label examples.
13–25 · Baseline／模型13–25 · Baselines and models
先用 LR／FTRL 和 user、ad、context、平滑歷史統計建立 baseline。固定資料比較 DeepFM 或 DCNv2 的交互收益；再以行為歷史是否有 candidate-specific 訊號決定加 DIN 或序列模型。解釋 BCE、sampling、OOV 和模型輸出。用 pooled／shuffled／ordered history、content ablation 與公平調參驗證假說，不因模型新就選它。
Start with LR/FTRL and user, ad, context, and smoothed history features. With fixed data, test interaction gains from DeepFM or DCNv2. Add DIN or sequence models only for a hypothesis about candidate-specific history or temporal signal. Explain BCE, sampling, OOV handling, and output semantics. Use pooled/shuffled/ordered histories, content ablations, and fair tuning budgets to test the hypothesis rather than selecting by novelty.
25–33 · 訓練與部署25–33 · Training and deployment
先批次重訓並獨立量 feature／model age，視收益再加近線更新。高 cardinality embeddings 估 optimizer memory、sharding 和 communication。Serving 批次處理候選、共用 user-only 計算，設定 feature deadline 與已測 fallback。把 vocabulary、encoder、模型與 calibrator 作相容 bundle 發布，經 parity、shadow、canary 後放量，資料污染時凍結更新。
Begin with batch training and measure feature/model age separately before adding nearline updates. Budget optimizer memory, sharding, and communication for high-cardinality embeddings. Batch candidates, share user-only computation, and define feature deadlines and evaluated fallbacks. Release vocabulary, encoders, model, and calibrator as a compatible bundle through parity checks, shadowing, and canaries; freeze updates when data is contaminated.
33–41 · 離線評估／線上實驗33–41 · Offline and online evaluation
離線以未來資料報 log loss、明確定義的 NE、AUC 和 calibration；按新廣告、版位、活躍度與 freshness 切片，使用適合相關結構的 uncertainty。線上先確認分流與 SRM，預先定義主要指標、MDE、觀察窗和 guardrails。共享 advertiser budget 可能造成干擾，需要評估隔離或不同 randomization。不能由 AUC 推導收入 lift。
On future data, report log loss, explicitly defined NE, AUC, and calibration, sliced by new ads, placement, activity, and freshness with dependence-aware uncertainty. Online, validate assignment and SRM, and prespecify the primary metric, MDE, observation window, and guardrails. Shared advertiser budgets may require isolation or a different randomization design. AUC does not establish revenue lift.
41–45 · 失敗案例與取捨41–45 · Failures and tradeoffs
若 click source 中斷，我會先凍結錯誤 labels 和更新，保留健康 bundle，再回補；若 user embedding 過期或版本不符，走相容 fallback。最後以資料新鮮度、模型容量與成本的實驗曲線決定下一步。我會明確區分已知證據、設計假設和仍需驗證的部分。
If the click source fails, freeze bad labels and updates, preserve the healthy bundle, and backfill. For stale or incompatible user embeddings, use a compatible fallback. Choose the next investment from measured freshness, capacity, and cost curves. Distinguish established evidence, design assumptions, and remaining validation.
追問：只有 click logs，沒有曝光紀錄？Follow-up: clicks but no impression logs?
沒有 denominator，也不知道哪些廣告有機會被點，不能直接訓練或評估標準 impression CTR。先改善 logging，或重新定義可識別的任務，例如 click-conditioned next-item prediction，並清楚說它不是 pCTR。隨機拿未點廣告當負例無法恢復缺失的曝光機制。
There is no denominator or record of click opportunities, so standard impression CTR is not identifiable from those logs alone. Improve logging or redefine an identifiable task such as click-conditioned next-item prediction, explicitly distinguishing it from pCTR. Random unclicked ads do not reconstruct the missing exposure mechanism.
追問：校準比例等於 1，還有問題嗎？Follow-up: calibration ratio is 1; are we done?
沒有。高估一群、低估另一群可能相互抵消；不同分數區間也可能失準。檢查 reliability plot、slice counts、Brier／loss 與 ranker 使用機率的區域。校準器的資料分布和時間也要匹配目標流量。
No. Overprediction in one group can offset underprediction in another, and score ranges can be miscalibrated. Inspect reliability plots, slice counts, Brier/loss, and the probability ranges used by the ranker. The calibrator's data distribution and time period must match target traffic.
追問：你如何設計一個有研究價值的下一步？Follow-up: what is a research-worthy next step?
提出可辨識的假說，例如多樣事件是否比同長度單一 click 序列更有用。控制事件數、時間跨度、模型容量與 tuning budget，分別消融 action type、順序和內容表示。用多時段、cold-start slices、uncertainty 與 cost-quality curve 判斷結果；保留不成立的假說和 failure analysis。
Propose an identifiable hypothesis, such as whether diverse event types outperform equally long click-only histories. Control event count, time span, capacity, and tuning budget; ablate action type, order, and content independently. Evaluate multiple periods, cold-start slices, uncertainty, and cost-quality curves, retaining negative results and failure analysis.
來源與程式碼導讀Sources and code reading
依序讀：FB14 的資料閉環 → FTRL／DeepFM／DCNv2 的 baseline → DIN 與 Meta 序列建模 → serving 與 robustness。教學文章用來練組織；原始論文與程式用來核對技術。各章架構、例子、數字與實驗方案是本篇的設計綜合。
Read in order: FB14's learning loop → FTRL/DeepFM/DCNv2 baselines → DIN and Meta sequence modeling → serving and robustness. Teaching articles help structure an answer; original papers and code ground the technical details. Architectures, examples, numbers, and experiment plans in this guide are a design synthesis.
- Facebook · Practical Lessons from Predicting Clicks on Ads ↗
讀 §2 metrics、§3 hybrid model、§4 online joiner，以及 Figure 4。本文借用問題意識，圖與整合方案重新設計；不能把 2014 的架構當成今日完整系統。
Read §2 metrics, §3 hybrid modeling, §4 online joining, and Figure 4. This guide redraws and extends the concepts; the historical architecture is not a specification of today's full system.
- Google · Ad Click Prediction: a View from the Trenches ↗
FTRL、稀疏性、記憶體與校準；也讀作者沒有成功的實驗。背景為搜尋廣告，需重新界定 social feed 的資料和 context。
FTRL, sparsity, memory, and calibration; read the unsuccessful experiments too. The search-ad setting requires adaptation to social-feed data and context.
- Meta · Sequence Learning for Personalized Ads Recommendations ↗
事件序列表示與效率設計。用它提出 feature／sequence 假說，不直接移植文章報告的提升數字。
Event-sequence representations and efficiency. Use it to form feature/sequence hypotheses rather than transferring reported gains.
- Meta · From User Sequences to Scaling Laws: A Multi-Stage Architecture ↗
重點是 async user model／online ranking 的分工。本文 Figure 3 是原創延伸；文章還連到 LLaTTE 原始論文。
Focus on asynchronous user modeling versus online ranking. Figure 3 here is an original extension; the article also links the LLaTTE paper.
- Meta · Adaptive Ranking Model: Bending the Inference Scaling Curve ↗
Request-oriented compute sharing、精度與 kernel／硬體效率。本文容量與 latency 數字另為假設，不是該文 benchmark。
Request-oriented compute sharing, precision, and kernel/hardware efficiency. Capacity and latency numbers in this guide are separate assumptions, not the article's benchmarks.
- Meta · Machine Learning Prediction Robustness ↗
特徵品質、snapshot validation、label／calibration 問題。用來設計「服務活著但預測壞掉」的診斷。
Feature quality, snapshot validation, labels, and calibration. Useful for diagnosing healthy endpoints with unhealthy predictions.
- DeepFM: A Factorization-Machine Based Neural Network for CTR Prediction ↗
對照共享 embeddings、FM branch 與 deep branch；再追 FuxiCTR 的模型與設定。
Inspect shared embeddings, FM and deep branches, then trace the FuxiCTR implementation and configuration.
- DCN V2: Improved Deep & Cross Network ↗
顯式 feature crosses 與 low-rank extensions；本篇給簡化 cross-layer 公式，完整變體讀原文。
Explicit crosses and low-rank extensions. This guide gives a simplified cross-layer equation; consult the paper for complete variants.
- Deep Interest Network for Click-Through Rate Prediction ↗
Local activation 依 candidate 聚合歷史；區分 candidate conditioning 與 order modeling，核對是否使用 normalized weights。
Local activation aggregates history conditioned on a candidate. Distinguish conditioning from order modeling and inspect weight normalization.
- Counterfactual Risk Minimization: Learning from Logged Bandit Feedback ↗
理解 logged feedback、propensity 與 variance；本篇不是完整反事實學習教科書，也不假設有理想 logging support。
Study logged feedback, propensities, and variance. This guide is not a full counterfactual-learning treatment and does not assume ideal logging support.
- Microsoft Research · Diagnosing Sample Ratio Mismatch ↗
先驗證分流與資料品質，再解讀效果；SRM 是症狀，需找原因。
Validate assignment and data quality before interpreting effects; SRM is a symptom requiring diagnosis.
- Microsoft Research · A Dirty Dozen: Metric Interpretation Pitfalls ↗
練習解釋 ratio、分母、觸發族群與指標變動；共享預算的設計取捨為本篇綜合建議。
Practice reasoning about ratios, denominators, triggered populations, and metric movement. Shared-budget design choices here are a synthesis.
- FuxiCTR · Configurable CTR Benchmarking ↗
從 demo、feature map、config 到 model zoo，練可重現比較。先核對版本與環境再執行。
Trace demos, feature maps, configs, and the model zoo for reproducible comparisons. Check versions and environment before execution.
- FuxiCTR · Build Dataset to Parquet ↗
追欄位定義、資料轉換與輸出；另設計自己的 temporal splits。
Trace field definitions, transformations, and outputs; design your own temporal splits.
- FuxiCTR · DeepFM with Parquet Input ↗
追 model config、data loader、訓練與評估呼叫；範例成功不等於實驗結論成立。
Trace model config, loaders, training, and evaluation calls; a working demo is not an experimental conclusion.
- DeepCTR-Torch · Criteo Classification Example ↗
範例在 split 前 fit LabelEncoder／MinMaxScaler。練習修成 train-only fit、OOV 和時序切分；不要直接沿用 demo 的評估。
The example fits LabelEncoder/MinMaxScaler before splitting. Practice train-only fitting, OOV handling, and temporal splits instead of copying the demo evaluation.
- DeepCTR-Torch · DIN Model Implementation ↗
追 query／keys、sequence length、mask 與 attention pooling，對照論文假設。
Trace queries/keys, sequence lengths, masks, and attention pooling against the paper's assumptions.
- Meta · Deep Learning Recommendation Model ↗
從 README 圖與 PyTorch forward 讀 dense／sparse interactions；不是完整 Meta 廣告 pipeline。
Read the README diagram and PyTorch forward path for dense/sparse interactions; this is not Meta's full advertising pipeline.
- DLRM · TorchRec Training Entry Point ↗
讀 argument、embedding configuration、sharding 與 train／evaluation loops，記錄所用 revision。
Inspect arguments, embeddings, sharding, and train/evaluation loops; record the revision used.
- PyTorch · Introduction to TorchRec ↗
Jagged inputs、embedding collections 與 distributed model parallel；把 memory 和 communication 分開量。
Jagged inputs, embedding collections, and distributed model parallelism; measure memory and communication separately.
- Monolith: Real Time Recommendation System With Collisionless Embedding Table ↗
Figure 1、streaming engine、online joiner、parameter synchronization；閱讀 freshness／reliability 取捨。
Figure 1, streaming engine, online joiner, and parameter synchronization; study freshness/reliability tradeoffs.
- ByteDance · Monolith Repository ↗
已封存；README 指向 Linux／Bazel 建置與 demos。適合讀系統設計，不能保證在目前環境直接跑。
Archived; the README describes Linux/Bazel builds and demos. Useful for systems reading, with no guarantee of working unchanged in a current environment.
- Criteo AI Lab · 1TB Click Logs Dataset ↗
確認 chronological ordering、匿名欄位與不同正負 subsampling rates；先理解資料能力才設計實驗。
Verify chronological ordering, anonymous fields, and different positive/negative subsampling rates before designing experiments.
- Khang Ich · Ad Click Prediction for Social Networks ↗
學 requirements → features → model → deployment 的組織。作者有簡化 cascade；本文架構另行重畫並加入資料／發布邊界。
Learn the requirements → features → model → deployment structure. The author simplifies the cascade; this guide redraws the architecture with data/release boundaries.
- AlgoMaster · Design Ad Click Prediction System ↗
用作完整回答的組織參考。其 NE 介於 0–1 的描述不成立：比 entropy baseline 差時可大於 1；容量與 latency 數字也須視為假設。
Useful for answer organization. Its claim that NE lies in 0–1 is incorrect: NE can exceed 1 when worse than the entropy baseline. Treat capacity/latency figures as assumptions.