閱讀方式：先限定推薦位置，再沿著資料 → 標籤 → 召回 → 排序 → 評估 → serving 閱讀。最後用 45 分鐘口說流程與反例檢驗設計。範例中的數字都是假設；程式碼入口已核對，沒有宣稱已訓練、部署或量測成效。
How to read: scope the recommendation surface, then follow data → labels → retrieval → ranking → evaluation → serving. Finish with the 45-minute rehearsal and counterexamples. Numerical examples are assumptions; code entry points were inspected, with no claim of training, deployment, or measured results.
產品目標與問題定義Product goals and problem definition
先定義使用者正在完成的購物任務，再決定要預測哪一種行為。首頁、商品頁、購物車與補貨推薦不應共用一個未加區分的目標。
Define the shopping task before choosing the prediction target. Homepage discovery, product pages, carts, and replenishment should not share an undifferentiated objective.
本篇的設計範圍Scope of this design
以電商首頁的 20 個個人化商品為主，支援登入與匿名 session；商品頁的替代品與互補品作為延伸。假設 1,000 萬商品、尖峰 5,000 QPS、推薦 API p95 150 ms。這些都是用來推導容量與取捨的面試假設，不是 Amazon 公開的流量、SLA 或現行內部架構。
Focus on 20 personalized homepage products for signed-in users and anonymous sessions, with substitutes and complements on product pages as extensions. Assume 10 million items, 5,000 peak QPS, and a 150 ms recommendation API p95. These are interview sizing assumptions, not disclosed Amazon traffic, SLAs, or its current internal architecture.
| 位置／任務Surface / task | 推薦什麼What to recommend | 容易犯的錯Common mistake |
|---|---|---|
| 首頁：探索與延續任務Home: discovery and mission continuation | 長期偏好＋最近 session，兼顧探索Long-term affinity plus recent session intent and exploration | 把剛買完的耐用品繼續排第一Keep ranking a recently purchased durable good first |
| 商品頁：比較與搭配Product page: comparison and pairing | 替代品與相容配件分開建模／展示Model and display substitutes separately from compatible accessories | 把圖像相似當作互補，忽略規格相容性Treat visual similarity as complementarity and ignore compatibility |
| 購物車：完成組合Cart: complete a bundle | 互補品、缺少的配件、可合併配送Complements, missing accessories, and shipping compatibility | 加太多交叉銷售干擾結帳Let cross-selling interfere with checkout |
| 回購：補貨Repeat purchase: replenishment | 依品類與個人消耗週期推估Estimate category-specific and personal consumption cycles | 所有已購商品一律永久排除Permanently exclude every purchased item |
主要成功指標可選每位分流使用者的成熟有效訂單數，並明確定義取消／退貨的觀察窗。CTR 是漏斗診斷；另看整站轉換、長期回訪、滿意度、取消與退貨、延遲及推薦缺貨率。目標若是 discovery，還要觀察新商品／新品類探索的後續品質，而不只是點了幾下。
A primary metric could be matured valid orders per assigned user, with explicit cancellation and return windows. CTR diagnoses the funnel. Guardrails include site-wide conversion, repeat visits, satisfaction, cancellations, returns, latency, and unavailable recommendations. For discovery, assess downstream quality of new-item and new-category exploration rather than clicks alone.
面試開場：你會先問什麼？Opening: what would you clarify first?
我會先確認推薦位置、使用者是探索還是完成購買、是否包含匿名流量，以及商品、variant 和 offer 的定義。接著對齊主要指標、延遲與庫存更新需求。我先以首頁個人化推薦為主，優化有效購買並保護體驗；廣告若加入，需保留來源與清楚標示，不能把贊助混入一般分數後失去可辨識性。
I would clarify the surface, discovery versus purchase completion, anonymous traffic, and the distinction between products, variants, and offers. Then I would agree on the primary metric, latency, and inventory freshness. I will start with personalized homepage recommendations that improve valid purchases while protecting experience. If ads are included, sponsorship provenance and visible labeling must survive ranking.
多階段架構與責任邊界Multistage architecture and responsibilities
先便宜地找到值得看的一小群商品，再把昂貴特徵與模型用在這一群。候選數與時間預算都必須由實測校準。
Find a small promising set cheaply, then spend richer features and model capacity on that set. Candidate counts and timing budgets require measurement.
| 階段Stage | 輸入 → 輸出Input → output | 設計責任Responsibility |
|---|---|---|
| 候選召回Candidate retrieval | 1,000 萬 → 約 1,000 個 ID10 million → about 1,000 IDs | 多路並行：item-to-item、two-tower、session、熱門／探索；保存來源與分數Parallel item-to-item, two-tower, session, and popular/exploration sources; preserve provenance and scores |
| 輕量排序Light ranking | 約 1,000 → 200About 1,000 → 200 | 便宜交叉特徵、去重與初步 eligibility，控制後段成本Cheap crosses, deduplication, and preliminary eligibility to bound downstream cost |
| 精排Rich ranking | 約 200 → 有效候選的效用分數About 200 → utility scores for eligible candidates | 候選相關歷史、多任務預測、最新商務特徵；批次取值Candidate-aware history, multitask prediction, fresh commerce features, and batched lookup |
| 結果組合Slate construction | 排序候選 → 最多 20 個Ranked candidates → up to 20 items | 最後資格查驗、family 去重、多樣性、疲勞與有界補取Final qualification, family deduplication, diversity, fatigue control, and bounded refill |
每個 retriever 先產生比配額略多的候選，合併後依 stable item ID 去重，保留所有來源。各來源原始分數通常不可直接比較；交給校準／ranker 使用來源、來源名次與分數。早期過濾減少浪費，最後再查驗會快速變動的庫存與可配送性。
Each retriever overfetches within a budget. Merge by stable item ID while retaining all source memberships. Raw scores from different retrievers are generally not comparable; expose calibrated scores, source indicators, and source ranks to the ranker. Early filtering reduces waste; a final check handles rapidly changing inventory and deliverability.
為什麼不直接對 1,000 萬商品跑最好的 ranker？Why not run the best ranker on all 10 million items?
精排通常有使用者與候選的交叉、序列 attention 和大量特徵查詢，成本隨候選數增加。先以可預計算的 item 表示召回，再用 query-dependent 交互精排，能讓延遲與成本可控。代價是 retrieval 形成上限；如果 relevant item 根本沒進候選，ranker 無法補救，所以要分開診斷召回損失與排序損失。
A rich ranker often uses user–candidate crosses, sequence attention, and many feature lookups, so cost grows with candidate count. Retrieval with precomputed item representations followed by query-dependent ranking bounds latency and cost. Retrieval becomes a ceiling: the ranker cannot recover an item that never entered the pool, so I would diagnose retrieval and ranking losses separately.
資料準備與時間一致性Data preparation and time consistency
一筆訓練資料應能回答：當下知道什麼、實際呈現了什麼、之後在完整觀察窗內發生什麼。資料的可用時間與事件時間同樣重要。
A training example should answer what was known, what was actually shown, and what happened during a complete observation window. Availability time matters as much as event time.
| 資料契約Data contract | 必要欄位Required fields | 品質檢查Quality check |
|---|---|---|
| 請求與候選Request and candidates | request_id、匿名／登入 key、session、surface、locale、來源與模型版本request_id, anonymous/signed-in key, session, surface, locale, sources, and model versions | 候選不等於曝光；保留可重播的版本與分流單位Candidates are not exposures; retain replayable versions and assignment units |
| 曝光與互動Exposure and interaction | item／offer ID、位置、render/viewability、click、cart、purchase、return、時間與去重 keyItem/offer ID, position, render/viewability, click, cart, purchase, return, timestamps, and deduplication key | API 回傳不等於看到；事件去重、bot 過濾、遲到與 session 邊界An API response is not a view; deduplicate events, filter bots, and handle late arrival and session boundaries |
| 商品與供給Catalog and supply | parent product → variant → seller offer；屬性、價格／幣別、地區、庫存與版本Parent product → variant → seller offer; attributes, price/currency, region, inventory, and versions | 不能用今天的價格重建過去；family 去重不能抹掉尺寸／相容性差異Do not reconstruct the past with today's price; family deduplication must preserve size and compatibility distinctions |
| 實驗與策略Experiment and policy | assignment、policy_id、探索機率（若已知）、feature/index versionAssignment, policy_id, exploration probability when known, and feature/index version | 跨裝置 identity 規則固定；遺失曝光或錯誤分流會扭曲結論Keep cross-device identity rules stable; missing exposure or faulty assignment distorts conclusions |
Point-in-time join 不只是 event_time ≤ tPoint-in-time is more than event_time ≤ t
事件在 10:00 發生，但 10:10 才到特徵服務，不能放進 10:05 的訓練特徵。若要忠實重播，需同時限制 event_time 與 available_time，並保留當時採用的轉換版本。所有 rolling aggregate 只看 cutoff 前可用的資料；未來購買、未來評論、全期間熱門度與新版本商品屬性都可能造成洩漏。
An event occurring at 10:00 but reaching the feature service at 10:10 cannot appear in a 10:05 training feature. Faithful replay constrains both event_time and available_time and preserves the transformation version. Rolling aggregates use only data available before the cutoff. Future purchases, later reviews, whole-period popularity, and newer catalog attributes can all leak information.
for exposure in logged_exposures:
t = exposure.decision_time
x = feature_snapshot(as_of=t, available_by=t, version=exposure.feature_version)
labels, masks = label_windows(exposure, observed_through=train_cutoff)
# masks[t] = 0 when the task's observation window is incomplete
emit(exposure.request_id, exposure.item_id, x, labels, masks)
# Choose chronological cutoffs; labels used in a release must mature before training.
# Fit vocabularies, imputers, and normalization only on the training partition.生產驗證採全域時間切分：先訓練，再以較晚區間驗證與測試，必要時留出標籤成熟的間隔。不要隨機拆開同一 session 的 prefix 與答案。詞彙表只用訓練集，OOV 與新商品另外報告；記錄資料快照、排除規則與樣本數。對遺失欄位建立 missing indicator，對刪除要求同步處理衍生特徵與索引。
For production validation, use global chronological splits: train first, validate and test later, with a maturity gap where needed. Do not randomly separate a session prefix from its answer. Fit vocabularies on training data and report OOV and new-item behavior separately. Record snapshots, exclusions, and counts. Add missing indicators and propagate deletion handling into derived features and indexes.
| 可用公開資料Public dataset | 可以做Supported tasks | 不能冒充Unsupported claims |
|---|---|---|
| Amazon-M2Amazon-M2 | 六個 locale 的匿名 session 與商品屬性；依官方協定做 next-item retrieval／rankingAnonymous sessions and product attributes across six locales; next-item retrieval/ranking under the official protocol | 沒有完整曝光／購買／退貨漏斗；不能擅自假設跨 session 使用者或絕對時間可用Not a complete exposure/purchase/return funnel; do not assume cross-session identities or absolute timestamps are available |
| Amazon Reviews 2023Amazon Reviews 2023 | 評論互動序列、商品文字／圖片表徵、冷啟動研究Review-interaction sequences, text/image representations, and cold-start research | review timestamp ≠ purchase timestamp；評分不是 CTR；metadata 未必是歷史快照Review timestamp ≠ purchase timestamp; ratings are not CTR; metadata may not be historical snapshots |
標籤、負例與偏差Labels, negatives, and bias
先定義觀察機制與分母，才有資格談 loss。未曝光的商品、曝光未點擊、點擊未購買，是三種不同的訊號。
Define the observation process and denominator before the loss. Unexposed items, exposed-but-unclicked items, and clicked-but-unpurchased items are different signals.
| 任務Task | 正例／負例Positive / negative | 成熟與遮罩Maturity and masking |
|---|---|---|
| CTRCTR | 已曝光且點擊／已曝光且在點擊窗內未點擊Clicked exposure / exposure without a click within its window | 完整觀察後才作負例；未曝光不是已知負例Mark negative after the complete window; unexposed items are not known negatives |
| 曝光後購買Post-exposure purchase | 依明確 attribution 規則連結購買；成熟但未購買為負例Link purchases using an explicit attribution rule; mature non-purchases are negative | 例如 7 天只是初始假設，需看延遲分布；未成熟遮罩Seven days is only an illustrative starting point; inspect delay distributions and mask immature labels |
| 退貨風險Return risk | 購買／履約且已進入可退貨觀察的 cohort 中建模Model within the purchased/fulfilled cohort eligible for return observation | 不曾購買的樣本應遮罩，而非標成未退貨；窗末尚未成熟也遮罩Mask non-purchases rather than labeling them non-returned; also mask immature observation windows |
| Next-itemNext item | session 下一個互動商品；抽樣商品是對比候選The next engaged item in a session; sampled items form a contrastive set | 這個分數不是已校準的購買機率This score is not a calibrated purchase probability |
最容易解釋的起步方案是等完整觀察窗成熟，再加入該任務訓練。若要吸收新鮮資料，可使用延遲回饋模型或明確的部分標籤策略，但不能把還沒買誤當永遠不買。Last-touch attribution 是記帳規則，不能證明推薦造成了額外購買；增量效果仍需可信的實驗設計。
The simplest defensible starting point is to train a task only after its observation window matures. Fresher data can use delayed-feedback models or explicit partial-label policies, but not-yet-purchased must not mean never-purchased. Last-touch attribution is an accounting rule, not proof that a recommendation caused an incremental purchase; incrementality needs credible experimentation.
三種偏差不能用同一個 correction 解決Three biases need different treatments
負例抽樣偏差：訓練看到的商品頻率受 sampler 改變。曝光／位置偏差：舊策略決定誰有機會被看見。延遲／選擇偏差：成熟 cohort 或只看點擊後購買會改變樣本分布。Log-Q 只針對特定抽樣機制；IPS 需要已知或可合理估計的 propensity、support 與方差控制；多任務 learning 也不會自動解決全部問題。
Negative-sampling bias comes from the sampler changing item frequency. Exposure/position bias comes from the old policy determining what can be seen. Delay/selection bias comes from maturity cohorts or observing conversion only after clicks. Log-Q addresses a specific sampling mechanism. IPS needs known or defensibly estimated propensities, support, and variance control. Multitask learning does not fix all three automatically.
如果為了訓練速度只保留 10% 負例？What if you retain only 10% of negatives to train faster?
保留抽樣機率與樣本權重，在符合抽樣假設下修正 loss 或 posterior，並以未改變母體比例的驗證集做機率校準。只看抽樣資料上的 AUC 或 log loss 不能保證線上機率正確。抽樣若依使用者、品類或難度而變，不能只用一個全域常數修正。
Retain sampling probabilities and weights, then correct the loss or posterior under the sampling assumptions. Calibrate on validation data that preserves the population distribution. AUC or log loss on sampled data alone does not guarantee correct serving probabilities. If sampling varies by user, category, or difficulty, a single global correction is insufficient.
特徵工程與商品表示Feature engineering and item representations
區分長期品味、當下意圖、商品內容與商務可用性；每個特徵都要有時間、來源、缺值策略和更新頻率。
Separate long-term taste, immediate intent, product content, and commerce availability. Every feature needs time semantics, provenance, missing-value handling, and a refresh policy.
| 特徵群Feature group | 例子與轉換Examples and transforms | 注意事項Caveat |
|---|---|---|
| 使用者長期Long-term user | 品類／品牌偏好、過去價格分位、尺寸、購買間隔；衰減聚合Category/brand affinity, historical price quantiles, sizes, purchase intervals, and decayed aggregates | 可撤銷／最小化資料；不同地區價格先處理幣別與分布Minimize retained data and support deletion; account for currency and regional price distributions |
| 短期 sessionShort-term session | 事件類型＋item 序列＋時間差、最近查詢／瀏覽；截斷與 padding maskEvent type, item sequence, time gaps, recent query/view context, truncation, and padding masks | 只用決策前的 prefix；購買完成代表意圖轉換Use only the prefix before the decision; purchase completion can change intent |
| 商品內容Item content | 類別、品牌、規格、文字與圖片 embeddings；數值 log／clip／normalizeCategory, brand, specifications, text/image embeddings; log/clip/normalize numeric values | 圖片相似不等於相容；需 variant 與型號級結構化屬性Visual similarity is not compatibility; retain variant- and model-level structured attributes |
| 行為統計Behavior statistics | 歷史窗曝光、點擊、購買、平滑後比率與樣本量Historical-window exposures, clicks, purchases, smoothed rates, and counts | 分子分母要同 cohort；只用過去已知資料；長尾不能信任 1/1 的 CTRUse matching numerator/denominator cohorts and past-known data; a tail item's 1/1 CTR is unreliable |
| 交叉與來源Crosses and provenance | 偏好相似度、價格差、相容性、來源 membership／名次／分數Affinity similarity, price gap, compatibility, source membership/rank/score | 強 query-dependent 特徵放 ranker；不同來源 raw score 不可直接相加Put strongly query-dependent features in ranking; raw source scores are not directly additive |
| 即時供給Live supply | 可配送區域、價格、庫存、交期、offer 品質Deliverability, price, inventory, delivery time, and offer quality | 使用最新 cache／服務查驗；不要因價格一變就重建所有 embeddingsCheck fresh caches/services; do not rebuild all embeddings for every price change |
smoothed_rate = (positive_count + alpha * prior_rate) / (exposure_count + alpha)
All counts use a defined historical window and the same eligible cohort.冷啟動商品用文字、圖片與結構化屬性生成表示；訓練可用 ID dropout 或 gating 降低模型只記住熱門 ID 的傾向。新使用者先依 locale、surface 與 session 內容個人化，再加入探索。不要把所有 OOV 映射為同一個無內容向量後，就宣稱已解決冷啟動。
For cold-start items, derive representations from text, images, and structured attributes. ID dropout or gating can reduce reliance on memorized popular IDs. For new users, start with locale, surface, and session context plus exploration. Mapping all OOV items to the same content-free vector does not solve cold start.
LLM 或多模態模型應放在哪裡？Where would an LLM or multimodal model help?
先用在離線商品屬性擷取、文字標準化或內容 embeddings，特別評估新品與長尾的增益。保留來源與驗證規則，避免把生成的錯誤規格當事實。只有當線上收益超過延遲與成本，才加入請求路徑；單純商品推薦並不需要先生成一段文字。
Start with offline attribute extraction, text normalization, or content embeddings, and measure gains on new and tail items. Preserve provenance and validation rules so generated specifications do not become unverified facts. Add models to the request path only when value justifies latency and cost; product recommendation does not inherently require text generation.
強 baseline：熱門與 item-to-itemStrong baselines: popularity and item-to-item
先做可解釋、可服務且有合理調參的 baseline，讓複雜模型必須證明額外價值。Item-to-item 是重要起點，不等於只能提供靜態推薦。
Begin with interpretable, servable, reasonably tuned baselines so complex models must demonstrate added value. Item-to-item is a useful starting point, not a requirement for static recommendations.
B0：分群熱門B0: segmented popularity
依 locale、品類與時間窗計算，先過濾供給；記錄這個 baseline 的 cold-start coverage、有效候選數與延遲。熱門不是隨便取全站銷量前 20。
Compute popularity by locale, category, and time window, then qualify supply. Measure cold-start coverage, eligible yield, and latency. Popularity is more than returning the site's top 20 sellers.
B1：Item-to-itemB1: item-to-item
離線建立每個商品的稀疏 top-L 鄰居；線上依最近瀏覽／購買與衰減權重合併。歷史變動即可改變結果，不必每次重算全部商品相似度。
Build sparse top-L neighbors per item offline. Online, aggregate neighbors using recent views/purchases and decay weights. Updated history changes the result without recomputing all item similarities.
B2：簡單 rankerB2: simple ranker
在同一候選池，以 logistic regression 或 GBDT 加入價格差、偏好、來源與過去行為；對照純相似度排序，建立可量測的起點。
On the same candidate pool, add price gap, affinities, provenance, and past behavior to logistic regression or GBDT. Compare against similarity-only ordering to establish a measurable baseline.
sim(i,j) = c_ij / sqrt(c_i * c_j) * c_ij / (c_ij + lambda)
score(u,j) = SUM over i in history(u): event_weight(i) * exp(-age(i)/tau) * sim(i,j)
Illustrative baseline only; not a disclosed Amazon implementation.共同出現次數需去除重複事件、異常超長 session 與 bot；可限制每個 session 對某 pair 的貢獻。Shrinkage 讓很少證據的 1 次共現不至於勝過穩定關係。保存稀疏 top-L 鄰居，避免 N×N dense matrix。每個候選來源都做增量 recall、coverage 與成本消融，防止加入很多來源卻沒有新訊號。
Deduplicate co-occurrences, remove bots and anomalously long sessions, and cap each session's contribution to a pair. Shrinkage prevents a single coincidental pair from outranking stable relationships. Store sparse top-L neighbors rather than a dense N×N matrix. Ablate each candidate source for incremental recall, coverage, and cost to detect redundant sources.
Co-view 就是替代品，co-buy 就是互補品嗎？Does co-view mean substitutes and co-buy mean complements?
它們只是弱監督訊號。共同瀏覽可能是比較，也可能是搭配；共同購買可能只是熱門度或促銷。互補常有方向性：買相機可能需要記憶卡，買記憶卡不代表需要相機。加入商品類型、相容性與方向性 P(j|i)，並按展示位置分開評估替代與互補品質。
They are weak supervision, not semantic truth. Co-views may reflect comparison or pairing; co-purchases may reflect popularity or promotions. Complements are often directional: buying a camera may imply a memory-card need, but the reverse need not hold. Use product types, compatibility, and directional P(j|i), and evaluate substitutes and complements separately by surface.
Two-tower、序列與負例抽樣Two towers, sequences, and negative sampling
Retrieval 的核心限制是商品向量必須可預計算；強烈依賴某位使用者與某個候選的交互作用留給精排。
Retrieval requires precomputable item vectors. Strong interactions tied to a particular user and candidate belong in rich ranking.
q = f(user_history, session_prefix, context)
v_i = g(item_id, text, image, attributes)
s(q,i) = dot(q, v_i) / temperature
L = -log [ exp(s(q,i+)) / SUM over j in sampled_set exp(s(q,j)) ]
Use the same normalization and similarity definition in training and ANN serving.Query tower 可先用衰減平均，再嘗試 GRU／Transformer 的 session encoder；item tower 融合 ID 與內容。若用 cosine，訓練與索引都做一致正規化；若用 inner product，要檢查向量 norm 是否只是熱門度捷徑。Multi-interest encoder 可以避免把多種興趣壓成一個向量，但會增加召回與合併成本，應由候選覆蓋缺口驅動。
Start the query tower with decayed pooling, then test a GRU/Transformer session encoder. Fuse IDs and content in the item tower. For cosine, normalize consistently in training and indexing. For inner product, inspect whether vector norms merely encode popularity. Multi-interest encoders can preserve distinct tastes, but increase retrieval and merging costs; adopt them for demonstrated coverage gaps.
| 負例Negative source | 優點Benefit | 必要處理Required treatment |
|---|---|---|
| In-batchIn-batch | 便宜、大量共享計算Cheap, with shared computation | 受 batch 商品頻率影響；遮掉同 ID accidental positives，確認多正例定義Affected by batch item frequency; remove same-ID accidental positives and define multiple positives |
| 隨機抽樣Random sampling | 可控制分布與長尾 coverageControl the distribution and tail coverage | 記錄 Q(i)；不能只用太容易的負例Record Q(i); avoid relying only on easy negatives |
| Hard negativesHard negatives | 迫使模型區分相似但不適合的商品Distinguish similar but unsuitable products | 採樣來源版本化；小心相同 family、未觀測喜好與有效替代品的假負例Version the mining source; watch false negatives from families, unobserved preferences, and valid alternatives |
若採用 sampling-probability correction，Q 必須對應實際 sampler；混合 in-batch、uniform 與 hard mining 時，不能隨手用全站熱門度代替。TFRS Retrieval.call 有 sampling probability、accidental-hit removal 與 score mask 的接口可讀。這些是訓練機制，不表示線上 ANN 分數永遠也要減 log Q。
If using sampling-probability correction, Q must correspond to the actual sampler. A mixture of in-batch, uniform, and hard mining cannot casually use global popularity as Q. TFRS Retrieval.call exposes sampling probability, accidental-hit removal, and score masking. These are training mechanisms; they do not imply that serving ANN scores should always subtract log Q.
Relevant-item recall 很差，先換更好的 ANN 嗎？If relevant-item recall is poor, should you replace ANN first?
先用同一模型做 exact nearest-neighbor 檢索。如果 exact 也找不到標的，是表示、訓練目標或資料問題；如果 exact 可以但 ANN 不行，才調索引參數、量化與搜尋預算。再量測 locale／庫存過濾後的 recall，因為過濾可能讓有效候選數大幅縮水。
First run exact nearest-neighbor retrieval with the same model. If exact search also misses the target, investigate representations, objectives, or data. If exact succeeds and ANN fails, tune indexing, quantization, and search budget. Then measure recall after locale and inventory filtering, which can sharply reduce eligible candidate yield.
精排、多任務與校準Rich ranking, multitask learning, and calibration
模型輸出的意義必須清楚：相似度、點擊機率、條件轉換率、預期效用不是同一件事。先選對目標，再增加模型複雜度。
Be explicit about output semantics: similarity, click probability, conditional conversion rate, and expected utility are different quantities. Choose the target before increasing model complexity.
以 GBDT 加交叉特徵為 baseline；資料與容量足夠後比較 embedding＋MLP／cross network。若使用者有多種興趣，可用候選相關 attention 聚合歷史，例如 DIN 的概念：看到相機候選時，更關注攝影歷史，而非所有歷史同權。DIN 來自 Alibaba 的廣告研究，這裡只是模型設計參考，不是 Amazon 模型揭露。
Start with GBDT and cross features, then compare embeddings plus an MLP/cross network when data and capacity justify it. For multiple interests, candidate-aware attention can weight relevant history, as in DIN: a camera candidate should emphasize photography interactions rather than all history equally. DIN comes from Alibaba advertising research; it is a design reference here, not an Amazon model disclosure.
L_total = SUM over tasks t: lambda_t * SUM_n [ mask_nt * loss_t(y_nt, p_nt) ] / max(1, SUM_n mask_nt)
Tasks: click, attributed purchase, return conditional on an eligible purchase cohort.
Masks encode label maturity and task eligibility, not just missing values.多任務共享表示可以讓稀疏購買訊號借用較密集的點擊訊號，但需監看 negative transfer：某任務樣本多或 loss 尺度大，可能壓過真正目標。依有效樣本正規化、調整權重，分別報告各 head 與最終效用；不要只看 total loss 下降。標籤遮罩、權重與觀察窗也必須跟著模型版本保存。
Shared multitask representations can help sparse purchase signals benefit from denser click signals, but monitor negative transfer: a large task or loss scale can dominate the intended objective. Normalize by eligible examples, tune weights, and report each head and final utility separately. A lower total loss is insufficient. Version label masks, weights, and observation windows with the model.
pCTR × pCVR 何時成立？When is pCTR × pCVR valid?
只有當 purchase 事件明確被定義為 click 之後的轉換，且 pCVR = P(purchase | click, exposure, x)，才可用 P(click | exposure, x) × pCVR 表示該路徑的購買率。若 head 已直接預測 P(purchase | exposure, x)，再乘 pCTR 會重複折扣；有 view-through purchase 時也不能直接套這個分解。ESMM 是理解條件機率與樣本選擇問題的參考。
The factorization is valid only when purchase explicitly means a conversion after a click and pCVR = P(purchase | click, exposure, x). Then P(click | exposure, x) × pCVR represents that path's purchase rate. If a head directly predicts P(purchase | exposure, x), multiplying by pCTR discounts it twice. View-through purchases also require different semantics. ESMM is a useful reference for conditional probabilities and sample selection.
Example policy objective, not an Amazon formula:
U(u,i) = P(buy | exposure,x) * [1 - P(return | buy,x)] - lambda * fatigue(u,i)
Use compatible cohorts/windows; normalize penalties before combining scales.
The policy score U is not itself a probability.若只在履約商品上有 return 標籤，就不能無條件把它當成所有購買的退貨率；需對齊 cohort，或另外建模履約／取消。金額效用可加入預期淨價值，但要監測高價商品偏置與長期體驗。校準用保持母體比例的時間驗證資料，檢查 reliability curve、log loss、Brier score 與 locale／新品／高低價格切片。校準後仍需線上驗證效用。
If return labels exist only for fulfilled purchases, do not treat that head as the return rate for all purchases without aligning cohorts or modeling fulfillment/cancellation separately. Monetary utility may use expected net value, but monitor bias toward expensive items and long-term experience. Calibrate on chronological validation data preserving population proportions; inspect reliability curves, log loss, Brier score, and locale/new-item/price slices. Calibration does not replace online utility validation.
為什麼更複雜的序列模型可能輸給 GBDT？Why might a complex sequence model lose to GBDT?
可能是成熟標籤太少、序列很短、強結構化特徵已足夠，或負例與 serving 候選分布不一致。我會在同一候選池、同一時間切分、合理調參與相近成本下比較，再拆解 session 長度與冷啟動切片。模型複雜度不是成果；有效的品質、延遲與成本才是。
There may be too few mature labels, short sequences, sufficient structured features, or a mismatch between training negatives and serving candidates. I would compare on the same candidate pool and chronological split, with reasonable tuning and comparable cost, then break down performance by session length and cold start. Complexity is not an outcome; quality, latency, and cost are.
Eligibility 與整份推薦清單Eligibility and the recommendation slate
單品分數再好，也不能保證整頁推薦有用。硬限制先決定能不能推薦；清單策略再決定如何一起呈現。
Strong item scores do not guarantee a useful page. Hard constraints determine what can be recommended; slate policies determine how items work together.
| 規則Rule | 實作方式Implementation | 取捨Tradeoff |
|---|---|---|
| 供給與相容性Supply and compatibility | 地區、可配送、庫存、必要尺寸／型號、品質限制；最後再次查驗Region, delivery, inventory, required size/model, and quality constraints; recheck before rendering | 未知狀態需明確政策；不能用高模型分數補償不相容Define unknown-state policy; a high score cannot compensate for incompatibility |
| Family 去重Family deduplication | 相同 parent 的多個顏色／seller offer 合併，保留適合的 variantConsolidate colors/seller offers under a parent while preserving the appropriate variant | 尺寸與配件規格不能粗暴合併Do not collapse meaningful size or accessory-specification differences |
| 多樣性與疲勞Diversity and fatigue | 品類／品牌上限、最近曝光懲罰、greedy relevance–similarity 組合Category/brand caps, recent-exposure penalties, and greedy relevance–similarity tradeoffs | 對明確購物任務過度多樣化會失焦Excessive diversity can dilute a focused shopping mission |
| 探索Exploration | 在合格商品內分配有限探索槽位，記錄選擇機率與 policyAllocate bounded exploration slots among eligible items; log selection probabilities and policy | 探索不是把未檢查的新品任意塞進結果；注意長期回饋循環Exploration does not bypass new-item qualification; monitor long-term feedback loops |
Greedy slate illustration:
next_item = argmax over eligible i not in S:
calibrated_utility(i) - alpha * max_similarity(i, S) - beta * recent_exposure(i)
Subject to: family uniqueness, category/brand caps, compatibility, and supply.相機已購買後，可暫時降低同類耐用品，改用相容記憶卡／鏡頭／保護套；咖啡、耗材則可依補貨週期重新推薦。完成任務的判斷要吃到最新 session 訊號，且允許事件延遲與取消。這比一條全站『排除所有已購品』更符合實際行為。
After a camera purchase, temporarily suppress similar durable goods and consider compatible cards, lenses, or cases. Coffee and consumables can reappear according to replenishment cycles. Mission completion should use fresh session signals while accounting for late events and cancellations. This better matches behavior than one site-wide rule excluding all purchased items.
最後查庫存後只剩 8 個，怎麼辦？What if only eight items survive the final stock check?
在剩餘 deadline 與補取上限內，從已排序備選或額外召回補足，再執行相同資格檢查。若供給服務失效，可使用符合 freshness 門檻的已驗證候選；否則回傳較短清單並記錄原因。不要為了湊到 20 個繞過硬限制。追蹤每階段淘汰率，改善源頭候選品質。
Within the remaining deadline and refill budget, use ranked reserves or additional retrieval and apply the same qualification checks. If the supply service fails, use validated candidates only within the freshness policy; otherwise return a shorter list and log why. Do not bypass hard constraints to fill 20 slots. Track rejection rates by stage to improve candidate quality upstream.
訓練、版本與部署閉環Training, versioning, and the release loop
部署單位不只是權重檔。特徵定義、query／item encoder、索引、校準器與策略必須形成相容且可回滾的版本集合。
The deployable unit is more than weights. Feature definitions, query/item encoders, index, calibrator, and policy must form a compatible, reversible release bundle.
Ranker 的候選分布應盡量接近實際 serving：保存線上候選或用當時可用的 retriever 重播。不要拿已用該測試答案訓練的 retriever 產生候選，再宣稱是泛化能力。必要時以時間外／out-of-fold retrieval 建訓練資料。精排診斷可固定候選，但端到端測試不能偷偷注入正確答案。
Match ranker candidates to serving by retaining logged candidates or replaying a retriever available at the relevant time. Do not generate candidates with a retriever trained on the test answers and call the result generalization. Use out-of-time/out-of-fold retrieval where needed. Fixed pools are useful for ranker diagnostics, but end-to-end evaluation must not inject the correct answer.
| 迭代Iteration | 變更Change | 需要證明Required evidence |
|---|---|---|
| M0 → M1M0 → M1 | 熱門／item-to-item → 增加 two-towerPopularity/item-to-item → add two-tower | 相同 budget 下的增量 recall、長尾與新品 coverageIncremental recall and tail/new-item coverage at the same budget |
| M1 → M2M1 → M2 | 加入新鮮 session 與內容表示Add fresh session context and content representations | 短期意圖與冷啟動切片改善；特徵延遲與索引成本可接受Gains for immediate intent and cold start with acceptable feature lag and indexing cost |
| M2 → M3M2 → M3 | GBDT → 多任務／候選 attentionGBDT → multitask/candidate attention | 固定候選的品質、校準、尾延遲與成本Quality, calibration, tail latency, and cost on fixed candidates |
| M3 → M4M3 → M4 | 清單多樣性／探索策略Slate diversity/exploration policy | 線上有效購買、長期體驗與 guardrails；不是只看離線分數Online valid purchases, longer-term experience, and guardrails, beyond offline scores |
release_id: rec-example-v4
data_snapshot: immutable-snapshot-id
label_policy: exposure_purchase_return_v2
feature_schema: features_v7
vocabulary: vocab_v3
query_encoder: query_v4
item_encoder: item_v4
ann_index: items_v4_snapshot_01
ranker: rank_v6
calibrator: population_validation_v6
slate_policy: policy_v3
# Compatibility checks and rollback operate on this bundle.先離線 gate，再 shadow traffic 檢查結果、延遲與特徵缺值，接著小流量 canary 驗證運作，最後進入有穩定分流的 A/B。Shadow 不會產生替代結果的真實使用者回饋。新的 item index 準備好後才切相容的 query encoder；切換前後均保留舊 bundle，避免 encoder 與索引座標系不同。
Use offline gates, then shadow traffic for output, latency, and missing-feature checks, an operational canary, and finally an A/B test with stable assignment. Shadow traffic cannot produce real user feedback for unshown alternatives. Prepare the new item index before switching to the compatible query encoder, retaining the old bundle for rollback. Mixed encoder/index coordinate systems can silently break retrieval.
每次點擊都需要重新訓練嗎？Must the model retrain after every click?
不需要。新點擊可更新 session state 與即時特徵，影響下一次 query 向量；模型權重可以較慢更新。新商品可增量計算向量並入索引，庫存由更快的供給服務處理。把狀態新鮮度、索引新鮮度與參數新鮮度分開，才能在成本與反應速度間做可控選擇。
No. A new click can update session state and online features, changing the next query vector while weights refresh less frequently. New items can enter an incremental embedding/index pipeline, while inventory uses a faster supply service. Separating state, index, and parameter freshness makes the cost–responsiveness tradeoff manageable.
離線評估與線上實驗Offline evaluation and online experiments
先確認評估究竟量到什麼，再解讀數字。離線已觀測正例的排序品質，不等於推薦造成的增量價值。
Determine what a metric measures before interpreting its value. Ranking observed positives offline is not the same as incremental value caused by recommendations.
| 層次Layer | 核心指標Core metric | 如何解讀Interpretation |
|---|---|---|
| 模型召回Model retrieval | Relevant-item Recall@K、target coverageRelevant-item Recall@K and target coverage | 是否找得到已觀測的 relevant item；切分新品／長尾／localeWhether observed relevant items are retrieved; slice new items, tail items, and locales |
| ANN 系統ANN system | ANN Recall@K against exact top-K、搜尋延遲ANN Recall@K against exact top-K and search latency | 近似搜尋保留多少模型鄰居；不代表使用者喜歡那些鄰居How many exact model neighbors approximation preserves, not whether users like them |
| 固定候選排序Fixed-pool ranking | NDCG／MRR、log loss、校準NDCG/MRR, log loss, and calibration | 對同一候選池比較 ranker；rank loss 與 retrieval loss 分開Compare rankers on the same pool; separate ranking loss from retrieval loss |
| 端到端清單End-to-end slate | 過濾後 Recall／NDCG、有效候選數、違規／重複率、coveragePost-filter Recall/NDCG, eligible yield, constraint violations, duplicates, and coverage | 包含漏召回、過濾、去重與最終組合；不注入答案Include retrieval misses, filtering, deduplication, and composition; do not inject targets |
| 真實產品價值Actual product value | 成熟有效訂單／使用者、整站轉換、回訪與 guardrailsMature valid orders per user, site-wide conversion, repeat visits, and guardrails | 透過可信的線上實驗判斷；觀察跨 surface 搬移與退貨Assess through credible online experiments, including surface cannibalization and returns |
100 個抽樣負例不是完整商品世界One hundred sampled negatives are not the full catalog
SASRec Amazon 範例使用每位使用者 leave-one-out 與 100 個測試負例；這適合重現其 benchmark，但不能冒充全量候選或全域時間切分的線上重播。抽樣評估甚至可能改變模型優劣排序。報告必須寫清楚候選全集、抽樣分布、是否排除歷史互動、時間切分與未登入／新品的處理。
The SASRec Amazon example uses per-user leave-one-out with 100 test negatives. That can reproduce its benchmark, but is not full-catalog evaluation or a global chronological production replay. Sampled evaluation can even change relative model rankings. Report the candidate universe, sampling distribution, historical-item exclusions, time split, and handling of anonymous users and new items.
比較模型時固定資料、候選生成與計算預算，給 baseline 合理調參；報告多個 seed 或以使用者／session 為群組的 bootstrap 信賴區間。若公開資料沒有絕對事件時間，就遵循它的官方協定，另外清楚標示不能驗證的生產假設。Reviews benchmark 的正例是評論互動，不應命名成真實購買或 CTR。
Hold data, candidate generation, and compute budgets fixed, and tune baselines reasonably. Report multiple seeds or confidence intervals from user/session-clustered bootstrap. If a public dataset lacks absolute event time, follow its official protocol and explicitly identify untested production assumptions. A review benchmark's positives are review interactions, not verified purchase or CTR labels.
A/B 使用穩定的使用者分流，匿名流量則定義 identity 與流失規則；先做 A/A、曝光完整性、SRM 與 instrumentation 檢查。由流量、變異、MDE、檢定力和標籤成熟時間估算樣本與時長，而非固定『跑七天』。預先指定主要指標與 guardrails，考慮多重比較與 sequential monitoring；退貨或長期回訪需延長觀察。
Use stable user-level A/B assignment, defining identity and attrition rules for anonymous traffic. Check A/A behavior, exposure completeness, SRM, and instrumentation first. Estimate sample size and duration from traffic, variance, MDE, power, and label maturity rather than prescribing seven days. Predefine the primary metric and guardrails; account for multiple comparisons and sequential monitoring. Returns or repeat visits require longer observation.
只有舊策略日誌，可以用 IPS 取代 A/B 嗎？Can IPS replace A/B testing when only old-policy logs exist?
只能在明確假設下估計。需要舊策略對欲評估行動有 support、propensity 可用且記錄正確；slate 的聯合選擇機率尤其困難。檢查極端權重、有效樣本數，必要時 clipping 並交代偏差／方差。對從未曝光的新商品或新策略區域，沒有資料就不能靠加權創造答案。
Only under explicit assumptions. The logging policy needs support for actions being evaluated, with usable and correctly logged propensities; joint slate probabilities are especially difficult. Inspect extreme weights and effective sample size, and disclose the bias–variance tradeoff if clipping. Weighting cannot invent evidence for never-exposed new items or unsupported policy regions.
Offline NDCG 上升、線上購買下降，怎麼排查？Offline NDCG rises but online purchases fall. What next?
先確認實驗有效與 serving 版本／特徵一致，再查離線洩漏、抽樣候選過於容易、負例與線上不同、庫存過濾及 latency 變化。接著按新品、locale、session 長度拆解，檢查是不是點擊誘人但購買價值差，或只是把其他 surface 的訂單搬過來。不要只重新調一個 loss 權重就結案。
First verify experiment validity and serving version/feature consistency. Then check leakage, easy sampled candidates, training–serving negative mismatch, inventory filtering, and latency. Slice by new items, locale, and session length. Determine whether the model attracts clicks without purchase value or moves orders from another surface. Do not stop after adjusting one loss weight.
Serving、容量與降級Serving, capacity, and degradation
用數量級推導瓶頸，再用真實流量與尾延遲驗證。每個降級路徑也都是使用者實際會看到的產品。
Use order-of-magnitude estimates to identify bottlenecks, then validate with realistic traffic and tail latency. Every fallback path is also a product users will experience.
Raw item vectors: 10,000,000 items * 128 dimensions * 2 bytes = 2.56 GB (decimal)
This excludes ANN structures, metadata, replicas, and simultaneous old/new indexes.
Light ranking: 5,000 requests/s * 1,000 candidates = 5,000,000 pairs/s
Rich ranking: 5,000 requests/s * 200 candidates = 1,000,000 pairs/s
All figures are hypothetical sizing inputs, not measured throughput.150 ms 是整個 API 的 p95 目標，不能把每階段 p95 相加就當作整體 p95。以 critical path 安排 deadline，多路 retrieval 並行，feature lookup 批次化，保留排隊、網路、最後資格查驗與序列化時間。容量測試要涵蓋尖峰、hot keys、cache miss、慢 partition、batch size、並發與 fallback；平均延遲不夠。
A 150 ms API p95 target is not obtained by adding stage p95s. Budget deadlines along the critical path, retrieve in parallel, batch feature lookups, and reserve time for queues, network, final qualification, and serialization. Load tests should cover bursts, hot keys, cache misses, slow partitions, batch sizes, concurrency, and fallback. Average latency is insufficient.
| 狀態State | 更新策略Refresh strategy | 失效處理Invalidation / failure |
|---|---|---|
| 長期偏好／統計Long-term affinity / aggregates | 批次＋必要的 stream aggregationBatch plus selected streaming aggregates | 版本與 freshness metadata；過期值帶 indicatorVersion/freshness metadata; mark stale values explicitly |
| SessionSession | 事件到達後更新短期狀態Update short-term state as events arrive | 晚到／重複事件處理，缺值退回長期偏好與 contextHandle late/duplicate events; fall back to long-term affinity and context |
| Item embeddings／ANNItem embeddings / ANN | 新商品增量更新，模型大改以新版本重建Increment new items; rebuild a version for major model changes | 兼容 query encoder、tombstones 與 deleted-item 過濾Compatible query encoder, tombstones, and deleted-item filtering |
| 價格／庫存Price / inventory | 更快的供給 cache／服務，freshness 門檻Faster supply caches/services with freshness thresholds | 渲染前再查驗；cache key 含 locale 與必要 offer contextRecheck before rendering; key caches by locale and relevant offer context |
def recommend(request, release, deadline):
ctx = get_context_and_features(request, deadline)
pools = parallel_retrieve(ctx, release, timeout=remaining_budget(deadline))
candidates = merge_ids_and_provenance(pools)
candidates = early_qualify(candidates, request.locale)
top = light_rank(batch_features(ctx, candidates), release)[:200]
scores = rich_rank_or_light_fallback(ctx, top, release, deadline)
slate = compose_and_recheck_supply(scores, request, deadline, limit=20)
slate = bounded_refill_if_needed(slate, candidates, deadline)
log_decision(request, slate, release, fallback_reason(), stage_counts())
return slate # may be shorter; render/view events are logged separatelyANN 失效時用 item-to-item＋分群熱門；精排超時時沿用輕量排序；特徵缺失時用明確的 missing path，避免把缺值解讀成零行為。Cache 可保存候選 ID 與模型版本、locale、歷史 signature，但庫存仍要新鮮查驗。設置 deadline、circuit breaker 與分階段 trace，並對 fallback 流量單獨追蹤品質。
On ANN failure, use item-to-item plus segmented popularity. On rich-ranker timeout, retain light-ranking order. Missing features use an explicit path rather than masquerading as zero behavior. Cache candidate IDs with model version, locale, and history signature, but revalidate supply freshness. Use deadlines, circuit breakers, and stage traces, and track quality separately for fallback traffic.
最先監控哪些指標？What would you monitor first?
系統看流量、錯誤、p50／p95／p99、timeout、fallback 與成本；資料看 freshness、缺值、OOV、事件到達延遲、候選數與資格淘汰率；模型看分數／向量 norm／品類分布、校準與成熟後結果。每條紀錄帶 release、index 和 feature 版本，讓品質下降可以追到具體變更。
For systems: traffic, errors, p50/p95/p99, timeouts, fallback, and cost. For data: freshness, missing values, OOV, event lag, candidate counts, and qualification rejection rates. For models: score/vector-norm/category distributions, calibration, and matured outcomes. Attach release, index, and feature versions so quality regressions can be traced to changes.
互動案例：失敗與取捨Interactive cases: failures and tradeoffs
切換五種情境，先口述你的判斷，再核對建議。情境皆為原創教學案例；數字與結果不是實驗報告。列印時會展開全部案例。
Switch between five scenarios, state your decision, then compare with the proposed response. These are original teaching cases, not experimental results. Printing expands every scenario.
同一位使用者有多種興趣One user, several interests
使用者長期買咖啡，今天連看三台相機；多路召回同時包含咖啡、相機與配件。
A frequent coffee buyer views three cameras today; retrieval returns coffee, cameras, and accessories.
提高當前 session 的相機意圖，但保留少量合理探索；商品頁將替代品與配件分組，首頁清單避免被同一 family 占滿。
Emphasize current camera intent while retaining bounded exploration. Separate substitutes and accessories on product pages; prevent one family from dominating the homepage.
檢查 session 特徵是否到達、來源增量 recall、family 重複與整頁任務完成率；不要只看相機 CTR。
Check session feature arrival, incremental source recall, family duplicates, and task completion, not just camera CTR.
買完相機還一直推薦相機Cameras keep appearing after purchase
購買事件已產生，但批次特徵仍停留在昨天；使用者需要相容記憶卡，候選卻以相似相機為主。
A purchase event exists, but batch features are still from yesterday. The user needs a compatible card while candidates mostly contain similar cameras.
用新鮮 session／購買狀態更新 query 與耐用品抑制；另開互補來源並做型號相容性檢查。取消訂單可解除抑制，耗材不套永久排除。
Update the query and durable-item suppression with fresh purchase/session state. Add complementary retrieval with model compatibility checks. Cancellations can lift suppression; consumables should not be permanently excluded.
驗證事件可用時間與 end-to-end lag，區分沒有召回配件和配件被錯排；量測完成任務後的重複曝光。
Verify event availability and end-to-end lag; distinguish missing accessories from misranking, and measure repetitive exposure after mission completion.
新商品有內容，卻沒有行為A new item has content but no history
新相機有文字、圖片、規格和有效庫存，但 ID 不在訓練 vocabulary，熱門與 co-occurrence 路徑不會召回它。
A new camera has text, images, specifications, and stock, but its ID is outside the training vocabulary and popularity/co-occurrence cannot retrieve it.
以內容 tower 生成向量並增量入索引，缺少 ID 時走已訓練的 cold path；合格商品可進入有限探索，不偽造歷史 CTR。
Embed content and incrementally index the item, using a trained cold path when its ID is missing. Allow bounded qualified exploration without fabricating historical CTR.
以 item cold-start split 比較 ID-only 與內容模型；監控新商品入索引延遲、coverage 與成熟後品質。
Compare ID-only and content models on an item cold-start split; monitor indexing lag, coverage, and matured quality.
高分清單多數不能買Most high-scoring items cannot be bought
快取清單 20 個商品中有 12 個已缺貨；距離 deadline 很短，供給服務有部分 timeout。
Twelve of 20 cached items are out of stock. Little deadline budget remains, and the supply service partially times out.
先移除已知不合格商品，依剩餘 budget 從已排序備選補取並查驗；unknown 依既定 freshness 規則處理，必要時回傳短清單。
Remove known-ineligible items, then refill from ranked reserves within budget and revalidate. Handle unknowns under the freshness policy and return a shorter slate when necessary.
監控各階段候選流失、供給 cache age、補取次數、短清單率與 p99；不能以填滿槽位作唯一成功條件。
Monitor stage attrition, supply-cache age, refill attempts, short-slate rate, and p99; filled slots are not the sole success criterion.
CTR 增加，但有效購買沒有變好CTR improves without better valid purchases
新版本吸引更多點擊，但訂單沒有增加，成熟 cohort 的退貨率偏高。離線測試只對 100 個抽樣負例算 NDCG。
The new version attracts more clicks, but orders do not increase and the mature cohort has more returns. Offline testing used NDCG against only 100 sampled negatives.
先檢查 SRM、曝光與標籤成熟，再拆解價格／品類與來源。重做 serving-like 候選評估，校準購買／退貨目標；guardrail 未過不擴大發布。
Check SRM, exposures, and label maturity first, then slice price, category, and sources. Reevaluate with serving-like candidates and calibrate purchase/return objectives; do not expand while guardrails fail.
檢查是否推薦高點擊低效用商品、是否 cannibalize 其他 surface，以及 sampling、洩漏或特徵版本差異。
Check for click-attractive but low-utility items, cannibalization of other surfaces, sampling artifacts, leakage, and feature-version mismatch.
失敗分析要定位在哪一層：訊號沒有到達、商品沒有進候選、特徵錯誤、目標不對、清單規則不合理，或實驗量測失真。每個修正都附一個能否證實的檢查，避免把所有問題都歸因於模型太小。
Localize failures: a signal never arrived, an item missed retrieval, features were wrong, the objective was misaligned, slate rules were poor, or measurement was invalid. Pair each fix with a falsifiable check instead of attributing every problem to insufficient model size.
程式碼導讀與可重現練習Code reading and reproducible practice
使用公開資料建立誠實的練習任務。沒有曝光與交易資料，就不能把模型包裝成已驗證的購買／退貨預測系統。本篇未執行模型訓練。
Build an honest practice task from public data. Without exposures and transactions, do not present a model as a validated purchase/return predictor. No model training was run for this guide.
| 程式來源Code source | 閱讀重點What to inspect | 改成商品練習時When adapting it |
|---|---|---|
| Recommenders SARRecommenders SAR | user affinity、item similarity、稀疏計算與推薦組合User affinity, item similarity, sparse computation, and aggregation | 強 baseline；不要宣稱是 Amazon 原始 item-to-item 的精確程式A strong baseline; not the exact original Amazon item-to-item implementation |
| TFRS basic_retrieval＋retrieval.pyTFRS basic_retrieval + retrieval.py | 兩個 tower、candidate dataset、loss correction、accidental hits、索引匯出Towers, candidate dataset, loss correction, accidental hits, and index export | 原教學是 MovieLens；重定義 item、時間切分、抽樣與完整候選評估The tutorial uses MovieLens; redefine items, splits, sampling, and full-candidate evaluation |
| SASRec Amazon notebookSASRec Amazon notebook | 序列切片、padding mask、leave-one-out、num_neg_test=100Sequence slicing, padding masks, leave-one-out, and num_neg_test=100 | 先重現原 protocol，再另建更接近 serving 的評估，不能混報分數Reproduce its protocol first, then build serving-oriented evaluation and report scores separately |
| AmazonReviews2023 / BLaIRAmazonReviews2023 / BLaIR | benchmark_scripts、blair、seq_rec_results 與商品語義表示benchmark_scripts, blair, seq_rec_results, and product semantic representations | 檢查 metadata 時間、資料版本、cold-item split；不要把 review 當曝光Check metadata timing, dataset versions, and cold-item splits; reviews are not exposures |
| NVIDIA Merlin session 範例NVIDIA Merlin session example | 資料處理、序列 next-item 與訓練流程Data processing, sequential next-item prediction, and training workflow | 重查相依版本與硬體需求，先以小資料 smoke run 驗證Recheck dependencies/hardware requirements and start with a small-data smoke run |
| Amazon DSSTNEAmazon DSSTNE | 歷史推薦模型／大規模 sparse input 的工程設計Historical recommender engineering and large sparse inputs | Repository 已封存，適合讀歷史設計，不作新專案的默認起點The repository is archived; read it for historical design, not as the default starting point for a new project |
實作 A：M2 next-itemBuild A: M2 next-item
保留官方 split 與 locale，先做 popularity＋co-occurrence，再 two-tower／session encoder，最後固定候選的 GBDT。輸出候選 recall、排序指標、有效候選數與來源消融，明確標成 next-item benchmark。
Preserve official splits/locales. Start with popularity and co-occurrence, then add a two-tower/session encoder and fixed-pool GBDT. Report candidate recall, ranking metrics, eligible yield, and source ablations, explicitly as a next-item benchmark.
實作 B：內容與冷啟動Build B: content and cold start
用 Reviews 2023 的互動與商品內容比較 ID-only、text-only、ID＋text／image。以明確商品冷啟動切分測試，排除測試答案文字洩漏；缺乏歷史 metadata 的限制要記錄。
Use Reviews 2023 interactions and content to compare ID-only, text-only, and ID plus text/image models. Define an item cold-start split, prevent answer-text leakage, and record limitations from missing historical metadata.
實作 C：服務契約Build C: serving contracts
用合成 fixture 測試庫存翻轉、重複 variant、未知特徵、index 版本錯配、deadline 與 fallback。這可以驗證服務行為，不能宣稱真實購買提升。
Use synthetic fixtures for inventory changes, duplicate variants, unknown features, incompatible index versions, deadlines, and fallback. This validates serving behavior, not real purchase lift.
experiment/
manifest.json # data versions, protocol, seeds, package versions
data_contract.md # observed labels vs unavailable business events
prepare.py # deduplication, split, vocab, masks
baseline.py # popularity + sparse item-to-item
train_retrieval.py # sampler + towers + exact/ANN comparison
train_ranker.py # candidate provenance + fixed-pool diagnostics
evaluate.py # full protocol, slices, uncertainty, ablations
serve.py # bounded retrieval/ranking/refill
fixtures/ # supply, missing features, timeouts, version errors
report.md # quality/latency/cost and explicit limitations下方來源清單提供直接連結。閱讀 repo 時先鎖定 commit 與套件版本，再確認資料取得、license 與硬體需求；依序檢查 data loader → label／sampler → forward／loss → evaluator → export。先追清楚一筆資料的生命週期，通常比先啟動大型訓練更能揭露假設。
Direct links appear in the source list below. Pin a commit and package versions, then check data access, licensing, and hardware requirements. Follow the data loader → labels/sampler → forward/loss → evaluator → export. Tracing one example through the lifecycle often reveals more assumptions than immediately launching a large training run.
45 分鐘面試口說流程A 45-minute interview walkthrough
沿用同一條回答主線：釐清產品目標 → 資料與標籤 → baseline／模型 → 訓練與部署 → 離線／線上評估 → 失敗與取捨。先說決策，再說原因與驗證方式。
Use one narrative: goals → data and labels → baselines/models → training and deployment → offline/online evaluation → failures and tradeoffs. State a decision, its rationale, and how you would validate it.
00–05 分鐘：釐清產品目標00–05 minutes: clarify goals
我先聚焦首頁 20 個個人化商品，支援登入與匿名 session。商品頁的替代／互補是不同任務，會分開評估。主要目標是每位使用者的有效購買，CTR 作為診斷，同時保護取消、退貨、回訪與延遲。假設 1,000 萬商品、尖峰 5,000 QPS、p95 150 ms；這些是設計假設，實務上會以需求與流量確認。
I will focus on 20 personalized homepage products for signed-in users and anonymous sessions. Substitutes and complements on product pages are distinct tasks to evaluate separately. The primary objective is valid purchases per user, with CTR as a diagnostic and guardrails for cancellations, returns, repeat visits, and latency. I will assume 10 million items, 5,000 peak QPS, and 150 ms p95, then validate these assumptions against actual requirements.
05–12 分鐘：資料與標籤05–12 minutes: data and labels
我會記錄 request、候選來源、實際曝光、位置、互動與版本，並區分 product、variant、offer。特徵只能使用當下已可用的資料，不能只檢查事件時間。點擊與購買負例需完整觀察窗，退貨只在合格購買 cohort 建模；未曝光不是負回饋。採時間切分、成熟遮罩與 train-only vocabulary，避免未來資料洩漏。
I would log requests, candidate provenance, actual exposures, positions, interactions, and versions, distinguishing products, variants, and offers. Features must have been available at decision time, not merely carry an earlier event timestamp. Click and purchase negatives need complete observation windows; return labels apply only to eligible purchase cohorts. Unexposed items are not negative feedback. I would use chronological splits, maturity masks, and training-only vocabularies to prevent leakage.
12–22 分鐘：Baseline 與模型12–22 minutes: baselines and models
我先建分群熱門與 item-to-item，再加入內容＋行為 two-tower，合併約 1,000 個候選。商品向量預計算，query 使用最新 session。負例混合 batch／random／hard，處理抽樣分布與假負例。輕量排序取約 200 個，精排先用 GBDT，再驗證多任務或候選 attention。最後以相容性、庫存、family 去重與多樣性組成最多 20 個。
I would start with segmented popularity and item-to-item, then add a content-and-behavior two-tower retriever to form about 1,000 candidates. Item vectors are precomputed; queries use fresh session context. I would mix batch, random, and hard negatives while handling sampling distributions and false negatives. Light ranking selects about 200 candidates. Rich ranking starts with GBDT before testing multitask or candidate-aware attention. Compatibility, inventory, family deduplication, and diversity produce up to 20 items.
22–32 分鐘：訓練與部署22–32 minutes: training and deployment
我會版本化資料、特徵、兩個 encoder、index、ranker、校準器和清單策略；用與 serving 接近的候選訓練 ranker。先離線評估，再 shadow、canary 和 A/B。新的 item index 必須與 query encoder 一起切換且可回滾。Session 可即時更新，模型不必每次點擊就訓練。線上批次取特徵、並行召回，保留硬限制查驗與有界降級。
I would version data, features, both encoders, index, ranker, calibrator, and slate policy, and train ranking on serving-like candidates. Releases pass offline checks, shadow traffic, a canary, and A/B testing. A new item index must switch with its compatible query encoder and support rollback. Session state can update immediately without retraining on every click. Serving batches features, retrieves in parallel, and preserves hard qualification checks and bounded fallback.
32–40 分鐘：評估與實驗32–40 minutes: evaluation and experiments
我會分開看 relevant-item recall 與 ANN 對 exact neighbors 的 recall，再看固定候選排序、校準及最後清單品質。抽樣負例的 NDCG 不能冒充完整候選評估。線上用穩定使用者分流，先檢查 A/A、SRM 與曝光完整性；依 MDE、變異和標籤成熟估算時長。主要看成熟有效訂單與整站價值，避免只把其他位置的點擊和訂單搬過來。
I would separate relevant-item recall from ANN recall against exact neighbors, then inspect fixed-pool ranking, calibration, and final slate quality. NDCG with sampled negatives must not masquerade as full-candidate evaluation. Online, I would use stable user assignment and first check A/A behavior, SRM, and exposure completeness. Duration depends on MDE, variance, and label maturity. I would measure mature valid orders and site-wide value to detect mere redistribution across surfaces.
40–45 分鐘：失敗案例與取捨40–45 minutes: failures and tradeoffs
若買完相機仍看到相機，我先查購買訊號延遲與 mission state，再調耐用品抑制和相容配件。若新品沒曝光，檢查 OOV、內容表示、索引更新與合格探索。若庫存大量淘汰，改善 early filter 與有界補取。若離線好但線上差，先查實驗與 train–serve skew，再查目標與偏差。我會保留強 baseline，只有收益足以支付延遲與維運成本才增加複雜度。
If cameras persist after purchase, I would inspect purchase-signal lag and mission state before changing durable-item suppression and compatible accessories. For unseen new products, check OOV handling, content representations, index freshness, and qualified exploration. For high stock rejection, improve early filtering and bounded refill. If offline gains fail online, first check experiments and train–serve skew, then objectives and bias. I would retain strong baselines and add complexity only when gains justify latency and operational cost.
來源與程式碼導讀Sources and code reading
以下區分原始研究、官方文件、程式與教學框架。來源支援各自概念；本篇整合架構、假設數字、案例與口說稿為自行整理。未複製來源圖，也未宣稱取得 Amazon 現行內部架構或執行模型訓練。外部連結需網路，正文與圖表可離線閱讀。
The list distinguishes original research, official documentation, code, and teaching frameworks. Each supports specific concepts; the integrated architecture, assumed numbers, scenarios, and scripts are original synthesis. No source artwork was copied, no current internal Amazon architecture is claimed, and no model training was run. External links need a network; text and diagrams work offline.
- Amazon Science · Two Decades of Recommender Systems ↗
讀 item-to-item 的離線關聯與線上組合，以及不同推薦位置的需求；歷史文章不能代表目前完整架構。
Read offline item relationships and online aggregation, with different recommendation surfaces. This historical account does not disclose the complete current architecture.
- Amazon Science · Improving Complementary-product Recommendations ↗
看互補／替代關係、類型多樣性與方向性；本文相機案例是自行設計。
Study complement/substitute relationships, type diversity, and directionality. The camera example here is an original teaching scenario.
- NVIDIA · Pro Tips for Building Multilingual Recommender Systems ↗
讀候選生成、session × candidate 資料表與排序特徵。文章介紹 KDD Cup 方法；不等於線上服務效能證據。
Read candidate generation, the session-by-candidate dataframe, and ranking features. The KDD Cup walkthrough is not evidence of online serving performance.
- Amazon-M2 · Challenge and Dataset Specification ↗
六個 locale 的匿名 session、商品屬性與官方切分。標的是下一個互動商品，不是完整曝光、購買與退貨紀錄。
Anonymous sessions, product attributes, and official splits across six locales. The target is the next engaged product, not a complete exposure, purchase, and return log.
- McAuley Lab · Amazon Reviews 2023 ↗
評論、商品 metadata、圖片與時間欄位；評論時間不等於購買時間，metadata 也未必有歷史版本。
Reviews, product metadata, images, and timestamps. Review time is not purchase time, and metadata may lack historical versions.
- TensorFlow Recommenders · Basic Retrieval ↗
Two-tower、完整候選評估與 ANN 匯出；原教學用 MovieLens，商品任務需改標籤與切分。
Two towers, full-candidate evaluation, and ANN export. The tutorial uses MovieLens; adapt labels and splits for commerce.
- TensorFlow Recommenders · retrieval.py ↗
已查核 candidate_sampling_probability、candidate_ids、score_mask 與 remove_accidental_hits 的用途；本篇未執行訓練。
Inspected candidate_sampling_probability, candidate_ids, score_mask, and remove_accidental_hits. No training was run for this guide.
- Google Research · Sampling-Bias-Corrected Neural Modeling ↗
理解 in-batch negatives 與長尾分布下的抽樣偏差；訓練校正必須對應實際 sampler。
Understand in-batch negatives and sampling bias under a long-tailed distribution. Training correction must match the actual sampler.
- Deep Interest Network for Click-Through Rate Prediction ↗
候選相關的歷史注意力；原始研究來自 Alibaba 廣告場景，不是 Amazon 的公開模型說明。
Candidate-aware attention over history. The original work concerns Alibaba advertising, not a disclosure of Amazon's models.
- Entire Space Multi-Task Model ↗
讀曝光→點擊→轉換的條件機率與選擇偏差；不能把多任務學習當作解決所有曝光偏差。
Read conditional probabilities and selection bias along exposure, click, and conversion. Multitask learning does not resolve every exposure bias.
- Feast · Point-in-time Joins ↗
依歷史時間重建特徵；精確重播線上已知狀態，還要處理遲到與可用時間。
Reconstruct historical features. Exact replay of online knowledge also requires late-arrival and availability-time semantics.
- Amazon Personalize · Use Case and Recipe Features ↗
區分即時行為適應、探索與模型更新；服務功能不能當成完整 Amazon 零售架構。
Separate real-time behavioral adaptation, exploration, and model updates. Service features do not describe the complete Amazon retail architecture.
- Amazon Personalize · Filtering Recommendations ↗
了解推薦結果如何受商品與互動條件限制；本文的庫存查驗與補取策略是設計建議。
Understand item and interaction constraints on recommendations. Live availability checks and refill policies in this guide are proposed designs.
- Google Research · On Sampled Metrics for Item Recommendation ↗
抽樣負例可能改變模型優劣排序；區分完整候選、抽樣 benchmark 與 ANN 評估。
Sampled negatives can change relative model rankings. Separate full-candidate, sampled-benchmark, and ANN evaluation.
- Google Research · On the Difficulty of Evaluating Baselines ↗
強 baseline 需要合理調參與一致的評估協定，不能以弱對照製造模型進步。
Strong baselines require reasonable tuning and consistent evaluation protocols; weak controls can manufacture apparent progress.
- Microsoft Research · Diagnosing Sample Ratio Mismatch ↗
先排查分流與資料品質，再解讀 lift。本文觀察窗、MDE 與 rollout 設定需由實際資料決定。
Check assignment and data quality before interpreting lift. Observation windows, MDE, and rollout settings require real data.
- Recommenders · SAR Deep Dive Notebook ↗
追 user affinity、item similarity 與推薦組合；SAR 不等於 Amazon item-to-item 的精確實作。
Trace user affinity, item similarity, and recommendation aggregation. SAR is not an exact implementation of Amazon's item-to-item system.
- Recommenders · SASRec on Amazon Notebook ↗
已查看程式：使用 leave-one-out，num_neg_test = 100。適合學流程，數值不可直接當全商品庫效果。
Code inspected: leave-one-out splitting and num_neg_test = 100. Useful for learning the workflow; metrics are not full-catalog results.
- Amazon Reviews 2023 / BLaIR · Processing and Models ↗
讀 benchmark_scripts、blair、seq_rec_results；追溯前處理、商品文字表示與序列推薦重現設定。
Read benchmark_scripts, blair, and seq_rec_results for preprocessing, item text representations, and sequential recommendation experiments.
- NVIDIA Merlin · Session-based Next-item Notebook ↗
看 session 資料如何進模型；安裝前核對套件與 GPU 相依版本，本篇沒有宣稱可直接重現。
Inspect how session data enters the model. Verify package and GPU dependencies before installation; no reproduction is claimed here.
- Amazon DSSTNE · Historical Sparse Recommendation Engine ↗
可讀稀疏推薦與多 GPU 設計；已封存，適合歷史工程導讀，不作新實作起點。
Study sparse recommendation and multi-GPU design. Archived; useful historical engineering material, not the starting point for a new implementation.
- Hello Interview · Recommendation Systems Pattern ↗
參考召回→排序→評估的說明方式；教學方案不代表公司內部架構或特定考題。
Learn how to explain retrieval, ranking, and evaluation. A teaching framework is not evidence of an internal architecture or a particular interview question.