Design a video recommendation feed
Design a TikTok-style video recommendation feed, step by step. Grow one architecture from a trending list to two-tower candidate retrieval, a multi-task ranker with a feature store, real-time session signals and exploration for new videos, with estimates, evals and code in Java, Python, C#, Go and TypeScript.
“Design the For You feed of a short video app. A user opens the app and swipes. Every next video should be one they want to watch.”
This is the classic machine learning design question. The model is the easy part to describe. The hard parts are around it:
- Choosing 10 videos out of a billion, in a fifth of a second.
- Reacting to a swipe within the same session.
- Giving 10 million new uploads a day a fair chance, when nobody has watched them yet.
We start with a plain trending list, and add one part at a time, only when a number forces us to. LLMs aren’t the core of this system, and the page shows why.
The answer in one minute
Section titled “The answer in one minute”If you only have a minute, this is the design. The rest of the page explains each part.
- A funnel. No model can score a billion videos per request. So a cheap step finds about 3,000 likely videos, a small model keeps 500, and a big model orders those. We show 10.
- The big model predicts several things: will they finish it, like it, share it, or tap “not interested”. A weighted sum, chosen by the product team, turns them into one score.
- Fresh inputs, not a fresh model. A stream turns each swipe into session signals within a second. The next page uses them.
- 5% of slots test new videos, so new uploads get an audience and the feed doesn’t narrow.
- Every step has a cheaper fallback. The last one is the trending list, which never fails.
Scope it
Section titled “Scope it”A few questions change the design a lot. For the rest, we assume an answer and move on.
Design the For You feed of a short video app.
Whether you ask what “good” means before anything else. The goal shapes the model, the labels and the metrics. Asking about video encoding or comments spends time on a different system.
These five questions matter most:
- What do we optimize? This one matters most. We aim for long-term satisfaction: watch time and finishing videos. “Not interested” and reports act as guardrails, limits a change must not break. Optimizing clicks alone builds a clickbait machine within weeks.
- How many users? 200 million a day, about 100 videos each.
- How big is the catalog? 1 billion videos, plus 10 million uploads a day.
- How fast must the feed react? A skip changes the next page. A new upload can reach viewers within an hour.
- Discovery or follows? Discovery: anything in the catalog. Followed creators are one source.
We leave out upload and video delivery (see video streaming), moderation, ads and search. We receive a “video ready” event once a video is processed and checked.
Go deeperThe full requirements and quality targets3 min
Must do. Return pages of 10 videos, endlessly, with no video the user saw in the last 30 days. Views, skips, likes, shares, follows and “not interested” change later pages, including the very next one. A new upload that passes checks can reach viewers within an hour. A new user gets a reasonable feed on the first open. Removed videos are never shown, and age and country rules hold.
Speed. Under 200 ms of server time per page for 99% of requests. The app fetches the next page early, so users never feel it.
Availability. 99.95%. The feed always returns something. A worse page beats an error.
Freshness. A skip reaches the next page within 2 seconds. Video counters lag under a minute. Models are under 36 hours old.
Quality.
- Retrieval finds tomorrow’s finished videos as often as production does, within 1%.
- Ranking is no worse than production on next-day logs, and each prediction is accurate in every score range, within 5%.
- Online, watch time per user wins its A/B test, and no guardrail gets worse by more than 1%.
- New videos: 95% of eligible uploads get 200 views within 24 hours.
- Variety: at most one video per creator on a page.
| More questions | We assume | If the answer were different |
|---|---|---|
| Regions and languages? | Global. Most users watch in their own language | One market fits one region, and trending is simpler |
| Ads in the feed? | An ad server fills reserved slots | Blending ads turns the last step into an auction |
| A small catalog? | No, 1 billion | 20,000 titles could be scored in full, with no retrieval step |
How much AI it needs
Section titled “How much AI it needs”Do you need machine learning for this? Couldn’t you just show what’s popular?
Whether you start from the cheapest thing that works and say exactly where it falls short. Jumping to a deep network without mentioning popularity reads as memorized. So does putting an LLM in the main path without doing the math.
Start from the cheapest option, and move up only where it falls short:
- Color key
- Too simple
- The funnel
- Picked
- Too costly
- Ruled out
Popular videos work as a first version and as the fallback. But everyone sees the same feed. “Watched this, also watched that” is cheap and strong, but a new video has no co-watches. So we pick a learned funnel: a step that finds a few thousand likely videos, then models that rank them. The models read features, like topic, language and creator, not just IDs. That is how they handle new users and new videos.
Why not an LLM? About 3,000 candidates at roughly 200 tokens each is 600,000 tokens per page. At 70,000 pages a second, that is 42 billion tokens a second. LLMs help offline instead: tags, captions and text summaries of videos that become features.
The numbers that decide it
Section titled “The numbers that decide it”How big is this? What does one feed request cost?
Whether the numbers size the funnel: requests per second, candidates and scores per request, GPUs, feature reads and events, and what each one decides. Storage for video files belongs to a different system.
- Daily active users
- 200 million
- Videos viewed per user per day
- 100
- Videos per feed page
- 10
- Peak over average
- 3×
- Candidates per request
- 3,000
- Videos the heavy ranker scores
- 500
- Heavy ranker scores per second per GPU
- 100,000
- Videos in the retrieval index
- 1 billion
- Bytes per indexed video (compressed vector + id)
- 72 bytes
- Bytes per event
- 200 bytes
- Days of raw events kept
- 90
200 million users × 100 videos ÷ 10 per page is 2 billion pages a day. Divide by 86,400 seconds. Evenings run about three times the average.
So we size every step for 70,000 pages a second, and keep the feed servers stateless.
70,000 requests × 500 videos. One GPU scores about 100,000 a second, so that is 350 GPUs fully busy, about 600 at 60% busy. At $2 an hour, about $29,000 a day, or 1.4 cents per 1,000 pages.
So we treat the big model's input size as a budget. Halving it from 500 to 250 halves the GPUs.
70,000 requests × 3,000 candidates, at about 2 microseconds each.
So we let a small CPU model see every candidate, so the GPU model sees only 500.
The big model needs a row for each of its 500 videos, plus the user and session rows. One key per feature would be 100 times more reads.
So we store features as one row per video and one per user, read in one batch per request.
About one event per video viewed (watch time, finished, skipped), plus likes and follows, at about 200 bytes each.
So we put events on a stream split by user, and keep 90 days for training.
Notice that feature reads and events are as big a job as the model. The design has to plan for all three.
Go deeperThe calculator, and the size of the search index2 min
- Feed requests, average
- 23,000/s
- Feed requests, peak
- 69,000/s
- Stateless feed servers scale out; the cost is in what each request does.
- Light ranker scores, peak
- 210,000,000/s
- At about 2 µs a score on CPU, a few hundred cores. Cheap per item, so it can see every candidate.
- Heavy ranker scores, peak
- 35,000,000/s
- GPUs for the heavy ranker, fully busy
- 350
- Provision about 1.7× this to run near 60% busy. The heavy ranker is the biggest serving cost.
- Feature rows read, peak
- 35,000,000/s
- A large in-memory cluster can serve this if features are packed into one row per video and per user, never one key per feature.
- Events (views, skips, likes), average
- 230,000/s
- Events, peak
- 690,000/s
- One event stream, partitioned by user, so one consumer owns each session.
- Raw events kept
- 360 TB
- Columnar files in object storage. Training reads a sample of negatives, not all of it.
- Retrieval index in memory
- 72 GB
- Fits in one large machine compressed. Shard it anyway, for query load and latency.
The search index. Each video is turned into 128 numbers. As 32-bit numbers, that is 512 bytes, or 512 GB for the catalog. Compression (see vector search) cuts each to 64 bytes plus an 8-byte ID: 72 GB. We split it 8 ways for query load, not for size.
Build the design
Section titled “Build the design”We start with a trending list. Then each step fixes the biggest problem left. The drawing grows with each step.
Step 1. A trending feed
Section titled “Step 1. A trending feed”Start simple. What’s the first version you’d ship?
Whether you can fill a feed with no model in two minutes, and name what it gets wrong. Starting with a two-tower diagram skips the baseline every later model is compared against.
An hourly job counts views and finishes per video per region, from the event stream. The feed service pages through the region’s top 500, skipping what the user has seen. That is a working feed for every user on day one, and the fallback for the rest of the page.
Stage 1: A trending feed
Stage 1 of 7
A trending feed
Why this stage
Nothing yet.
This is the smallest design that fills a feed for every user on day one.
What we add
A feed service that pages through a per-region trending list, minus what the user has seen, and an hourly job that builds the lists from view events.
What it costs
Everyone in a region sees the same feed, a video needs hours to trend, and a new upload with no views can never get in.
At 70k feed requests per second, every part has headroom. Raise the traffic until something turns red.
The numbers behind the busy bars
How busy it gets
- Each feed request produces about 10 events.
What one machine can do
- One feed server handles about 2,000 requests a second.
- One ANN replica set (8 shards) answers about 5,000 queries a second.
- One GPU makes about 100,000 heavy-ranker scores a second. That is 200 requests a second at 500 videos each, or 400 at 250.
How big the shared systems are
- The feature store is sized for 150,000 requests a second of about 500 rows each. Caching video rows on the ranker hosts lets it serve ten times that.
- The event stream takes 3 million events a second, and 10 million with client batching and more partitions.
- Instance counts aim for 60% busy.
// Stage 1: one trending list per region, rebuilt every hour from view counts. Everyone in a// region gets the same order, minus the videos they have already seen.class TrendingList { // After 6 hours a view counts half, so last night's hit fades instead of staying on top. private static final double HALF_LIFE_HOURS = 6.0; private static final double DROP_BELOW = 1.0; // keeps the map to videos watched recently private final Map<String, Double> score = new HashMap<>();
void addHour(Map<String, Integer> views, Map<String, Integer> completions) { // One number per video, decayed once an hour, instead of keeping every view. double decay = Math.pow(0.5, 1.0 / HALF_LIFE_HOURS); score.replaceAll((id, s) -> s * decay); views.forEach((id, n) -> { // A completion counts five times a start. Ranking by starts alone rewards // clickbait: a thumbnail that gets the tap and a video that loses you in a second. double add = 0.2 * n + completions.getOrDefault(id, 0); score.merge(id, add, Double::sum); }); score.values().removeIf(s -> s < DROP_BELOW); }
List<String> top(int k) { return score.entrySet().stream() .sorted(Map.Entry.<String, Double>comparingByValue().reversed()) .limit(k) .map(Map.Entry::getKey) .toList(); }
// The only personal touch stage 1 has: never show the same person a video twice. static List<String> page(List<String> trending, Set<String> seen, int size) { return trending.stream().filter(id -> !seen.contains(id)).limit(size).toList(); }}# Stage 1: one trending list per region, rebuilt every hour from view counts. Everyone in a# region gets the same order, minus the videos they have already seen.class TrendingList: HALF_LIFE_HOURS = 6.0 # after 6 hours a view counts half, so last night's hit fades DROP_BELOW = 1.0 # keeps the map to videos watched recently
def __init__(self) -> None: self._score: dict[str, float] = {}
def add_hour(self, views: dict[str, int], completions: dict[str, int]) -> None: # One number per video, decayed once an hour, instead of keeping every view. decay = 0.5 ** (1 / self.HALF_LIFE_HOURS) for vid in self._score: self._score[vid] *= decay for vid, n in views.items(): # A completion counts five times a start. Ranking by starts alone rewards # clickbait: a thumbnail that gets the tap and a video that loses you in a second. add = 0.2 * n + completions.get(vid, 0) self._score[vid] = self._score.get(vid, 0.0) + add self._score = {v: s for v, s in self._score.items() if s >= self.DROP_BELOW}
def top(self, k: int) -> list[str]: return heapq.nlargest(k, self._score, key=self._score.__getitem__)
def page(trending: list[str], seen: set[str], size: int = 10) -> list[str]: # The only personal touch stage 1 has: never show the same person a video twice. return [v for v in trending if v not in seen][:size]// Stage 1: one trending list per region, rebuilt every hour from view counts. Everyone in a// region gets the same order, minus the videos they have already seen.class TrendingList{ const double HalfLifeHours = 6.0; // after 6 hours a view counts half, so old hits fade const double DropBelow = 1.0; // keeps the map to videos watched recently private readonly Dictionary<string, double> _score = new();
public void AddHour(IReadOnlyDictionary<string, int> views, IReadOnlyDictionary<string, int> completions) { // One number per video, decayed once an hour, instead of keeping every view. double decay = Math.Pow(0.5, 1.0 / HalfLifeHours); foreach (var id in _score.Keys.ToList()) _score[id] *= decay; foreach (var (id, n) in views) { // A completion counts five times a start. Ranking by starts alone rewards // clickbait: a thumbnail that gets the tap and a video that loses you in a second. double add = 0.2 * n + completions.GetValueOrDefault(id); _score[id] = _score.GetValueOrDefault(id) + add; } foreach (var id in _score.Where(kv => kv.Value < DropBelow).Select(kv => kv.Key).ToList()) _score.Remove(id); }
public List<string> Top(int k) => _score.OrderByDescending(kv => kv.Value).Take(k).Select(kv => kv.Key).ToList();
// The only personal touch stage 1 has: never show the same person a video twice. public static List<string> Page(IEnumerable<string> trending, ISet<string> seen, int size) => trending.Where(id => !seen.Contains(id)).Take(size).ToList();}// TrendingList is stage 1: one list per region, rebuilt every hour from view counts. Everyone// in a region gets the same order, minus the videos they have already seen.type TrendingList struct { score map[string]float64}
const ( halfLifeHours = 6.0 // after 6 hours a view counts half, so last night's hit fades dropBelow = 1.0 // keeps the map to videos watched recently)
func NewTrendingList() *TrendingList { return &TrendingList{score: map[string]float64{}}}
func (t *TrendingList) AddHour(views, completions map[string]int) { // One number per video, decayed once an hour, instead of keeping every view. decay := math.Pow(0.5, 1/halfLifeHours) for id := range t.score { t.score[id] *= decay } for id, n := range views { // A completion counts five times a start. Ranking by starts alone rewards clickbait: // a thumbnail that gets the tap and a video that loses you in a second. t.score[id] += 0.2*float64(n) + float64(completions[id]) } for id, s := range t.score { if s < dropBelow { delete(t.score, id) // deleting while ranging over a map is safe in Go } }}
func (t *TrendingList) Top(k int) []string { ids := make([]string, 0, len(t.score)) for id := range t.score { ids = append(ids, id) } sort.Slice(ids, func(i, j int) bool { return t.score[ids[i]] > t.score[ids[j]] }) if len(ids) > k { ids = ids[:k] } return ids}
// Page is the only personal touch stage 1 has: never show the same person a video twice.func Page(trending []string, seen map[string]bool, size int) []string { out := make([]string, 0, size) for _, id := range trending { if len(out) == size { break } if !seen[id] { out = append(out, id) } } return out}// Stage 1: one trending list per region, rebuilt every hour from view counts. Everyone in a// region gets the same order, minus the videos they have already seen.class TrendingList { private static readonly HALF_LIFE_HOURS = 6; // after 6 hours a view counts half private static readonly DROP_BELOW = 1; // keeps the map to videos watched recently private score = new Map<string, number>();
addHour(views: Map<string, number>, completions: Map<string, number>): void { // One number per video, decayed once an hour, instead of keeping every view. const decay = 0.5 ** (1 / TrendingList.HALF_LIFE_HOURS); for (const [id, s] of this.score) this.score.set(id, s * decay); for (const [id, n] of views) { // A completion counts five times a start. Ranking by starts alone rewards clickbait: // a thumbnail that gets the tap and a video that loses you in a second. const add = 0.2 * n + (completions.get(id) ?? 0); this.score.set(id, (this.score.get(id) ?? 0) + add); } for (const [id, s] of this.score) if (s < TrendingList.DROP_BELOW) this.score.delete(id); }
top(k: number): string[] { return [...this.score.entries()] .sort((a, b) => b[1] - a[1]) .slice(0, k) .map(([id]) => id); }}
// The only personal touch stage 1 has: never show the same person a video twice.function page(trending: string[], seen: Set<string>, size = 10): string[] { return trending.filter((id) => !seen.has(id)).slice(0, size);}Its problems are the reasons for the next steps. Everyone in a region sees the same feed. A video needs an hour of views to trend, so the list is always behind. And a new upload with no views can never get in.
Step 2. Personal candidates
Section titled “Step 2. Personal candidates”Make it personal. There’s a billion videos. How do you find the few hundred worth considering for this user?
Whether you separate finding candidates from ranking them, and know why: first don’t miss good ones, then order them. “Run the model over all videos” with no math is the red flag.
Scoring a billion videos per request is impossible. So the first job is to find a few thousand plausible ones, fast. This is called retrieval. We judge it by one question: did the good videos make it into the set at all?
The main source is a two-tower model. It has two halves:
- Color key
- Inputs
- Models (towers)
- Vectors: 128 numbers
- Search
The two towers are trained together, so a user lands close to the videos they finished. Video vectors are computed ahead into a search index. Per request, we run the user tower and ask the index for the 2,000 closest videos. The index finds them in milliseconds because it searches approximately. This is called approximate nearest-neighbor search, or ANN. And because the video tower reads content, a new video gets a sensible vector before anyone watches it.
Two more sources cover what the towers miss: new uploads from followed creators (500), and the trending list (500). A merger removes duplicates and seen videos, and caps each source so one can’t crowd out the rest.
Stage 2: Personal candidates
Stage 2 of 7
Personal candidates
Why this stage
One list for 200 million people wastes most of their watch time, and the index holds 1 billion videos: nothing can score them all per request.
What we add
A retrieval service that runs a user tower and an ANN search over video embeddings, a follow-graph source, the trending list as a third source, and a daily training pipeline over logged events.
What it costs
Candidates come back unordered and of mixed quality, and the embeddings are a day old.
At 70k feed requests per second, every part has headroom. Raise the traffic until something turns red.
The numbers behind the busy bars
How busy it gets
- Each feed request produces about 10 events.
What one machine can do
- One feed server handles about 2,000 requests a second.
- One ANN replica set (8 shards) answers about 5,000 queries a second.
- One GPU makes about 100,000 heavy-ranker scores a second. That is 200 requests a second at 500 videos each, or 400 at 250.
How big the shared systems are
- The feature store is sized for 150,000 requests a second of about 500 rows each. Caching video rows on the ranker hosts lets it serve ten times that.
- The event stream takes 3 million events a second, and 10 million with client batching and more partitions.
- Instance counts aim for 60% busy.
// One video a source proposes. The source travels with it: the ranker uses it as a feature.record Candidate(String videoId, String creatorId, String source) {}
// Stage 2: several sources propose videos. The merger drops duplicates and anything the user// has seen or must not see, and caps each source so one cannot crowd out the others.class CandidateMerger { // Counted after filtering, so a source full of seen videos still fills its share. static final Map<String, Integer> QUOTA = Map.of("follow", 500, "embedding", 2000, "trending", 500); // When two sources propose the same video, the first one here gets the credit. static final List<String> ORDER = List.of("follow", "embedding", "trending");
List<Candidate> merge(Map<String, List<Candidate>> bySource, Predicate<String> seen, Set<String> blockedCreators) { List<Candidate> out = new ArrayList<>(); Set<String> taken = new HashSet<>(); for (String source : ORDER) { // A source that timed out is simply missing: the page is built from the rest. int kept = 0; for (Candidate c : bySource.getOrDefault(source, List.of())) { // best first if (kept == QUOTA.get(source)) break; // `seen` is a Bloom filter. It wrongly says "seen" for about 1% of videos, // which costs a candidate; it never says "not seen" for a video already shown. if (seen.test(c.videoId()) || blockedCreators.contains(c.creatorId())) continue; if (!taken.add(c.videoId())) continue; out.add(c); kept++; } } return out; }}# One video a source proposes. The source travels with it: the ranker uses it as a feature.@dataclass(frozen=True)class Candidate: video_id: str creator_id: str source: str
# Stage 2: several sources propose videos. The merger drops duplicates and anything the# user has seen or must not see, and caps each source so one cannot crowd out the others.class CandidateMerger: # Counted after filtering, so a source full of seen videos still fills its share. QUOTA = {"follow": 500, "embedding": 2000, "trending": 500} # When two sources propose the same video, the first one here gets the credit. ORDER = ("follow", "embedding", "trending")
def merge( self, by_source: dict[str, list[Candidate]], seen: Callable[[str], bool], blocked_creators: set[str], ) -> list[Candidate]: out: list[Candidate] = [] taken: set[str] = set() for source in self.ORDER: # A source that timed out is simply missing: the page is built from the rest. kept = 0 for c in by_source.get(source, []): # each source returns its best first if kept == self.QUOTA[source]: break # `seen` is a Bloom filter. It wrongly says "seen" for about 1% of videos, # which costs a candidate; it never says "not seen" for a video already shown. if seen(c.video_id) or c.creator_id in blocked_creators: continue if c.video_id in taken: continue taken.add(c.video_id) out.append(c) kept += 1 return out// One video a source proposes. The source travels with it: the ranker uses it as a feature.record Candidate(string VideoId, string CreatorId, string Source);
// Stage 2: several sources propose videos. The merger drops duplicates and anything the user// has seen or must not see, and caps each source so one cannot crowd out the others.class CandidateMerger{ // Counted after filtering, so a source full of seen videos still fills its share. static readonly Dictionary<string, int> Quota = new() { ["follow"] = 500, ["embedding"] = 2000, ["trending"] = 500 }; // When two sources propose the same video, the first one here gets the credit. static readonly string[] Order = { "follow", "embedding", "trending" };
public List<Candidate> Merge(IReadOnlyDictionary<string, List<Candidate>> bySource, Func<string, bool> seen, ISet<string> blockedCreators) { var output = new List<Candidate>(); var taken = new HashSet<string>(); foreach (var source in Order) { // A source that timed out is simply missing: the page is built from the rest. if (!bySource.TryGetValue(source, out var candidates)) continue; int kept = 0; foreach (var c in candidates) // each source returns its best first { if (kept == Quota[source]) break; // `seen` is a Bloom filter. It wrongly says "seen" for about 1% of videos, // which costs a candidate; it never says "not seen" for a video already shown. if (seen(c.VideoId) || blockedCreators.Contains(c.CreatorId)) continue; if (!taken.Add(c.VideoId)) continue; output.Add(c); kept++; } } return output; }}// Candidate is one video a source proposes. The source travels with it: the ranker uses it// as a feature.type Candidate struct { VideoID, CreatorID, Source string}
// Counted after filtering, so a source full of seen videos still fills its share.var quota = map[string]int{"follow": 500, "embedding": 2000, "trending": 500}
// When two sources propose the same video, the first one here gets the credit.var order = []string{"follow", "embedding", "trending"}
// Merge is stage 2: several sources propose videos. It drops duplicates and anything the user// has seen or must not see, and caps each source so one cannot crowd out the others.func Merge(bySource map[string][]Candidate, seen func(string) bool, blockedCreators map[string]bool) []Candidate { out := []Candidate{} taken := map[string]bool{} for _, source := range order { // A source that timed out is simply missing: the page is built from the rest. kept := 0 for _, c := range bySource[source] { // each source returns its best first if kept == quota[source] { break } // seen is a Bloom filter. It wrongly says "seen" for about 1% of videos, which // costs a candidate; it never says "not seen" for a video already shown. if seen(c.VideoID) || blockedCreators[c.CreatorID] || taken[c.VideoID] { continue } taken[c.VideoID] = true out = append(out, c) kept++ } } return out}// One video a source proposes. The source travels with it: the ranker uses it as a feature.interface Candidate { videoId: string; creatorId: string; source: 'follow' | 'embedding' | 'trending';}
// Stage 2: several sources propose videos. The merger drops duplicates and anything the user// has seen or must not see, and caps each source so one cannot crowd out the others.class CandidateMerger { // Counted after filtering, so a source full of seen videos still fills its share. static readonly QUOTA = { follow: 500, embedding: 2000, trending: 500 } as const; // When two sources propose the same video, the first one here gets the credit. static readonly ORDER = ['follow', 'embedding', 'trending'] as const;
merge( bySource: Partial<Record<Candidate['source'], Candidate[]>>, seen: (videoId: string) => boolean, blockedCreators: Set<string>, ): Candidate[] { const out: Candidate[] = []; const taken = new Set<string>(); for (const source of CandidateMerger.ORDER) { // A source that timed out is simply missing: the page is built from the rest. let kept = 0; for (const c of bySource[source] ?? []) { if (kept === CandidateMerger.QUOTA[source]) break; // `seen` is a Bloom filter. It wrongly says "seen" for about 1% of videos, which // costs a candidate; it never says "not seen" for a video already shown. if (seen(c.videoId) || blockedCreators.has(c.creatorId) || taken.has(c.videoId)) continue; taken.add(c.videoId); out.push(c); kept++; } } return out; }}The towers are retrained daily, and the whole catalog is re-embedded. The user tower and the index must come from the same training run, or the vectors live in different spaces and the search returns noise. So they ship together, as one versioned pair.
Step 3. A ranking model
Section titled “Step 3. A ranking model”You have 3,000 candidates and room for 10. How do you order them?
Whether you size the ranking steps from the time budget, predict several outcomes, not one, and know where the features come from at serving time. “A deep network predicts watch time” with no word on features or cost is a mid-level answer.
Ranking runs in two steps:
- A small model with a few dozen cheap features scores all 3,000 on CPUs in about 6 ms, and keeps 500.
- A big model with hundreds of features scores those 500 on a GPU in about 25 ms. 500 is what 600 GPUs can afford at 70,000 requests a second.
The big model predicts several things at once: will the user finish the video, watch for a few seconds, like, share, follow, or tap “not interested”? A value model combines the predictions into one score, with weights the product team chooses. Those weights decide what the feed is for.
// What the heavy ranker predicts for one user and one video: several heads, one network.record Predictions(double pComplete, double watchSeconds, double pLike, double pShare, double pFollow, double pNotInterested) {}
// Chosen by the product team, versioned and A/B tested like a model, because these numbers// decide what the feed optimizes for.record Weights(double watch, double complete, double like, double share, double follow, double notInterested) {}
// Stage 3: turns several predictions into the one number the feed sorts by.class ValueModel { // Watch time alone favors long videos: a 3 minute video watched halfway beats a 15 second // one watched twice. Capping the seconds stops length from winning on its own. private static final double WATCH_CAP_S = 60; private final Weights w;
ValueModel(Weights w) { this.w = w; }
// A weighted sum of probabilities only means something if each one is calibrated // (a predicted 0.3 happens 30% of the time). The release gate checks that per head. double score(Predictions p) { double watch = Math.min(p.watchSeconds(), WATCH_CAP_S) / WATCH_CAP_S; return w.watch() * watch + w.complete() * p.pComplete() + w.like() * p.pLike() + w.share() * p.pShare() + w.follow() * p.pFollow() // Rare but strong: a small chance of "not interested" with a large weight // keeps a video this user would hide off the page. - w.notInterested() * p.pNotInterested(); }}# What the heavy ranker predicts for one user and one video: several heads, one network.@dataclass(frozen=True)class Predictions: p_complete: float watch_seconds: float p_like: float p_share: float p_follow: float p_not_interested: float
# Chosen by the product team, versioned and A/B tested like a model, because these# numbers decide what the feed optimizes for.@dataclass(frozen=True)class Weights: watch: float complete: float like: float share: float follow: float not_interested: float
# Stage 3: turns several predictions into the one number the feed sorts by.class ValueModel: # Watch time alone favors long videos: a 3 minute video watched halfway beats a 15 second # one watched twice. Capping the seconds stops length from winning on its own. WATCH_CAP_S = 60.0
def __init__(self, w: Weights) -> None: self.w = w
def score(self, p: Predictions) -> float: # A weighted sum of probabilities only means something if each one is calibrated # (a predicted 0.3 happens 30% of the time). The release gate checks that per head. w = self.w watch = min(p.watch_seconds, self.WATCH_CAP_S) / self.WATCH_CAP_S return ( w.watch * watch + w.complete * p.p_complete + w.like * p.p_like + w.share * p.p_share + w.follow * p.p_follow # Rare but strong: a small chance of "not interested" with a large weight keeps a # video this user would hide off the page. - w.not_interested * p.p_not_interested )// What the heavy ranker predicts for one user and one video: several heads, one network.record Predictions(double PComplete, double WatchSeconds, double PLike, double PShare, double PFollow, double PNotInterested);
// Chosen by the product team, versioned and A/B tested like a model, because these numbers// decide what the feed optimizes for.record Weights(double Watch, double Complete, double Like, double Share, double Follow, double NotInterested);
// Stage 3: turns several predictions into the one number the feed sorts by.class ValueModel(Weights w){ // Watch time alone favors long videos: a 3 minute video watched halfway beats a 15 second // one watched twice. Capping the seconds stops length from winning on its own. const double WatchCapS = 60;
// A weighted sum of probabilities only means something if each one is calibrated // (a predicted 0.3 happens 30% of the time). The release gate checks that per head. public double Score(Predictions p) { double watch = Math.Min(p.WatchSeconds, WatchCapS) / WatchCapS; return w.Watch * watch + w.Complete * p.PComplete + w.Like * p.PLike + w.Share * p.PShare + w.Follow * p.PFollow // Rare but strong: a small chance of "not interested" with a large weight keeps // a video this user would hide off the page. - w.NotInterested * p.PNotInterested; }}// Predictions is what the heavy ranker predicts for one user and one video: several heads,// one network.type Predictions struct { PComplete, WatchSeconds, PLike, PShare, PFollow, PNotInterested float64}
// Weights are chosen by the product team, versioned and A/B tested like a model, because// these numbers decide what the feed optimizes for.type Weights struct { Watch, Complete, Like, Share, Follow, NotInterested float64}
// Watch time alone favors long videos: a 3 minute video watched halfway beats a 15 second one// watched twice. Capping the seconds stops length from winning on its own.const watchCapS = 60.0
// Score is stage 3: it turns several predictions into the one number the feed sorts by.// A weighted sum of probabilities only means something if each one is calibrated (a predicted// 0.3 happens 30% of the time). The release gate checks that per head.func Score(w Weights, p Predictions) float64 { watch := math.Min(p.WatchSeconds, watchCapS) / watchCapS return w.Watch*watch + w.Complete*p.PComplete + w.Like*p.PLike + w.Share*p.PShare + w.Follow*p.PFollow - // Rare but strong: a small chance of "not interested" with a large weight keeps a // video this user would hide off the page. w.NotInterested*p.PNotInterested}// What the heavy ranker predicts for one user and one video: several heads, one network.interface Predictions { pComplete: number; watchSeconds: number; pLike: number; pShare: number; pFollow: number; pNotInterested: number;}
// Chosen by the product team, versioned and A/B tested like a model, because these numbers// decide what the feed optimizes for.interface Weights { watch: number; complete: number; like: number; share: number; follow: number; notInterested: number;}
// Stage 3: turns several predictions into the one number the feed sorts by.class ValueModel { // Watch time alone favors long videos: a 3 minute video watched halfway beats a 15 second // one watched twice. Capping the seconds stops length from winning on its own. private static readonly WATCH_CAP_S = 60;
constructor(private readonly w: Weights) {}
// A weighted sum of probabilities only means something if each one is calibrated // (a predicted 0.3 happens 30% of the time). The release gate checks that per head. score(p: Predictions): number { const { w } = this; const watch = Math.min(p.watchSeconds, ValueModel.WATCH_CAP_S) / ValueModel.WATCH_CAP_S; return ( w.watch * watch + w.complete * p.pComplete + w.like * p.pLike + w.share * p.pShare + w.follow * p.pFollow - // Rare but strong: a small chance of "not interested" with a large weight keeps a // video this user would hide off the page. w.notInterested * p.pNotInterested ); }}The features come from a feature store: one row per user, per video and per session, read in one batch. Every page served is logged with the exact features and model versions used. So the training data is exactly what the model saw.
Stage 3: A ranking model
Stage 3 of 7
A ranking model
Why this stage
About 3,000 candidates arrive per request and the page shows 10.
Their order decides watch time, and it needs hundreds of features no retrieval model can use.
What we add
A ranking service (a light ranker on CPU, a multi-task heavy ranker on GPUs), an online feature store, and a log of the exact features served.
What it costs
About 600 GPUs at peak, and a feature store on the critical path of every request.
At 70k feed requests per second, every part has headroom. Raise the traffic until something turns red.
The numbers behind the busy bars
How busy it gets
- Each feed request produces about 10 events.
What one machine can do
- One feed server handles about 2,000 requests a second.
- One ANN replica set (8 shards) answers about 5,000 queries a second.
- One GPU makes about 100,000 heavy-ranker scores a second. That is 200 requests a second at 500 videos each, or 400 at 250.
How big the shared systems are
- The feature store is sized for 150,000 requests a second of about 500 rows each. Caching video rows on the ranker hosts lets it serve ten times that.
- The event stream takes 3 million events a second, and 10 million with client batching and more partitions.
- Instance counts aim for 60% busy.
Go deeperOne page, step by step, and the data model3 min
- Color key
- Queue or stream
- Database
- Service
The data. Almost every read on the request path is a batch of rows by key.
-- Feature store: one packed row per thing, in memoryvideo:{id} views_1h, views_24h, completion_rate_24h, like_rate_24h, skip_rate_24h, age_minutes, content_emb[64] written by stream + batchuser:{id} long_term_emb[128], topic_affinity{}, creator_affinity{}, language, country written daily by batchsession:{id} topic_affinity{} (fades over 5 min), recent_videos[50], skips_by_creator{} written by stream, gone after 30 minseen:{user_id} a compact filter of videos shown in the last 30 days, about 4 KB
-- Training logs: column files in object storage, split by hourimpressions request_id, user_id, video_id, position, source, model_versions, features (exactly as served), served_atevents event_id, request_id, video_id, type, watch_ms, completed, tsThe seen filter is a Bloom filter, a compact set that can say “maybe seen” by mistake, but never “not seen” by mistake. It may wrongly hide 1% of candidates, but it never lets a seen video through.
The feature store is an in-memory key-value store, like Redis, so 500 rows come back in about 10 ms. Training reads cheap files in Amazon S3 without touching serving.
Step 4. React within the session
Section titled “Step 4. React within the session”I skip three cooking videos in a row. When does the feed notice?
Whether session features come from a stream and reach the next page within seconds, and whether you handle repeated and late events. “We retrain the model every hour” is the red flag. The model doesn’t need to change. Its inputs do.
With daily features, the answer is “tomorrow”. A session lasts minutes. The fix is fresher inputs, not a faster model.
A stream processor reads the event stream. The stream is split by user, so one worker owns each session. It keeps a session row: how much the user likes each topic right now (fading over about 5 minutes), the last 50 videos, and skips per creator. The next page’s ranking reads it like any other feature. The same stream keeps hourly counts per video, so a video going viral now ranks higher now.
- Color key
- Service
- Database
- Queue or stream
Stage 4: Real-time signals
Stage 4 of 7
Real-time signals
Why this stage
Features refreshed daily cannot see that the user skipped three cooking videos a minute ago, and a video going viral now looks cold until tomorrow.
What we add
A stream processor that turns events into session features and video counters in the online store within about a second.
What it costs
A stream job to run and keep correct, with duplicate and late events to handle.
At 70k feed requests per second, every part has headroom. Raise the traffic until something turns red.
The numbers behind the busy bars
How busy it gets
- Each feed request produces about 10 events.
What one machine can do
- One feed server handles about 2,000 requests a second.
- One ANN replica set (8 shards) answers about 5,000 queries a second.
- One GPU makes about 100,000 heavy-ranker scores a second. That is 200 requests a second at 500 videos each, or 400 at 250.
How big the shared systems are
- The feature store is sized for 150,000 requests a second of about 500 rows each. Caching video rows on the ranker hosts lets it serve ten times that.
- The event stream takes 3 million events a second, and 10 million with client batching and more partitions.
- Instance counts aim for 60% busy.
record WatchEvent(String eventId, String videoId, List<String> topics, long watchMs, boolean completed, boolean liked, long tsMs) {}
// Stage 4: kept per active session by the stream processor, in the online feature store. The// ranker reads it on the next page, so a skip a second ago changes what comes next.class SessionState { private static final double HALF_LIFE_MS = 5 * 60_000; // a session's mood shifts fast private static final long QUICK_SKIP_MS = 2_000; // swiped away before it got going private static final int REMEMBER = 200; // event ids kept for dedup
private final Map<String, Double> topicAffinity = new HashMap<>(); private final Deque<String> recentIds = new ArrayDeque<>(); private final Set<String> recentIdSet = new HashSet<>(); private long asOfMs;
void apply(WatchEvent e) { // The stream delivers at least once, so an event can arrive twice. Counting a skip // twice would push its topics down twice as hard. if (!recentIdSet.add(e.eventId())) return; recentIds.addLast(e.eventId()); if (recentIds.size() > REMEMBER) recentIdSet.remove(recentIds.removeFirst());
// Decay lazily, only when the session changes, so there is no timer per session. A // late event is applied without moving the clock backwards. if (e.tsMs() > asOfMs) { double f = Math.pow(0.5, (e.tsMs() - asOfMs) / HALF_LIFE_MS); topicAffinity.replaceAll((topic, v) -> v * f); asOfMs = e.tsMs(); } // A quick skip is the strongest negative signal a feed gets without asking. double signal = e.watchMs() < QUICK_SKIP_MS ? -1.0 : (e.completed() ? 1.0 : 0.3) + (e.liked() ? 1.0 : 0.0); for (String t : e.topics()) topicAffinity.merge(t, signal, Double::sum); }
// Read by the ranker as features. A copy, so a concurrent update never tears a read. Map<String, Double> features() { return Map.copyOf(topicAffinity); }}@dataclass(frozen=True)class WatchEvent: event_id: str video_id: str topics: tuple[str, ...] watch_ms: int completed: bool liked: bool ts_ms: int
# Stage 4: kept per active session by the stream processor, in the online feature store.# The ranker reads it on the next page, so a skip a second ago changes what comes next.class SessionState: HALF_LIFE_MS = 5 * 60_000 # a session's mood shifts fast QUICK_SKIP_MS = 2_000 # swiped away before it got going REMEMBER = 200 # event ids kept for dedup
def __init__(self) -> None: self.topic_affinity: dict[str, float] = {} self._recent_ids: deque[str] = deque() self._recent_id_set: set[str] = set() self._as_of_ms = 0
def apply(self, e: WatchEvent) -> None: # The stream delivers at least once, so an event can arrive twice. Counting a skip # twice would push its topics down twice as hard. if e.event_id in self._recent_id_set: return self._recent_id_set.add(e.event_id) self._recent_ids.append(e.event_id) if len(self._recent_ids) > self.REMEMBER: self._recent_id_set.discard(self._recent_ids.popleft())
# Decay lazily, only when the session changes, so there is no timer per session. A # late event is applied without moving the clock backwards. if e.ts_ms > self._as_of_ms: f = 0.5 ** ((e.ts_ms - self._as_of_ms) / self.HALF_LIFE_MS) for t in self.topic_affinity: self.topic_affinity[t] *= f self._as_of_ms = e.ts_ms
# A quick skip is the strongest negative signal a feed gets without asking. if e.watch_ms < self.QUICK_SKIP_MS: signal = -1.0 else: signal = (1.0 if e.completed else 0.3) + (1.0 if e.liked else 0.0) for t in e.topics: self.topic_affinity[t] = self.topic_affinity.get(t, 0.0) + signalrecord WatchEvent(string EventId, string VideoId, IReadOnlyList<string> Topics, long WatchMs, bool Completed, bool Liked, long TsMs);
// Stage 4: kept per active session by the stream processor, in the online feature store. The// ranker reads it on the next page, so a skip a second ago changes what comes next.class SessionState{ const double HalfLifeMs = 5 * 60_000; // a session's mood shifts fast const long QuickSkipMs = 2_000; // swiped away before it got going const int Remember = 200; // event ids kept for dedup
private readonly Dictionary<string, double> _topicAffinity = new(); private readonly Queue<string> _recentIds = new(); private readonly HashSet<string> _recentIdSet = new(); private long _asOfMs;
public void Apply(WatchEvent e) { // The stream delivers at least once, so an event can arrive twice. Counting a skip // twice would push its topics down twice as hard. if (!_recentIdSet.Add(e.EventId)) return; _recentIds.Enqueue(e.EventId); if (_recentIds.Count > Remember) _recentIdSet.Remove(_recentIds.Dequeue());
// Decay lazily, only when the session changes, so there is no timer per session. A // late event is applied without moving the clock backwards. if (e.TsMs > _asOfMs) { double f = Math.Pow(0.5, (e.TsMs - _asOfMs) / HalfLifeMs); foreach (var t in _topicAffinity.Keys.ToList()) _topicAffinity[t] *= f; _asOfMs = e.TsMs; } // A quick skip is the strongest negative signal a feed gets without asking. double signal = e.WatchMs < QuickSkipMs ? -1.0 : (e.Completed ? 1.0 : 0.3) + (e.Liked ? 1.0 : 0.0); foreach (var t in e.Topics) _topicAffinity[t] = _topicAffinity.GetValueOrDefault(t) + signal; }
// Read by the ranker as features. A copy, so a concurrent update never tears a read. public IReadOnlyDictionary<string, double> Features() => new Dictionary<string, double>(_topicAffinity);}type WatchEvent struct { EventID, VideoID string Topics []string WatchMs, TsMs int64 Completed, Liked bool}
const ( halfLifeMs = 5 * 60_000.0 // a session's mood shifts fast quickSkipMs = 2_000 // swiped away before it got going remember = 200 // event ids kept for dedup)
// SessionState is stage 4: kept per active session by the stream processor, in the online// feature store. The ranker reads it on the next page, so a skip a second ago changes what// comes next. One goroutine owns each session (the stream is partitioned by user), so no lock.type SessionState struct { TopicAffinity map[string]float64 recentIDs []string recentIDSet map[string]bool asOfMs int64}
func NewSessionState() *SessionState { return &SessionState{TopicAffinity: map[string]float64{}, recentIDSet: map[string]bool{}}}
func (s *SessionState) Apply(e WatchEvent) { // The stream delivers at least once, so an event can arrive twice. Counting a skip twice // would push its topics down twice as hard. if s.recentIDSet[e.EventID] { return } s.recentIDSet[e.EventID] = true s.recentIDs = append(s.recentIDs, e.EventID) if len(s.recentIDs) > remember { delete(s.recentIDSet, s.recentIDs[0]) s.recentIDs = s.recentIDs[1:] }
// Decay lazily, only when the session changes, so there is no timer per session. A late // event is applied without moving the clock backwards. if e.TsMs > s.asOfMs { f := math.Pow(0.5, float64(e.TsMs-s.asOfMs)/halfLifeMs) for t := range s.TopicAffinity { s.TopicAffinity[t] *= f } s.asOfMs = e.TsMs } // A quick skip is the strongest negative signal a feed gets without asking. signal := -1.0 if e.WatchMs >= quickSkipMs { signal = 0.3 if e.Completed { signal = 1.0 } if e.Liked { signal += 1.0 } } for _, t := range e.Topics { s.TopicAffinity[t] += signal }}interface WatchEvent { eventId: string; videoId: string; topics: string[]; watchMs: number; completed: boolean; liked: boolean; tsMs: number;}
// Stage 4: kept per active session by the stream processor, in the online feature store. The// ranker reads it on the next page, so a skip a second ago changes what comes next.class SessionState { private static readonly HALF_LIFE_MS = 5 * 60_000; // a session's mood shifts fast private static readonly QUICK_SKIP_MS = 2_000; // swiped away before it got going private static readonly REMEMBER = 200; // event ids kept for dedup
readonly topicAffinity = new Map<string, number>(); // A Set iterates in insertion order, so its first entry is the oldest id. private readonly recentIds = new Set<string>(); private asOfMs = 0;
apply(e: WatchEvent): void { // The stream delivers at least once, so an event can arrive twice. Counting a skip // twice would push its topics down twice as hard. if (this.recentIds.has(e.eventId)) return; this.recentIds.add(e.eventId); if (this.recentIds.size > SessionState.REMEMBER) { this.recentIds.delete(this.recentIds.values().next().value!); }
// Decay lazily, only when the session changes, so there is no timer per session. A late // event is applied without moving the clock backwards. if (e.tsMs > this.asOfMs) { const f = 0.5 ** ((e.tsMs - this.asOfMs) / SessionState.HALF_LIFE_MS); for (const [t, v] of this.topicAffinity) this.topicAffinity.set(t, v * f); this.asOfMs = e.tsMs; } // A quick skip is the strongest negative signal a feed gets without asking. const signal = e.watchMs < SessionState.QUICK_SKIP_MS ? -1 : (e.completed ? 1 : 0.3) + (e.liked ? 1 : 0); for (const t of e.topics) this.topicAffinity.set(t, (this.topicAffinity.get(t) ?? 0) + signal); }}The stream may deliver an event twice. So the update ignores an event ID it has already applied (this is called idempotency). The fading is computed only when a new event arrives, so millions of idle sessions cost nothing.
Step 5. Variety, and room for new videos
Section titled “Step 5. Variety, and room for new videos”Your top 10 has three videos by the same creator, and nothing uploaded today. Fix both.
Whether you treat the page as a set, not 10 separate scores, and see testing new videos as a budget that pays for future quality. “Add some randomness”, without saying how much, where and what it costs, is the red flag.
The big model scores each video alone, so similar high scores cluster. A final pass builds the page as a whole:
- At most one video per creator.
- A penalty for each repeated topic.
- Age and country rules.
- 5% of slots go to new videos, from uploads under 48 hours old that match the user’s topics. Never the first two slots, because those decide whether the user keeps scrolling. Testing new things like this is called exploration.
Stage 5: Re-ranking and exploration
Stage 5 of 7
Re-ranking and exploration
Why this stage
The ranker scores each video alone, so a page can hold three videos by one creator, and 10 million uploads a day get no impressions without a way in.
What we add
A re-rank step that builds the page as a whole (creator caps, topic spread, policy) and spends 5% of slots on new videos from a fresh pool.
What it costs
A little short-term watch time in exchange for a catalog that keeps renewing itself.
At 70k feed requests per second, every part has headroom. Raise the traffic until something turns red.
The numbers behind the busy bars
How busy it gets
- Each feed request produces about 10 events.
What one machine can do
- One feed server handles about 2,000 requests a second.
- One ANN replica set (8 shards) answers about 5,000 queries a second.
- One GPU makes about 100,000 heavy-ranker scores a second. That is 200 requests a second at 500 videos each, or 400 at 250.
How big the shared systems are
- The feature store is sized for 150,000 requests a second of about 500 rows each. Caching video rows on the ranker hosts lets it serve ten times that.
- The event stream takes 3 million events a second, and 10 million with client batching and more partitions.
- Instance counts aim for 60% busy.
record Scored(String videoId, String creatorId, String topic, double score) {}
// Stage 5: the heavy ranker scores each video alone. This builds the page as a whole: no// creator twice, topics spread out, and some slots for new videos still being tested.class ReRanker { private static final double TOPIC_PENALTY = 0.15; // each repeat of a topic costs 15% // The first two slots are never exploration: they decide whether the user keeps scrolling. private static final int PROTECTED = 2; private final double exploreRate; // share of all slots, 0.05 on this page private final Random rng;
ReRanker(double exploreRate, Random rng) { this.exploreRate = exploreRate; this.rng = rng; }
List<Scored> page(List<Scored> ranked, Deque<Scored> fresh, int size) { List<Scored> pool = new ArrayList<>(ranked); List<Scored> page = new ArrayList<>(); Set<String> creators = new HashSet<>(); Map<String, Integer> topics = new HashMap<>(); // Spread the exploration share over the slots allowed to hold it, so the overall // rate still comes out at exploreRate. double pExplore = exploreRate * size / (size - PROTECTED); while (page.size() < size) { Scored pick = null; if (page.size() >= PROTECTED && !fresh.isEmpty() && rng.nextDouble() < pExplore) { pick = fresh.pollFirst(); // a new video, already matched to this user's topics } else { // Greedy: score what is left against what the page already holds. 10 slots // times 500 videos is nothing next to the ranker. double best = Double.NEGATIVE_INFINITY; for (Scored s : pool) { if (creators.contains(s.creatorId())) continue; int repeats = topics.getOrDefault(s.topic(), 0); double adjusted = s.score() * Math.pow(1 - TOPIC_PENALTY, repeats); if (adjusted > best) { best = adjusted; pick = s; } } if (pick == null) break; // a short page beats a repetitive one pool.remove(pick); } page.add(pick); creators.add(pick.creatorId()); topics.merge(pick.topic(), 1, Integer::sum); } return page; }}@dataclass(frozen=True)class Scored: video_id: str creator_id: str topic: str score: float
# Stage 5: the heavy ranker scores each video alone. This builds the page as a whole: no# creator twice, topics spread out, and some slots for new videos still being tested.class ReRanker: TOPIC_PENALTY = 0.15 # each repeat of a topic costs 15% PROTECTED = 2 # the first two slots decide whether the user keeps scrolling: never explore
def __init__(self, explore_rate: float, rng: random.Random) -> None: self.explore_rate = explore_rate # share of all slots, 0.05 on this page self.rng = rng
def page(self, ranked: list[Scored], fresh: deque[Scored], size: int = 10) -> list[Scored]: pool = list(ranked) page: list[Scored] = [] creators: set[str] = set() topics: Counter[str] = Counter() # Spread the exploration share over the slots allowed to hold it, so the overall rate # still comes out at explore_rate. p_explore = self.explore_rate * size / (size - self.PROTECTED) while len(page) < size: if len(page) >= self.PROTECTED and fresh and self.rng.random() < p_explore: pick = fresh.popleft() # a new video, already matched to this user's topics else: # Greedy: score what is left against what the page already holds. 10 slots # times 500 videos is nothing next to the ranker. allowed = [s for s in pool if s.creator_id not in creators] if not allowed: break # a short page beats a repetitive one pick = max( allowed, key=lambda s: s.score * (1 - self.TOPIC_PENALTY) ** topics[s.topic], ) pool.remove(pick) page.append(pick) creators.add(pick.creator_id) topics[pick.topic] += 1 return pagerecord Scored(string VideoId, string CreatorId, string Topic, double Score);
// Stage 5: the heavy ranker scores each video alone. This builds the page as a whole: no// creator twice, topics spread out, and some slots for new videos still being tested.class ReRanker(double exploreRate, Random rng) // exploreRate: share of all slots, 0.05 here{ const double TopicPenalty = 0.15; // each repeat of a topic costs 15% // The first two slots are never exploration: they decide whether the user keeps scrolling. const int Protected = 2;
public List<Scored> Page(IReadOnlyList<Scored> ranked, Queue<Scored> fresh, int size) { var pool = new List<Scored>(ranked); var page = new List<Scored>(); var creators = new HashSet<string>(); var topics = new Dictionary<string, int>(); // Spread the exploration share over the slots allowed to hold it, so the overall rate // still comes out at exploreRate. double pExplore = exploreRate * size / (size - Protected); while (page.Count < size) { Scored? pick = null; if (page.Count >= Protected && fresh.Count > 0 && rng.NextDouble() < pExplore) { pick = fresh.Dequeue(); // a new video, already matched to this user's topics } else { // Greedy: score what is left against what the page already holds. 10 slots // times 500 videos is nothing next to the ranker. double best = double.NegativeInfinity; foreach (var s in pool) { if (creators.Contains(s.CreatorId)) continue; double adjusted = s.Score * Math.Pow(1 - TopicPenalty, topics.GetValueOrDefault(s.Topic)); if (adjusted > best) { best = adjusted; pick = s; } } if (pick is null) break; // a short page beats a repetitive one pool.Remove(pick); } page.Add(pick); creators.Add(pick.CreatorId); topics[pick.Topic] = topics.GetValueOrDefault(pick.Topic) + 1; } return page; }}type Scored struct { VideoID, CreatorID, Topic string Score float64}
const ( topicPenalty = 0.15 // each repeat of a topic costs 15% // The first two slots are never exploration: they decide whether the user keeps scrolling. protectedSlots = 2)
// ReRank is stage 5: the heavy ranker scores each video alone. This builds the page as a// whole: no creator twice, topics spread out, and some slots for new videos still being// tested. exploreRate is the share of all slots, 0.05 on this page.func ReRank(ranked []Scored, fresh []Scored, size int, exploreRate float64, rng *rand.Rand) []Scored { pool := append([]Scored(nil), ranked...) page := make([]Scored, 0, size) creators := map[string]bool{} topics := map[string]int{} // Spread the exploration share over the slots allowed to hold it, so the overall rate // still comes out at exploreRate. pExplore := exploreRate * float64(size) / float64(size-protectedSlots) for len(page) < size { var pick Scored if len(page) >= protectedSlots && len(fresh) > 0 && rng.Float64() < pExplore { pick, fresh = fresh[0], fresh[1:] // a new video, already matched to this user's topics } else { // Greedy: score what is left against what the page already holds. 10 slots times // 500 videos is nothing next to the ranker. bestAt, best := -1, math.Inf(-1) for i, s := range pool { if creators[s.CreatorID] { continue } adjusted := s.Score * math.Pow(1-topicPenalty, float64(topics[s.Topic])) if adjusted > best { bestAt, best = i, adjusted } } if bestAt < 0 { break // a short page beats a repetitive one } pick = pool[bestAt] pool = append(pool[:bestAt], pool[bestAt+1:]...) } page = append(page, pick) creators[pick.CreatorID] = true topics[pick.Topic]++ } return page}interface Scored { videoId: string; creatorId: string; topic: string; score: number;}
// Stage 5: the heavy ranker scores each video alone. This builds the page as a whole: no// creator twice, topics spread out, and some slots for new videos still being tested.class ReRanker { private static readonly TOPIC_PENALTY = 0.15; // each repeat of a topic costs 15% // The first two slots are never exploration: they decide whether the user keeps scrolling. private static readonly PROTECTED = 2;
constructor( private readonly exploreRate: number, // share of all slots, 0.05 on this page private readonly random: () => number = Math.random, // injected so tests are repeatable ) {}
page(ranked: Scored[], fresh: Scored[], size = 10): Scored[] { const pool = [...ranked]; const page: Scored[] = []; const creators = new Set<string>(); const topics = new Map<string, number>(); // Spread the exploration share over the slots allowed to hold it, so the overall rate // still comes out at exploreRate. const pExplore = (this.exploreRate * size) / (size - ReRanker.PROTECTED); while (page.length < size) { let pick: Scored | undefined; if (page.length >= ReRanker.PROTECTED && fresh.length > 0 && this.random() < pExplore) { pick = fresh.shift(); // a new video, already matched to this user's topics } else { // Greedy: score what is left against what the page already holds. 10 slots times // 500 videos is nothing next to the ranker. let best = -Infinity; for (const s of pool) { if (creators.has(s.creatorId)) continue; const adjusted = s.score * (1 - ReRanker.TOPIC_PENALTY) ** (topics.get(s.topic) ?? 0); if (adjusted > best) [best, pick] = [adjusted, s]; } if (!pick) break; // a short page beats a repetitive one pool.splice(pool.indexOf(pick), 1); } page.push(pick!); creators.add(pick!.creatorId); topics.set(pick!.topic, (topics.get(pick!.topic) ?? 0) + 1); } return page; }}This is enough for a mid-level answer. Trending as a fallback, two towers to find candidates, two models to rank them, a stream for session signals, and a final pass for variety and new videos. The rest of the page is what makes it a senior answer: proving quality, the size of the exploration budget, ten times the users, and hidden bugs in training.
Go deeperFollow a page and a skip, and the classes3 min
Follow one full page. Use Next hop to move along. The bar is the server time budget.
Trace a request: Next feed page
Next feed page: step 1 of 12
Users to API gateway
The app asks for the next page while the user is still three videos from the end of this one. The gateway checks the token and routes by user id.
- Users → API gateway3 ms
- API gateway → Feed service2 ms
- Feed service → Retrieval service2 ms
- Retrieval service → User tower6 ms
- Retrieval service → ANN index15 ms
- Retrieval service → Follow graph0 ms
- Feed service → Trending lists0 ms
- Feed service → Ranking service6 ms
- Ranking service → Feature store35 ms
- Feed service → New video pool3 ms
- Feed service → Event streamasync
- Response
And one skip, from the swipe to the session row:
Trace a request: A skip reaches the next page
A skip reaches the next page: step 1 of 5
Users to API gateway
The user swipes away after 1.2 seconds. Skips, likes and "not interested" are sent at once; plain view progress is batched every few seconds. Each event has an id.
- Users → API gateway100 ms
- API gateway → Event stream20 ms
- Event stream → Stream processor500 ms
- Stream processor → Feature store5 ms
- Response
- Color key
- Facade
- Entity
- Value object
- Enum
- Interface
- Implementation
A new candidate source or ranker is a new class, not an edit to the feed service.
The API. GET /v1/feed?cursor=... returns 10 videos and a requestId. The cursor holds the
session and a page number, not a list of videos, so each page is ranked fresh. POST /v1/events
reports what the user did. Each event carries the requestId and the video’s position, so training
can tie a later skip to the exact ranking decision. Each event also has its own ID, so a retry is
safe.
Prove it works
Section titled “Prove it works”You have a new ranking model. How do you know it’s better, and what stops a bad one from shipping?
Whether you know offline metrics only pick models for an A/B test, and you name guardrails and position bias. “AUC went up, ship it” is the red flag. Offline logs only contain what the old model chose to show.
Here the labels are what users did, not labels written by people. The check still runs in two parts:
- Offline, on yesterday’s logs. Does the model rank the videos people finished above the ones they skipped? Do its predictions match what really happened? These tests only pick which models deserve a live test.
- Online, in an A/B test for two weeks, on 1% of users per group. Watch time per user must win. The guardrails (“not interested”, reports, and users still active after 7 days) must not get worse by more than 1%.
Why offline isn’t enough. The logs contain only videos the old model chose. A new model that likes different videos is graded on a test it can’t pass. The videos it would have shown were never shown, so they have no labels.
Position bias. A video in slot 1 gets watched more because it is in slot 1. During training, the model learns position separately. At serving time, position is set to the same value for every video, so the model ranks the video, not the slot.
A long-term holdout keeps 1% of users on last quarter’s models. It shows whether many small wins add up, or slowly wear the feed down.
Go deeperAll the release metrics2 min
| Metric | How we measure it | Target | Blocks a release? |
|---|---|---|---|
| Retrieval recall at 3,000 | Replay yesterday’s requests. Share of next-day finished videos found | No drop over 1% | Yes |
| AUC per prediction | Next-day logs the model never trained on | Finish prediction: no drop over 0.002 | Yes |
| Calibration per prediction | Predicted against real rate, in every score range | Within ±5% everywhere | Yes |
| NDCG at 10 | Re-rank logged sessions. Finished videos count as relevant | No drop over 0.5% | Yes |
| Watch time per daily user | A/B test, 2 weeks, 1% of users per group | Wins at 95% confidence | Yes: the main one |
| 7-day retention | The same A/B, plus the long-term holdout | No drop over 1% | Yes |
| “Not interested” and reports | The same A/B | No rise over 1% | Yes: guardrail |
| New videos reach 200 views | Eligible uploads within 24 hours | 95% or more | Yes |
| Server time per page | Shadow traffic before the A/B | Under 200 ms for 99% | Yes |
AUC measures how often the model ranks a finished video above a skipped one. NDCG measures whether the best videos sit near the top of the page. Calibration means a predicted 30% chance really happens about 30% of the time, which the weighted sum needs.
The release runs in order, and each step can stop it: offline metrics, then the model scores live traffic without being used, then 1% of traffic for a day, then the two-week A/B.
Interviewer asksWatch time is up 2% but 'not interested' is up 3%. Ship it?
No. Guardrails exist for this case. The model found videos people watch and regret, and users usually leave weeks later. Look at what changed. Often one prediction is off, or the weights reward length. Fix it, and test again.
Stress it
Section titled “Stress it”The design works at today’s size. Now we push it through four hard situations: how much to explore, a new user and a new video, ten times the users, and a model that wins offline and loses online.
How much exploration is enough
Section titled “How much exploration is enough”Your ranker learns from what it showed. Isn’t that a loop? And how much exploration is enough?
Whether you can name the bias and size exploration with math, not taste. “A few percent” with no reasoning is weak. Deriving 5% from uploads and test size is strong.
The model only gets labels for videos it chose to show. A video it underrates is never shown, so the mistake is never corrected, and the feed narrows. This is called selection bias. The cure is to show some videos the model is unsure about.
The budget is math. The feed shows 20 billion videos a day. About half of the 10 million daily uploads pass the quality checks. If judging a video takes about 200 views, testing them all costs 5 million × 200 = 1 billion views. That is 5% of slots. Try other values:
Ten feed pages, ten slots each. Amber slots go to new videos.
Fixed: 20 billion feed slots and 10 million uploads a day. Illustrative assumptions: an exploration slot earns about 70% of the watch time of a ranked slot, and about 1 in 100 tested videos turns out better than the average ranked video.
If a test slot earns about 70% of a normal slot’s watch time, 5% costs about 1.5% of today’s watch time. (The 70% is an assumption. The A/B test measures the real number.) It buys a catalog that renews itself, and creators who keep uploading.
A new user, and a new video
Section titled “A new user, and a new video”A user installs the app for the first time. Another uploads their first video. What does each see in the first hour?
Whether you treat the two as different problems with different fixes, and whether the models read features, not only IDs. “Show popular videos” for both is half an answer.
A new user has no history. But they have a device, a language, a country and a time of day. The first page is trending in their region and language. Then the session row does the work. By the fifth video, skips and finishes give the user tower something to work with. The tower reads features, not a user ID, so it can place a stranger.
A new video has no views, so nothing ranks it high, so it gets no views. Four things break that loop:
- Color key
- The upload
- Model
- Ready to be found
- Test slots
- Outcome
The fourth is a head start for creators whose videos usually do well. The test uses finishes and “not interested”, not taps, so a misleading thumbnail fails its test fast.
Ten times the users
Section titled “Ten times the users”Users grow ten times. What breaks, and what does it cost?
Whether you scale from the numbers: GPUs, feature reads and events. “Add more servers” is true and useless. The strong answer cuts work per request, and says what each cut risks.
At 700,000 pages a second, the big model needs about 5,800 GPUs, about $100 million a year. The feature store and the event stream fall behind, and the search index needs over 230 copies. Cutting work per page is cheaper than buying all that.
Step 6: cheaper pages. Four cuts, each aimed at one number:
- Find candidates once per few pages, rank every page. The 3,000 candidates are kept for three pages. Ranking still runs on every page with fresh session signals, so a skip still counts. Search load falls to about a third.
- The big model sees 250 videos, not 500. First check offline how often the big model’s top 10 sit in the small model’s top 250. At 98% or more, the cut halves the GPUs and loses almost nothing.
- Popular video rows kept on the ranking servers. The hot 10 million video rows (about 10 GB) live in each server’s memory, refreshed every minute. The feature store then serves ten times the traffic with the same cluster.
- The app sends events in batches, and the stream gets more partitions.
Stage 6: Cheaper pages at 10×
Stage 6 of 7
Cheaper pages at 10×
Why this stage
At 700,000 requests a second the feature store and the event stream pass 100%, and the heavy ranker would need about 5,800 GPUs.
What we add
A per-session candidate cache, a heavy ranker that sees 250 videos instead of 500, video feature rows cached on ranker hosts, and client-side event batching.
What it costs
Candidates up to three pages old, a funnel that must be checked for lost recall, and more caches to keep warm.
At 70k feed requests per second, every part has headroom. Raise the traffic until something turns red.
The numbers behind the busy bars
How busy it gets
- Each feed request produces about 10 events.
What one machine can do
- One feed server handles about 2,000 requests a second.
- One ANN replica set (8 shards) answers about 5,000 queries a second.
- One GPU makes about 100,000 heavy-ranker scores a second. That is 200 requests a second at 500 videos each, or 400 at 250.
How big the shared systems are
- The feature store is sized for 150,000 requests a second of about 500 rows each. Caching video rows on the ranker hosts lets it serve ten times that.
- The event stream takes 3 million events a second, and 10 million with client batching and more partitions.
- Instance counts aim for 60% busy.
Step 7: regions. A user in Jakarta shouldn’t cross the Pacific for every page. So every region runs the full serving path, including a copy of the 72 GB index. Training stays in one place, because it needs every region’s logs. It ships each day’s models and index everywhere as one versioned bundle.
Stage 7: Regions
Stage 7 of 7
Regions
Why this stage
A user in Jakarta should not pay 200 ms to reach Virginia, and one region is one failure domain for the whole feed.
What we add
The full serving path copied to every region, with its own index replica, feature store and stream.
Training stays central.
What it costs
Models and indexes shipped to every region, and a user who travels starts with a cold session.
At 70k feed requests per second, every part has headroom. Raise the traffic until something turns red.
The numbers behind the busy bars
How busy it gets
- Each feed request produces about 10 events.
What one machine can do
- One feed server handles about 2,000 requests a second.
- One ANN replica set (8 shards) answers about 5,000 queries a second.
- One GPU makes about 100,000 heavy-ranker scores a second. That is 200 requests a second at 500 videos each, or 400 at 250.
How big the shared systems are
- The feature store is sized for 150,000 requests a second of about 500 rows each. Caching video rows on the ranker hosts lets it serve ten times that.
- The event stream takes 3 million events a second, and 10 million with client batching and more partitions.
- Instance counts aim for 60% busy.
Wins offline, loses online
Section titled “Wins offline, loses online”The new ranker beat production offline by a wide margin, and lost the A/B test. What happened?
Whether “training and serving skew” is your first suspect, and you can say where the leak usually is. Blaming the A/B test, or retraining with more data, misses the most common cause.
The usual cause: the model trained on information it doesn’t have at serving time. This is called training and serving skew.
Here is a typical leak. A job computes “finish rate” from the full day’s logs. A video that went viral at 22:00 already looks viral at 09:00. Offline, the model looks brilliant, because it can see the future. Online, those values don’t exist yet.
The fix: log the features exactly as served, with each page. Training joins those logs with what the user did next. Then training sees exactly what the model saw.
- Color key
- Logged when served
- Joined data
- What users did
- Training
The cost: a new feature needs weeks of logging before the model can train on it. To speed that up, we rebuild a new feature’s past values “as of” each page’s time, only to fill the gap.
Go deeperA page in 200 ms for 99% of requests2 min
The median is 72 ms, but the slowest 1% take 350 ms. Where does the tail come from? A page waits for the slowest of many calls. If each of 8 index shards is slow 1% of the time, a query that needs all 8 is slow about 8% of the time. The fixes spend a little quality to cap the time:
| Technique | What it does | Gives up |
|---|---|---|
| A deadline per source | Each source gets 40 ms. The merger uses what arrived | A few candidates on a bad request |
| Partial search results | Merge the shards that answered in time | One eighth of the catalog, that once |
| Hedged requests | Ask a second copy if the first is slow | About 5% more search load |
| GPU batch window | Wait at most 5 ms to fill a batch, then run it | Some GPU efficiency |
| A fallback ladder | Big model late: use the small model’s order. Everything late: trending | Quality on that page only |
A slightly worse page is invisible to the user, and an error is not. So every step has a deadline, and a cheaper answer behind it. These are the usual resilience patterns.
When things break
Section titled “When things break”The feature store is down. What does the user see?
Whether every part has a fallback that still returns a page, and whether you list quality failures (a bad model, broken labels) beside outages. Only listing server crashes is the red flag in a machine learning design.
The feed gets worse before it gets empty. The order of fallbacks is the design: the full pipeline, then the small model only, then saved candidates, then the trending list. If the feature store is down, the models use the default values they were trained with, and the small model’s order is served.
Click any box to break it and see what happens:
Break it: what fails, who notices, what users see
Failure: click any box to break it
Feature store goes down
What breaks
The heavy ranker gets no user, session or video rows.
How we notice
Read error rate and p99 latency.
What users see
A less personal order if rows are missing; slow pages if reads hang.
What we do
A tight deadline on reads, defaults for missing features (the model was trained with some missing), and the light ranker fallback.
Some failures belong only to a learned system. No server is down, but the feed quietly gets worse:
- Broken labels. An app release stops sending watch time. The model learns that every view was a skip. Training refuses data whose label rates move too far.
- Tower and index mismatch. The user tower is from today and the index from yesterday. Search returns noise. They ship as one bundle, and a recall check runs on deploy.
- The feed narrows over weeks. Fewer topics and creators per user. Exploration, variety rules and the long-term holdout catch it.
Go deeperEvery failure, and what we do2 min
| Part | Failure | What users see | What we do |
|---|---|---|---|
| Feature store | Down or slow | A slightly worse order | Tight deadline. Default values. Small model’s order |
| Big model GPUs | Overloaded or down | A worse order | Small model’s order, then saved candidates, then trending |
| Search index | A shard down | Fewer candidates | Copies. Merge partial results |
| Event stream | Down or behind | The feed stops reacting in the session | The last session row keeps serving. Replay safely |
| Training | Fails | Models age | Alarm at 36 hours. Keep serving the last good model |
| A bad model | Passes offline, hurts users | Watch time or guardrails drop | Canary and A/B guardrails. Roll back the bundle in minutes |
| Fake views | Bots inflate a video | Trending is poisoned | Count only views from trusted sessions. Finishes count more |
| A whole region | Outage | A minute of errors, cold sessions | DNS to the nearest region. Same model bundle everywhere |
Wrap up
Section titled “Wrap up”The whole design
Section titled “The whole design”Here is everything we built, in the order we built it. Use the buttons to step through it.
The full design, stage by stage
Stage 7 of 7
Regions
Why this stage
A user in Jakarta should not pay 200 ms to reach Virginia, and one region is one failure domain for the whole feed.
What we add
The full serving path copied to every region, with its own index replica, feature store and stream.
Training stays central.
What it costs
Models and indexes shipped to every region, and a user who travels starts with a cold session.
At 70k feed requests per second, every part has headroom. Raise the traffic until something turns red.
The numbers behind the busy bars
How busy it gets
- Each feed request produces about 10 events.
What one machine can do
- One feed server handles about 2,000 requests a second.
- One ANN replica set (8 shards) answers about 5,000 queries a second.
- One GPU makes about 100,000 heavy-ranker scores a second. That is 200 requests a second at 500 videos each, or 400 at 250.
How big the shared systems are
- The feature store is sized for 150,000 requests a second of about 500 rows each. Caching video rows on the ranker hosts lets it serve ten times that.
- The event stream takes 3 million events a second, and 10 million with client batching and more partitions.
- Instance counts aim for 60% busy.
The decisions, and what each one costs
Section titled “The decisions, and what each one costs”| Decision | We chose | We gave up | We’d choose differently if |
|---|---|---|---|
| Approach | Two towers to find candidates, then two ranking models | The simplicity of popular lists | The catalog were small enough to score in full |
| Goal | Capped watch time and finishes, with guardrails | The easy growth of optimizing taps | The product sold ads per view |
| Funnel size | 3,000 → 500 → 10 | Some good videos the big model never sees | GPUs were cheap |
| Freshness | Session signals from a stream, daily training | A model that adapts within the hour | Trends turned over in minutes |
| Training data | Log features as served | Training on a new feature from day one | Features changed weekly and logs cost too much |
| Exploration | A fixed 5%, matched by topic, not in the first two slots | About 1.5% of today’s watch time | Uploads were few |
| At ten times | Candidates kept for 3 pages, big model input 250 | Candidates up to a minute old | GPUs weren’t the limit |
| Regions | Serve everywhere, train in one place | Models per region | Regions had very different laws or catalogs |
How to spend 45 minutes
Section titled “How to spend 45 minutes”- 0 to 6. Scale, catalog, and above all the goal. Popular, co-watch, then the funnel. The token math that keeps an LLM out.
- 6 to 11. The numbers: 70,000 pages a second, 3,000 candidates, 500 big-model scores, about 600 GPUs, 35 million feature rows a second.
- 11 to 25. Build it: trending, two towers, two rankers, the stream, variety and new videos. A mid-level answer can stop here.
- 25 to 29. Offline tests, the A/B test, guardrails, position bias.
- 29 to 42. Stress it: exploration, cold start, ten times, skew.
- 42 to 45. Failures, then your own questions.
Never skip the funnel math, the goal, or exploration.
Follow-up questions
Section titled “Follow-up questions”Interviewer asksWhere could an LLM actually help?
Offline, around the funnel. It can write topics, captions and text summaries of new videos, which become better features for cold start. It can explain why a video was recommended, or clean up noisy labels. But not score candidates per request. The token math rules that out at this traffic.
Go deeperMore follow-up questions3 min
Why not one big model instead of finding candidates and then ranking? Cost per video. The big ranker mixes user and video features freely: accurate and expensive. The two towers meet only in one multiplication: weaker, but video vectors can be computed ahead and searched in milliseconds. The funnel uses each where it is affordable.
How often do you retrain, and could you train online? Daily, with streamed features, gets most of the value. Training online, updating the model from the stream every few minutes (as in ByteDance’s Monolith), catches trends faster. But a bad update reaches everyone in minutes, so it needs automatic checks and rollback first.
Product wants “show me fewer videos like this”. Where does it go? An event like any other, sent at once. The stream processor applies a strong negative to the video’s topics and creator in the session row, and a longer one in the user row. The big model’s “not interested” prediction already learns from it.
A video is taken down. How fast does it leave every feed? The takedown marks the video row removed. The merger, the new video pool and the final pass all filter on it, and ranking servers refresh cached rows every minute. The vector can stay in the index until its daily rebuild, because the filter runs after search. Worst case, about a minute.
How do you stop creators gaming the new video pool? Only uploads that pass checks, from accounts that aren’t flagged, get in, with a cap per creator per day. The test audience is matched by content, so an off-topic thumbnail gets skipped fast. Graduating uses finishes and “not interested”, not taps.
The seen filter is lost. What happens? Users see repeats for a while. Rebuild it from 30 days of logs. Meanwhile, the session row’s last 50 videos stop the most visible repeats.
Further reading
Section titled “Further reading”- Recommendation and ranking, embeddings and vector search and evaluation and guardrails on this site, and choosing an approach for the ladder of options.
- Deep Neural Networks for YouTube Recommendations, Covington et al. 2016: the two-step design this page follows, and why watch time beat clicks.
- Recommending What Video to Watch Next: A Multitask Ranking System, Zhao et al. 2019: a model that predicts several outcomes, and a small branch for position bias.
- Monolith: Real Time Recommendation System With Collisionless Embedding Table, ByteDance 2022: online training for a short video feed.
The designbeyondcode digest, every Saturday
New deep dives, interview tips and site news in one short email. Free. Leave with one click.
We never sell or share your email. We only use it to send this newsletter.