Recommendation and ranking

A feed, a “you might also like” row, search results, the next video: each one is a ranking system. It picks a handful of items out of millions for one person, in well under a second.

Ranking is the most widely used kind of machine learning. It is older than LLMs, and it is still behind most of what people see online. Interviewers ask about it in feed, video, marketplace and ads designs. And more and more LLM products use the same machinery to choose which documents, tools or examples go into a prompt.

The problem

A video platform has a billion videos. A user opens the app. Within 200 ms, the home screen must show the ten videos they’re most likely to watch.

The best way to predict “will this user watch this video” is a large model with hundreds of features (inputs): the user’s history, the video’s engagement, the time of day, the device. That model takes about 50 microseconds per item, even when batched on a GPU. Scoring a billion items would take 14 hours. Per request.

How it works: a funnel

Every large recommender solves this the same way. It uses several stages. Each stage is cheaper per item than the one before, and sees fewer items.

The ranking funnel

Scroll or pinch to zoom, drag to pan

• Color key
• Service

Try it. Change how many candidates each stage passes on. Watch the latency, and how many of the user’s truly best items survive.

Retrieval returns candidatesLight ranker keeps Score all 1 billion items with the heavy model

With too few candidates, the best items are thrown away before the good model ever sees them. With too many, the heavy ranker blows the latency budget. So the design work is choosing the stage sizes. Each stage must fit its share of the budget. And each stage must be good enough not to drop items the next stage would have ranked highly.

Step 1: Retrieve candidates

Retrieval has to be fast and have high recall: get the good items into the set, and don’t worry about their order yet. Real systems merge several sources:

• Embedding retrieval. A two-tower model learns one embedding for users and one for items. A user’s vector ends up close to the vectors of items they engage with. Item vectors are computed ahead of time and indexed. At request time, compute the user’s vector and run an approximate nearest-neighbor search (vector search).
• Graph and co-occurrence. “People who watched this also watched”, and accounts the user follows.
• Fresh and trending. New items have no engagement history, so the embedding model won’t find them yet. They need their own source.
• Business rules. Items the platform must show, like a live event or a paid promotion.

Each source returns a few hundred to a few thousand items. Together that’s around 10,000. Duplicates are removed, and items the user already saw or blocked are filtered out.

Step 2: Rank in stages

A light ranker is a small model with a few dozen cheap features. It scores all the candidates and keeps the top few hundred.

A heavy ranker is a large neural network with hundreds of features. It scores those few hundred. It usually predicts several things at once: the chance of a click, a long watch, a like, a share, or a “not interested”. It combines them into one score, with weights the product team chooses. Those weights are where a company decides what it’s optimizing for.

Features are the inputs:

• user: watch history embedding, language, country, how active they are
• item: age, length, creator, engagement rate over the last hour, day and week
• user and item together: has this user watched this creator before, and how often
• context: time of day, device, network speed

Features must be the same at training time and at serving time. A feature store serves precomputed values fast. It also logs exactly what the model saw for each request. So the training data matches what was served.

Step 3: Re-rank

The heavy ranker scores each item on its own. So its top ten might be nine videos from the same creator. A final pass applies rules across the whole list:

• diversity of creators and topics
• no duplicates
• freshness
• policy filters
• slots for exploration (Step 5)

Step 4: Learn from what users do

Training labels come from implicit feedback: what users did with what they were shown. Watched for more than 30 seconds is a positive. Shown and scrolled past is a negative.

There’s a lot of this data, billions of impressions a day. That is why recommenders work well. But the data is biased:

• Position bias. Items at the top get clicked more, just because they’re at the top. Train with position as a feature. Then, when serving, set it to a fixed value.
• Selection bias. You only learn about items you showed. An item the model never ranked highly never gets the chance to prove it’s good.

Step 5: Explore, and evaluate online

Because of selection bias, a ranker left alone gets narrower over time. It shows what it already believes in, and learns only from that. So a few percent of slots go to exploration: items the model is less sure about. That gives new items and new interests a chance.

Offline metrics are useful for comparing models. Examples are AUC, precision at k, and normalized discounted cumulative gain on held-out logs. But they’re computed on data the old model chose. The real test is an A/B test on online metrics: watch time, return visits, and guardrail metrics such as “not interested” clicks and reports.

Options and tradeoffs

TradeoffMore candidates and a bigger heavy ranker

We gain

Better recommendations: fewer good items lost early, and better ordering of the rest.

We give up

Latency and GPU cost grow with the number of items scored. Doubling the heavy ranker’s input doubles its share of serving cost.

TradeoffOptimizing for a short-term signal like clicks

We gain

Plenty of data, fast feedback, and easy to move.

We give up

It rewards clickbait. Blend in longer-term signals (watch time, return visits, satisfaction surveys) and guardrail metrics.

Where LLMs fit. LLMs are too slow and too expensive to score millions of items per request. So they rarely sit in the main funnel. They help around it instead:

• writing item descriptions and tags that become features
• embedding text for retrieval
• explaining a recommendation
• re-ranking the final handful, where traffic is low and each item is worth a lot

It works the other way too. A RAG system’s retrieve-then-rerank is a two-stage ranking funnel.

How it fails

Failure	What happens	Fix
Cold start: new user	No history to embed	Onboarding choices, popular in their region, fast learning from the first few actions
Cold start: new item	Never retrieved, never gets engagement	A fresh-items source and exploration slots
Training and serving features differ	Great offline, poor in production	One feature store for both; log served features
Feedback loop	The feed narrows to a few topics	Exploration, diversity in re-ranking, long-term metrics
Stale features	Trending items surface an hour late	Streaming feature updates for fast-moving signals
Retrieval recall is low	No ranker can fix items that never arrive	Measure retrieval recall separately; add sources

Common mistake

Only designing the ranking model

Interview answers often jump to “a deep neural network predicts engagement” and stop. The model is only one stage. The design is the funnel: where candidates come from, how many each stage sees, where features come from at serving time, and how the system learns without trapping itself.

At scale

Take 100,000 feed requests a second at peak:

• Retrieval runs against an approximate nearest-neighbor index of a billion item vectors. The index is sharded across machines. Each query fans out to the shards and merges the results (see vector search).
• Heavy ranking at 500 items per request is 50 million item scores a second. A GPU does roughly 100,000 scores a second for a large model. So that’s about 500 GPUs. That is why the heavy ranker’s input size is a budget line, not a tuning knob.
• Features are read from an in-memory store: about 500 items times dozens of features per request. So they’re stored per item, in batches, not as one key per feature.
• Training never stops. Models are retrained daily or more often on the latest logs. New items get embeddings within minutes of upload.
• Precompute where you can. For users who aren’t active right now, candidate lists can be built in batch and cached. Only the final ranking is left for request time.

In the interview

Interviewer asksDesign the home feed for a video app. How do you pick the videos?

A funnel, sized by the latency budget. Candidates come from several sources: a two-tower embedding model with nearest-neighbor search, the creators the user follows, and trending and fresh items. A light ranker cuts 10,000 down to a few hundred. A heavy ranker predicts several engagement signals and combines them. A re-rank pass adds diversity and exploration. It is trained on implicit feedback with position bias handled, and judged by A/B tests on long-term metrics.

Interviewer asksHow do new videos get recommended if no one has watched them?

Three things. A candidate source just for fresh content. Item features that don’t need engagement: creator, topic, and text and visual embeddings, so a new video lands close to similar older ones. And exploration slots that show new items to a small, relevant audience and learn from the result.

Mid-level covers

Describes retrieval, then ranking. Uses a two-tower model with nearest-neighbor search for candidates, and a model that predicts engagement from user and item features.

Senior adds

Sizes the funnel from the latency budget and GPU cost. Uses several candidate sources. Handles cold start, position bias and selection bias. Keeps training and serving features the same. Blends short- and long-term objectives. Builds exploration and diversity into the design.

Recap

The funnel. Retrieval (about 10,000) → light ranker (hundreds) → heavy ranker → re-rank (10). Each stage is cheaper per item and sees fewer items.

Retrieval needs recall. Use several sources, including two-tower embeddings with nearest-neighbor search, and a fresh-items source.

Ranking predicts several engagement signals from user, item and context features. It combines them with weights the product team chooses. One feature store serves both training and serving.

Learning from implicit feedback is biased by position, and by what was shown. Correct for position, explore, and judge by A/B tests on long-term metrics.

Further reading

• Deep Neural Networks for YouTube Recommendations, Covington et al. 2016: the two-stage design most systems still follow.
• Embeddings and vector search for the retrieval stage.