Katkat
Back

Explain and Reduce Position Bias in Ranking

RedditMachine Learning Engineer · Technical Screen
HardMachine Learning EngineerTechnical ScreenMachine Learning

Last updated

Assess understanding of position bias in click-based ranking systems, including formalizing the examination hypothesis, estimating observation propensities, training with Inverse Propensity Scoring (IPS), and designing counterfactual evaluation metrics.

Scenario

You are designing a ranking model for a personalized social feed. The system logs user interactions (clicks, upvotes, dwell time) across ranked lists of posts. Historical analysis reveals that items displayed at rank 1 achieve an empirical click-through rate (CTR) roughly four times higher than items at rank 5, even when post quality and user intent are controlled.

Training directly on raw click logs introduces a feedback loop: lower-ranked relevant posts receive fewer clicks, leading the model to deprioritize them further.

Task

Walk through an end-to-end strategy to formalize, estimate, and correct for position bias:

  1. Formal Formulation:

    • Define the standard examination hypothesis relating the observed click indicator C_i, post relevance R_i, and examination indicator E_i at position k.
    • State the conditional independence assumptions required for this factorization.

  2. Propensity Estimation:

    • Describe at least two distinct approaches to estimate the examination propensity P(E_i = 1 | position = k): one intervention-based (e.g., randomized experiments or top-k swapping) and one observation-based (e.g., dual learning algorithm or EM-based click model).
    • Compare their trade-offs regarding user experience impact, sample efficiency, and implementation complexity.

  3. Debiased Model Optimization:

    • Formulate the Inverse Propensity Scoring (IPS) objective for an empirical loss function L(y_pred, y_true).
    • Address the high-variance problem of IPS when propensities for deep positions are very small. How do you stabilize training (e.g., propensity clipping, normalized IPS)?

  4. Counterfactual Offline Evaluation:

    • Explain how to compute an unbiased estimate of NDCG or Average Precision on logged bandit feedback without deploying the model to production.
    • Identify key failure modes where the counterfactual estimator becomes invalid (e.g., violation of common support / positivity).

Hints

Hint 1

Under the examination hypothesis, assume a user clicks if and only if they both examine the item and find it relevant: P(C = 1) = P(E = 1) * P(R = 1).

Hint 2

Randomizing post rank positions on a small fraction of traffic isolates the effect of rank from relevance, yielding an empirical propensity curve.

Hint 3

Small propensities in the denominator of IPS cause extreme gradients; bounding propensities from below or using self-normalized weights mitigates this variance.

Follow-up questions

  • How does position bias interact with contextual factors, such as device viewport size, image aspect ratios, or post presentation formats?
  • If user attention drops off dynamically based on whether preceding items were relevant (cascade model), how does your propensity formulation change?
  • How do you validate in an online A/B test that an IPS-debiased ranker actually outperforms the biased baseline without hurting short-term engagement metrics?