Design a model to choose dynamic K

The matching group is ranking-related. The interview had one open-ended ML System/Model Design question. I'm not very familiar with retrieval-related topics and didn't come up with a great answer during the interview. Feel free to discuss in the comments.

In the recall stage of a recommender system, K candidates are typically retrieved and sent to the heavy ranker. Design a system or model to decide what K should be.

K should not be a static number — the goal is to use a model to determine K dynamically.

Some thoughts (not necessarily correct, open to discussion):

• Define the goal metric: how do you determine whether the chosen K is good? What do you measure by?
• The proportion of candidates that pass the heavy ranker?
• What should the model output?
• Directly output the optimal K → seems very hard to implement
• Output the probability of each candidate being selected by the heavy ranker (i.e., the goal metric from above)? → then design a threshold, dynamically set K based on how many candidates pass the threshold? → but how do you design the threshold — it seems like defining K just becomes defining the threshold
• What features and what model to use. This is closely related to the output question above.