Katkat
Back

How to rank statements by likelihood

OptiverData Scientist · Online Assessment
HardData ScientistOnline AssessmentMachine Learning

Last updated

This problem tests your ability to translate statistical distributions and categorical data into ranked probabilistic likelihoods, accounting for uncertainty and sample size variance.

Problem Statement

You are provided with a dataset containing performance metrics for 5 proprietary trading strategies (A, B, C, D, E) observed over the last 100 trading days. Each strategy has a mean daily return mu and a standard deviation sigma of returns. Additionally, each strategy has a 'Confidence Score' based on the number of observations (N), where N_A=100, N_B=80, N_C=50, N_D=20, N_E=10.

Your task is to rank the likelihood of the statement: 'Strategy X will yield a positive return tomorrow', given the following data:

StrategyMean Return (%)Std Dev (%)Observations (N)
A0.150.8100
B0.201.280
C0.100.550
D0.302.520
E0.404.010

Assuming daily returns follow a normal distribution, rank these strategies from most likely to least likely to yield a return > 0 tomorrow. You must account for the uncertainty of the mean estimate, which is proportional to sigma / sqrt(N).

Implementation

Provide a function rank_strategies(data) that returns the strategy labels sorted by the probability P(return > 0). Explain your reasoning regarding the trade-off between the observed mean and the uncertainty introduced by low sample sizes (the 'look-back' bias).

Example

Input: The table provided above. Output: ['C', 'A', 'B', 'D', 'E'] (Hypothetical) Explanation: Even though E has the highest mean, its high standard deviation and low N make the probability of a negative outcome higher compared to C, which has a very stable mean.

Hints

Hint 1

Consider the return distribution as a t-distribution since the population variance is unknown and estimated from a sample.

Hint 2

Calculate the z-score (or t-score) for the threshold of 0 for each strategy, defined as `(mean - 0) / (std_dev / sqrt(N))`.

Hint 3

Notice that as N decreases, the interval for the potential mean expands, significantly penalizing strategies with high volatility and low sample counts.

Follow-up questions

  • How would your ranking change if you discovered that Strategy E is actually a non-stationary process that changes its regime every 20 days?
  • If we were to incorporate a Bayesian prior (e.g., assuming all strategies have a mean return of 0 initially), how would that influence the ranking of the low-N strategies (D and E)?
  • How does the choice of confidence interval affect the robustness of this ranking in a real-time trading environment?