Katkat
Back

Mean and Variance Through Maximum Likelihood

AmazonApplied Scientist · Onsite
MediumApplied ScientistOnsiteStatistics & Math

Last updated

Derive the maximum likelihood estimators (MLE) for the mean and variance of a univariate Gaussian distribution, and analyze the bias of the resulting variance estimator.

Scenario

At Amazon, you are analyzing server latency metrics for a critical payment gateway service. Preliminary exploratory data analysis shows that latency measurements under steady-state traffic approximately follow a univariate normal (Gaussian) distribution.

Suppose you observe an independent and identically distributed (i.i.d.) sample of n continuous latency measurements: X = {x_1, x_2, ..., x_n}, where each x_i ~ Normal(mu, sigma^2) with unknown mean mu and unknown variance sigma^2 > 0.

Tasks

  1. Likelihood & Log-Likelihood Construction: Write down the joint probability density function (likelihood function L(mu, sigma^2)) of the sample, and compute the natural log-likelihood function ell(mu, sigma^2) = ln L(mu, sigma^2).

  2. Derive the MLE Estimators:

    • Find the maximum likelihood estimator for mu, denoted mu_hat_MLE.
    • Find the maximum likelihood estimator for sigma^2, denoted sigma^2_hat_MLE.
    • Show your step-by-step calculus (partial derivatives set to zero) and justify that this critical point represents a maximum.

  3. Bias Analysis of the Variance Estimator:

    • Compute the expected value E[sigma^2_hat_MLE] in terms of the true variance sigma^2 and sample size n.
    • State whether sigma^2_hat_MLE is unbiased. If biased, explain intuitively why this bias occurs (degrees of freedom / estimation of the mean).
    • Show how to obtain the standard unbiased sample variance s^2 from sigma^2_hat_MLE (Bessel's correction).

Hints

Hint 1

Start by expressing the log of the product of Gaussian densities as a sum of individual log densities, separating terms dependent on mu and sigma^2.

Hint 2

When computing E[sigma^2_hat_MLE], rewrite (x_i - mu_hat) as ((x_i - mu) - (mu_hat - mu)) before expanding the squared term.

Hint 3

Recall that Var(mu_hat) = sigma^2 / n, which will help evaluate E[(mu_hat - mu)^2].

Follow-up questions

  • Is the maximum likelihood estimator for the standard deviation, sigma_hat_MLE = sqrt(sigma^2_hat_MLE), an unbiased estimator of sigma?
  • How does the Mean Squared Error (MSE) of sigma^2_hat_MLE compare to the MSE of the unbiased estimator s^2?
  • Under a Bayesian framework with a conjugate normal-inverse-gamma prior, how do the posterior point estimates for mu and sigma^2 relate to the MLE solutions?