Analyze Town Temperatures and Select Predictors for NYC
CitadelData Scientist · Technical ScreenLast updated
Implement exact pandas and NumPy functions for temperature statistics, single- and two-town regressions, deterministic return values, and greedy five-feature forward selection.
A pandas DataFrame contains (N) daily temperature observations for NYC and (P\ge5) town columns. Implement:
q1_q5(df) -> list
q6(df) -> list[str]
q1_q5(df) must return exactly this five-element list:
[
largest_std_column,
rounded_conditional_median,
rounded_absolute_slope_sum,
best_single_town,
[best_pair_town_1, best_pair_town_2]
]
q6(df) must return exactly five town names in greedy selection order.
Constraints & Assumptions
- The DataFrame has at least one row, unique string column names, a column named
NYC, a column namedTown2, and at least five non-NYCcolumns. - All cells are finite, numeric, and nonmissing.
NYCis the response and every other column is a candidate town. - Use population standard deviation (
ddof=0). If columns tie for the largest standard deviation, return the lexicographically smallest column name. This comparison includesNYC. - Every regression includes an intercept and is scored by in-sample mean squared error on all supplied rows.
- Break exact MSE ties lexicographically by town name or by the ordered two-town tuple.
- “Round” means nearest integer, with exact half values rounded away from zero.
- If no row has
Town2between 90 and 100 inclusive, raiseValueError. - A constant predictor has slope zero. For rank-deficient multivariate designs, use the minimum-norm least-squares solution; scoring is based on its predictions.
Part 1 — Variability and conditional median
Set largest_std_column to the column with the largest population standard deviation. Set rounded_conditional_median to the median NYC value among rows where 90 <= Town2 <= 100, rounded once.
What This Part Should Cover
- Population standard deviation across rows for every column, including
NYC, with the stated lexicographic tie rule. - Inclusive
Town2filtering, the required empty-subset error, and one half-away-from-zero rounding step. - Placement of these two outputs in positions 0 and 1 of the returned list.
Part 2 — Single-town regressions
Fit one separate simple linear regression per candidate town. Set rounded_absolute_slope_sum to the sum of the absolute slope coefficients, rounded once after summing. Set best_single_town to the town whose single-predictor model has the lowest MSE.
What This Part Should Cover
- A distinct intercept-bearing fit for each non-
NYCcolumn and a zero slope for a constant predictor. - Absolute slopes summed before the single required rounding operation.
- In-sample MSE comparison with the stated town-name tie rule and outputs in positions 2 and 3.
Part 3 — Best pair
Fit every unordered two-town linear model. Return the lowest-MSE pair as two lexicographically ordered names.
What This Part Should Cover
- Enumeration of each unordered town pair exactly once and a joint two-predictor fit with an intercept.
- Minimum-norm least squares for a rank-deficient pair and MSE scoring from its predictions.
- Lexicographic ordering within the pair, tuple-based score ties, and the nested list in position 4.
Part 4 — Approximate best five-town subset
For q6, begin with no predictors. At each of five steps, refit each model formed by the already selected towns plus one remaining town. Append the town producing the lowest MSE, using its name to break an exact tie. Greedy selection is an approximation and need not equal the globally best five-town subset.
What This Part Should Cover
- Five forward-selection rounds, with every remaining candidate refit alongside the already selected towns at each round.
- In-sample MSE and the exact town-name tie rule at every selection step.
- Exactly five distinct names returned in selection order, plus a clear statement that the greedy result is not guaranteed globally optimal.
Hints
Hint 1
Reuse one scorer: Build a deterministic least-squares scoring helper, a half-away-from-zero rounding helper, and one sorted candidate-town list, then use them for every output.
Follow-up questions
- How would nested cross-validation change feature selection and final scoring?
- Why can a town with a weak univariate model become useful in a multivariate model?
- How could sufficient statistics reduce repeated fitting when \(P\) is large?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.