Decide When Automated Tests Are Necessary

Interview Prompt

"Engineers often debate the right balance between moving quickly and maintaining strict test coverage. Walk me through your philosophy on automated testing:

1. When is comprehensive automated testing non-negotiable, and when is lighter testing (or even manual verification) justified?
2. How do you assess whether a test suite actually protects user-facing behavior, rather than simply satisfying a code coverage percentage target?"

Context & Expectations

High-reliability software environments—especially in fintech and crypto—handle irreversible actions such as balance updates, order matching, and blockchain transactions. In these domains, a bug can mean immediate financial loss or security breach. At the same time, dogmatic pursuit of 100% line coverage on low-risk, fast-changing prototypes can paralyze engineering velocity.

A strong answer should avoid generic platitudes ("tests are always good") and instead articulate a risk-weighted, pragmatic engineering framework.

What a Strong Answer Demonstrates

• Risk-Based Prioritization: Distinguishes between critical paths (financial calculations, authorization, data persistence, state transitions) where exhaustive automated tests (unit, integration, property-based) are mandatory, versus ephemeral code (one-off scripts, throwaway UI spikes, early discovery prototypes) where manual validation or minimal smoke tests are sufficient.
• Quality Beyond Metrics: Explains why vanity metrics like line or branch coverage can create a false sense of security (e.g., executing code without meaningful assertions). Advocates for behavioral testing, boundary-value analysis, fault injection, or mutation testing to measure actual resilience.
• Production-Minded Pragmatism: Demonstrates an understanding of the total cost of ownership of tests, including flakiness, maintenance overhead, and CI runtimes. Shows an ability to balance engineering velocity with system stability.