Katkat
Back

Testing a New Feature: Planning Test Cases and Handling Flaky Tests

CiscoSoftware Engineer · Onsite
EasySoftware EngineerOnsiteSoftware Engineering Fundamentals

Last updated

A two-part QA question on how to test a new feature and plan its test cases, and how to find, diagnose, and prevent flaky automated tests. It evaluates requirement analysis, test design techniques, risk-based prioritization, test levels, release exit criteria, and keeping a test suite trustworthy.

You own quality for a team, and a new feature is about to enter testing. Explain how you test it and how you plan its test cases, then explain how you deal with flaky tests in the team's automated suite. If no specific feature is given, pick a concrete one and use it throughout your answer.

Clarifying Questions

  • What kind of product and feature is it (a web UI, an API, software running on network devices), and is there a written requirement or only a verbal description?
  • What test infrastructure already exists: developer unit tests, an automated API or UI suite, CI on every change?
  • What is the release cadence, and how costly is a defect that reaches customers?

Part 1 — Testing a new feature and planning its test cases

Walk through how you test the feature, from the moment you see its requirements until it ships: what you do before any code exists, how you derive and prioritize test cases, which levels of testing you use, and how you decide it is ready to release.

What This Part Should Cover

  • Requirement analysis, with testability questions raised early
  • Systematic test-design techniques, including negative cases, boundaries and state changes
  • Prioritization by risk, and the choice of test level for each case
  • Entry and exit criteria, and how release readiness is decided

Part 2 — Handling flaky tests

Some automated tests pass and fail intermittently without any code change. How do you find them, diagnose them, fix them, and keep them from eroding the team's trust in the suite?

Clarifying Questions for this Part

  • How large is the suite, how often does it run, and does CI already retry failed tests automatically?
  • Who owns a failing test today: QA, or the developers whose code it covers?

What This Part Should Cover

  • Detection and confirmation of flakiness using repeated runs and failure history
  • The main root-cause categories and how to diagnose each
  • Fixes that remove the nondeterminism rather than hide it
  • A process for quarantine, ownership and measurement

Hints

Hint 1

Start before the code: Think about what you can review, and which questions you can raise, when only the requirement or the design exists.

Hint 2

Not every case is equal: You cannot run every combination of inputs and states. Think about how you choose which cases to write first and which ones to automate.

Hint 3

Prove it is flaky: Before fixing anything, consider how you would show that a failure is intermittent rather than a real regression.

Hint 4

What differs between two runs: List what a test depends on that can be different between two runs of exactly the same code.

Follow-up questions

  • The release date is fixed and you only have time for a third of your planned cases. Which do you run?
  • A test fails once in about 200 runs. Is it worth fixing, and how do you decide?
  • How would you test a feature whose behavior depends on time, such as a task that becomes overdue after its deadline?
  • How do you know whether your test suite is effective, not just large?