Katkat
Back

Design a Test Strategy for a System That Counts the Leaves on a Large Tree

GleanSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSoftware Engineering Fundamentals

Last updated

An open-ended testing scenario: design a test strategy for a system that counts the leaves on a large tree. Tests clarifying an ambiguous system, finding a test oracle, functional and boundary cases, scale and robustness testing, and setting release criteria for counts that may be estimates.

In a QA engineer interview, the testing scenario was: how would you test a system that counts the number of leaves on a large tree?

The prompt is deliberately underspecified. Part of the exercise is establishing what the system is before designing tests for it.

Clarifying Questions

  • Is it a physical tree photographed or scanned by a vision system, or a software tree data structure whose leaf nodes are counted?
  • What accuracy is acceptable: an exact count, or a count within some tolerance?
  • How large is "large", and are there time or memory limits?
  • What are the inputs and outputs: images or a stream of nodes, a single number or a per-branch breakdown?

Hints

Hint 1

Find the oracle: Decide how you would know the correct count for a test input. Most of the strategy follows from that.

Hint 2

Vary one thing at a time: List the properties of the tree and of the input that could change the count or break the counter, and design inputs that vary them deliberately.

Follow-up questions

  • You cannot count the leaves of a real large tree by hand. How do you still estimate the system's accuracy?
  • The tree is too large for one machine's memory and is stored across machines. What new failure modes do you test?
  • How do you set release criteria for a count that is never exactly right?