Choose a Safe Polling Strategy for Delayed Records

Data store A emits one record every 30 seconds. A is a black box. Each record eventually appears in database B, with an individual lag between zero and five minutes, and no record is lost. You can read B but cannot change it. You control database C and a poller that copies records from B.

In a boot or test state, you may observe arrivals and adjust the design. In live state, the business asks for the shortest possible polling interval while also requiring every poll to retrieve exactly one previously unprocessed record and never fail.

Clarifying Questions to Ask

• Does B retain history, and can records be read by a stable sequence or cursor?
• Can one query request at most one unprocessed record?
• Can records appear in B out of generation order?
• Does a poll returning zero records count as failure?
• May C buffer records separately from the live one-at-a-time consumer?

Part 1: Analyze the Requirement

Determine whether the stated generation period and maximum lag are sufficient to choose a fixed polling interval that guarantees exactly one newly arrived record per poll.

What This Part Should Cover

• A worst-case arrival argument rather than an average-rate argument
• The difference between generation cadence and observed arrival cadence
• Recognition of impossible or underspecified guarantees

Part 2: Use the Boot State

Explain what to measure or validate before live operation and when observation is evidence versus a proof.

What This Part Should Cover

• Stable record identity, ordering, and lag measurement
• Validation of cursor and idempotency behavior
• A warm-up or backlog policy
• Why a finite test cannot prove an unstated production bound

Part 3: Define the Live Strategy

Propose a design and polling rate under explicit assumptions. Preserve the requirement that downstream processing handles one record at a time without pretending B always receives one record per interval.

What This Part Should Cover

• Decoupling ingestion from one-at-a-time consumption
• Durable cursors, deduplication, and retries
• Buffer sizing or warm-up based on the five-minute bound
• Monitoring and a response to bound violations