Lease-Based Job Queue for Flaky Render Workers

A photo service renders thumbnails on a pool of worker threads. The workers are unreliable: some segfault in the image library, some get stuck on huge files, and some raise errors. Design and implement the queue between the upload handler and those workers.

Requirements

1. No silent loss. Every submitted job must end up either done or dead (deliberately parked after too many failures). A worker crashing or hanging must not make a job disappear.
2. Crash detection without help from the worker. A crashed worker never calls back. The queue has to notice on its own.
3. Long jobs. A slow but healthy worker must be able to keep its job.
4. Zombie protection. A worker that was presumed dead might finish later, after its job went to someone else. Its report must be rejected, so it can't overwrite the current holder's outcome.
5. Poison jobs. A job that keeps failing must stop being retried after max_attempts and go to a dead-letter list along with its error history.
6. Thread safety. Many workers call the queue at the same time. No job may be handed to two workers at once.
7. Testability. Tests must not use sleep. Time has to be controllable.

API to implement (adjust if you can justify it)

class LeaseQueue:
    def __init__(self, lease_seconds=30.0, max_attempts=3, clock=time.monotonic): ...
    def submit(self, payload) -> int: ...                 # returns job id
    def claim(self) -> Lease | None: ...                  # None if nothing ready
    def heartbeat(self, lease) -> None: ...               # extend the lease
    def complete(self, lease, result=None) -> None: ...
    def fail(self, lease, reason: str) -> None: ...
    def status(self, job_id) -> str: ...                  # waiting|leased|done|dead
    def dead_letters(self) -> list[tuple[int, list[str]]]: ...

A report on a lease that is no longer current must raise StaleLeaseError.

Deliverables

• A working class.
• Unit tests that simulate each failure mode: a crash (silence), a hang that ends after the lease expired, an explicit failure, a poison job, and concurrent claimers.
• A short explanation of the delivery guarantee you provide (at-most-once, at-least-once or exactly-once) and why.

Example timeline (lease = 10 s)

t=0   submit(job 1)
t=0   worker A claims job 1      -> lease token 1, attempt 1
t=11  worker B claims            -> job 1 again, token 2, attempt 2
t=12  A calls complete(token 1)  -> StaleLeaseError
t=13  B calls complete(token 2)  -> job 1 is done