Design distributed parallel job processing

Design a distributed system to execute many independent jobs in parallel across a cluster. Specify the job model and APIs, task partitioning/sharding strategy, scheduler and worker architecture, coordination (e.g., leases or heartbeats), fault tolerance and retries with backoff, idempotency and deduplication, progress tracking and result aggregation, scaling and resource management, backpressure and prioritization, and ordering guarantees. Discuss consistency trade-offs (at-least-once vs exactly-once), monitoring, and capacity planning.