Design a Core Dump Collection and Analysis System
AmazonSoftware Engineer · Onsite
Last updated
This problem tests the ability to design a distributed, scalable ingestion and analysis pipeline for massive binary crash files while maintaining system performance and data availability.
Design a Core Dump Collection and Analysis System
Design a system that collects, stores, and analyzes core dumps generated by a fleet of 100,000 microservices running across multiple regions. When a service crashes, it generates a core dump file (ranging from 100 MB to 10 GB in size).
Functional Requirements
- Ingestion: Efficiently upload core dumps from production nodes to a centralized storage.
- Deduplication/Grouping: Group similar crashes together (e.g., same stack trace or signal) to reduce noise for developers.
- Analysis: Extract stack traces and identify the faulty function/module from the binary.
- Retention: Define a policy for how long these large files should be stored.
Non-Functional Requirements
- High Availability: The ingestion service must not block the production host's health or stability.
- Scalability: Handle thousands of concurrent crash uploads during a major deployment failure (thundering herd problem).
- Security: Core dumps contain sensitive memory data; ensure PII scrubbing and encryption at rest/transit.
Deliverables
Provide a high-level architecture diagram (described in text), API designs for the ingestion service, storage strategies for varying file sizes, and a strategy for the analysis pipeline. Address how you would prioritize which core dumps to process first during high-load periods.
Hints
Hint 1
Consider implementing a local agent on the production host that performs initial filtering or compression before attempting an upload.
Hint 2
Use a tiered storage approach: move raw binary files to cold storage (e.g., S3 Glacier) once the metadata and stack traces are extracted.
Hint 3
To handle a thundering herd, implement backpressure and rate limiting on the ingestion servers using a message queue or a prioritized task buffer.
Follow-up questions
- How would you modify the system to handle PII (Personally Identifiable Information) masking within the core dump before it is stored?
- If a new 'impossible' crash occurs that the automated analyzer cannot parse, what manual intervention or debugging tool would you provide for the engineer?
- How would you measure the cost of this system, and what strategies would you use to optimize storage costs for petabytes of core dump data?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.