Katkat
Back

Explain auth, key rotation, secrets, and incident response

RobinhoodSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSoftware Engineering Fundamentals

Last updated

This question tests your knowledge of secure microservice communication, the lifecycle management of sensitive credentials, and the operational procedures for mitigating security breaches.

Scenario

You are a senior engineer at a fintech platform. You are tasked with designing the security architecture for a new microservice that processes sensitive payment data. This service must communicate with several downstream internal services.

Task

  1. Authentication: Propose a mechanism for service-to-service authentication. Explain how you would ensure that Service A knows it is talking to the genuine Service B.
  2. Secret Management: Where should API keys and database credentials reside? Describe a workflow for storing and retrieving these secrets that minimizes the risk of exposure.
  3. Key Rotation: Explain how you would implement automated key rotation for a long-lived database credential without causing downtime for the dependent services.
  4. Incident Response: Your team receives an alert that a service token was accidentally committed to a public repository. Outline the steps you would take to contain the incident and remediate the risk.

Requirements

  • Provide a high-level architecture for your proposed solution.
  • Address how you handle 'secret zero' (the problem of authenticating the service to the secret manager itself).
  • Ensure your incident response plan covers identification, containment, eradication, and post-mortem analysis.

Hints

Hint 1

Consider using Short-Lived Tokens (like SPIFFE/SPIRE or Mutual TLS) to replace long-lived static credentials for internal traffic.

Hint 2

For secret management, think about how dynamic secrets (secrets created on-the-fly with a TTL) can eliminate the need for traditional rotation.

Hint 3

During incident response, prioritize invalidation (revocation) before investigation to stop potential active exploitation.

Follow-up questions

  • How would you change your strategy if the secret management tool itself experienced an outage?
  • What metrics would you monitor to detect potential credential stuffing or unauthorized access attempts?
  • How does your design change if the services span across multiple cloud regions or hybrid-cloud environments?