Katkat
Back

Design a bidirectional data sync dashboard

NVIDIASoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

This question assesses the ability to design a distributed system that synchronizes data between a central dashboard and multiple external third-party services in real-time.

Problem Statement

Design a bidirectional data synchronization platform that allows a central dashboard to keep its data in sync with multiple third-party SaaS integrations (e.g., Jira, Salesforce, GitHub). Users should be able to view and update data in the dashboard, with changes propagating to the third-party tools and vice versa.

Functional Requirements

  • Real-time Sync: Updates in the dashboard should reflect in the third-party tool, and external updates should reflect in the dashboard.
  • Conflict Resolution: Handle cases where the same entity is updated simultaneously in both systems.
  • Idempotency: Ensure that network retries do not result in duplicate actions.
  • Scalability: Support thousands of active integrations with varying throughput requirements.

Non-Functional Requirements

  • Reliability: Data consistency is critical; no updates should be lost during outages.
  • Latency: Aim for sub-second propagation where APIs allow.
  • Security: Securely store and manage authentication tokens for third-party services.

Expected Deliverables

  1. A high-level architecture diagram or component description.
  2. The mechanism used to detect and handle conflicts (e.g., Last-Write-Wins, version vectors).
  3. Handling of rate limits imposed by third-party APIs.
  4. Database schema or data model for tracking sync state and pending tasks.

Hints

Hint 1

Consider using a message queue or event bus to decouple the dashboard updates from the external API calls.

Hint 2

Think about how you would distinguish between a system-generated sync update and a user-triggered update to avoid infinite circular update loops.

Hint 3

Use a 'Sync State' table to track versions or timestamps of records across both systems to identify drift.

Follow-up questions

  • How would you handle a third-party service that does not provide webhooks and instead requires periodic polling?
  • If a sync task fails repeatedly due to a data validation error, how would you prevent it from blocking the queue for other users?
  • How would you implement a 'backfill' feature for a new user who connects an account with thousands of existing records?