Katkat
Back

Design Reliable Command Delivery Between Autonomous Vehicles and the Cloud

WaymoSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

Design a resilient, secure command dispatch and execution system between a cloud fleet management platform and autonomous vehicles operating over unreliable cellular networks.

Scenario

Autonomous vehicle (AV) fleets rely on cloud-based remote operations for path suggestions, dispatch instructions, diagnostics, configuration rollouts, and safety pull-overs. However, vehicles operate in environments with intermittent cellular connectivity (e.g., tunnels, urban canyons, dead zones).

You are tasked with designing the command delivery and execution platform connecting the cloud control plane to the AV fleet. The system must guarantee that commands are safely received, validated, and executed in order, without duplicate execution or outdated actions taking effect.

Requirements

Functional Requirements

  • Targeting: Ability to route commands to a single vehicle, a geofenced area, or a dynamic cohort (e.g., software version canary).
  • Lifecycle & Acknowledgements: Track states explicitly: Created -> Delivered -> Safety-Validated -> Executing -> Completed / Rejected / Expired.
  • Ordering & Expiry: Support strictly ordered sequences per vehicle when required, alongside configurable command time-to-live (TTL).
  • Vehicle Safety Arbitration: The on-vehicle autonomous stack must locally validate and hold authority to reject commands that violate safety constraints.
  • Idempotency: Prevent duplicate executions caused by retries or intermittent network drops.

Non-Functional Requirements

  • Latency: Sub-200ms round-trip delivery for connected vehicles under normal network conditions.
  • Security: Mutual authentication (mTLS), tamper-proofing, replay prevention, and non-repudiation for all command payloads.
  • Reliability & Auditability: Every command issuance, state change, and rejection must be durably logged for safety and compliance audits.

Scale Assumptions

  • Fleet Size: 100,000 active vehicles.
  • Throughput: Peak generation of 10,000 commands/sec cloud-wide; average of 2-5 commands/minute per vehicle.
  • Payload Size: Typical command payload is 1 KB to 50 KB.

What the Answer Should Cover

  1. Architecture & Communication Protocol: Cloud ingress/egress, edge daemon on the vehicle, and the transport protocol choice (e.g., gRPC streaming, MQTT, WebSockets) balancing battery/data overhead and latency.
  2. Command State Machine & Storage: Cloud queue/mailbox design, vehicle-side storage, and the ACK protocol over flaky networks.
  3. Safety & Execution Layer: On-vehicle arbitration mechanism, replay prevention (nonces, timestamps), and idempotency keys.
  4. Fleet Rollouts & Blast-Radius Mitigation: Phased delivery mechanisms and circuit breakers for fleet-wide commands.

Hints

Hint 1

Consider whether a push, pull, or persistent bi-directional streaming approach best handles connection drops while minimizing latency when the vehicle is online.

Hint 2

Think through how the vehicle should handle an in-flight command whose TTL expires while the vehicle is executing it or waiting for safe execution conditions.

Hint 3

Decouple cloud-side command delivery from on-vehicle execution; the vehicle's onboard safety supervisor must have the final veto authority.

Follow-up questions

  • How do you prevent replay attacks if a malicious actor captures and replays a signed, high-priority command like 'pull over and stop' hours later?
  • If a broadcast command (e.g., geofenced emergency reroute) is sent to 20,000 vehicles at once, how do you prevent an ACK storm from overwhelming your cloud intake?
  • How should the system behave if a newer command supersedes an older queued command (e.g., new destination while vehicle was offline)?