Design a computer-use agent end-to-end

Problem Statement

Design an autonomous agent that can perform tasks on a desktop OS (e.g., 'Book a flight to Seattle on this website' or 'Draft a meeting summary from these emails') by observing the screen and controlling the mouse and keyboard.

Requirements

1. Perception: How will you represent the screen state for the model? Discuss the trade-offs between raw pixel inputs (screenshots) and structured data (DOM/Accessibility tree).
2. Decision Making: Propose an architecture for the agent. Should it be a purely reactive policy, or do you need a planning module? How do you handle long-horizon tasks?
3. Action Space: How do you map the agent's output (e.g., coordinates, text tokens, clicks) to OS-level inputs?
4. Robustness & Safety: How does the agent handle UI changes (e.g., responsive web design) or non-deterministic task environments? How do you implement guardrails to prevent the agent from performing unintended destructive actions (e.g., deleting files)?

Scale and Constraints

• The agent should handle screen resolutions up to 4K.
• Latency should be low enough for a fluid user experience (target < 500ms per inference step).
• The agent must be capable of multi-step reasoning (e.g., navigate to a site, fill a form, click submit).

Expected Output

Provide a high-level system architecture diagram (in text description), explain the training strategy (e.g., imitation learning vs. reinforcement learning), and describe how you would evaluate the agent's performance in a real-world setting.