Katkat
Back

Design an Entitlement-Aware Agentic Portfolio Workflow

J.P. MorganMachine Learning Engineer · Technical Screen
MediumMachine Learning EngineerTechnical ScreenSystem Design

Last updated

This problem tests the ability to architect an LLM-based agentic system that balances autonomous task execution with strict role-based access control (RBAC) and auditability.

Scenario

You are designing a portfolio management agent for internal investment analysts at a financial institution. The agent receives natural language requests (e.g., 'Compare the risk-adjusted returns of Tech holdings against the S&P 500') and executes tools to fetch data or perform analysis.

Requirements

  1. Entitlement-Awareness: The system must ensure the LLM never accesses data for accounts or instruments the specific user is not authorized to see. The authorization check must happen outside the model's logic.
  2. Orchestration: Define how the system manages tool discovery, state, and decision loops.
  3. Auditability: Every step, including the model's reasoning and the raw tool output, must be logged for regulatory compliance.
  4. Safety: Differentiate between 'Read-only' queries (market data) and 'Consequential Actions' (e.g., rebalancing a portfolio).

Design Constraints

  • Assume 500 active analysts making ~5,000 requests per day.
  • The system must be durable: if a process crashes mid-step, it should be able to resume without repeating external API calls.
  • Latency should be under 5 seconds for read-only queries.

Deliverables

  • Describe the high-level system architecture.
  • Explain how you enforce authorization boundaries without exposing PII to the model.
  • Define the interaction between the LLM and the tool-executor regarding 'Consequential Actions'.
  • Describe the data model for audit logging.

Hints

Hint 1

Consider implementing a middleware 'Policy Enforcement Point' (PEP) that intercepts tool outputs based on the user's IAM token before the model processes them.

Hint 2

Use a state machine or workflow engine (e.g., temporal/state-based orchestration) to handle multi-step reasoning chains for failure recovery.

Hint 3

For consequential actions, implement a human-in-the-loop (HITL) gate that requires a cryptographically signed approval before execution.

Follow-up questions

  • How would you handle a scenario where the model hallucinates an authorization check or attempts to circumvent the tool-calling schema?
  • If the portfolio data source is latency-sensitive, how do you incorporate caching without violating data entitlement rules?
  • How do you measure the 'correctness' of an agent's workflow beyond simple response accuracy?