Katkat
Back

Walk Through an Agentic PDF Evaluation Codebase

AppleMachine Learning Engineer · Technical Screen
HardMachine Learning EngineerTechnical ScreenSoftware Engineering Fundamentals

Last updated

This question tests your ability to navigate, critique, and extend a modular codebase designed for evaluating agentic PDF processing pipelines, focusing on evaluation metrics and robust system architecture.

Context

You are presented with a codebase that evaluates an agentic system tasked with extracting structured data from unstructured PDF invoices. The system uses an LLM-based agent that iteratively calls tools (e.g., extract_text, search_table, verify_total) to populate a JSON schema.

The current evaluation suite consists of:

  1. A GroundTruthLoader: Loads PDFs and JSON ground truth.
  2. An AgentRunner: Orchestrates the LLM calls.
  3. An Evaluator: Compares the agent's output against the ground truth using standard metrics (F1-score, Edit Distance).

The Task

Analyze the current architecture and answer the following:

  1. The current Evaluator treats all fields in the JSON schema with equal weight. How would you modify the system to handle hierarchical or nested fields where a missing parent object should penalize the score more heavily than a missing leaf attribute?
  2. Agentic systems often produce non-deterministic output traces. How would you design a validation layer to ensure the agent’s intermediate tool calls are 'safe' and 'logical' before final output comparison?
  3. Propose a mechanism to identify if the agent is hallucinating data by utilizing the 'trace' logs generated during the execution loop.

Constraints

  • Assume the system processes 1,000 PDFs per batch.
  • The codebase is Python-based using pydantic for schema validation and langgraph or similar frameworks for state management.
  • Your answer should focus on modularity and testability.

Hints

Hint 1

Consider using a weighted tree traversal algorithm to assign importance scores to nodes in the JSON schema.

Hint 2

Think about implementing a state-machine or a 'guardrail' module that validates tool arguments against a predefined policy before execution.

Hint 3

For hallucination detection, leverage cross-referencing: verify if the cited text in the trace log actually exists in the raw PDF context provided to the model.

Follow-up questions

  • How would you handle cases where the ground truth itself might be ambiguous or contain OCR errors?
  • If the evaluation process takes 30 seconds per PDF, how would you re-architect the `Evaluator` to scale to 100,000 PDFs?
  • How do you distinguish between an agent failing due to 'reasoning error' versus a 'tool capability error'?