Extract W-2 PDF Data and Evaluate It Against Reference JSON

Problem Statement

You are building an evaluation pipeline for an AI system that extracts tax information from W-2 PDF forms. You are given a list of extracted_records (a list of dictionaries produced by the AI) and a reference_data dictionary containing the ground-truth values.

Task

Implement a function evaluate_extraction(extracted_records, reference_data) that performs the following:

1. Record Matching: Match the extracted W-2 records to the reference data based on a unique identifier (e.g., employer_ein). If multiple records exist for an EIN, merge them or identify the best match.
2. Normalization: Convert currency strings (e.g., "$1,200.50") to float, and standardize keys to lowercase.
3. Error Calculation: Calculate the precision and recall of fields found versus expected. Report fields that are missing or mismatched.

Function Signature

def evaluate_extraction(extracted_records: list[dict], reference_data: dict) -> dict:
    # Returns a report of matching accuracy

Constraints

• extracted_records: 1 to 50 entries.
• reference_data: Contains key-value pairs representing the ground truth.
• Handle potential None or missing keys in the extracted records gracefully.

Example

Input:

extracted = [{'ein': '12-3456789', 'wages': '$50,000.00'}]
reference = {'12-3456789': {'wages': 50000.00, 'tax': 5000.00}}

Output:

{
    'match_rate': 1.0,
    'errors': {'12-3456789': {'missing_fields': ['tax'], 'mismatched_values': {}}}
}

Edge Cases

• The extracted data has a different key name (e.g., 'tax_withheld' vs 'tax').
• The extracted record includes fields not present in the reference schema (potential leakage).