Document Type

Report

Publication Date

5-6-2026

Abstract

Federal Artificial Intelligence (AI) initiatives have increasingly emphasized accelerating AI adoption across government operations while maintaining appropriate governance and oversight mechanisms (Office of Management and Budget [OMB], 2025; The White House, 2025). AI systems are increasingly being integrated into research and operational workflows to support the large-scale analysis of unstructured information. Across high-impact environments, AI-enabled systems are being used to accelerate document review, organize information repositories, extract structured indicators, and support operational and analytic decisionmaking. However, despite rapid adoption, many organizations still lack practical frameworks for determining when AI-generated outputs are sufficiently reliable to support operational use and when human oversight remains necessary. This report evaluates an AI-assisted analytic system designed to support structured coding and analysis of mass-casualty incidents using open-source information. Rather than focusing solely on predictive accuracy, this report evaluates AI performance through an operational reliability framework centered on consistency, stability, and alignment with expert human judgment. Specifically, we propose that AI-enabled analytic performance is systematically shaped by two factors:

1. Information Density: Whether relevant information is explicitly and consistently available across input materials.

2. Construct Transparency: Whether variables reflect concrete observable attributes or require interpretive judgment.

To evaluate this framework, we applied an AI-assisted coding system to a structured analytic framework consisting of 56 variables across 15 mass-casualty incidents using multiple open source documents per case.

Comments

PDF passed Adobe accessibility checker prior to upload. 

Download

Files over 3MB may be slow to open. For best results, right-click and select "save as..."

Share

COinS