Failure Taxonomies for Agent Workflows

A failure category is only useful if someone can act on it. Group failures by where the fix likely lives: retrieval, tool schema, state handling, policy, escalation, prompt, model behavior, or product requirements.

A label such as 'bad answer' is too broad. A label such as 'tool call missing required account scope' points to a fix.

For each replay row, capture the trigger, expected behavior, observed behavior, severity, likely owner, and whether the issue blocks release.

This prevents high-volume low-risk issues from hiding low-volume failures that should block rollout.

The first version should be practical, not exhaustive. A team can usually act on a compact set of recurring failure modes faster than a long academic list.

When a category keeps collecting unrelated failures, split it only after the split changes the engineering response.