Insights
Failure Taxonomies for Agent Workflows
How to group agent failures by trigger, owner, severity, and release impact so evaluation results lead to engineering action.
A taxonomy should assign ownership
A failure category is only useful if someone can act on it. Group failures by where the fix likely lives: retrieval, tool schema, state handling, policy, escalation, prompt, model behavior, or product requirements.
A label such as 'bad answer' is too broad. A label such as 'tool call missing required account scope' points to a fix.
Record the trigger and release impact
For each replay row, capture the trigger, expected behavior, observed behavior, severity, likely owner, and whether the issue blocks release.
This prevents high-volume low-risk issues from hiding low-volume failures that should block rollout.
Keep the taxonomy small enough to use
The first version should be practical, not exhaustive. A team can usually act on a compact set of recurring failure modes faster than a long academic list.
When a category keeps collecting unrelated failures, split it only after the split changes the engineering response.