Skip to content
Turing College Logo
Submit projectSign In

Multi-Agent Failure Triage for LLM Applications

Multi-Agent Failure Triage for LLM Applications

About this project

LLM applications rarely fail loudly. A retrieval returns nothing, a tool reports an error to the model, an upstream model times out, and the user just gets a worse answer. Finding these failures usually means reading traces one by one. TraceScan automates that triage.

The target application sends its OpenTelemetry traces to TraceScan’s own collector, which stores them in Postgres. A LangGraph pipeline then works through them:

  1. Judge: an LLM scores each trace in parallel and blames the span where it went wrong.

  2. Cluster: the failures are embedded and grouped with HDBSCAN, so failures with the same cause land together.

  3. Diagnose: a model diagnoses each cluster against a failure taxonomy. Every diagnosis must cite real spans from the traces, and the code checks those citations. The diagnoser also sees healthy calls of the same step and causes that reviewers confirmed earlier.

  4. Review: the pipeline pauses for a human. The reviewer approves or rejects each drafted GitHub issue and regression case. Nothing is written without that approval.

Created byMattia Smacchia
Published atNovember 16, 2026
CourseAI engineering
Looking for
New job