Wiring an LLM Agent Into Airflow to Triage Failed Data-Quality Tests

A failing data-quality test tells you almost nothing on its own. not_null_orders_customer_id failed. Okay. Is that a genuine regression in the source system, a schema change upstream that nobody flagged, a test that was always too strict and finally caught an edge case, or a backfill that legitimately has nulls for old records? A human answers that by opening five tabs: the failing rows, the recent commits to the model, the upstream freshness, and the last time this test flaked. That triage takes fifteen minutes and it is the same fifteen minutes every time.

This is a good fit for an LLM agent, precisely because it is a context-gathering and classification task rather than a decision with consequences. The agent reads what a human would read, forms a hypothesis, and routes the alert to the right person with that hypothesis attached. It does not fix anything. That boundary is what makes it safe to run unattended.

Where it hooks into the DAG

The natural insertion point is failure, not success. In Airflow you have two clean options. The first is an on_failure_callback on your dbt test or Great Expectations task, which fires the triage logic whenever that task fails. The second, which I prefer for anything beyond a single test, is a dedicated triage task wired downstream with a trigger rule of all_done, so it runs after the test task regardless of outcome and inspects what failed.

The dedicated-task approach keeps the triage logic testable in isolation and gives you a normal task instance with logs, retries, and a place to see the agent's output in the UI. The callback approach is lighter if you only have one gate. Either way, the agent runs inside your orchestration, with your warehouse credentials scoped to read-only, and its output is an alert, not a mutation.

Give the agent the tabs a human would open

An agent is only as good as the context it can pull, so the engineering work is mostly in the tools you expose. I keep the tool set small, read-only, and specific:

• Fetch a sample of failing rows. For a dbt test, the compiled SQL is in the run results; run it with a LIMIT and hand the agent a handful of rows. For Great Expectations, the validation result already contains unexpected_values and counts. This is the single most informative input.
• Read the test definition and recent history. What the test asserts, and how often it has failed in the last thirty days. A test failing for the first time in a year is a different story than one that flakes weekly.
• Check upstream freshness and volume. If the source table did not load, the failure is upstream and the null column is a symptom. A row count that dropped by half is a louder signal than any individual test.
• Look at recent changes. The last few commits touching the model or its sources, from the dbt manifest and git. A test that broke right after a model change points at the change.

Each of these is a plain Python function the agent can call. Keep them read-only at the credential level, not just by convention. The database role the triage task uses should have no write grants at all. That way a prompt injection in some failing row of text data cannot turn into a DELETE, because the permission simply is not there.

Ask for a classification, not a paragraph

The output that makes this useful is structured. I have the agent return a fixed schema so the result can drive routing logic, not just read nicely in Slack.

{
  "category": "source_data_issue | test_too_strict | real_regression | upstream_failure | transient",
  "confidence": 0.0-1.0,
  "summary": "one sentence a human can act on",
  "evidence": ["failing rows all have order_ts before 2024", "..."],
  "suggested_owner": "team or person",
  "suggested_action": "what to check first"
}

The evidence array is the part I care about most. It forces the agent to ground its classification in the specific rows and facts it retrieved rather than pattern-matching on the test name. When I review triage output, I read the evidence first and the category second. If the evidence is thin, I distrust the category regardless of the stated confidence.

Routing then becomes deterministic code on top of the structured output. High-confidence upstream_failure pages the platform team and links the freshness check. test_too_strict opens a low-priority ticket to the model owner with the failing edge cases attached. real_regression at high confidence escalates. Anything below your confidence threshold goes to a general channel labeled clearly as an unverified guess.

The failure modes you have to design against

An agent that triages incidents can create new incidents if you are careless. Three things matter.

It will sometimes be confidently wrong. The mitigation is that being wrong is cheap here. The worst case is a misrouted alert that a human re-routes, which is strictly better than the current worst case of a fifteen-minute manual triage on every failure. Never let the classification suppress an alert entirely. The agent adds context to a notification; it does not decide whether to notify.

Alert fatigue can get worse, not better. If the agent posts a wall of reasoning for every flaky test, people stop reading. Keep the Slack message to the one-sentence summary, the category, and the suggested first check, with the full evidence behind a thread or a link. The goal is that a responder reads one line and knows where to go.

Cost is per failure, so it scales with your worst days. On a healthy pipeline this is a handful of calls a day and trivial. On the day a schema change breaks forty tests at once, you do not want forty independent agent runs each pulling context and reasoning separately. Batch related failures, especially ones sharing an upstream source, into a single triage call. One good hypothesis about the root cause beats forty correct-but-redundant ones about the symptoms.

What this actually buys you

The honest framing is that this does not improve your data quality at all. Your tests already caught the problem; that is why they failed. What it improves is time-to-diagnosis and the routing of human attention. The engineer who gets paged arrives with a hypothesis and the relevant rows already in hand instead of a bare test name and a cold start.

Keep the boundary sharp and it stays safe: the agent reads, classifies, and routes, always with read-only access, and a human still owns every fix. Give it the same context a good on-call engineer would gather, make it show its evidence, and let deterministic code turn its classification into routing. Done that way, a failed data-quality test stops being the start of a scavenger hunt and becomes a well-annotated ticket that lands on the right desk.