Static DAGs are great until the source schema changes, the upstream API rate-limits, or a column rename breaks 30 downstream models. Agentic pipelines let the orchestration adapt instead of paging the on-call. Here is what that actually looks like.
The shape of an agentic pipeline
Three changes from classical DAGs:
- Tasks emit signals, not just success/failure.
- An agent monitors signals and decides reactions (retry, repair, reroute, escalate).
- The agent can rewrite the next step, within bounded authority.
Classical DAGs say what to do. Agentic pipelines describe what should happen and let the agent figure out how.
Where it wins
Five concrete use cases:
Schema drift adaptation
Source adds a column. Static pipeline ignores; agentic pipeline detects, decides whether to map it, alerts the owner if uncertain.
Self-healing on transient failure
The third-party API is slow. Agent backs off, retries, swaps to a fallback if available. Less paging.
Data quality intervention
A nightly load comes in with 10x the usual nulls. Static: passes through. Agentic: pauses, investigates, alerts with diagnosis.
Test-driven repair
A dbt test fails. The agent looks at the failing rows, hypothesises a cause, proposes a fix, opens a PR. The data engineer reviews.
Capacity routing
A backlog forms. The agent shifts work between regions or scales workers within budget. Within bounds.
Where it loses
Three counter-indications:
- Strictly compliant pipelines — change control mandates that no automated mutation happens.
- Sub-second latency requirements — adding LLM calls adds seconds.
- Pipelines where wrong data is more dangerous than no data — payments, regulated reporting.
Apply agentic patterns to operational data; keep critical financial pipelines deterministic.
Architecture
Three layers wrapping the existing orchestrator:
Airflow / Dagster / Prefect (existing)
↑
signal bus (events from tasks)
↑
monitoring agent (reads signals, makes decisions)
↑
action surface (retries, mutations, escalations)
↑
human-in-the-loop for high-blast-radius actions
You do not replace the orchestrator. You add a brain on top.
The bounded authority pattern
The agent must not be allowed to do everything. A working scope:
agent: data-pipeline-bot
allowed_actions:
- retry_task: max_per_pipeline_day: 3
- scale_workers: max_workers: 50
- route_to_fallback: if_fallback_in_allowlist
- open_pr_for_fix: never_auto_merge
forbidden_actions:
- mutate_production_schema
- alter_downstream_models
- skip_dq_tests
escalation:
- on_data_loss_signal: page_oncall
- on_authority_breach: hard_stop_and_alert
Without bounded authority, the agent becomes a new failure source.
Data quality reasoning
The agent's day job is reading dq signals and reasoning. A working prompt template:
Recent task: nightly_orders_load
Status: succeeded
Anomaly signals:
- null_rate(customer_id): 12% (typical 0.3%)
- row_count: 84,200 (typical 86k-88k)
- upstream_lag: 15 minutes (typical 2 minutes)
Decide: continue, pause, or escalate?
Provide: reasoning, suggested next steps, owners to notify.
The agent then reasons over the signals and proposes an action. Human reviews high-stakes recommendations; low-stakes execute automatically within bounds.
Cost reality
Adding agentic monitoring to a typical mid-size data stack:
- 2–10 LLM calls per task (signal interpretation).
- 1 LLM call per anomaly investigation.
- Monthly cost: hundreds to low thousands of dollars for typical orgs.
Compare to: hours of on-call time saved, fewer broken-downstream incidents.
Audit and observability
Every agent decision logs:
- Signals it saw.
- Reasoning chain.
- Action taken.
- Outcome (whether the action helped).
Feeds the audit trail. Quarterly review of agent decisions catches drift.
Common mistakes
- No bounded authority — agent runs out of guard rails into production damage.
- Same agent for ops and data quality — split roles. See role specialization.
- No human escalation path — the moment the agent is uncertain, who do they ping?
- Over-trusting the agent's reasoning — verify against ground truth on a regular sample.
Where this is heading
Three trends by 2027: native agent hooks in Airflow, Dagster, and Prefect, dq-vendor agent integrations (Monte Carlo, Anomalo), and pipeline-aware MCP servers. Build the bounded-authority pattern now.