Splunk survey: 62% of teams use AI agents to troubleshoot, but half cannot verify answers

AI agents have moved into the core of enterprise incident response: 62% of IT and engineering professionals now rely on them frequently to work out why a system broke. Splunk surveyed 2,050 of them across ten countries, and Help Net Security reported the findings.

The work agents get is narrow and consequential: finding root causes, tuning alerts and drafting runbooks, the step-by-step guides engineers follow during an incident. Writing new code sits at the bottom of the list, and about a quarter of respondents never hand it to an agent. A bad alert threshold takes a minute to roll back; bad code in production is a different problem.

Most teams say agents help them spot problems sooner, and many report faster recovery. But 14% say recovery actually got slower after agents arrived. Sahinaz Safari, who leads AI for Splunk's observability business, compares agents to new hires: enormous potential, plus a period of heavier oversight while they are integrated.

That oversight now has a name — the verification tax, the time engineers spend double-checking an agent's reasoning before trusting its answer. The gap shows up in the numbers: 85% say it is important to see when a model invents an answer, but only 58% say they can reliably catch it.

Tooling and ownership are the two holes. About half say their observability tools were not built to watch AI, and 38% say their organization has not decided whether a misbehaving agent belongs to the data science team, the SRE team or security. Splunk's Mark Woods put it this way: when agent observability is everyone's job, it becomes nobody's job.

A concrete failure shows why the ambiguity matters: an agent stuck in a loop hammering an internal API thousands of times a minute looks like a performance failure to engineering and like a denial-of-service attack to security. It is one incident, and 81% of respondents say monitoring AI has pushed their teams to work more closely with security.

Money is the other pressure point. In 57% of organizations, observability now costs more than the infrastructure it monitors, and half received a surprise telemetry bill this year. Splunk's Greg Leffler compares it to grocery shopping where the price of a tomato could jump from two dollars to ninety overnight, with the bill arriving at the end of the month.

Taken together, the survey points to a single conclusion: once agents move from pilots into production, the constraint is not model capability but observability, ownership and cost control. The next question is whether monitoring vendors make AI-specific coverage standard, and whether enterprises start assigning agents a clear owner and a budget line.