In a growing number of enterprise deployments, humans are no longer the sole decision makers. Guidance from ISACA® on AI risk and governance frames this change as a governance problem rather than a purely technical one. When IT auditors evaluate automated systems, the control objective is typically clear: Verify that the system executes its defined logic correctly, access to the system is appropriately restricted, changes go through a formal management process, and that outputs are logged. These procedures assume that the system's logic is deterministic and that human decision-making authority sits above the automation layer.
Agentic artificial intelligence (AI) systems, however, break several of these assumptions simultaneously. An AI agent does not execute fixed logic; it reasons through a task using a language model and selects its own sequence of steps. The agent’s outputs are probabilistic, not deterministic. Its scope of action can extend across multiple connected systems.
Auditing these systems with frameworks designed for deterministic automation will produce audit reports that appear complete while lacking the specific failure modes that agentic AI introduces. Conventional IT audit approaches fall short and additional measures, such as boundary testing and third-party review, are needed if organizations want to proactively address this growing threat.
Where Conventional IT Audit Falls Short
There are 4 gaps where conventional IT audit assumptions fall short once a system can reason for itself: How its logic is documented, how its outputs are sampled, how its scope of action is bounded, and how changes to it are governed. Each gap identifies where the conventional procedure stops working and why it fails.
- The logic documentation gap—For an agentic AI system, the relevant logic is partly encoded in the large language model's (LLM) weights, which the organization typically does not own or control, and partly in the system prompt that instructs the agent. System prompts are often managed informally, stored in code repositories without version control discipline, and changed by engineers without formal change management approval. Auditors who accept standard software documentation as evidence of what an agentic system does will have an incomplete picture and thus an incomplete audit. Under ISACA's IT Audit Framework (ITAF), sufficiency and reliability of evidence remain the auditor's responsibility, which means prompt artifacts and model configuration must be obtained directly rather than inferred from system documentation.1
- The determinism assumption gap—When auditing a payment processing system, an auditor can select a sample of transactions and verify that each was processed according to defined rules. The same input should always produce the same output. An agentic AI system processing the same input on 2 separate occasions can produce different outputs. The documented failure modes of LLMs, including hallucination under distribution shift, mean that output variance is a property of the system rather than an anomaly that sampling can isolate.2 Sampling-based testing that evaluates whether individual transactions were processed correctly does not detect whether the system's general behavior is within acceptable bounds. The relevant question for an agentic system then becomes: Does the system produce acceptable outputs with sufficient reliability across the range of inputs it encounters?
- The scope-of-action gap—Access reviews for human users examines what permissions a user has and whether those permissions are appropriate for their role. For AI agents, the same review process applies in principle but requires additional scope. Auditors must evaluate not just what data and systems the agent can access, but what actions it can take across those systems. An agent that can read a customer record, update a contract, and trigger an approval workflow has a combined action scope that represents a specific risk profile. Standard access reviews that check read and write permissions at the system level will not capture the business impact of that combined scope.
- The change management gap—When the underlying language model is updated by a third-party provider, the agent's behavior can change without any organizational change management process being triggered. A model update from an AI vendor is not a change to an internally managed system—it comes through an application programming interface (API). That update can materially alter how the agent interprets instructions, how it handles edge cases, and which outputs it produces. Conventional change management controls do not address this exposure.
Audit Procedures for Agentic AI Systems
There are several procedures that extend conventional IT audit methodology to address agentic AI-specific risk. These procedures are intended to complement, not replace, standard controls testing:
- System prompt documentation review—Auditors should obtain and review the system prompt for each production AI agent, verifying that it exists in version control, that changes to it follow a formal approval process, and the prompt itself aligns with the agent's documented purpose and authority scope. Discrepancies between the documented agent role and the actual system prompt instructions are a relevant audit finding.
- Authority boundary testing—Auditors should design test cases that probe the boundaries of the agent's authority, including inputs that would require the agent to take an action outside its defined scope, inputs that involve sensitive data categories not in the agent's intended data diet, and inputs that simulate edge cases that the agent was not designed to handle. The harm categories documented in the language model risk literature, including unauthorized disclosure and unsafe delegated action, are a practical starting point for designing these cases.3 The agent's response to these boundary cases should be logged and evaluated against the organization's defined acceptable behavior.
- Output distribution analysis—Rather than sampling individual transactions, auditors should request aggregate output data for a defined period of time and analyze the distribution of outcomes. For a quoting agent, this involves reviewing the distribution of pricing exceptions approved, discount levels granted, and contract terms modified over time. Significant shifts in these distributions without corresponding shifts in business inputs are a signal that the agent's behavior has changed. Auditors should document the baseline distribution and establish what deviation would constitute a reportable anomaly.
- Third-party model update log review—Auditors should request records of all model updates received from AI providers during the audit period, confirm that the organization received advance notice for material updates, and verify that regression testing was performed following updates that affected production agents.
- Escalation and override testing—Well-designed agentic systems include conditions under which the agent stops and escalates to a human rather than proceeding autonomously. Auditors should verify that these escalation conditions are documented, test whether the agent escalates when those conditions are met, and review logs for instances where escalation should have occurred but did not.
The Evidence Challenge
A common challenge for agentic AI audits is ensuring reliable audit evidence. Log data from AI agents can potentially mix tool call records, model outputs, and user interactions in formats that conventional audit evidence management tools typically cannot handle. Auditors should establish at the outset what log data is available, what retention period applies, and whether logs are stored in a tamper-evident system. The absence of adequate logging is itself a finding. The NIST AI RMF identifies explainability and traceability as characteristics of trustworthy AI systems and calls for documentation sufficient to support them. Furthermore, comparable transparency and record-keeping expectations are appearing in AI regulation and management system standards.4 The EU AI Act requires high-risk AI systems to technically allow the automatic recording of events over the lifetime of a system and obliges providers and deployers to retain those logs for at least 6 months.5 The International Organization for Standardization (ISO) and International Electrotechnical Commission (IEC) ISO/IEC 42001 imposes a parallel expectation through documented operational controls and retained documented information.6 For an agentic system, the practical implication is that log scope, retention period, and tamper-evidence should be assessed against whichever of these applies to the deployment, rather than against an internal logging standard alone.
Conclusion
Auditing agentic AI requires extending traditional IT audit frameworks rather than replacing them. The core principles, including control existence, design effectiveness, and operating effectiveness, still apply. What changes is how auditors test for each. The nondeterministic nature of these systems requires distribution analysis rather than transaction sampling; delegated authority requires scope testing rather than access reviews alone; third-party model updates require a new category of change management review. Auditors who develop proficiency in these procedures now will be well positioned in the future as agentic AI deployments become a standard feature of enterprise technology environments.
Endnotes
1 ISACA, IT Audit Framework (ITAF™️): A Professional Practices Framework for IT Audit, 5th Edition, USA, 2026
2 Huang, L.; Yu, W.; et al.; "A Survey of Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions," arXiv, 2024
3 Weidinger, L.; Mellor, J.; et al.; "Ethical and Social Risks of Harm from Language Models," arXiv, 2021, arXiv:2112.04359
4 National Institute of Standards and Technology (NIST), Artificial Intelligence Risk Management Framework (AI RMF 1.0), USA, January 2023
5 European Parliament, “EU AI Act: First Regulation on Artificial Intelligence,” 14 June 2023
6 International Organization for Standardization (ISO) and International Electrotechnical Commission (IEC), ISO/IEC 42001:2023, Information Technology—Artificial Intelligence—Management System, 2023
Eshaan Jain
Is a lead product owner for Salesforce and Vlocity CPQ at T-Mobile (via Mphasis). With 15 years of experience across enterprise AI, applied machine learning, and agentic system design at organizations including Amazon, PwC, and Accenture. Jain co-built an machine learning (ML) clause extraction system that reached 95% accuracy across a US$40 billion annual contract portfolio in last-mile supply chain logistics. He has published peer-reviewed research in IEEE and Elsevier with 55+ citations and is a Forbes Tech Council member and IEEE Senior Member.