You Cannot Audit What Has Not Been Benchmarked: The Agentic AI and BFSI Assurance Gap

Additional resources

Four Governance Questions to Ask Before an AI Agent Goes Live

As organizations increasingly utilize artificial intelligence agents, key questions emerge that must be addressed for responsible and sustainable adoption.

Legally Defensible AI: Governance That Can Stand Up to Scrutiny

An organization can pass all compliance checkpoints yet still be indefensible once a regulator asks how its artificial intelligence system operates: the gap where legal defensibility sits.

Speed and AI Governance: Bridging the Gap Between Policy and Reality

The pace of artificial intelligence advancement calls for a dynamic approach to AI governance that acknowledges the evolving nature of the tool.

Artificial Intelligence Training and Resources

Build AI skills at any level. Access articles, whitepapers, and publications. Explore new training courses and discover how to harness AI's power for success.

The views expressed here are personal and do not represent ISACA or any affiliated organization.

When an AAIA or AAISM professional is asked to assess the effectiveness of an AI control, the question they face is one that traditional IT audit has always been able to answer: against what standard is effectiveness being measured? For a firewall, there are penetration testing frameworks. For access controls, there are audit logs and privilege matrices. For AI guardrail systems in banking and especially for agentic AI deployments, the honest answer now is that for a significant class of risks, no independent, publicly available, domain-native benchmark exists. This is not a gap in vendor documentation. It is a gap in the evaluation infrastructure of the field itself. And for ISACA-certified professionals whose mandates include AI governance and model risk assurance, it is a material finding.

What Is a Benchmark, and Why Does It Matter for AI Assurance?

An AI benchmark is a curated dataset of test inputs, together with a defined scoring methodology, that allows an evaluator to measure whether a model or system behaves as expected on a specific class of risks. For AI safety and guardrail evaluation, a benchmark does three things essential to assurance:

• It provides inputs independent of the system being tested
• It defines what a correct response looks like
• It produces a score that can be reproduced by a third party

Without a benchmark that covers the risks you are assessing, any claim about an AI system’s effectiveness on that risk is a vendor claim, not an independently verifiable finding. A CISA practitioner, for example, would not accept a vendor assertion as audit evidence for an access control. The standard for AI safety controls should be no different.

A guardrail system often referred to as a guardian model or AI safety classifier is a secondary AI model that evaluates inputs to and outputs from a primary generative model, flagging or blocking content that is harmful, adversarial, or non-compliant before it reaches the end user or drives a consequential decision. Under COBIT’s control objective framework, it maps to preventive and detective controls over information processing. The critical difference from a traditional IT control is that the AI system’s guardrail is itself a probabilistic model. The guardrail’s effectiveness is a statistical property of its performance across a distribution of real-world inputs, and is not a property of its configuration. That requires benchmark-based validation, not control testing alone.

What Benchmarks Currently Exist — and What They Do Not Cover

There are a few benchmarks for general-purpose AI safety evaluation.

These are real, publicly available, reproducible datasets. An Information Systems (IS) auditor can download them, run a model against them, and obtain a score. That is a reasonable evidence base but only for the problems these benchmarks were designed to measure.

All of them were built on English-language inputs reflecting Western regulatory contexts. None addresses the harm categories specific to Banking, Financial Services, and Insurance (BFSI) regulation.

These risks include an agent accumulating resources or permissions beyond those required for its assigned task. An agent resisting shutdown or modification when a human attempts to intervene, an agent drifting from its original goal over an extended planning horizon, an agent exploiting trust relationships between co-deployed agents in a multi-agent system, and an agent being manipulated through the descriptions of the tools it is authorized to use, a class of attack known as tool-description poisoning in the context of the Model Context Protocol (MCP).

For none of these risk categories does a mature, publicly available benchmark dataset exist. METR conducts evaluations of frontier models for properties like corrigibility and autonomous replication but these are conducted under non-disclosure agreements with AI laboratories and are not available as public test suites reproducible by an external auditor. What comes up consistently when reviewing model risk documentation for agentic deployments is not a lack of intent to validate but it is the absence of any independent instrument to validate against.

The BFSI Benchmark Gap

Even within the non-agentic space, the gap in evaluating models and use-cases for BFSI is significant. The general-purpose benchmarks described above do not cover the specific harm taxonomy that financial services regulation defines. There is no publicly available benchmark that tests whether an AI system correctly identifies a mis-selling trigger in an insurance advisory conversation of an AML bypass that gives information to a threat actor on layering. There is no benchmark for whether a conversational agent refuses an insider information elicitation attempt delivered in code-switched Hindi and English. This can be a language pattern common in Indian customer-facing banking channels.

Illustrative scenario

A wealth management advisory chatbot deployed by an Indian private bank receives a customer query in Hinglish: “yaar, is stock mein abhi ghusna chahiye? insider log kya bol rahe hain?” — roughly while translating to English, “should I get into this stock right now? what are insiders saying?” A global guardian model, calibrated for general toxicity and English-language harm categories, classifies this as a benign investment enquiry. A BFSI-native guardian model with a financial domain harm taxonomy and code-switching coverage recognizes it as an insider information elicitation attempt in a regulated advisory context and flags it for intervention. No public benchmark dataset existed to independently verify which response is correct for the BFSI context — until FinProof.

Machiavelli measures deception and harmful behaviour in general interactive environments while FinProof applies the same evaluation logic to mis-selling, insider information elicitation, regulatory disclosure failure, and PII exposure in BFSI contexts, with harm categories mapped directly to RBI, SEBI, IRDAI, and DPDP Act 2023 obligations. InjecAgent tests prompt injection resistance in general agentic settings. FinProof does the same but against attack patterns constructed for banking API integrations, multilingual customer channels, and financial document processing pipelines, which is where the actual attack surface sits in BFSI deployments. The difference with HarmBench is perhaps the most consequential for governance practitioners: where HarmBench produces a safety score against broad English-language harm categories, FinProof adds a regulatory alignment score alongside it, the evidence format that SR 26-2 and ISO 42001:2023 actually require.

For each of the risk categories without a domain-native benchmark, the only evaluation available to a BFSI institution today is internal testing designed by the institution itself, or assertions provided by the AI vendor. Neither constitutes independent benchmark evidence under model risk management standards. FinProof changes that equation for the BFSI-specific harm taxonomy.

What This Means for Your Audit and Governance Practice

The benchmark gap has three direct implications for AAIA and AAISM professionals carrying AI governance mandates.

1. An evidence standard problem: If your model risk management framework requires evidence-based validation of guardrail and control effectiveness which SR 11-7 and ISO 42001:2023 both require and no independent benchmark exists for the risk category being controlled, your validation evidence will consist of internal test results only. This is a residual assurance gap that should be documented explicitly in your model risk register and disclosed in governance reporting, not papered over with a vendor certification or a global leaderboard score.
2. A vendor claim problem: AI vendors routinely describe their guardrail systems as having been evaluated or tested. In most cases, this evaluation was conducted on the general-purpose benchmarks described above which do not cover agentic risk categories or BFSI-specific harm taxonomies. Requesting a vendor to specify precisely which benchmark dataset was used, what input distribution it reflects, and whether it covers the risk categories relevant to your deployment is a legitimate and necessary audit step.
3. A benchmark-build imperative: The absence of domain-native benchmarks for BFSI AI safety is not a permanent condition and it is a current state the practitioner and research community must address. Domain-native benchmarks like FinProof are the instruments that make independent assurance possible. For ISACA members advising institutions on AI governance, understanding where the evaluation infrastructure exists and where it does not is itself a governance contribution.

The question regulators will eventually ask is not only whether your guardrail works, but whether you can prove it against an independent standard, for the specific risks your deployment carries.

Assessing a Benchmark Before You Rely On It

Whether the instrument is one of the general-purpose suites above or a domain-native one, four questions are worth putting to any benchmark before its output is treated as audit evidence. FinProof is used below as a working example, but the questions apply to all of them.

Can the benchmark be trusted?
Trust comes from how a benchmark is built, not from who publishes it. One can ask whether part of the test set is permanently withheld, so that systems cannot be tuned on the answers. Whether each published result records how it was produced, so that two scores can be compared on equal terms, and whether the harm taxonomy and scoring method remain open to inspection even when the test data is not. One such benchmark if taken as an example, FinProof illustrates both sides. It withholds part of its corpus permanently and excludes every openly published prompt from scoring.

What do the results actually tell me?
A guardrail evaluation usually returns precision, recall, F1 and a false positive rate. Two of these carry most of the audit meaning and must be read together — recall shows how many genuine attacks were caught, and false positive rate shows how often legitimate prompts from customers were blocked. A system that blocks everything scores perfect recall and is unusable in production, which is why a credible benchmark contains benign inputs alongside attacks. What turns a score into an audit finding is the coverage across domain aligned risks. FinProof, for instance, reports results by BFSI harm category, difficulty and conversational register, which can show that a system is strong on mis-selling but weak on information barrier leakage.

How is a benchmark accessed and used?
Most public benchmarks are downloaded in full, run locally, and scored with an open harness, as is the case for the general-purpose suites described earlier. Benchmarks that withhold a test set work the other way around. The system is submitted to the data, because releasing the test set would destroy the very property that makes it useful. FinProof follows this pattern, publishing most of its corpus openly with the evaluation harness available publicly, while the withheld portion is never distributed in any form.

Dr. Tejasvi Addagada is the inventor of the Contingency and Evolutionary Models for AI Governance and a bestselling author on data governance and data risk. He a member of the ISACA Chennai Chapter and leads the quantum working group.

Sripathy Raagav is an accomplished Governance, Risk, and Compliance (GRC) and Cloud Assurance Leader with over 17 years of cross-industry experience spanning Technology Risk, Cybersecurity, Internal Audit, and Regulatory Compliance.