Editor’s note: The following is a sponsored blog post from QA.
In the interconnected financial services world, artificial intelligence (AI) isn’t just a tool for efficiency: it’s at the heart of innovation and growth. While AI enhances fraud detection, decision-making, and customer experience, it also creates new vulnerabilities.
Adversarial AI, the manipulation of AI systems to behave in unintended or harmful ways, is an emerging hidden threat – and one that the banking system is not ready for. To stay ahead of this fast-evolving problem, we should update regulatory frameworks in the financial sector, adding protections against adversarial AI.
Adversarial AI
It’s no secret the world’s multinational financial services (FS) organizations are using AI, allowing large FS institutions to innovate at the pace of a FinTech start-up.
But they are still not up to speed when it comes to new security risks this will create. Chief among them is “adversarial AI.”
Adversarial AI is not just about breaking into systems – what you would call the traditional “hack.” (Of course, this is a problem, too, but most existing IT security best practices should cover that). Adversarial AI is something more subtle: manipulating a company’s AI algorithms, or the data that feeds it, in order to influence its outputs.
This is a growing problem for any organization that relies on AI outputs. In the world of financial services – banks, insurance companies, FinTechs – it is potentially catastrophic. By my reckoning, there are no fewer than five different threats for FS from adversarial AI.
AI hallucinations
One of the most interesting challenges of large language models (LLMs) is their ability to “hallucinate.” This is when they create outputs that look and sound credible but are factually incorrect. In finance, this isn’t just inconvenient; it’s dangerous for our economy.
Let me give an example. Imagine an AI-driven market analysis tool that advises on high-stakes investments. When prompted to assess a specific company, AI confidently generates optimistic earning projections. But if those outputs are based on hallucinations that were intentionally created by a bad actor, those projections could contradict verified regulatory filings. This misstep could lead to substantial losses and damaged trust.
Data poisoning
AI is only as good as the data it learns from. Poison the data, and you poison the outputs, too. Attackers could target publicly accessible or poorly monitored datasets, injecting carefully designed adversarial samples that skew model behavior over time.
It's not hard to imagine how this could look with FS. They increasingly rely on large AI models to detect fraud – having trained these models with millions of examples of both legitimate and fraudulent transactions. A malicious actor hoping to bypass these systems might seek to inject poisoned data during the model’s retraining cycle, convincing it that fraudulent transactions are legitimate.
With no provenance tracking built into the model, or real-time verification of training datasets, the AI-enabled fraud detection service is at risk.
Model theft
AI models are high value intellectual property: crown jewels to protect from theft. However, a technique known as “model extraction” has been used to essentially reverse engineer AI models. This is where an attacker repeatedly queries a model – and uses the outputs it receives to re-create its own version. Chaining these model extraction techniques, adversaries can replicate proprietary models, which could strip businesses of their commercial competitive edge.
It’s not uncommon to see API architecture vulnerabilities exploited in complex technical supply chains. Exposing an insecure API of an advanced trading algorithm by systematically querying the system, attackers could begin to reverse engineer its behavior, replicating its strategic insight to mirror the architecture.
Ethical bias
Ethical bias in AI is nothing new. I’ve written and spoken about the ethical use of autonomous AI agents at the ISACA 2025 virtual conference, which has many parallels in this sector. AI outputs are driven by the data they are fed– and that can quite easily lead to unfair, unethical, or even immoral outputs.
In the FS world, AI outputs could lead to unfair outcomes in approvals, credit scoring, or fraud detection. Imagine a credit scoring AI agent multi-tasking with other AI agents in the flow of customer service, consistently flagging applicants from a specific demographic group as high-risk, despite identical financial profiles to other applicants.
AI supply chain attack
Not all FS institutions – even the big ones – have the resources and expertise to design their own in-house AI models. Much of the modern AI infrastructure ecosystem depends on third-party libraries, and what’s called pre-trained models. But these can be hijacked, as bad actors replace legitimate packages with malicious versions, or insert dormant code set to remotely activate under specific conditions.
This type of attack is becoming more common: Kaspersky uncovered a year-long attack involving masked python interfaces for ChatGPT and Claude AI, which looked like real chatbots but were secretly carrying malware that then infected company devices.
Improving adversarial AI resilience
Adversarial AI isn’t a future threat: it’s here now. As financial institutions push further into AI-driven innovation, understanding and mitigating these risks will become ever more important.
I advise anyone dealing with AI security to build a multi-disciplinary AI Red Team to constantly test your own exposure to adversarial AI threats. Break the silo mentality, be vendor agnostic, and create a multi-disciplinary team, including security testers, data scientists, governance professionals, engineers, and architects.
There are a few AI-specific approaches we know work. For example, skilled synthetic adversarial threat modelling – essentially mimicking an attacker to help work out your own weaknesses to improve resilience, by exposing models to an adversarial AI attack. And greater emphasis is needed to verify data quality going into training the models.
It’s time to modernize security frameworks and regulations to reflect this new type of threat and include adversarial AI attacks – which really matters because this is a systemic threat that transcends any single firm. I fully expect – and support – the continued widespread adoption of AI in our financial services globally. But for that to work, our adoption of best practice AI security and governance must evolve at the same rate.
