Editor’s note: The following is a sponsored blog post from QA.
I’d like to frame a governance question “Who decides when AI is too dangerous?” as a security question and take you through some of the practicalities of where we are today.
We continue to develop AI systems with greater autonomy, with access to tools and very capable cybersecurity skills. At the same time, we’re trying to learn how those systems really behave when operating independently and the controls begin to fail.
As a practitioner, security professionals should be forgiven for reaching for trusted playbooks, attempting to reduce privilege, restrict access, isolate systems and monitor behavior – all in an attempt to limit the blast radius when it goes wrong.
I don’t believe we really understand the risk or required controls well enough; in my view it will take a sufficiently serious incident before this is treated differently.
The risk predictors of doom
There are now daily AI risk predictions making the headlines – from those risks that can be managed with good engineering and security best practice, to the other end of the spectrum, with the frontier AI development community, in particular, predicting catastrophic outcomes.
As cybersecurity practitioners, we shouldn’t choose between these two extremes, aside from the reality that this is a very fast-changing space, and rather we should attempt to find supporting evidence for these claims.
Not all evidence is necessarily dramatic. Models are becoming more capable of acting autonomously, using tools and interacting with external systems as part of a cybersecurity task. At the same time, researchers release evidence that they’re seeing behaviors they didn’t expect or explicitly authorize.
While I do have many concerns and worries about AI capability exceeding controls, for those of you who have read my earlier articles, that will ring true. I do not subscribe to the notion that we’re approaching human extinction. This is probably the innate optimist in my nature, but dismissing the whole discussion because some predications are scary and extreme isn’t unhelpful.
We can’t ignore the evidence
The UK’s AI Security Institute (AISI) continues to report numerous instances of AI agents taking unsanctioned actions during cyber evaluations. OpenAI has separately disclosed examples of models creating additional instructions for themselves, attempting to hide mistakes, searching for credentials, sharing files without permission and communicating through various novel ways that researchers had not foreseen.
None of this demonstrates that AI has become uncontrollable. It does prove that AI can become malevolent, and very capable systems can find ways of achieving objectives that their designers didn’t anticipate (see my article, Managing Malevolent AI Agents).
At the risk of repeating myself from previous articles, for security professionals, that will sound familiar. Our adversaries have always looked for the path the defender didn’t consider. The difference now is that we’re deliberately building systems capable of finding paths through complex environments themselves. The cybersecurity profession rarely waits for an attack technique to become catastrophic before taking action and collaborating to take the underlying weakness seriously.
What does ‘too dangerous’ actually mean?
If we saw dangerous activity, could we pause, reflect and/or stop? How about possess the ability to autonomously identify and exploit vulnerabilities? Circumvent the controls within a test bed or sandbox? Acquire credentials and deceive the operator to do what it takes, and if necessary, coordinate with another agent, replicate and resist intervention?
There is no simple answer, and there may never be. Risk in this context depends on the capability, access, autonomy, environment and consequences. A capability that’s benign inside an isolated research lab could become much more significant when it is connected to a production system, with credentials, code repositories or our critical infrastructure.
The frontier developers are beginning to define “Dangerous AI” for us. Hold back those cynical thoughts of IPO and stock valuations driven by fear. Consider capability classifications, preparedness frameworks and responsible scaling policies, as well as governments and independent evaluators, although we’ve already seen when those assessments don’t align and these bodies profoundly disagree. I discussed one of these and their novel index ranking system in this article.
Are the AI controls we have sensible?
The UK National Cyber Security Centre (NCSC) recommends principles that security practitioners will recognize and are worth bookmarking.
I would add that an AI agent shouldn’t receive more autonomy just because more autonomy makes it more useful. Access should be directly related to the task, with external connectivity restricted – is that too obvious to make sense? Sensitive actions should have stronger controls, so the potential blast radius is considered before the agent is allowed to operate.
Nothing over the top here: best practice and good proportionate security engineering. We mustn’t wait for renewed regulation or the perfect AI security framework to be released and ratified when there is so much we can already do now.
Why controls are not the same as assurance
A sandbox is a control; evidence that the sandbox continues to contain a capable agent is assurance. Human approval (in-the-loop) is a control, including evidence that the human genuinely understands what they’re approving, has enough time to intervene and cannot easily be manipulated or bypassed is assurance. We should be looking further than simply checking if the control exists and demanding evidence that it remains effective against the capability of the system we’re deploying.
An assurance assessment undertaken by an independent AI auditor, conducted against one generation of model, cannot automatically tell us how the next generation will behave. As security practitioners we have so much experience. I can remember asking for robust evidence that a control was actually in place, only to spend time proving the false paper trail that auditors can spot a mile off.
Who should provide the evidence?
Model developers inevitably understand their systems better than most external organizations. After all, they do have access to training information, external evaluations, telemetry and emerging capability evidence that independent researchers and auditors may never see. A good example of being blindsided is when the AISI wasn’t given access to the latest Anthropic model, without justification aside from elevating the mystique of the version itself. Perhaps that was the intention.
Here we start to see the unravelling of the beginning of the conflict of interest, where the developer is also responsible for deciding whether the evidence is sufficient.
OpenAI has now argues in their recent AI policy announcement for independent verification and mandatory national safety requirements for the small number of organizations developing the most capable frontier systems. They also acknowledged that frontier labs have also built their own rules for managing these frontier risks.
We must insist on having transparency, which is not the same as independent assurance. The Evaluator Forum published an open letter with over 100 respected AI expert signatories calling for genuine independent assessment of models and for external evaluators to be embedded directly inside frontier labs with much deeper access to models, systems, staff and the development processes.
They’re insisting on introducing conditions, genuine independence with deep access and the freedom to report findings with legal protection from any backlash. This will definitely remove the information disadvantage and even give them the chance of seeing and reporting an emerging risk during the build instead of just checking at the release gate.
Who should have the authority to say no?
This is an important question and where the issues become most difficult.
I’ve tried to break it down. Let’s say an embedded evaluator, independent I hope, with deep access identifies a problem, or material and unacceptable security risk. Security teams can recommend additional controls, and the model developer, in theory, can decide that a capability requires stronger safeguards.
Do they or should they be obliged to tell the board and inform the regulator? When do we view the findings as more than just advice? (Assuming there is the freedom to publish its findings, although the closer the frontier models get to the defense industrial base, the quicker the national security card will be played). Our regulators should and could set expectations, but who gets to say that a system should not be deployed, especially as these parties are highly likely to disagree?
Sound familiar? Every security professional who has ever challenged a risky production deployment will recognize the problem where the other side of the table would say that a specific security issue isn’t a product defect, so with CTO blessing, it’s still going live. Identifying the risk is one thing, having the authority to do something about it is another. I see and hear lots of superficial AI governance rhetoric. Accountability must be explicit and governance needs the authority to demand additional controls where risk cannot reasonably be accepted.
Do we need to resolve P(doom)?
For the record I’m not a fan of this end of the world probability term but couldn’t avoid mentioning it here. I think some of the debate around P(doom) risks are becoming a distraction. If someone believes there is a 10% probability of AI causing human extinction and somebody else believes it is effectively zero, or finding ourselves competing with a human-made silicon species, as a Microsoft exec noted, we’re unlikely to conclude the argument anytime soon.
Fortunately, security professionals make decisions under uncertainty all the time, not always with available evidence, full view of the consequence or the confidence that they have sufficient and effective controls in place.
So, if the people building these systems assign meaningful probabilities to catastrophic outcomes, those claims should at least influence the level of scrutiny we can expect. If only we could influence the administrations (that come and go) to arm the regulators with this distinction, perhaps even giving the AISI / CAISI statutory or federal regulator status.
Who owns the risk?
How do we avoid the risk which cannot fall through the virtual cracks between the developer of the model, the organization deploying the model, the security team, independent evaluator AI auditor, and/or regulator? Someone must own the decision.
We have a well-trodden path and established concept of risk acceptance in the field of cybersecurity, without having frontier AI risks silently imposed upon us.
For those deploying AI that should mean clear accountability for autonomy, access and acceptable use, with well-understood escalation paths when systems behave unexpectedly, as we know they will. Security, risk, governance and assurance functions, let alone the AI auditors conducting the evaluations, must have sufficient authority to challenge an AI deployment when the evidence doesn’t support the risks being taken.
If the consequences of getting the decision wrong reach through the supply chain (we’ve already seen evidence of this in the Hugging Face breach), beyond the organization deploying the model, should that company or the developers accept the risk on everyone’s behalf?
We can’t and shouldn’t design regulation around every technical AI implementation. Just think of the frontier AI innovation cycle speed pre-IPO and think about the need to satisfy shareholders. But we should establish clear expectations for safety, evidence and accountability, ultimately ensuring we have accountability when those expectations aren’t met. Apart from the obvious, why can’t this be part of the Form 1 SEC due diligence process?
The real governance question isn’t who owns the risk, but who is entitled to accept it?
Who gets to decide?
I don’t think we have a clear and useful answer yet. Security and assurance professionals must continue to raise the bar. Build the guardrails, restrict autonomy enforce least privilege, sandbox rigorously. Continue to monitor behavior, test assumptions, demand evidence and be brave enough to escalate (with or without the physiological safety to do so) when that evidence isn’t sufficient.
Are you seeing this too? It’s not a done deal; no one should be a passive bystander watching AI agents flexing frontier capability as the risks are imposed on us and the model developers whisper for more controls.
We mustn’t confuse having controls with having control. As AI systems become more capable and autonomous, it’s not going to be about safeguards – it will be if the evidence still justifies allowing the systems to operate at all.
And when that moment comes, evaluators need the independence, evidence and authority to say no. If we don’t establish who can make that decision before a serious incident, the first real threshold for “too dangerous” may end up being defined for us after something has already gone wrong. Ultimately this will come down to whether the person who recognizes the danger has the authority to do anything about it.
