I once saw a crisis start with a harmless-looking file. A model artifact. Someone pasted it into a pipeline as if it were a slide deck. No questions. No checks. Two weeks later, a customer support assistant began “helping” by leaking account details to anyone who sounded convincing. The outage was short. The trust damage stayed.
That’s the lesson. When models run production decisions, integrity becomes continuity. If you can’t trust the model, you can’t trust the service. If you can’t trust the service, you don’t have a process. You have a bet.
Boards hear “model” and think math. CISOs hear “model” and think new attack surface. Both are right. But the bigger shift is operational. Models behave like software, with a longer supply chain and stranger ways to fail. Treat them like production software or they will treat your business like a toy.
Discover. Know what you are actually running
Your first problem is amnesia, not an attacker.
Ask one blunt question. How many models are running in your company right now? Not pilots. Running and touching customers, money, safety or regulated data.
Most organizations can’t answer without calling three teams and opening five spreadsheets. That is already a continuity risk. If something goes wrong and you can’t even find the thing, you can’t contain it.
Start with an inventory that reads as an ops register. Model name. Owner. Business use. Where it runs. What data does it touch? What systems can it call? What version is live? Who can change it? Who can roll it back?
Then widen what counts as “the model.” It’s not only weight. It’s prompts, retrieval indexes, embeddings, feature pipelines, evaluation sets, fine-tuning data and the wrapper code. If it can change the output, it belongs in your chain of trust.
Diagnose. Map how integrity fails
Once you know what exists, ask how it can be bent.
Integrity breaks in a few clean ways. Each one hits continuity differently.
Data poisoning. A small set of records nudges behavior while your average scores still look fine. Like salt in tea, you notice too late.
Pipeline compromise. A library update pulls in a bad package. A notebook copied from the internet carries a surprise. Training is complete, yet you now ship a backdoor.
Artifact tampering. Someone swaps a model version in the registry. A storage bucket leaks. A model file gets replaced with one that waits for a trigger phrase.
Runtime abuse. Prompt injection. Retrieval poisoning. Agents are calling tools with more privileges than they deserve. The model may be intact. The system around it is not.
Feedback loop decay. Teams retrain on user inputs without filters. Attackers seed those inputs. Drift creeps in. The model gets worse, then weird, then unsafe.
None of this requires movie-style hacking. It requires access, speed and your habit of trusting what “looks normal.”
Design. Build a trust spine
Design is where programs go to die. They fall in love with words. “Secure.” “Responsible.” “Governed.” Nice. Still useless at 2 a.m.
Keep the spine short and testable.
Provenance. Trace a deployed model back to data sources, code version, training run and approver. If you can’t answer “Where did this come from?,” you can’t answer "Should we trust it?”
Integrity. Treat artifacts like release binaries. Hash them. Sign them. Store them in an immutable registry. Enforce checks at deploy time. If the checks fail, it does not ship.
Least privilege. Training should not have broad access to production data. The registry should not be writable by everyone. Agents should not have access to god-mode tools. Permission mistakes become outages.
Change control. Version everything. Require reviews for high-impact changes. Keep rollback simple. If rollback takes a week, you won’t do it when it matters.
Assurance. Test abuse paths, not only accuracy. Test for data leakage, unsafe tool calls, prompt injection and retrieval abuse. Then keep testing after release because systems drift.
Deliver. Put controls into the delivery flow
Controls in a document don’t protect systems. Controls in pipelines do.
Start at the beginning. Data. Version datasets. Record lineage. Strip sensitive fields. Track licensing. If you can’t explain what data trained the model, your audit trail is broken before you deploy.
Secure training runs. Isolate compute. Lockdown secrets. Log every run. Make metadata easy to reproduce, even if you can’t reproduce weights.
Protect the registry. Keep it boring and strict. Limited write access. Every upload is logged. Every download is traceable. Artifacts immutable. A change means a new version, not an edit.
Add release gates. Dependency checks. Policy checks. Abuse tests. Privacy checks. If the model touches customers or money, require an approval with a name attached, not a shared mailbox.
Watch the runtime behavior. Spikes in tool calls. New prompt patterns. Unusual data access. Sudden changes in refusal rates. Strange outputs that show up in tickets before dashboards.
And rehearse rollback. Pull a model version. Restore the previous one. Measure the time. The first attempt shouldn’t be during an incident.
Drive. Run it like continuity
This is where boards and CISOs meet. Ownership and speed.
Assign roles. A model owner for the business outcome. A data owner for sources. A platform owner for pipelines and registries. Security for policy and assurance. No gaps.
Set cadence. Monthly reviews for the most sensitive models. Quarterly supplier checks for third-party models and data. Regular red team exercises focused on prompt injection and tool misuse.
Measure what matters. Percentage of models inventoried with owners. Percentage of deployed artifacts signed and verified. Time to revoke registry access. Time to roll back. Exceptions past expiry. Incidents where model behavior drove customer impact.
Do this, and you get something rare. Calm. The calm that lets teams ship and lets boards sleep.
Continuity is not a promise. It’s a habit. Models will keep joining the party. Your job is to know who brought them, what changed and how fast you can escort them out when they start a fight.
