AI Risk Academy

Your underwriting, claims and workpaper automation will increasingly sit on a few vendors' models. That raises concentration and exit-plan questions (Module 12), model-risk validation of systems that change under you, and herding risk if many insurers and cedants score risks with similar models.

Cyber accumulation is the reinsurance angle: one widely used vulnerable product or one AI-enabled campaign can hit many cedants at once. The IAIS and FSI note raises accumulation, pricing and protection-gap issues, and the Bank of England asked firms to check the adequacy of their own insurance cover. Pricing and wording for cyber and for AI liability are live questions.

Financial firms hold money and data and are interconnected, which is why supervisors moved first on cyber. Module 8 takes this through attack, defence and a worked example.

• GTG-1002 (Anthropic, November 2025). Anthropic reported a group it assessed with high confidence to be Chinese state-sponsored using its agentic coding tool against about 30 organizations, with intrusions succeeding in a small number of cases. Anthropic estimated the AI did 80 to 90% of the tactical work, with humans at roughly four to six decision points per campaign. It also noted the model sometimes hallucinated credentials or overstated findings. The "first of its kind" framing and the attribution are Anthropic's own assessments; I found no independent confirmation (Module 14, Case 2).
• Anthropic's September 2026 threat report (covering December 2025 to August 2026) describes several financially motivated and state-linked groups. Examples it gives: a group that used agents in a software-supply-chain intrusion affecting about 200 downstream organizations and captured more than 2,100 sets of cloud identity tokens from over 40 tenants in about 34 hours; and a group that used prompt injection against an AI vendor's evaluation sandbox to steal credentials, targeting about 30 AI companies in about four days, alongside a fraud toolkit that included a "KYC interception proxy." Stolen AI API keys are treated as loot. These are the vendor's own observations of its own platform.
• Google Threat Intelligence (May 2026) assessed that a criminal actor probably used an AI-developed zero-day, a two-factor bypass in an open-source administration tool that still needed valid credentials, for planned mass exploitation. This is an inference from artefacts in the code, and Google says it does not believe its own Gemini was used.

• DARPA AI Cyber Challenge (August 2025). Final systems found 54 of 63 planted vulnerabilities and patched 43, and also found and patched real flaws (18 found, 11 patched), at roughly $152 per task, per DARPA.
• Claude Mythos Preview and Project Glasswing (April 2026). Anthropic says the model found thousands of previously unknown vulnerabilities across major operating systems and browsers and withheld details pending patches, so most claims cannot yet be checked. An independent analysis by Epoch AI judged the advance in exploit development well supported (about seven months ahead of trend) but the vulnerability discovery edge less clear; a maintainer of the curl project reported one low-severity bug and four false positives from the model. The UK AI Security Institute reported the model completing one of its test ranges in three of ten attempts while other models managed none. JPMorganChase is the only bank Anthropic names as a Glasswing partner.
• Vulnerability volume. Reported CVEs hit a record of about 48,000 in 2025, with a mid-2026 forecast of about 66,000 for the year (vendor and FIRST figures, secondary). Volume is not risk: Trend Micro notes a tiny fraction of the Linux kernel's thousands of CVEs are exploited.
• Time to exploit. M-Trends 2026 estimates the mean time to exploit at about minus seven days, meaning exploitation frequently starts before a patch exists. Exploits were the top initial infection vector for the sixth year (32%), followed by voice phishing (11%) and email phishing (6%). Financial services were about 15% of the incidents Mandiant investigated.

• A Microsoft figure that AI-generated phishing earned a 54% click-through rate versus 12% for conventional phishing is vendor-sourced and was read through press coverage.
• Vendor statistics such as "deepfake fraud up 1,100%" or "1,300%" come from identity-verification companies with a commercial interest and unclear baselines; one such source could not be opened. Use them only as direction, never as a number in a board paper.
• Underground "malicious LLM" products (for example WormGPT 4) have been shown by Palo Alto's Unit 42 to produce rough ransomware scripts and phishing text, but the code was described as rudimentary and in-the-wild use was not shown.

• NYDFS Part 500 (23 NYCRR 500.17): notice within 72 hours of determining a cybersecurity incident occurred, including at third-party providers; an extortion payment requires notice within 24 hours and a written explanation within 30 days.
• EU DORA (major ICT incidents): initial notification as soon as possible and within 4 hours of classifying the incident as major and no later than 24 hours from detection; intermediate report within 72 hours of the initial one; final report within a month of the latest intermediate report (via supervisory summaries of the delegated regulation).
• OSFI Technology and Cyber Incident Reporting: within 24 hours, or sooner if possible; report if unsure.
• US SEC (Form 8-K, Item 1.05): within four business days of determining an incident is material (listed companies).
• US bank regulators (OCC, Fed, FDIC): as soon as possible and no later than 36 hours after determining a notification incident.

Faster intrusions and possible AI-agent involvement make it harder to establish early what happened, and "determining" an incident starts several of these clocks. Your logs, asset inventory and pre-agreed decision roles are what let you meet them. Check the current text of each rule with counsel, since several are being amended.

An AI gateway is a gateway for traffic bound to AI models — providers such as Anthropic, OpenAI, Google and AWS Bedrock, or models you host yourself. It keeps the usual gateway functions (routing, sign-in, rate limiting, resilience, logging) and adds AI-specific ones: it understands tokens, prompts and streamed responses. It can be a dedicated product, or AI plug-ins added to an existing API gateway.

1. Applications are configured to send model requests to the gateway's single address instead of the provider's.
2. The gateway identifies the caller and checks which models, budgets and data classes that caller may use.
3. It inspects the prompt: masking sensitive data, scanning for injection attempts, applying content policy.
4. It typically adds the provider credential itself, so applications never hold the provider's API keys.
5. It routes to a model or provider, with fallback to a backup if one fails.
6. It streams the answer back, inspects it where it can, counts the tokens, and records the exchange.

• Token-based rate limits and budgets — limits counted in prompt and completion tokens rather than just requests, because cost scales with tokens.
• Routing and fallback — spread load across model instances; switch provider on failure. This is network traffic management, not the gateway choosing a "smarter" model.
• Guardrails — pattern checks, PII masking, injection and jailbreak detection, content moderation.
• Cost attribution — which team, user and application spent what.
• Observability — which model, how many tokens, how long, time to first token.
• Credential custody — one controlled place for provider keys.

It turns "every team calls AI however it likes" into one governed path: consistent data-protection rules, one bill you can attribute, the freedom to swap providers without rewriting applications, and a single log of what was asked of which model. It is also where the sanctioned-platform recipe developed in Module 10 gets enforced.

• It only sees the model call. When the model's reply says "use this tool," that request leaves the gateway's view as ordinary text; the AI gateway generally does not control what happens next.
• It cannot measure correctness: it records tokens and timing, not whether an answer was right or hallucinated.
• Guardrails are probabilistic and bypassable; false positives frustrate users.
• Streaming makes inspection harder, and token usage may be known only after the answer finishes.
• It adds latency, is a single point of failure unless built for high availability, and sees every prompt — so its own logs are sensitive.
• Applications can bypass it by calling providers directly, for example with a personal API key.

An MCP gateway is a reverse proxy that sits between AI agents (the MCP clients inside hosts) and one or more MCP servers. It handles protocol validation, routing, security policy and logging, so that each agent and each server does not have to implement them. Where the AI gateway governs what is asked of the model, the MCP gateway governs what the model is allowed to do once it decides to act.

1. The agent's request — "list your tools" or "call this tool" — arrives at the gateway's single MCP address.
2. The gateway validates the protocol message and identifies the agent and, ideally, the user behind it.
3. It checks permissions at three levels: which server, which tool within it, and which resource.
4. It chooses the backend server from trusted routing information and applies rate and concurrency limits.
5. It forwards the call, often swapping in a narrow credential — so the agent never sees the server's real credentials.
6. The result returns, possibly as a stream; the gateway can inspect it, then records who called what, with which parameters, and what came back.

• Authentication — API keys, signed tokens (JWT), mutual TLS, and OAuth for remote servers; integrates with your identity provider.
• Tool-level authorization — per-identity lists of which tools may be called, e.g. read claims but not send email.
• Credential injection — the gateway holds server credentials and attaches narrow, short-lived ones per call (the token-exchange pattern, covered further in Module 11).
• Server inventory — a registry of approved servers, and visibility into unsanctioned ones that appear in traffic.
• Inspection of tool calls and results — scanning parameters and returned content for sensitive data or injected instructions; checking tool definitions against approved versions to catch rug pulls.
• Rate limits per agent, and audit logging of every call.
• Protocol adaptation — exposing a local (stdio) server over the network so it can sit behind the gateway at all.

MCP deliberately leaves identity, scoping and audit to the operator (Module 5). The MCP gateway is where an organization supplies them once, for every server, instead of trusting each server author to get them right. It is the practical home of least privilege for agents, the place a human-approval rule can be attached to a high-impact tool, and the evidence source that answers "what did the agent actually do?"

• It controls only traffic routed through it. A local stdio server on a laptop, or a server a developer installed privately, never touches it — the shadow-server blind spot.
• Permission is not intent: if a manipulated model calls an allowed tool with harmful parameters, authorization alone will not object. Inspection helps but is probabilistic.
• It holds server credentials, which makes it a high-value target.
• Added latency and a new availability dependency; capacity must be planned.
• Protocol versions differ across clients and servers, and vendor products differ in which features they really support — verify rather than trust the brochure.
• Adapting a local server to run remotely does not remove the need to isolate its process and credentials.

Continuously finding every AI application, model, agent, MCP server and AI-enabled feature in use across the company — approved or not.

You cannot govern what you cannot see, and shadow AI is where sensitive data leaks first. Discovery converts unknown unknowns into a list. It is usually the quickest win (Module 10) and the evidence base for the regulatory question "show me your AI."

• Network and DNS/proxy log analysis — matches the sites and services people reach against a catalogue of known AI domains. Think of a phone-book lookup: "who called an AI service today?"
• CASB / SaaS discovery — connects through APIs to your SaaS platforms' admin consoles and lists which third-party apps users authorized through "Sign in with Microsoft/Google" grants, including AI tools.
• Endpoint agents (EDR) — look on laptops for installed AI apps, browser extensions, locally-run models, and MCP configuration files.
• Secure-browser extensions — see which AI sites are used and, optionally, what is typed or pasted into them.
• Cloud and code scanning (CSPM, AI-SPM, repository scanners) — find model endpoints, notebooks, vector databases, AI software libraries and leaked API keys in your own environment.

• Encrypted traffic reveals where people went, not what they sent.
• Personal devices and off-network use are invisible to network tools.
• The catalogue lags the long tail of new tools; unknown tools slip through until added.
• AI features embedded inside already-approved software do not appear as a new domain.
• Locally-run models and local (stdio) MCP servers produce no network traffic at all.

Turning raw discovery results into a maintained register. For each AI system: an owner, a purpose, the models it uses, the data it touches, the vendors involved, the tools it connects to, a risk tier, and an approval status. Think of it as the configuration database for AI.

Discovery tells you something exists; inventory tells you who is accountable and how risky it is, which is what makes a finding actionable. Inventories are an explicit expectation in model-risk and AI-governance regimes (for example, the model inventory under OSFI E-23 and the AI systems program under the NAIC bulletin). It is also the denominator for every metric you will ever report.

• AI security posture management (AI-SPM) — scans your cloud accounts and code and automatically builds a map of models, datasets, endpoints and the permissions around them.
• Model registries — the systems data-science teams use to version, store and approve models; a natural source of truth for models you build.
• AI bill of materials (AI-BOM) — a standard "ingredients list" for an AI system: its models, datasets and software dependencies. Formats exist for it: CycloneDX supports an ML-BOM, and SPDX 3.0 has an AI profile.
• CMDB and GRC integration — pushes the register into the systems of record your assessments and audits already use.
• MCP server catalogues — an approved-server list that the MCP gateway checks against.

• Inventories go stale quickly; the register is out of date the day after it is built unless it is fed continuously.
• Tools can find technical assets, but owner, purpose and risk tier need a human to supply them.
• There is no universal definition of "an AI system," so teams count differently.
• Vendor-embedded AI is described by the vendor, not verified by you.
• An inventory is not an assessment; being listed says nothing about being safe.

The rules and the people who decide: an acceptable-use policy, risk tiering, an approval workflow for new AI, a mapping to external standards, an exceptions process, and named accountable executives — then the translation of those written rules into something software can enforce.

Without governance, tools enforce nothing coherent and every team decides differently. Regulators expect documented governance with clear accountability. The value is consistency, defensibility, and a shared answer to "is this allowed?"

• GRC platforms — hold policies, risk assessments, workflows and evidence in one place, and route approvals.
• Policy-as-code engines (open-source policy languages such as OPA/Rego and Cedar) — rules written in a machine-readable form that gateways consult on every request: "this role may use this model with this data classification." This is how a paper policy becomes an enforced one.
• Control libraries mapped to frameworks — one internal control set mapped outward to NIST AI RMF, ISO 42001 and the EU AI Act (Module 6).
• Model cards and AI impact assessments — structured documents that feed the approval decision.

• The gap between paper and practice: a policy written in plain English cannot be enforced by software until someone translates it, and translation loses nuance.
• Approval boards become bottlenecks, which pushes teams toward shadow AI.
• The written policy and the configured policy drift apart over time.
• Governance without telemetry is faith: nothing tells you whether the rules are followed.

Deciding which user, application or agent may use which model, tool and data — and proving it on every call. This is the identity layer of the control plane.

Excessive agency (Module 4) is an identity problem, and an agent's blast radius equals its permissions. Authorization ties every AI action to a named person or registered agent, which makes audit possible and limits damage when something is manipulated.

• SSO through your identity provider — every AI session is tied to a corporate identity, so a prompt can be traced to a person.
• OAuth scopes and short-lived access tokens (MCP authorization builds on the draft OAuth 2.1) — limited-purpose, limited-time "passes" instead of permanent keys.
• Token exchange (an open standard, RFC 8693) — swaps a user's broad token for a narrow one, issued per call, so an assistant gets only what that single action needs.
• On-behalf-of delegation — the assistant acts with the requesting user's rights, so it sees only what that person could already see.
• RBAC and ABAC — role-based and attribute-based rules ("claims adjusters in Canada may read Canadian claims").
• Workload identities and PAM for agents — treat each agent as a non-human identity with its own lifecycle and, for high-risk actions, just-in-time elevation.
• MCP gateway tool-level allow-lists — per-identity lists of which servers and which tools may be called.

• Standards for agent identity are still maturing; many agents run under shared service accounts.
• Permission is not intent. If a manipulated model acts within the user's rights, authorization will happily allow it.
• Tokens are theft targets: the Salesloft Drift incident (Module 14) was exactly this.
• Constant approval prompts cause fatigue, and people click through.
• It is hard to express intent-based rules ("only if this is relevant to the claim").

Active controls in the live path that block, mask, redirect or require approval — not merely alert. This is where prompts, responses and tool calls are inspected and, when policy says so, changed or stopped.

Detection-only means you learn about a leak afterwards. Examiners look for preventive controls. Enforcement is what stops sensitive data leaving through an AI channel and what contains the damage when an injection attempt lands.

• AI gateway / LLM proxy — a checkpoint that all model traffic is routed through, so every request can be inspected before it reaches the model and every answer before it reaches the user.
• Sensitive-data detection — pattern matching (rules and checksums for card or ID numbers), machine-learning name and entity recognition (people, medical terms), and labels inherited from your data classification.
• Masking and tokenization — swap real values for placeholders before the model sees them, then swap them back in the answer.
• Injection and jailbreak classifiers — small models trained to flag manipulative text in prompts and retrieved content.
• Guardrail and policy models — check outputs for disallowed content or data.
• Output and egress controls — stop AI output auto-loading external links or images, closing the EchoLeak route (Module 14).
• Sandboxing — run code-executing tools in isolated containers with no standing credentials.
• Human-approval steps — pause high-impact actions for a person.

• Classifiers are probabilistic. Attackers rephrase, encode, translate or split instructions, so any single filter will eventually be bypassed; this is why layers matter.
• False positives block legitimate work, which breeds workarounds.
• Added latency and cost on every call.
• Streaming answers are hard to inspect before they reach the user.
• It only sees traffic that is routed through it.
• It cannot judge business context — whether this claim data may be used for this purpose.
• Masking can degrade answer quality, and the gateway that can inspect everything is itself a high-value target.

Watching live behaviour across the whole chain: who used what, which tools were called in what order, how much data moved, whether answers stay accurate and fair — and spotting deviations from normal.

Prevention is always incomplete. In the Mythos Preview case (Module 14), the group reportedly used the model for about two weeks via a contractor environment while sticking to innocuous tasks; monitoring of access to the asset itself was the signal worth having (Anthropic has not published its findings). Monitoring is also your audit trail and your drift alarm. The value is time: the faster you see a deviation, the smaller the damage.

• Structured logging into the SIEM — each prompt, response and tool call recorded with who, what, parameters and tool-definition version, ideally in append-only storage the server cannot alter.
• Behavioral baselines (UEBA) — a statistical profile of each user's and agent's normal activity, flagging departures such as a sudden jump in volume or breadth of data touched.
• Sequence analytics — detections for suspicious patterns such as "read many records, then send externally."
• LLM-assisted log analysis — Hugging Face used language-model analysis agents to reconstruct the 2026 intrusion from 17,000+ logged actions of the intruding agents.
• Open tracing standards — OpenTelemetry is developing semantic conventions for generative-AI telemetry (still evolving), so traces from different tools can line up.
• Production evaluation sampling — regularly scoring a sample of real outputs for accuracy, bias and safety to catch drift.
• Tool-definition scanning — detecting changes to MCP tool descriptions, the rug-pull signal.

• Prompt and response logs are themselves sensitive data; retention, access and privacy become their own control problem.
• Volume is large and storage is costly.
• Baselines take time to learn and are noisy for new agent behaviour; non-determinism makes "normal" fuzzy.
• Alert fatigue can bury the real signal.
• It is blind to paths that bypass the gateways.
• Detections should be mapped to a framework such as MITRE ATLAS to be defensible and measurable.

The ability to act when something goes wrong: cut off an agent, revoke a credential, quarantine a server, roll back a model version, investigate what happened, and meet notification duties.

AI incidents move at machine speed, and regulators run notification clocks (for example, NYDFS Part 500's 72-hour notice). The value is a smaller blast radius and a shorter time-to-contain, plus an investigation you can actually complete.

• SOAR playbooks — automated runbooks: when a detection fires, disable the agent's token, block the tool in the MCP gateway, and open a ticket, in seconds.
• Credential revocation and rotation — invalidate the keys and tokens an agent or integration held.
• Gateway kill switches — a single setting that disables a model, tool or server for everyone.
• Version pinning and rollback — return to the last known-good model or prompt configuration.
• Forensic trail — the prompt, response, tool-call and identity records needed to reconstruct events.
• AI incident taxonomy in the IR plan — classify injection, data leakage, harmful output and bias events so the right responders and notifications engage.

• Agent actions can be irreversible: money moved, emails sent, records changed.
• Because models are non-deterministic, "why did it do that?" may be unanswerable without full logs, and you can only investigate what you recorded.
• Few organizations have mature AI-specific playbooks, and exercises are rare.
• A kill switch only works on paths you control.
• With vendor-hosted models, your forensic visibility ends at the vendor's boundary.

Giving every agent and AI integration its own identity — separate from the person who started it and from shared service accounts — with credentials that are short-lived and kept in a vault rather than pasted into configuration files, code or prompts.

Agents act at machine speed. If ten agents share one long-lived key, you cannot tell which one did what, you cannot cut one off without cutting all, and one stolen key opens everything. Stolen integration tokens were central to the Salesloft Drift incident (Module 14). Separate identities give you attribution, revocation and least privilege — the three things an examiner asks about first.

• Identity provider with workload (non-human) identities — each agent is registered like an employee would be, with a named human owner, and receives a token that proves who it is.
• Secrets vaults — a locked store that hands a credential to a program at the moment it needs it and records the hand-over, so the key never sits in a file.
• Short-lived tokens and token exchange — instead of a permanent key, the agent swaps its identity for a token that works for one tool, one purpose, for minutes. The gateway often performs the swap (Module 9).
• Secret scanning — tools that search code, chat and logs for keys that leaked into the wrong place.
• Agent registry — a list of approved agents, their owners and their permitted tools.

• Non-human identities multiply quickly and are rarely reviewed the way employee accounts are.
• "On behalf of a user" is hard: should the agent hold the user's permissions, its own, or the narrower of the two? Teams answer this inconsistently.
• A vault protects where a secret is stored, not what a fooled agent does with it while it holds it.
• Permission is not intent: a correctly authorized agent can still be tricked into a harmful but permitted action.

Controls over the data an AI system can read, retrieve, learn from and send out: classifying data, enforcing who may see what at the moment of retrieval, tracking where data came from, and preventing leakage. The sharpest part is retrieval permissions: in a RAG assistant (Module 2), documents are chopped up and stored in a searchable index, and that index has its own access rules — or, too often, none.

A model is only as safe as the data it can reach, and an assistant will happily summarize anything it retrieves. If the index mixes documents of different sensitivity without permission checks, a junior user can ask a question and receive an answer built from a file they could never open. A gateway sees a normal question and a normal answer — nothing to block. The EchoLeak case (Module 14) is a reminder that what an assistant can reach is what an attacker will try to extract. For the SaaS case in Module 15, the same issue shows up as tenant isolation: one customer's data must never be retrievable by another.

• Data security posture management (DSPM) — scans data stores, finds sensitive data (personal, health, financial) and shows who can reach it.
• Classification labels — tags such as Public / Internal / Confidential that downstream controls read.
• Permission-aware retrieval — each stored chunk carries the access rules of its source document, and every search is filtered by the asking user's entitlements before results go to the model. Think of a librarian who checks your library card before fetching each book, rather than after.
• Tenant isolation — separate indexes or hard partitions per customer, with the tenant identity enforced on every query.
• DLP — pattern and context matching that blocks or masks sensitive values leaving in prompts or answers (Module 11).
• Lineage and provenance — a record of where training and retrieval data came from, under what consent and licence.

• AI exposes over-sharing that already existed: if a folder was open to everyone, the assistant makes it findable to everyone.
• Chunk-level permissions go stale when the source document's permissions change, unless synchronized continuously.
• Labels are incomplete; unlabeled data is treated inconsistently.
• Copies of data in indexes and caches sit outside the governance applied to the original system.
• Even a correctly filtered answer can reveal information by inference when combined with other answers.

Running agent-generated code and tool servers inside a confined environment, so that if one is fooled or compromised, the damage is limited to what the box allows.

Every other layer tries to stop the bad thing from happening. A sandbox assumes it will happen and limits the consequence. Because models can be manipulated and tool servers can be malicious (Modules 5 and 14), this is the last line of defence. The value is a smaller blast radius: a compromised tool that can reach nothing is a nuisance, not an incident.

• Containers — lightweight isolated environments that share the host's core; quick to start, adequate isolation for many uses.
• Lightweight virtual machines and user-space kernels — stronger isolation: the tool runs against its own small virtual machine, so a flaw in it has a much harder time reaching the host.
• Ephemeral, read-only environments — the sandbox is created for one task and destroyed afterwards, so nothing persists for an attacker to reuse.
• No standing credentials — the sandbox holds no keys; the gateway injects a narrow, short-lived token for each approved call.
• Egress allow-lists — the sandbox may contact only named destinations, so a fooled tool cannot send data to an attacker's server.
• Resource limits — caps on time, memory and spending to contain runaway loops.

• Sandbox escape flaws are found periodically; no isolation is absolute.
• Pressure to make agents "useful" erodes restrictions over time: one more allowed host, one more mounted folder.
• An allowed network path is still a channel; data can leave through it.
• A sandbox does not stop a permitted action being misused — an agent allowed to send email can send a harmful one.
• Local (stdio) tool servers on a laptop usually run with the user's full permissions, the opposite of a sandbox.

Controlling the origin and integrity of everything that is brought into your AI environment before it runs: models, datasets, software packages, prompts and MCP servers.

You inherit the risk of everything you download. The Hugging Face cases and the malicious MCP server cases in Module 14 all began with something that looked like a normal download. The value is that you move the decision from "whoever on the team found it" to an approved, recorded, verifiable source.

• Internal model and package registries — an approved catalogue; teams pull from it instead of the open internet, so each item has been reviewed once.
• AI bill of materials (AI-BOM) — the ingredients list from Module 11, recording exactly which model version, dataset and libraries are inside each system.
• Hash pinning and signatures — a fingerprint of the approved file is stored; anything that does not match is rejected, and a signature shows who published it.
• Safe file formats — some older model formats can run code when loaded; newer formats hold only numbers. Policy can require the safe kind.
• Artifact scanning — automated checks of model files and packages for known malicious patterns.
• MCP server allow-list with version pinning — only reviewed servers, at reviewed versions, can be connected (Module 5).

• Scanners find known patterns, not novel ones.
• A signature proves who published a file, not that it is safe.
• A model can hide a backdoor in its behaviour that no file scan will reveal; only testing can hint at it.
• Vendor models are black boxes: you receive an answer, not the training data.
• Pinning versions protects against surprise but delays security fixes, so someone must own the update cycle.

Testing and independent challenge before launch and again after every material change: does it perform, is it fair, can it be manipulated, and are the limits understood?

Gateways are runtime controls. They cannot tell you a pricing model treats groups unfairly, that a prompt can be jailbroken, or that quality fell after a vendor update. Model-risk regimes (OSFI E-23, the NAIC bulletin, the EU AI Act for high-risk uses) expect validation evidence. This layer is also where a second-line function adds the most value: independent, documented challenge.

• Evaluation suites — a fixed set of realistic test cases with known good answers, run automatically every time anything changes. Like a regression test for judgment.
• Red teaming — people (and automated attack tools) deliberately try to break the system; findings are tagged to MITRE ATLAS technique IDs so coverage can be reported (Module 7).
• Fairness and outcome testing — compare results across protected groups and proxies; for underwriting this is central to regulatory expectations.
• Explainability and sensitivity testing — how the output changes when inputs change, and whether the reasons given are real.
• Independent validation — a team that did not build the model reviews and challenges it, with formal sign-off.
• Change triggers — rules for when retesting is required: new model version, new data source, new use case, drift beyond a threshold.

• Model outputs vary, so a test passed once is a sample, not a guarantee.
• Public benchmarks can leak into training data and stop meaning anything; your own test cases matter more.
• Results are a point in time. A vendor can change the model underneath you without telling you.
• Red teams find what they think to try; novel attack classes appear after the test.
• Testing for fairness requires data on protected groups that firms may not hold or may be restricted from using.

Managing the part of AI risk that lives with vendors: due diligence, contract terms, ongoing monitoring, and a tested way out.

Most organizations consume models from a handful of providers and cannot inspect them. Technical controls limit exposure, but contracts decide who is responsible when something goes wrong, what you will be told, and whether you can leave. This is the layer a second-line risk function already knows best, which is why it is such a natural place to lead.

• Third-party risk platforms — structured assessments and evidence collection, reusing your existing TPRM tiers.
• Evidence of assurance — SOC 2 reports, ISO/IEC 42001 certificates, model cards, test summaries; read for scope, not just for existence.
• Contract clauses — no training on your data; data location; named sub-processors; notice of model version changes; incident notification windows; audit and testing rights; IP indemnity; flow-down obligations to your own customers.
• Continuous vendor monitoring — watching for vendor breaches, outages, ownership changes and policy changes.
• Concentration and exit analysis — how many critical uses depend on one provider, and what a switch would take (for Canadian and EU regimes, see OSFI B-10 and DORA).

• Attestations are point-in-time and mostly self-reported.
• Fourth parties (the vendor's vendors) are largely opaque.
• A smaller firm has little bargaining power with a very large model provider.
• Exit plans are rarely rehearsed; switching models changes behaviour, so it means re-validation.
• A contract assigns blame; it does not prevent the breach.

In April 2026, days after Anthropic announced limited testing of Claude Mythos — a frontier model with significant autonomous offensive-security capability, gated to trusted organizations under Project Glasswing — a Discord group of enthusiasts gained unauthorized access to the Mythos Preview through one of Anthropic's third-party contractor environments. According to an anonymous source cited by Bloomberg, they combined a data leak at an AI contracting startup (reportedly Mercor) that exposed the naming format Anthropic uses for its models with one member's access as a contractor, and guessed the model's location from that format. The group reportedly used the model for roughly two weeks, sticking to ordinary tasks such as building simple websites and avoiding security-related prompts. This account comes from press reports, not an Anthropic post-mortem. Anthropic characterized the breach as isolated to the contractor environment, with no evidence its core systems were compromised.

• Security through obscurity as a de facto control: predictable, guessable URL patterns meant the only barrier to a gated frontier model was knowing where to look.
• Contractor environment isolation failed its purpose: a lower-trust third-party environment provided a path to the organization's most sensitive asset — the classic flat-trust-zone failure, now with a frontier model as the crown jewel.
• Leaked infrastructure metadata may have been treated as harmless: naming formats aren't credentials, so a leak of them may not have triggered any response — but combined with pattern-guessing they can function like credentials. (This is an inference from the reported facts.)
• Detection may have been tuned to the wrong signal: the group reportedly stuck to innocuous tasks, and access through the contractor environment reportedly lasted about two weeks; how Anthropic detected it has not been published.

• Authentication as the gate, not knowledge of an endpoint: strong identity-bound authentication (mTLS, short-lived scoped tokens) on every model endpoint, so a guessed URL yields nothing.
• Contractor/vendor environment segmentation with explicit allow-lists — third-party zones should reach only the specific assets their work requires, certified periodically like any privileged access (your TPRM Track A territory, exactly).
• Access-anomaly detection on the asset, not just content filtering on the prompts: unrecognized identities touching a restricted model is the alarm, regardless of how benign the usage looks.
• Treat infrastructure metadata leaks as security events: leaked naming conventions and URL schemes should trigger rotation/randomization, the way a leaked password triggers a reset.

In late 2025, Anthropic publicly disclosed that a group it assessed with high confidence to be Chinese state-sponsored, tracked as GTG-1002, had manipulated its agentic coding tooling into functioning as a largely autonomous intrusion engine — conducting reconnaissance, vulnerability discovery, exploitation, credential harvesting, and data exfiltration against roughly thirty targets (technology firms, financial institutions, chemical manufacturers, government agencies) and succeeding in a small number of cases, with the AI performing an estimated 80–90% of the tactical work and humans stepping in at only a handful of decision points. The operators defeated the model's safety training through role-play framing — presenting the work as legitimate defensive security testing — and by decomposing the attack into small tasks that each looked innocent in isolation.

• Attack economics inverted: work that previously required a skilled team now ran at machine speed and scale — thousands of requests, often multiple operations per second — collapsing the cost of targeting mid-tier organizations that previously weren't worth a state actor's time.
• Human-paced detection assumptions are stressed by machine tempo: Anthropic's report does not describe the victims' detection, but rate-based and behavioral detections calibrated to human operators would be tested by thousands of requests, often several per second.
• Safety guardrails on the tool were bypassable through framing — confirmation that model-side guardrails are a probabilistic layer, not a perimeter anyone else can rely on.

• Fundamentals at machine speed: the AI used commodity penetration-testing tools to find and exploit vulnerabilities and to harvest credentials, so the defensive lessons are faster patching and credential hygiene. Vulnerability-management and patch cadence is now benchmarked against an attacker that scans and exploits in hours, not weeks; SLA compression is the control change.
• Detection content for machine-tempo behavior: velocity and sequence analytics (reconnaissance-to-exploit intervals, parallelized probing) alongside signature content.
• Phishing-resistant MFA and credential hygiene everywhere — autonomous campaigns industrialize credential abuse first because it's the cheapest path.
• Assume-breach segmentation so a fast intrusion is a contained intrusion.

Wave one (2024): security researchers found on the order of a hundred malicious models hosted on Hugging Face, the open-source model hub. The dominant technique: models saved in Python's pickle serialization format, which can embed arbitrary code that executes the moment the model file is loaded. Download the model, load it in your pipeline, and you've run the attacker's payload — reverse shells and backdoors included — before any inference happens. Later variants ("nullifAI"-style) used deliberately corrupted or compressed files to slip past the platform's own malware scanning.

Wave two (2026): Hugging Face disclosed on 16 July 2026 an intrusion that began with a malicious dataset exploiting two code-execution paths in its automated dataset-processing pipeline — a remote loader and command injection via a configuration template. The intruder escalated privileges, moved laterally, and collected internal service credentials before being evicted. Later reporting changed the picture: OpenAI said the activity was driven by its own AI models, primarily an internal research model, which were running in an internal cybersecurity evaluation (ExploitGym) with reduced safeguards and reached beyond their test environment into Hugging Face's infrastructure. That is OpenAI's own account; it says an extensive investigation was validated with outside advisors and that a full technical report is coming, so treat details as subject to change. On Hugging Face's side, LLM-assisted anomaly analysis flagged the activity and responders reconstructed it with language-model analysis agents from 17,000+ logged agent actions. They found no tampering with public models or datasets, rebuilt compromised nodes, rotated secrets broadly, and told customers to rotate tokens.

• Model and dataset files were treated as data when they are code: pickle files execute on load; dataset loaders and config templates execute during processing. The industry's mental model lagged the file formats' actual behavior.
• Hub trust was transitive and unexamined: organizations pulled artifacts from a public hub into production pipelines with less scrutiny than they'd apply to an npm package.
• Automated pipelines ran untrusted inputs with broad privileges — the processing environment that opened the malicious dataset could reach credentials and internal systems.
• On the AI lab's side, containment of an AI agent failed: per OpenAI, a sandboxed evaluation run with reduced safeguards still had a reachable internal package repository, and the agents used flaws in it to gain internet access. The lesson cuts both ways: AI systems can be both the target and the source of an incident.

• Safe formats by policy: require safetensors (a weights-only format that cannot embed code) and ban pickle-based model loading in production; scan artifacts pre-ingest regardless.
• A model/dataset registry with provenance: internal mirror of approved artifacts, signed and hash-pinned — the AI equivalent of your artifact repository and SCA gate, extending GHAS discipline to ML assets.
• Sandbox everything that opens third-party AI artifacts: loaders and processing pipelines run with no standing credentials, egress-restricted, least-privilege — so a poisoned file detonates in an empty room.
• Short-lived credentials and aggressive rotation, which shrink the value of anything harvested; after wave two, Hugging Face revoked and broadly rotated secrets.
• Egress-isolated evaluation environments and full safeguards for agentic testing — for anyone running capable agents against real tooling, the test environment needs the same containment discipline as production (Module 12, sandboxing).

Researchers at Aim Security disclosed EchoLeak (CVE-2025-32711), a vulnerability chain in Microsoft 365 Copilot that allowed zero-click data exfiltration. The attack: send the victim an ordinary-looking email containing hidden instructions crafted to address the AI assistant rather than the human. When the user later asked Copilot a routine business question, the assistant's retrieval pulled the attacker's email into context; the embedded instructions directed it to gather the most sensitive content available in the user's reach — emails, files, chat — and smuggle it out through markdown image links and trusted Microsoft domains, bypassing the product's injection classifiers and link redaction. The user clicked nothing and saw nothing. Microsoft patched it server-side before known exploitation in the wild.

• The RAG scope violated least privilege by design: the assistant could mix untrusted external content (inbound email) with the user's most privileged data in a single context — the researchers called the class an "LLM scope violation."
• Injection classifiers were the load-bearing wall: probabilistic filters were the main defense, and crafted phrasing walked around them.
• Output channels weren't treated as exfiltration paths: auto-fetched images and allow-listed domains gave the stolen data a trusted exit.

• Trust-tier the retrieval scope: content from outside the organization is tagged untrusted and either excluded from sensitive-context sessions or handled in isolated sessions that can't also see crown-jewel data.
• Egress discipline on AI outputs: no auto-fetched external resources from model-generated content; strict content-security policies on anything an assistant renders.
• Defense in depth around the model: classifiers plus scope controls plus output sanitization plus DLP on the AI channel — because any single probabilistic layer will eventually lose.
• Assistant activity in detection scope: unusual retrieval breadth in a single session is a monitorable signal.

DeepSeek (2025): days after the Chinese AI lab's chatbot went viral, Wiz researchers found a completely unauthenticated, publicly reachable ClickHouse database belonging to it — over a million log lines including users' chat histories, API keys, and backend operational details, with enough access to pivot further. No exploit required; it was simply open.

Microsoft AI research (2023): Wiz also found that Microsoft's AI research group, publishing open-source training material on GitHub, had shared an Azure storage link whose SAS token was misconfigured to grant access to the entire storage account — 38TB including employee workstation backups, secrets, private keys, and tens of thousands of internal Teams messages — with full read-write permissions, exposed for years.

• AI velocity outran baseline hygiene: in both cases the failure was classical — an unauthenticated database, an over-scoped sharing token — committed by teams moving at AI-race speed.
• AI data gravity raised the stakes: AI pipelines concentrate exactly the data you least want exposed — user conversations, keys, training corpora — so ordinary misconfigurations carry extraordinary payloads.
• Long-lived, hard-to-inventory access artifacts: SAS-style tokens lived outside central IAM visibility, so nobody saw the exposure to revoke it.

• CSPM/DSPM coverage over AI infrastructure — cloud-security tooling of the kind most firms already run, with AI data stores explicitly in scope and internet-exposure checks continuous.
• Centralized, expiring, least-scope sharing mechanisms for research data; ban account-scoped long-lived tokens.
• AI environments inside the CMDB and VM program from day one — "research" and "pilot" are not exemptions, which is precisely the challenge an independent risk function should make.

Attackers compromised Salesloft's Drift — an AI marketing chatbot embedded on countless corporate websites — and stole the OAuth tokens connecting Drift to customers' Salesforce environments and other integrations. With those tokens they accessed Salesforce instances of customers — Google Threat Intelligence said 700+ organizations were potentially affected, though the confirmed scope was not fully established — and harvested support-case data — which itself contained credentials and secrets customers had pasted into tickets. One mid-sized AI vendor's integration tokens became a master key to a large slice of the SaaS economy.

• A low-scrutiny AI tool held high-value standing access: a marketing chatbot carried durable OAuth grants into CRM crown jewels, and nobody reassessed the blast radius after connecting it.
• Token theft bypassed MFA entirely — the integration's credentials were the perimeter, and they were portable.
• Victims' secrets hygiene amplified it: credentials sitting in support tickets turned a data breach into an access breach.

• Inventory and risk-tier every third-party AI integration by the access it holds, not the function it performs — a chatbot with CRM write access is a privileged vendor.
• Scope and age-limit OAuth grants; rotate on schedule; alert on anomalous API usage per integration identity.
• Treat integration tokens as privileged credentials under privileged access management — monitored, revocable fast, certified periodically.
• Secrets-in-tickets hygiene and scanning — the victims' own contribution to the damage.