Match power with evidence.
Ask for stronger safety evidence as a system gains the power to do more harm.
Prevents A system given authority to act on the strength of tests that never examined that authority.
Proposed
What it asks
The more consequential a system’s capabilities and permissions, the stronger the evidence required before expanding them. That evidence should identify the version tested, the conditions of the test, the failures observed, and the limits of what the results establish.
A model that suggests a database query and an agent that executes it against a hospital’s records have different opportunities to cause harm. Adding credentials, persistent memory, external communication, or the ability to delegate can change the risk even if the model remains identical. The assessment should follow those changes.
A useful safety case is an argument someone else can inspect: here is what could go wrong, here is why we believe our controls address it, and here is the evidence that might prove us wrong. Passing a benchmark is one input. It cannot establish that an entire deployment is safe in every environment.
For the most powerful systems, this obligation should reach decisions about further development when development itself creates material risk. The relevant thresholds would need public justification and technical revision. I do not think we can responsibly invent a universal number in an essay and call everything below it safe.
The requirement must also be affordable in proportion to the risk. A small tool with narrow permissions should have a straightforward way to demonstrate compliance. A large company should face demanding requirements when the capabilities warrant them. Revenue and headcount are poor substitutes for examining what a system can do.
Proposed
What compliance looks like
- A safety case specific to the version deployed, with realistic tests and documented limitations.
- Explicit conditions that would invalidate the assessment.
- For research claims, precise assumptions, reproducible checks and explanations others can use.
Illustrative
The case that tests it
It can suggest an action. Is it ready to carry one out?
A person asks an AI assistant for help with a payment. Suggesting what to do is one thing; permission to move their money is another. The same underlying model gains different opportunities for harm once it has tools and permissions. What should the developer show before people rely on it to act?
What should happen before wider release?
People want useful tools sooner. They also need reasons to trust what those tools can do.
Openness and deployment control are separate questions. Open weights are not automatically open source; a closed API is not automatically safe. Openly released weights can enable outside research and adaptation, but copies can persist after release. Provider-controlled access can support limits, updates and withdrawal, but scrutiny then depends more heavily on the provider’s discretion.
Our inferenceTesting the product includes examining the actions people take because of its output.
Reported
What the case draws on
Regulator’s allegations
Testing is an obligation with human stakes.
- Reported
- December 2023
The FTC alleged that Rite Aid’s facial-recognition system generated thousands of false-positive matches and that the retailer failed to adequately test accuracy before deployment. The agency described customers being wrongly accused and proposed a five-year ban on surveillance facial recognition as part of a settlement.
This is an account of the agency’s allegations and announced proposed settlement, not a claim that every allegation was adjudicated.
US Federal Trade Commission: Rite Aid banned from using AI facial recognition
Reported
On the record
- Research announcement
A map of possible changes.
Google DeepMind introduced AlphaGenome Atlas, a resource containing predicted molecular effects for nine billion single-letter DNA variants.
These are predictions. The announcement reports experimental checks of selected findings, not validation of every variant or proof of clinical benefit.
- Research announcement
A proof offered for scrutiny.
OpenAI announced a claimed solution to the Navier-Stokes problem and released a written proof and Lean formalization.
This is the developer’s claim, not a statement of independent mathematical acceptance. A September 10 update addresses provenance; attribution and verification remain separate questions.
Proposed
Where it can fail
- A benchmark can become a target that substitutes for evaluating the actual deployment.
- Testing costs can exclude smaller builders.
- This will slow down useful tools.
- The requirement is proportionate. A small tool with narrow permissions should have a straightforward way to demonstrate compliance; demanding requirements are for capabilities that warrant them.
- Passing the benchmarks should be enough.
- A benchmark is one input. It cannot establish that a whole deployment is safe in every environment it will meet.