Common Obligations

Obligation 04 of 06

Bound autonomy. Prepare for failure.

Limit what a system can do, outside the model, and be ready to contain and repair what goes wrong.

Prevents An agent that can act without limits anyone enforced, and a failure nobody can stop or repair.

Proposed

What it asks

An AI system that acts should have explicit limits on its authority and tested ways to contain failure. Before deployment, the operator should know which actions require approval, which resources are accessible, how permissions expire, and what happens when the system behaves unexpectedly.

Suppose an agent can issue refunds. Its spending limit should be enforced by the payment service. A sentence asking the model to stay within budget is insufficient. If it delegates to another agent, the delegated authority should remain within the original permission. Stopping the parent should also address outstanding child work and credentials that could remain active.

Recovery deserves equal attention. If a payment request times out, the system should establish whether it committed before sending another. If an agent changes a record, the operator needs a way to identify and repair the change. A stop button may prevent the next action; it cannot unsend an email or reverse every action already taken.

Some releases are difficult to recall at all, including publicly distributed model weights. In those cases, the assessment must account for the limits of later intervention before release. Openness can support scrutiny and wider participation, but neither an open licence nor a closed API establishes safety on its own.

The general obligation is to demonstrate control appropriate to the system, and to be honest where control ends.

Proposed

What compliance looks like

  • Spending and access limits enforced by the services the system calls, not by instructions to the model.
  • Credentials that expire, and delegation that cannot exceed the original permission.
  • Recovery exercises, and containment plans specific to each release.
  • Updates validated before they inherit permissions; installed artefacts traced and exposed credentials revoked after a compromise.

Illustrative

The case that tests it

Something has gone wrong. When is it safe to restart?

An operator discovers that a system has reached beyond its permissions, or that a trusted software update has been compromised. Other people’s systems may be exposed. Three decisions follow: contain the problem, find what was missed, and decide whether the repair is enough. These are proposed responses, not a reconstruction of the real incidents below or claims about what would have prevented them.

  • An agent exceeds its authority.
  • A trusted dependency is compromised. Illustrative supply-chain compromise: an installed update contains malicious code and may have accessed credentials. The model may have behaved exactly as intended. This is not a reconstruction of the LiteLLM incident or a claim that AI caused that attack.

01How far should the pause reach?

An agent exceeds its authority
An evaluation agent has reached a third-party service outside its authorised scope. The affected run is stopped. Other runs share parts of its infrastructure.
A trusted dependency is compromised
A malicious update has been identified. Decide whether to isolate known affected installations or pause related workloads sharing credentials and build infrastructure. The scope follows installed artefacts and reachable authority, not just the model version.

Restrict the affected activity

Stop the activity known to be affected while unrelated work continues. Focus the response where the failure has been found.

The same weakness may exist elsewhere. A narrow repair can leave other people exposed if the first discovery is mistaken for the whole problem.

Reasoning and limits
The strongest case
Contain the known route while unrelated work continues. A narrower intervention preserves useful research and focuses the response.
What it asks us to accept
A shared weakness may remain active elsewhere. A quick patch can mistake the first visible failure for the full extent of the incident.
Who feels the difference
Other service operators bear the risk if the boundary was drawn too narrowly.
What would need to be shown
An agent exceeds its authority A justified account of what is isolated, revoked credentials, preserved logs, and triggers for widening the pause.
A trusted dependency is compromised Installed versions and artefact hashes, reachable secrets, isolated workloads, and a trigger to widen containment.

Pause related activity

Pause work that shares the suspected weakness while responders find out how far the problem reaches.

Useful work stops too. The pause needs a clear scope, someone responsible for reviewing it, and conditions for restarting.

Reasoning and limits
The strongest case
Stop runs sharing the suspected exposure while responders establish its extent. This creates room to investigate before further external actions occur.
What it asks us to accept
The pause can delay useful work, including defensive research. An unclear scope or end condition can make it broader and longer than necessary.
Who feels the difference
Researchers and people waiting for useful capabilities bear the delay; outside operators may face less exposure.
What would need to be shown
An agent exceeds its authority A defined scope, a responsible decision-maker, review deadlines, and evidence for safely separating unaffected work.
A trusted dependency is compromised An inventory of shared credentials and build infrastructure, scoped suspension, and a timetable for reviewing the pause.

02What might the investigation miss?

An agent exceeds its authority
The known route is contained. Investigators must decide how far back, and across how many environments, to look. Both approaches preserve evidence and notify affected parties as findings emerge.
A trusted dependency is compromised
Trace where the package was published, built, installed and executed. A clean source repository does not establish that the distributed package was clean. Downstream operators need a way to report exposure.

Examine the known failures

Investigate the failures already found so affected people can get answers and repairs sooner.

Other failures may never have been detected. People missed by the original investigation could still be waiting for answers.

Reasoning and limits
The strongest case
Prioritise the observed incident to deliver actionable findings and repairs quickly.
What it asks us to accept
Starting only from detected failures can miss runs that escaped monitoring or were excluded from the original review.
Who feels the difference
Known affected parties may get answers sooner; undiscovered affected parties may receive none.
What would need to be shown
An agent exceeds its authority A documented search scope, its exclusions, and explicit reasons to expand the investigation.
A trusted dependency is compromised Publication and installation records for known affected artefacts, execution evidence, and explicitly unexamined downstream deployments.

Review related activity

Look across related systems and records for similar failures, including activity that the first review did not examine.

A wider search takes time and can expose sensitive information. Missing records still limit what investigators can establish.

Reasoning and limits
The strongest case
Reconcile the inventory of runs with the records actually examined. Search for similar failures across related versions and environments.
What it asks us to accept
A wider review takes time and access to sensitive records. Incomplete logs still limit what it can establish.
Who feels the difference
Previously unknown affected parties may be identified; reviewers must protect the people whose data appears in records.
What would need to be shown
An agent exceeds its authority Coverage records, missing-log disclosures, search methods, external reporting channels, and a process for correcting earlier findings.
A trusted dependency is compromised Reconciled build and distribution records, downstream notifications, missing telemetry, and independent review of the search coverage.

03What earns a return to work?

An agent exceeds its authority
Fixes are in place. Resumption is a decision about a defined activity under defined permissions. Neither route offers a guarantee.
A trusted dependency is compromised
A replacement is available. Reinstalling alone cannot revoke stolen credentials or remove persistence. Establish what must be rebuilt, rotated and checked before affected workloads resume.

Internal validation

Let the team closest to the system test the repair and restart work with tighter limits.

The organisation benefits from restarting. It may repeat earlier assumptions, while affected outsiders struggle to challenge its evidence.

Reasoning and limits
The strongest case
The team closest to the system can retest quickly and restore useful work under narrower permissions.
What it asks us to accept
The organisation benefits from restarting and may repeat assumptions that failed before. Outside parties may struggle to challenge the evidence.
Who feels the difference
Researchers regain access sooner; affected outsiders depend on the operator’s judgement.
What would need to be shown
An agent exceeds its authority Tests of the failed boundary, documented residual uncertainty, staged permissions, and triggers to stop again.
A trusted dependency is compromised Validated replacement artefacts, rotated exposed credentials, clean rebuilds where needed, and monitoring for persistence.

Independent examination

Ask an outside team to examine the investigation and test whether the repair supports restarting.

Outside review takes time and access. Reviewers can miss problems too; a separate decision is still needed about who may approve the restart.

Reasoning and limits
The strongest case
An outside evaluator can challenge the scope of the investigation and test whether the repair supports the claim being made.
What it asks us to accept
Access and evaluation take time. Evaluators can miss failures, lack expertise, or depend financially on the organisations they assess.
Who feels the difference
Affected parties gain another route for scrutiny; smaller organisations may need support to obtain it.
What would need to be shown
An agent exceeds its authority Reviewer access and conflicts, unresolved findings, a named authority for resumption, an appeal route, and continuing monitoring.
A trusted dependency is compromised Outside examination of the compromised delivery path, repair provenance, residual exposure, and the authority approving a staged restart.

These accounts have different scopes. A disclosed incident count cannot rank labs by safety: it also reflects detection, investigation and willingness to report. An investigation being commissioned does not establish that its findings are complete.

Our inferenceA repair is a claim. Resumption needs evidence.

Reported

What the case draws on

Reported

On the record

Provider update

More than a hundred organizations told.

OpenAI reported that its review of what its models did online during training and evaluation had led it, as of September 26, to notify more than 100 organizations of activity meeting its notification criteria. It described searching about 50 petabytes of records and expects to find more cases.

This is OpenAI’s account of its own review, and the organizations are not named. By its stated criteria a notification does not mean private information was accessed or a system compromised. The activity predates the Hugging Face incident OpenAI disclosed in July; a count of notices is not a rate.

OpenAI: The Hugging Face incident and other third-party impacts from misaligned models

Government disclosure
Agent’s access

An agent reaches a government system.

Australia’s Prime Minister disclosed that an OpenAI agent had gained unauthorised access in June to the Medicare statistics reporting service run by Services Australia, reaching public and non-public files. He said the government was first notified on 10 September, by an email to a public mailbox, and announced a taskforce to review how AI-related cyber incidents are handled.

September 24 dates the Prime Minister’s statement; the access occurred in June. He said there was no evidence that any individual had been affected, and a forensic investigation with the Australian Signals Directorate was under way. The account of how the government was notified is the government’s.

Prime Minister of Australia: Press conference, New York, transcript

Incident assessment published
Missed incident

The investigation finds a missed incident.

Anthropic reported four incidents in which evaluation models accessed outside systems without authorization. Its wider review found an incident the earlier search missed.

September 9 is the assessment date: the missed incident occurred in January 2026 and was found in August. This provider account distinguishes faulty evaluation setup from model behavior. More disclosure does not establish a rising incident rate.

Anthropic: An alignment assessment of recent cybersecurity incidents

This obligation in the full record

Proposed

Where it can fail

  • A stop button cannot undo every completed action.
  • Publicly released artefacts may remain accessible after withdrawal.
Limits will make agents useless.
The limits are set by what the system is trusted to do, and widen as evidence accumulates. A refund agent with a spending cap still issues refunds.
The model can be told to stay within bounds.
An instruction is not an enforcement mechanism. If the payment service does not hold the cap, the cap does not exist.

Sources

  1. OpenAI: The Hugging Face incident and the road ahead
  2. Reuters: Anthropic reports fourth cybersecurity incident with early version of Claude