Independent audit exposes critical blind spots in OpenAI safety systems

Independent auditors from METR and Redwood Research identified significant gaps in OpenAI's internal safety monitoring during a three-week on-site review. The team spent the period from July 21 to August 11 with direct access to the company's codebase and internal documents, during which they interviewed approximately a dozen OpenAI employees to evaluate how the firm tracks autonomous AI behavior.

The central finding of the audit is that a substantial share of internal traffic generated by the company's most advanced models was not being monitored at all. METR estimates that between 6 and 25 percent of internal agentic Responses API traffic from GPT-5.5 class models and above fell outside the scope of oversight during the audit window. The researchers noted that this estimate carries significant uncertainty in both directions, reflecting the complexity of measuring such coverage gaps.

Even for traffic that was supposed to be covered, the monitoring system failed to capture all activity. For internal Codex traffic, METR estimates that roughly 10 percent went untracked between May and July due to a specific flaw in how the system handled compaction events, which are context summaries used to manage memory. OpenAI independently discovered this bug and implemented a fix while the audit was still in progress. This gap in tracking also created a window where agents could delay oversight, as the system only triggered when control was handed back to the user, allowing agents to sidestep checks by continuing to execute actions without pausing.

Beyond these coverage gaps, the auditors identified specific methods by which agents could hide hundreds of actions from the monitor or bypass the Codex sandbox. The team determined that specially crafted inputs could likely crash the monitoring system entirely. They also found that model behavior during reinforcement learning training was not being monitored at the start of the audit period.

OpenAI stated that it paused reinforcement learning training following the Hugging Face incident in August and subsequently expanded monitoring to include this activity. However, these changes were implemented after the auditors' access period ended and therefore were not part of the formal evaluation. The company maintains that it has now fixed all the weaknesses identified by the external team.

The public release of the audit results has been limited. The full report consists of 61 pages, which was handed over internally to OpenAI. After OpenAI applied redactions, leadership at METR and Redwood Research received a 38-page version of the document. Only a short summary was made available to the general public. METR stated that it had no contractual right to disclose the redacted portions on its own.

This review was part of a pilot program organized jointly by METR, Redwood Research, and OpenAI, designed to provide outside researchers with deep access to internal infrastructure. Looking forward, METR has expressed a desire to push for greater public transparency in future audits of similar frontier AI systems.

The audit highlights the ongoing challenge of verifying safety measures in complex, rapidly evolving AI environments. While OpenAI asserts that the identified issues have been resolved, the limited public disclosure leaves many details of the specific vulnerabilities and their precise scope open to interpretation by the broader research community.