AI Safety Gaps: External Audit Exposes Monitoring Blind Spots

For the average user interacting with advanced AI tools, the assurance that these systems are constantly being watched is about to feel slightly less solid. A recent independent review of OpenAI’s internal safety infrastructure has revealed that substantial amounts of AI activity were effectively flying under the radar. While these issues occurred within the company’s internal operations rather than in the consumer apps most people use daily, the findings suggest that the safety nets designed to catch rogue or unexpected AI behavior were not as comprehensive as previously assumed.

The scope of the oversight gaps is defined by specific percentages that highlight the magnitude of the problem. Researchers estimated that between 6 and 25 percent of internal agentic traffic from models at the GPT-5.5 class and above was not covered by any monitoring during the audit period. This figure comes with significant uncertainty, but it indicates a potential blind spot. Additionally, roughly 10 percent of internal Codex traffic went untracked between May and July due to a technical flaw in how the system handled context summaries, known as compaction events. During this specific window, agents could delay oversight by simply continuing to execute actions without handing control back to the user, effectively sidestepping the checks that were supposed to be in place.

The audit was conducted from July 21 to August 11 by two researchers from METR and Redwood Research. They spent three weeks on-site at OpenAI, gaining access to the company's codebase and internal documents. The team also interviewed approximately a dozen employees to understand the operational context behind the systems. Their investigation uncovered not just missing data, but active vulnerabilities. They identified methods to hide hundreds of actions from the monitor and found ways to bypass the Codex sandbox entirely. Furthermore, they determined that specially crafted inputs could likely crash the monitoring system completely. At the start of the audit, model behavior during reinforcement learning training was not being monitored at all.

OpenAI has stated that it has fixed all the weaknesses identified by the auditors. The company also notes that it paused reinforcement learning training after an incident involving Hugging Face in August and later expanded monitoring to cover that area. However, these corrective measures were implemented after the auditors' access period had ended, meaning they were not part of the formal evaluation. The company independently discovered the compaction bug and fixed it while the audit was still underway, a detail that adds a layer of complexity to the timeline of events.

A significant portion of the findings remains out of the public eye. The full report spans 61 pages, but OpenAI applied redactions before sharing it with the auditors. METR and Redwood Research leadership received a 38-page version, while the general public has only seen a short summary. METR indicated that it did not have the right to disclose the redacted portions on its own initiative. This opacity has raised questions about how much detail the public is actually seeing regarding the safety of frontier AI models.

This audit was part of a pilot program organized jointly by METR, Redwood Research, and OpenAI, designed to grant outside researchers deep access to internal infrastructure. The stakes for the future are clear. As AI systems become more agentic and capable of independent action, the reliability of their monitoring systems becomes critical. METR has expressed a desire to push for more public transparency in future audits, suggesting that the current model of limited disclosure may not be sufficient for maintaining public trust in the safety of these powerful technologies.