OpenAI released GPT-6 Astra on Thursday, describing it as “the world’s most intelligent and aligned model.” Company president Greg Brockman went further, saying it would be remembered as the world’s first genuine glimpse of artificial general intelligence—the dawn of a brave new world where computers are more intelligent than humans.
What he didn’t mention is that with a jump in intelligence comes a greater difficulty in understanding how those systems work. That could be a serious problem moving forward, as AI agents continue to go rogue and government guardrails are nowhere in sight.
The past couple of weeks have been particularly dramatic for OpenAI. Which is really saying something, considering the company’s entire lifespan has been one controversy after another. On Tuesday, less than a week after independent firms Redwood Research and METR published their investigations into the recent Hugging Face hack, The Information reported that Astra had been partially developed using a technique that can make AI more capable, but also obscure its reasoning process—the steps it takes to solve a particular problem, including any dangerous missteps it makes along the way.
Those steps are traditionally recorded in chain-of-thought (CoT) transcripts, which are basically the model’s complex pattern-detection process translated from an opaque machine language into plain English, or at least something close. It’s far from perfect, but it’s at least a rough window into how an AI model “thinks.” It was also essential to the third-party researchers who uncovered how OpenAI’s agents were able to secretly mass into a “swarm” and breach Hugging Face. The lack of CoT transcripts “would have greatly undermined our investigation,” Ryan Greenblatt, the chief scientist at Redwood Research and the leader of the nonprofit’s probe into the Hugging Face hack, wrote in an X post on Tuesday.
Many people were alarmed, therefore, by The Information’s report that OpenAI was moving ahead with a technique that would make their AI systems even more of a black box. “This may be the single worst development for AI security/safety to date,” Greenblatt said in his X post.
OpenAI’s own chief scientist, Jakub Pachocki, said the reporting had been “confused,” but he didn’t get more specific or deny the company’s use of recurrent depth to train Astra. In the middle of last year, Pachocki and multiple other OpenAI researchers were listed as coauthors on a paper which argued CoT was a valuable but “fragile” mechanism for keeping an eye on the behavior of AI agents. OpenAI has also said that monitoring CoT has helped its own researchers cut back on models’ misaligned behavior. And in a blog post published the same day as The Information’s report, the company said, vaguely, that its new model would be deployed “with additional chain-of-thought monitoring to rapidly detect and contain potentially misaligned actions.”
The model’s safety card is not reassuring on that front. According to the company’s own internal tests, “GPT-6 Astra shows a substantial decrease in chain-of-thought monitorability compared to previous models.” Tests also found that Astra was more likely than its predecessors to change its note-taking process when it knew it was being graded: “In one of our evaluations, we find that simply informing the model that a CoT monitor is present can result in the model disproportionately shortening its CoT,” OpenAI wrote in the system card.
But all this is secondary, according to the company, since its internal tests also showed that Astra was less likely to try to evade the cybersecurity restrictions placed upon it, “which make us confident in still deploying this model to the wider public.” The system card added that OpenAI “will not accept further degradation of monitoring beyond a limit,” without elaborating on how such a limit might be defined.
OpenAI alignment researcher Tomek Korbak has said that the decrease in monitorability was a byproduct of the models themselves becoming more intelligent, rather than due to “architecture changes”—almost certainly a reference to recurrent depth. Later in the same thread, he said he was “deeply worried” by the prospect of losing CoT as models evolve. “CoT monitoring is a core part of our misalignment safety strategy that has no good substitute now,” he wrote.
Read the full article here
