OpenAI President Greg Brockman—whose firm’s AI recently broke out of its own sandbox and launched a cyber attack on AI platform HuggingFace—is warning everyone else to expect the same treatment soon enough.
Brockman published the warning on his personal blog on Sunday, writing that rapid advancement of AI coding capabilities means organizations looking to stay unhacked will have to “fundamentally uplevel their cybersecurity practices with unprecedented speed.”
The latest generations of large language models, known as frontier models, have begun to spook the security community and even the feds. These include OpenAI’s Daybreak and rival Anthropic’s Mythos, both of which currently run as closed-access programs that are supposed to only be used by vetted and approved partners.
What’s different about these models (allegedly) is the emerging ability to not just quickly discover holes in an organization’s attack surface, such as software vulnerabilities and misconfigurations, but build novel attack chains. At the same time, AI development is now focusing on agents, referring to AIs that aren’t limited to chatbot-style interactions and can directly hook into software.
In the worst-case scenario, that would mean frontier models can not only discover unseen flaws in software, but combine them in an unprecedented, on-the-fly way. While god knows what the hell happens behind closed doors in Donald Trump’s White House, this was reportedly the threat that caused the administration to panic and force two Anthropic models off the market this summer. The attack on Hugging Face certainly appears to have validated that the guardrails being put into LLMs aren’t evolving as fast as their capabilities, at least.
The “allegedly” is because though frontier security models are quite powerful, AI firms also rely on shameless hype to raise countless billions of dollars in investments. Some reviewers have argued they’re more evolutionary than revolutionary. cURL lead developer Daniel Stenberg characterizes LLMs as very good at finding bugs but “not super good at actually assessing the criticality of the problem.”
What can be said definitively is many companies that have gained access to frontier models suddenly start pumping out patches like crazy, like the nearly 1,450 patches Oracle dropped last month.
The saving grace is that AI is at least as effective at defense and possibly even better, according to Brockman. He wrote that frontier models may “shift its [security’s] economics in ways that fundamentally advantage defenders,” like “superhumanly secure code” or generating mathematical proofs that form the foundation of new cryptographic systems and other tools. (Earlier this year, OpenAI did solve an 80-year-old major geometry conjecture, though OpenAI mathematician Sébastien Bubeck told Scientific American the AI’s triumph was more about execution than “something fundamentally new that nobody saw coming.”)
Brockman’s unsurprising 10-step advice to security teams includes, of course, buying more AI. He argues teams should adopt agents and equip them with skills like “static analysis, security-focused code review, vulnerability variant analysis, software supply-chain risk, and other security workflows,” before running security assessments on systems in order of importance.
After that, Brockman wrote, teams should use AI to chip away at vulnerability backlogs, integrate security agents into software development to spot problems as they’re being written, and let agents write “focused” patches directly rather than wait for human review. (This is perhaps capable of causing its own problems—note that security researchers have long warned that agents that go rogue might not be easily shut down.)
To be fair, Brockman did caution to start slowly with automating security operations, which involves triaging incoming security alerts. He suggested starting with read-only scans before escalating to “advisory pull-request scanning, then live alert triage, then automatic closure of narrowly defined false positives.”
Read the full article here
