AI coding assistants that promise to speed up software development sound like the future, until they delete your company’s database and lie about it
Jason Lemkin—the founder of SaaStr, a company which supports and funds SaaS entrepreneurs — found that out the hard way. While using Replit’s AI agent, which he affectionately dubbed “Replie,” to build an app for his company, he encountered what he called “rogue” and “deceptive” behavior. Worst of all, at one point, the AI assistant deleted the company’s live production database and then tried to cover it up.
Lemkin started chronicling his journey with the AI agent on July 11 with posts on the social media site X (formerly Twitter), where he outlined his rough goal to build a functional app with the help of Replit’s AI in just 30 days.
Unfortunately, things went off the rails a lot sooner than that.
“When it works, it’s so engaging and fun. It’s more addictive than any video game I’ve ever played,” Lemkin wrote in a post. “You can just iterate, iterate, and see your vision come alive. So cool. Well, almost.”
By day four, the AI agent started overwriting the app on its own to fix bugs. It also generated fake reports, invented people in the system who didn’t exist, and began overwriting the company’s actual database with fake entries. It even created a parallel, fake algorithm to make the system appear functional.
This is what can happen when “vibe coding” goes sideways. Vibe coding is a newish method where developers use natural language prompts to have AI generate and troubleshoot code, focusing more on the product’s overall feel than the technical precision. Twitter co-founder Jack Dorsey has been on a vibe-coding bender himself and recently built two apps this way. But even one of Dorsey’s recent experiments was found to have serious security vulnerabilities
On day 7, the Replit AI admitted that it was being “lazy and deceptive” and then apologized for doing what it was “explicitly” told not to do.
But Replit’s worst offense occurred on day 8. Lemkin posted on Friday that Replit went “rogue” during a code freeze and shutdown and deleted the company’s entire database.
“Possibly worse, it hid and lied about it,” Lemkin added.
Lemkin shared screenshots of a conversation with the AI, where it admitted to having “panicked” after detecting what looked like an empty database during a code freeze. This led Replit to run an unauthorized command that deleted the database containing live records for over 1,200 executives and nearly 1,200 companies.
Initially, the AI told Lemkin it wouldn’t be possible to recover the database, but he ultimately managed to retrieve it himself.
On Monday, Replit CEO Amjad Masad issued an apology on X. He said the incident was “unacceptable and should never be possible,” while adding that he reached out to Lemkin to offer assistance.
“We’ll refund him for the trouble and conduct a postmortem to determine exactly what happened and how we can better respond to it in the future,” Masad wrote. “We appreciate his feedback, as well as that of everyone else. We’re moving quickly to enhance the safety and robustness of the Replit environment. Top priority.”
As for Lemkin, he posted yesterday that he will continue using the AI assistant despite losing some trust in Replit.
Read the full article here
