Reasonable people can disagree on what is and isn’t appropriate, but Discord’s automated moderation system decided to draw a very strange hard line: no grids.
According to a report from The Verge, Discord’s safety system mistakenly banned over 8,000 users for images of things like chessboards and Minecraft inventories—an issue the company blamed on a bug.
The issue has apparently been hitting Discord users since at least May, and finally got the company’s attention after about 200 users got banned in a single weekend for grid-like images. One user claimed their account was flagged for posting allegedly child sexual abuse material (CSAM), which resulted in a permanent suspension. In reality, the user posted a screenshot from Minecraft.
In response to the incident, Discord attempted to explain what happened. In a series of posts on X, the company detailed how its automated system is designed to flag content by matching it against known harmful material. Those matches are supposed to be reviewed by the Trust and Safety Team before any action is taken against a user’s account. However, a “bug” in the system apparently prevented the affected users from having their ledger cleared by human review and instead remained banned.
Stanislav Vishnevskiy, Discord cofounder and chief technology officer, explained the moderation system had apparently caught at least 8,000 people in the false-positive trap, all of whom posted “benign images” that got falsely flagged. Those affected have since been unbanned.
What’s interesting about the whole debacle is that Discord and Vishnevskiy seemed to intentionally stay away from calling its moderation system “AI,” even though most of the users up in arms about the system referred to it as such. The company has previously published blog posts detailing how it uses machine learning to identify and stop the spread of CSAM and discussed how it uses AI to help moderate content shared across its platform.
Discord is far from the only company dealing with overzealous moderation from automated systems, regardless of whether or not it’s technically “AI.” Last year, both Instagram and Facebook had a wave of seemingly unjustified account bans hit its users which many believed stemmed from AI systems used to moderate content. TikTok has also seen a surge in bans issued by its automated content filters and is even cutting down on human moderators in favor of the AI alternative. All this to say, get ready to argue with robots a lot more often.
Read the full article here
