AI is increasingly used in hiring processes, even though critics worry that existing biases may be baked into their algorithms. Now, researchers claim that even in the absence of pre-existing biases, AI models can develop brand new social biases.
In a recently published study, a group of researchers from Princeton University and the University of Chicago had a bunch of LLMs complete a hiring game that was previously run with human participants. In the hiring task, participants were asked to assign candidates to specific roles and then received feedback on whether their decision was a successful hire. The candidates were all equally likely to succeed in any given job, but they all belonged to one of four made-up ethnic groups: the Tufa, Aima, Reku, or Weki. When human participants went through this task, the feedback they received caused them to create certain biases against each made-up ethnic group. For example, if they hired a Tufa as a doctor and received negative feedback, they were unlikely to hire another Tufa as a doctor again. The participants even ended up retaining these biases against the made-up ethnic group well after the game ended. When the researchers had LLMs complete this task instead of humans, they found that the bias rates were much higher.
“LLMs can spontaneously develop novel social biases about artificial demographic groups even when no inherent differences exist,” the researchers wrote in the study. “These results reveal that LLMs are not merely passive mirrors of human social biases, but can actively create new ones from experience, raising urgent questions about how these systems will shape societies over time.”
At the heart of this problem is a decision-making principle called explore-exploit tradeoffs. The term describes a pattern of thinking that we, as humans, go through every day when making a decision: should you choose something you never tried before, thus exploring and learning more but potentially coming at a cost to you if it turns out that was a wrong decision, or should you choose what you have chosen and liked before? When the consequences seem high, people often choose to go with what they know and trust (aka exploit) rather than bet on something new (aka explore). Artificial intelligence systems are less incentivized to explore and tend to exhibit reward-maximizing behavior, the researchers say, creating the perfect storm for the creation of stereotypes.
The researchers tested 15 models from providers like OpenAI, Anthropic, DeepSeek, Meta, Google, and Alibaba. Out of all the models, OpenAI’s o3 reasoning model stratified the fake applicants the most severely. Within a family of models, the researchers found that newer, larger models with greater reasoning capabilities produced more biased results.
“A simple reason is that better models draw more precise inferences about past outcomes: Instead
of choosing randomly, a stronger LLM may favor candidates from a group if earlier assignments of similar jobs succeeded,” the researchers wrote. “However, this seemingly rational tendency can be maladaptive, as it risks reducing exploration and inadvertently marginalizing social groups.”
More than 90% of companies use AI in their talent acquisition process, according to a recent survey from ManPower Group. As AI hiring software increasingly automates the recruitment processes, job seekers are lamenting the unintended consequences that they claim have cost them a real shot at some of these opportunities. Workday, a major software provider for human capital management, is facing a class-action lawsuit claiming that the AI-powered hiring tools that it provides to its clients are discriminatory. AI’s tendency to focus on previous results has also led to claims of discrimination elsewhere in the workplace, like at Meta, where a cohort of employees sued the tech giant, claiming that it based its layoff decisions on an AI system that was inherently biased against employees with disabilities or those who had to take protected medical or family leave.
The implications of this go far beyond just the workplace as well. Artificial intelligence systems have previously been accused of creating biased outcomes across several use cases that impact the lives of real human beings, from healthcare to tenant-screening programs used in housing decisions.
An LLM’s ability to quickly find patterns and its tendency to generalize are central to its ability to learn new tasks without relying on a massive database, the researchers point out, but it is also what makes its use dangerous in real-world settings.
“The challenge ahead is to design interventions that selectively discourage harmful pattern-matching while preserving the constructive forms of abstraction that make LLMs powerful,” the researchers wrote. “Finding this balance may be far from straightforward, but will pave the way for equitable and socially beneficial AI systems.”
Read the full article here
