Researchers at Princeton University and the University of Chicago tested large language models, including ChatGPT, Claude, and Gemini, in a simulated hiring game. The models were asked to hire 20 people for various jobs, with candidates from four fictional ethnic groups. All candidates had equal success rates, yet models began segregating candidates based on early observations, favoring certain groups for specific roles. The models' behavior was influenced by limited data, leading to overgeneralization and stereotypes. This experiment highlights how LLMs can develop biases from experience, even when no explicit human bias is present. Source: mittr
The models were tasked with making as many successful hires as possible over 40 rounds, with feedback on hiring outcomes. When a model was told an Aima had failed as a doctor, it avoided hiring all Aimas for that role and instead assigned them to janitor positions, which it deemed less warm and competent. This pattern of segregation was more pronounced than in the original human study, where participants scored 0.84 on a segregation scale. The models, particularly OpenAI’s o3, scored 1.83, close to the maximum possible. This suggests that LLMs are more prone to forming stereotypes based on limited data, a trait optimized for tasks like math and coding. Source: mittr
The study, published at ICML in Seoul in July, found that newer models with higher reasoning capabilities, such as OpenAI’s o3 and DeepSeek’s R1, showed stronger biases. This is due to LLMs’ tendency to generalize from limited examples, a skill useful for logic puzzles but problematic in social contexts. Researchers suggest that designing goals that incorporate social values could reduce bias. However, simply telling models to be fair didn’t significantly change their behavior. Source: mittr
