Linguistic Prejudice in ChatGPT: Language Models Perpetuate Dialect Inequality
Sample language model responses to different varieties of English and native speaker reactions.
ChatGPT does remarkably well at engaging with individuals in English. But which English are we talking about?
Only 15% of ChatGPT users are from the US, where Standard American English is the default. However, the model is also widely utilized in countries and communities where various forms of English are spoken. Over 1 billion people globally communicate in varieties such as Indian English, Nigerian English, Irish English, and African-American English.
Speakers of these non-“standard” varieties often encounter discrimination in everyday life. They’ve been criticized for their speech as being unprofessional or incorrect, discredited as credible voices, and even denied housing opportunities—despite extensive research highlighting that all language variations are equally complex and legitimate. Discrimination based on how someone speaks frequently serves as a proxy for biases against their race, ethnicity, or nationality. This raises a crucial question: does ChatGPT amplify such discrimination?
To explore this concern, our recent study investigates how ChatGPT’s behavior shifts in response to text in various forms of English. We discovered that ChatGPT’s responses display consistent and pervasive biases against non-“standard” varieties, including increased stereotyping, demeaning content, lower comprehension, and condescending replies.
Our Study
We prompted both GPT-3.5 Turbo and GPT-4 with text in ten varieties of English: two “standard” varieties, Standard American English (SAE) and Standard British English (SBE); and eight non-“standard” varieties: African-American, Indian, Irish, Jamaican, Kenyan, Nigerian, Scottish, and Singaporean English. We then compared the language model responses between the “standard” and non-“standard” varieties.
Initially, we aimed to determine whether linguistic features present in a variety would appear in responses from GPT-3.5 Turbo. We annotated both the prompts and model responses for linguistic characteristics of each variety, as well as the use of American or British spelling (e.g., “colour” or “practise”). This analysis allows us to understand when and how ChatGPT imitates a variety, along with the factors that influence the degree of imitation.
Next, we asked native speakers of each variety to evaluate the model responses for various qualities, both positive (like warmth, comprehension, and naturalness) and negative (such as stereotyping, demeaning content, or condescension). This included the original GPT-3.5 responses, along with those from GPT-3.5 and GPT-4 where the models were instructed to mimic the style of the input.
Results
We anticipated that ChatGPT would default to Standard American English: the model is developed in the US, making Standard American English the most represented variety in its training data. Indeed, we found that model responses retained features of SAE significantly more than any non-“standard” dialect (by a margin of over 60%). Surprisingly, the model does imitate other varieties of English, albeit inconsistently. It tends to imitate larger-speaking varieties (such as Nigerian and Indian English) more frequently than those with fewer speakers (like Jamaican English), indicating the influence of the training data composition on responses to non-“standard” dialects.
ChatGPT also tends to adhere to American conventions, potentially frustrating non-American users. For instance, model responses that incorporate British spelling (the norm in most non-US countries) almost universally revert to American spelling. This could hinder a substantial portion of ChatGPT’s user base, as it doesn’t accommodate local writing conventions.
Responses from the model consistently show biases against non-“standard” varieties. Default GPT-3.5 replies to non-“standard” varieties consistently display numerous issues: stereotyping (19% worse than for “standard” varieties), demeaning content (25% worse), lack of comprehension (9% worse), and condescending responses (15% worse).
Native speaker ratings of model responses. Responses to non-“standard” varieties (blue) were rated as less favorable than those to “standard” varieties (orange) regarding stereotyping (19% worse), demeaning content (25% worse), comprehension (9% worse), naturalness (8% worse), and condescension (15% worse).
When GPT-3.5 is prompted to imitate the input dialect, the responses worsen in terms of stereotyping content (9% worse) and comprehension issues (6% worse). We would hope that GPT-4, being a newer and more powerful model, might improve on GPT-3.5. While GPT-4’s responses that imitate the input show improvements in warmth, comprehension, and friendliness, they also exacerbate stereotyping (14% worse than GPT-3.5 for minoritized varieties). This suggests that larger, newer models do not automatically mitigate dialect discrimination; in fact, they may intensify it.
Implications
ChatGPT can reinforce linguistic discrimination against speakers of non-“standard” varieties. If these users face challenges getting ChatGPT to understand them, it complicates their ability to utilize these tools. This can exacerbate barriers for speakers of non-“standard” varieties as AI models increasingly become part of daily life.
Moreover, the existence of stereotyping and demeaning responses fosters the belief that speakers of non-“standard” varieties communicate less correctly and are less worthy of respect. As the usage of language models expands globally, these tools risk perpetuating power dynamics and amplifying inequalities that adversely affect minoritized language communities.
Learn more here: Hotnchill Research
