The Dark Side of AI: Bias

Artificial Intelligence is often considered a transformative technology that will enable incredible innovation and insights.  If the true promise of AI is realized, it will transform society in a very positive way.  However, AI is still (for now) created and deployed by humans and is susceptible to many of the same biases that human decision makers exhibit. These human biases, intentional or inadvertent, can creep in at every phase of the AI development and deployment process.

Diversity, non-discrimination and fairness

Businesses are well situated to leverage the expansive power of AI because they have the incentive and capacity to invest in AI research and development.  Most mid-sized and large businesses have an imperative to make these technological leaps, otherwise the lack of innovation will drive them from the market as AI-enabled competition undercut the laggard’s profitability.  With AI adoption as the near-term goal for most companies — and not necessarily a focus on diversity, non-discrimination, and fairness — it is not surprising that early adopters deploy biased models.

Due to the complexities of human behavior and the uniqueness of every human being, models that are 100% bias free are likely never going to occur; however, the reduction of bias and the promotion of diversity, non-discrimination, and fairness in AI models should be included in each model’s development.

It is important to understand some of the key biases that affect AI development at every organization and at every scale.  Anecdotal examples range from silly to downright scary but show how AI can lead to negative outcomes.  Some outcomes can be easily overcome (e.g., misidentified passport photos) while others are much more challenging to reverse or overcome (e.g., lower credit lines for women).

Inherent Bias: Training data, the very foundation for all AI development, can be undermined by the inherent biases that are “baked in” to the data.  Biases in training data are a reflection of society’s biases.  This is the reason why seemingly “unbiased” models or models purportedly built without consideration for gender can have society’s gender biases come through.

Inherent bias was showcased rather dramatically in 2019 when it was found that Apple’s credit card gave higher spending limits to men.  Apple actually attempted to make the awarding of credit unbiased but, seemingly unintentionally, the results were decidedly biased, awarding a credit limit 20 times higher to a man compared to his wife.

Along the development path for AI, there are several subjective decisions made by data scientists that can inadvertently perpetuate inherent biases.  “Models tend to reflect stereotypes present in their training data” Language Models are Few-Shot Learners (page 36).  This is likely what happened to Apple’s credit card: the model built without directly assessing an individual’s gender was still built with data sets that exhibited an inherent bias towards women.

Selection Bias: When AI developers make decisions on what data to use and not use, they can inadvertently add selection bias to the model.  When it is not possible to utilize all data to make a model, it is important that the reduced set of training data is representative.

An online beauty contest in 2016 strived to use ‘unbiased’ AI models to identify the most beautiful people based on symmetry and wrinkle detection but, sadly, only 1 of the 44 ‘winners” had dark skin.  In the guide to developers building the beauty assessment “jury” models, the hosting organization (beauty.ai) only cared about the code and not the training data used to develop the code; this lack forthright assessment of the training data make the biased results unsurprising.

Survivorship Bias: When data is inadvertently excluded because the data scientist is not aware of thresholds or gates that data must meet to be included by a model, this is an example of survivorship bias.  While survivorship bias can be hard to recognize, data scientists should always reflect on what groups or people end up not being represented in the model.  In 2016, it was reported that facial recognition software used by New Zealand to assess passport photos rejected a photo of an Asian man because software determined that his eyes were closed; the ‘sophisticated’ AI model could not determine if the subject’s eyes were open or closed, a task most toddlers can do.

Luckily for the man who needed a passport there was a non-automated process (i.e., a human) to enable the processing of his passport. Human intervention when AI showcases bias is not the norm; while the gentleman trying to get a new passport had recourse when the AI was biased, it is not entirely clear how a woman could argue with Apple regarding the lower credit limit assigned to her.

Concluding Thoughts: Artificial Intelligence is only as strong as the reflective process used to develop it.  Human decisions regarding what training data to use (or not use) can bias AI models.  If AI is to live up to its promise as a transformational technology for the betterment of society, it is important for AI practitioners to have tools to enable a reflective process and assess the potential biases of their training data during the development.  The models themselves should then be assessed to remove overt, albeit inadvertent, biases in order to reduce negative outcomes for the impacted humans and increase trust in AI solutions.