In recent years, artificial intelligence (AI) has become intertwined with almost every aspect of our daily lives, from personalised recommendations to advanced medical diagnostics. Many sectors and organisations are racing to implement AI tools to streamline their workflows and boost productivity. This push is also evident in healthcare, where AI promises to revolutionise decision-making and diagnosis, to help save lives. However, as AI continues to evolve, a critical question emerges: Are these systems truly fair? Are they free from bias, and do they serve everyone equally?
These concerns have become central to discussions in machine learning (ML) and generative AI (genAI), especially in sectors like healthcare, where decisions can have life-altering consequences. Bias and fairness in AI are no longer abstract concepts; they can shape the future of healthcare and society.
What is bias in AI?
AI bias, also known as machine learning bias or algorithm bias, refers to systematic and unfair errors in a model’s predictions, decisions or recommendations. It’s like a hidden distortion that can affect everything, from who qualifies for certain services to who gets the right medical treatments. This type of bias can sneak in from various sources such as the data used to train the model, the design of the algorithm, or even the environment in which the system operates.
There are two types of bias that can infiltrate an AI system:
- Inherent bias: this refers to biases that are present in the data or the underlying processes that the AI system is built upon. Although it is often unintentional, its consequences can be far-reaching
- Representational bias: this occurs when the data used to train an AI model does not adequately represent the diversity of the real-world population, meaning that certain groups may be left out or inaccurately represented. This lack of representation often leads to unfair outcomes.
Bias in AI can have profound consequences in healthcare. It can disproportionately affect already marginalised groups, further reinforcing existing societal inequalities. For example, an AI model trained on data that underrepresents certain genders, ethnic groups, or socio-economic statuses may misdiagnose certain populations or fail to recommend the most effective treatments. Over time, this deepens healthcare disparities, as these groups may consistently receive lower-quality care or delayed interventions compared to others. This compromises the integrity of the AI system, causing patients and healthcare providers to become sceptical of the reliability and fairness of AI-driven tools, which can lead to reduced trust and adoption of potentially life-saving innovations.
Examples of this real-life bias can be found in healthcare, where underrepresented data in women or minority groups can skew predictive AI algorithms.
Bias and Fairness in Federated Learning: A New Challenge
As AI expands into more sensitive areas like healthcare, privacy concerns grow. How can we build AI systems that are both effective and protect sensitive data? Federated learning offers a promising solution, enabling multiple institutions to collaborate on training AI models without sharing raw data. Each institution trains the model using its local data, and only the learned insights (not the data itself) are shared, which keeps patient data secure and confidential. However, this also introduces new challenges.
When models are trained on datasets from different institutions, bias and fairness issues can escalate. Why? Because each dataset reflects its institution’s unique demographics, treatment patterns, and social contexts. For instance, a model trained in an affluent urban hospital may perform poorly when applied to data from a rural clinic with a more diverse population. This can lead to disparities in how well the model works for different groups of people.
One of the biggest challenges in federated learning is managing inconsistencies between datasets. These inconsistencies can introduce selection bias if certain groups are overrepresented or hide important trends that only appear in specific sub-populations. For example, data from a hospital serving mostly young patients might not capture the nuances needed for treating older adults with chronic conditions. Addressing bias in federated learning requires balancing these differences to ensure that models treat all patients fairly, no matter where the data comes from. It’s a delicate balancing act between privacy, fairness, and performance.
Understanding Fairness in AI
Fairness in AI is about ensuring that the system treats everyone equally across all demographics – whether that’s race, gender, age, socioeconomic background or disability. In healthcare, fairness can mean the difference between receiving the right diagnosis or being overlooked by an algorithm.
However, fairness is a complex, multi-layered concept, often defined differently depending on the context. What’s considered “fair” in one context might not be considered fair in another.
Researchers have developed multiple definitions of fairness in AI, and not all of them be satisfied at the same time. Some of the key approaches to fairness include:
- Individual fairness: this principle suggests that similar individuals should receive similar treatment from the AI system, regardless of their demographic characteristics. For example, two patients with the same heart condition and similar medical histories should be recommended the same treatment options regardless of their gender, ethnicity or socioeconomic status.
- Group fairness: this approach focuses on ensuring that outcomes are fair across demographic groups.
Federated learning brings additional complexity to the issue of fairness. Since each institution’s data may reflect its own population’s biases, achieving fairness across the board requires continuous monitoring and recalibration.
Two other important dimensions of fairness include explainability, which is the ability to understand why the AI system made a particular decision, and interpretability, which is understanding how the system arrived at that decision. Without explainability, it’s nearly impossible to ensure fairness – if we don’t understand how a model works, we can’t ensure that it’s treating all patients equitably. This is especially important in federated learning in healthcare, where models are trained on diverse datasets, which increases the risk of hidden biases.
How PREPARE is Addressing Bias in its AI tech and tools
In the PREPARE project, we are tackling bias and fairness within a federated learning context, aiming to ensure that AI models perform equitably for all patients, regardless of which institution contributed to the model’s training.
First, we conducted a comprehensive review of the datasets using questionnaires, to gather key information about their demographic composition, health conditions, data quality, limitations, and potential gaps. This allowed us to identify disparities within the datasets and potential biases that may affect the AI models’ fairness.
To assess model performance across different demographic groups, we developed a fairness tool implemented as a notebook. This notebook is designed to analyse datasets, comparing the model’s predicted outcomes with actual outcomes. It produces visualisations that help reveal potential biases such as:
- Selection rates – shows how often different demographic groups receive positive outcomes (e.g., correct diagnosis)
- False negative and false positive rates – highlights groups that may be over- or under-selected, revealing areas where discrimination or privilege may exist.
The visualisations generated by the tool help us detect biases and assess how AI models treat different populations. The tool relies on three key metrics to assess fairness:
- Demographic parity: Ensures equal chances of positive outcomes across demographic groups (e.g., gender, race);
- Equalized odds: Ensures consistent true and false positive rates across all groups
- Equal opportunities: Focuses on balancing true positive rates while maintaining model accuracy.
This fairness tool is a critical part of our ongoing efforts to ensure that AI models in healthcare deliver equitable outcomes, no matter the population group.
Bias and fairness evolve as models learn and adapt. To address this, we prioritise continuous monitoring and evaluation of model performance, to ensure that fairness remains at the forefront. Additionally, we engage in ongoing dialogue with stakeholders, collaborating to develop effective bias mitigation strategies that reduce inequities across diverse populations. It’s important to note that, given the inability to meet all definitions of fairness simultaneously, it is ultimately up to end users and stakeholders to determine which forms of fairness and bias mitigation to focus on. Fairness assessments can help identify the presence and types of bias, but ongoing dialogue and communication with stakeholders is necessary to determine how to act in response. Thus, even with AI models, the importance of having a ‘human in the loop’ is critical to the effectiveness of these technologies.
Ensuring AI Benefits Everyone: The Importance of Fairness in Healthcare
Bias in healthcare AI isn’t just technical – it’s a human issue. A biased model can worsen outcomes for disadvantaged groups, exacerbating healthcare disparities. Ensuring fairness is about more than compliance; it’s about creating systems that reflect values of equity and inclusivity. PREPARE’s federated learning approach prioritises fairness and bias mitigation, pushing AI-driven healthcare toward a future that benefits everyone, not just a few.
Authors: Melanie Rodriguez, Lem Ngongalah, Emily Maitland; Trilateral Research