AI & bias at work: who gets designed in?

Why AI is not a neutral tool

There is a comforting idea that algorithms are more objective than people. They're not — they're a reflection of the data, assumptions and dominant perspectives baked into them during training. If the data collected historically over-represents one demographic, the model will treat that demographic as the default. Everyone else becomes an edge case: served less well, assessed less accurately, or simply not considered.

This matters enormously at work, where AI is now embedded in recruitment screening, performance management, pay modelling and learning recommendations. When bias is invisible in the system, the decisions it shapes can appear objective even as they quietly entrench inequality.

Who gets designed out

The groups most exposed to AI bias tend to be those already underrepresented in the mainstream data that AI is trained on. That includes:

• LGBTQIA+ colleagues, whose identities and lived experiences are frequently absent from or misrepresented in large data sets.
• Neurodivergent employees, whose communication styles, working patterns and responses may not fit the behaviours the model was trained to recognise as high-performing.
• Disabled people, where AI-driven processes may not accommodate difference in input or output.
• Anyone outside the majority demographic at the time the training data was gathered — because the majority becomes the implicit standard against which everyone else is measured.

The problem is not that these groups are unusual. The problem is that the system was designed without them in mind — or designed by a team that didn't include them at all.

The risk of treating AI as an authority

One of the most significant risks in deploying AI at work is the authority it is granted. When a system produces a ranking, a recommendation or a risk score, there is a strong organisational pull to treat that output as more reliable than a human judgement. The algorithm is assumed to have no agenda. But every algorithm inherits the agendas of the people and processes that shaped its training — they are just harder to see.

Using AI as a proxy decision-maker — especially for decisions that affect people's careers, pay or dignity — hands power to a system that may not represent the people it is assessing. It also removes accountability: if the AI decides, no one person signed off. That is not efficiency; it is diffused responsibility.

Five questions every organisation should ask

Before deploying any AI tool that affects people, ask these questions and require honest answers:

• Who benefits? Which groups does this tool serve most effectively — and is that the full range of people in your workforce?
• Who is excluded? Which groups might be underserved, misrepresented or actively disadvantaged by the tool's outputs?
• Who decided? Was the procurement and design process representative of the people most affected?
• Who is heard? Whose feedback shapes ongoing review — including from those the tool assessed unfavourably?
• Where must human judgement remain central? Which decisions are too consequential, too personal or too context-dependent to delegate to an automated system?

AI as augmenter, not authority

None of this means AI has no place at work. Used well, it can be a powerful augmenter: a thinking partner, an accessibility aid, a way to reduce cognitive load and give colleagues more capacity for the work that requires human connection and judgement. The distinction that matters is between AI as a coach or sounding board — helping a person think, draft or prepare — and AI as a proxy decision-maker — replacing human judgement on matters that directly shape someone's opportunities and experience.

Leaders who get this right use AI to augment the quality of their decisions, not to outsource them. They keep a human in the loop wherever the stakes are high — and they actively review whether AI-assisted processes are producing equitable outcomes across different groups, not just on average.

Practical steps for leaders and HR

• Audit any AI-driven process that affects hiring, promotion or pay for disparate impact across protected characteristics.
• Involve diverse employees in procurement, piloting and ongoing review of AI tools — especially those from groups most likely to be edge cases.
• Document where AI is used in decisions and ensure there is always a named human responsible for the final call.
• Train managers to scrutinise, not simply accept, AI-generated recommendations.
• Build psychological safety so employees can raise concerns about AI-generated outcomes without fear of being dismissed as "anti-technology".

Take it further

Explore the speaking topic AI, Belonging and the Future of Work, read the companion guide AI, Belonging and the Future of Work, or hear these ideas explored across episodes of the Inclusion Bites podcast.