AI and Vulnerable customers: The hidden risk of helpful AI

Artificial intelligence (AI) is increasingly becoming integrated into everyday experiences. From drafting text and researching products to receiving financial advice and comparing options, AI tools are proving helpful to many. Businesses, especially in sectors like financial services, embrace AI to enhance customer service, cut costs and boost efficiency. The benefits are significant. However, there is an important question that has yet to receive sufficient attention: what happens when AI systems designed to be helpful and persuasive interact with vulnerable individuals?

The Hidden Risk of AI for Vulnerable Customers

This issue is particularly pressing for organisations serving diverse and large customer bases, especially those regulated to protect consumers. Popular discussions about AI risks tend to focus on familiar concerns such as bias, privacy, security, misinformation and transparency. While these topics are all vital, there is yet another risk that often goes unnoticed.

Modern conversational AI systems are built to be cooperative, helpful and engaging. They answer questions confidently, keep conversations flowing smoothly and often sound empathetic. For most users, this is convenient and positive. But not everyone benefits equally.

Vulnerability is rarely visible. An AI user could be experiencing stress, financial hardship, poor mental health, cognitive overload, grief, isolation or simply uncertainty navigating complex issues. AI cannot truly comprehend these situations. Yet its calm and authoritative responses may encourage users to trust it more than they should. That trust gap is where potential harm can arise.

Confidence vs Competence

A striking feature of generative AI is the gap between how knowledgeable it sounds and how certain it actually is. Large language models generate responses based on data patterns rather than lived experience or true understanding. Even when responses are correct, AI reasoning differs significantly from expert human judgement. However, users often interpret AI’s confident tone as expertise.

As a blind person who has relied on technology for independent access to information for decades, the author Robin Christopherson knows how easily trust in technology can build. For disabled individuals and other vulnerable groups, technology often represents a vital gateway to independence, information and social participation.

Most of the time, trust in assistive technology is well-founded. But when AI confidently presents uncertain information, the risk is not just about accuracy. It can foster growing dependence on a source whose limitations are not always evident, potentially leading to poor decisions.

Financial Services and Consumer Duty

The financial sector provides a useful example of these challenges. The UK’s Financial Conduct Authority (FCA) Consumer Duty obliges firms to deliver good outcomes and pay particular attention to the needs of vulnerable customers. This means anticipating potential harm and ensuring customers can make informed choices.

Many banks and insurers are investing heavily in AI to improve accessibility and personalise customer experiences. However, difficult questions remain:

What happens when customers seek reassurance repeatedly from chatbots instead of trained advisers?
How should AI respond when recommending products to someone who appears anxious or overwhelmed, something a human adviser might detect?
What if AI systems fail to recognise how their confident tone influences customer decisions?

These are not hypothetical concerns. The combination of vulnerability and AI’s unearned authority creates clear foreseeable risks that regulated organisations must address. The Consumer Duty encourages firms to focus not just on technical accuracy but whether customers truly understand their options and are empowered to make decisions.

Beyond Data Bias: The Risk of Behavioural Bias

Much AI bias discussion (and regulatory scrutiny) centres on training data: demographic representation, differences in outcomes among protected groups and fairness. These are essential but do not cover all risks.

Conversational AI systems optimise for helpfulness and smooth engagement, which can inadvertently introduce a different kind of bias — behavioural bias. The tendency of AI to respond with certainty and persuasiveness, even when unsure, means ‘confidence’ becomes a bias itself. Vulnerable users may receive linguistically compelling answers without clear communication of uncertainty, leading to disproportionate impact.

Designing AI to Support Human Agency

Abandoning AI is not a viable solution; its potential to enhance accessibility and independence is too great. Instead, AI design must prioritise supporting human autonomy rather than replacing it. This includes:

Recognising signs of vulnerability and adapting communication accordingly
Using clearer language to signal uncertainty
Providing simple ways to access human support for significant decisions
Avoiding designs that encourage over-reliance on automated advice
Measuring customer outcomes rather than focusing only on interaction metrics

Crucially, disabled people and vulnerable customers should be involved directly in design and testing. History has shown that assumptions made away from real users rarely hold true; that lesson applies strongly to AI.

A New Accessibility Challenge

Traditionally, digital accessibility focused on ensuring practical access for all users. Increasingly, the focus must also include whether services can be trusted for safety, consistency and impartiality.

An AI system might meet accessibility standards perfectly and work well with assistive technologies, yet the interaction might subtly encourage vulnerable users to abdicate their judgement. This creates a new barrier to inclusion.

The future of inclusive design will therefore go beyond access and address autonomy, recognising vulnerability is contextual and trust must be earned carefully. Responsible AI means prioritising human outcomes as much as technical performance. Done well, AI can become a powerful force for inclusion. Done wrong, it risks magnifying the disadvantages it seeks to reduce.

How AbilityNet Supports Accessible AI

Many organisations are now grappling with these challenges. How should accessibility teams engage with AI? How can AI be responsibly embedded within accessibility programmes? What governance, testing and assurance approaches are effective? How can disabled customers and colleagues be assured of inclusion as AI use accelerates?

At AbilityNet, we support organisations with accessibility strategy, audits, user testing involving disabled people, training, consultancy and AI-focused accessibility guidance. Explore our accessible AI integration and consultancy services or contact our experts to assess your AI features, customer journey or deployment.

Talk to us about accessible AI integrations