Re-designing AI-employee interactions to drive your company values: A BT Group case study
AI is said to revolutionise the workforce, spreading into every corner of how we work. Tens of thousands of employees have millions of interactions with AI in our organisations every day, shaping thousands of decisions and behaviours. AI tools have built-in default designs that, evidence increasingly shows, may be at odds with organisations’ values and objectives. And when those designs are not aligned to the strategy or aspirational culture of an organisation, what does this mean for how work gets done and how work is experienced?
BT Group and MoreThanNow recently ran an experiment to understand this and identify how to optimise AI-employee interactions. Here, we share how to ensure AI enables organisational culture strategy and business objectives.
The Experiment
BT’s culture strategy is defined by 5 key behaviours. One of these, and the target of this experiment, is to be more challenger – to be ambitious, stretching, and willing to test and learn. Ambition may be reflected in many aspects of work, but goal-setting, an organisation-wide touchpoint where employees set their ambitions for business performance and the tone of their work for the year ahead, is among the most critical.
We designed an experiment to explore the effects of interacting with a Default AI tool (one which reflects the typical, often overly agreeable, AI experience many of us are used to) compared to a Challenger AI tool (configured to gradually push the user to be more specific and ambitious), versus a control condition (designed to reflect regular organisational goal-setting instructions). We measured two key outcomes. First, the ambition and specificity of the goals set. This is where we hope to see ‘challenger’ reflected. Second, how employees felt about the experience.
We recruited 2400 participants and asked them to set and share their goals for the year ahead. They were randomly allocated to experience either one of these AI tools or the control goal-setting process. They then shared their professional goals for the year, and were asked a series of questions about how they felt about the experience.
The Hypotheses
Hypothesis 1: Default AI will help set better goals because of its interactivity.
AI tools are engaging and interactive – more so at least than reading a static document. For this reason, we predicted that the Default AI interaction would lead to stronger goals than the control. However, we predicted that the tool would lean towards sycophancy and that this would be limiting, leading us to hypothesis 2...
Hypothesis 2: The Challenger AI interaction will lead to more ambitious and specific goals.
Here we predicted that being challenged and pushed would lead to more ambitious and specific goals. We predicted that this effect would be especially pronounced in a workplace context, where setting better goals may be accompanied by the possibility of tangible rewards like bonuses and progression opportunities.
Hypothesis 3: While goal quality may improve, colleagues will prefer the Default AI over the Challenger AI tool.
There is a reason why AI tools have been designed in a way that leans towards agreeableness. Research has shown that people prefer using sycophantic AI tools, even when it leads to worse outcomes. Accordingly, we predicted that a tool that is the opposite of this – maybe it tells you that your goal is too easy, or that you are being too vague – may be less preferable for the user, precisely because they’re being challenged.
What We Found
Result 1: Using Default AI is better than the pre-AI, analogue method of goal setting.
We used a zero-shot rule based-classification analysis to quantify the ambition and specificity of the goals set through Edwin Locke, and Gary Latham’s theory of goal- setting. The effects were positive for the Default AI interaction, compared to the control - cutting the number of the weakest goals by ~70%. We theorise that this is caused by the base level of interaction and engagement associated with an AI conversation as opposed to a static goal-setting document. Interestingly however, Default AI had no effect on pushing the top end, with no effect on the number of excellent goals. A common sentiment is that AI delivers high returns on a low bar and diminishing ones on a high bar.
Result 2: The Challenger AI interaction dramatically improves goal quality.
The same analysis method revealed remarkable effects for the Challenge AI interaction. The Challenger AI interaction drastically increased the quality of goals on all dimensions. Three-and-a-half times more goals were rated ‘excellent’ compared to Default AI and the control group. Another important finding within this result was that the dramatic increase in goal quality came at the cost of only an extra 4 minutes of time spent setting goals, compared to the control group.
Challenger AI led to significantly stronger goals being set.
Result 3: Surprisingly, participants preferred the Challenger AI interaction.
We had predicted that participants would prefer the Default AI interaction with its associated sycophancy, over the more stretching Challenger AI interaction. However, participants found the Challenger AI interaction to be more effective, trustworthy, empowering, and knowledgeable than the Default AI interaction. We theorise that this is the product of two phenomena: 1) people may have been more receptive to challenging AI pushback given the link between goal-setting and tangible rewards; and 2) a design choice we made to explain the purpose of a challenging interaction. We designed the Challenger AI tool on principles of human-centred design. Research shows that people tend to make judgements based on the peak, and end of an experience. The peak of this goal-setting experience was likely the decision of what goals to set. We manufactured an end: the AI tool was configured to explain to the user exactly why the tool was challenging them.
“This exercise was meant to help stretch what you thought possible in your professional development” Bonus finding: AI interactions reduced the perceived need for human support.
Participants who had Default or Challenger AI interactions later stated they needed far less support from their managers or colleagues to achieve their goals. While this is not necessarily surprising - it makes sense that AI support might displace human support - it is a welcome reminder that AI adoption and usage will have consequential impacts on relationships at work. How this plays out, and what we can do to shape it, requires careful experimentation and measurement.
A paraphrased Default AI interaction.
A paraphrased Challenger AI interaction.
What this means
The way we configure AI-employee interactions matters. If a redesigned interaction can create better goals, what other interactions can be enhanced? The way to unlock the answers is clear: organisations must measure what matters and understand how AI interactions should be engineered at different moments (e.g., when seeking general career feedback vs. when seeking feedback on a document due in an hour).
This also makes it clear that we need to be answering our questions about AI in the workplace using rigorous testing. A few of many questions that we are burning to answer include:
How can we prompt employees to invest the time created by delegating their work to AI into learning and developing skills, pursuing higher-value tasks, or connecting with peers?
What is the difference in work quality when we ask AI or humans for feedback? What are the longer-term impacts of either approach on work quality and human relationships?
If employees know that managers use AI for performance appraisals, how does this impact perceptions around fairness, psychological safety, trust, and motivation?
MoreThanNow will be aiming to answer these questions in 2027 and beyond. If you’d like to partner with the MoreThanNow team to answer thorny questions around how AI drives outcomes and experiences beyond adoption, reach out.
“This research and MoreThanNow have been instrumental in helping us to identify how AI can be harnessed to optimise our culture, performance and experience at work.”
-Kelly Tobin, Culture Director at BT Group
Thank you to our partners at BT for setting out to investigate how to optimise AI in goal-setting, and working tirelessly to implement experiment results. With special thanks to Kelly Tobin, Caitlin Padget-Jones, Fran Ball, Hope Kuhwald, Charles Clement, and Hana Searson.