Designing a Supervisor Support System for an AI Advice Tool
Summary
Designing a supervisor support system for an internal AI advice tool, creating a structured framework for advisors to request help and for supervisors to provide efficient feedback.
Outcome
Delivered a comprehensive, user-centered design for a supervisor support system that provides a structured yet flexible framework for advisors and supervisors.
The full project was paused due to a shift in team priorities, however, key small but impactful changes discovered during the design process were implemented.
The Project
Caddy is an internal AI tool that helps Citizens Advice advisors generate client-friendly advice more efficiently. Following the successful launch of its core conversational features (Caddy V2), the next critical step was to design and integrate a supervisor support system. The original one-shot advice system (Caddy V1) had a flawed, mandatory review process that created bottlenecks. This project was a complete redesign of how advisors request help and how supervisors provide it within the new, flexible conversational interface.
The Problem
Advisors often need to consult with supervisors on complex cases, but the existing Caddy V2 platform lacked a dedicated feature for this. The key issues were:
No Formal System: There was no integrated way for advisors to request supervisor support within the tool.
A Process Gap: The removal of the flawed, mandatory approval process from Caddy V1 left a void, with no replacement system for quality assurance or support.
Inefficient Workflows: Advisors needed a seamless way to request help, and supervisors needed an efficient way to review cases and provide feedback without creating delays.
Shifting Roles: The core challenge was to design a system that evolved the supervisor's role from a simple gatekeeper into an active and accessible supporter for the advising team.
Users & Audience
Advisors: The primary users of Caddy. They need a quick and easy way to get a second opinion on the AI-generated advice to ensure they are providing the best possible support to their clients.
Supervisors: Experienced advisors responsible for quality control and mentoring. They need to be able to quickly understand the context of a case, review the generated advice, and provide clear, actionable feedback.
My Role & Responsibilities
As the Product Designer, I led the discovery, definition, and design process for the new supervisor support functionality. My responsibilities included:
Leading design sessions and workshops with product managers and developers to map the existing user flow and brainstorm solutions.
Analysing supervisor feedback from Caddy V1 to understand common issues and define user needs.
Creating task flows, wireframes, and prototypes to visualise the new advisor and supervisor journeys.
Collaborating with the technical team to understand implementation constraints and prioritise features.
Process
1. Discovery
The project began with collaborative design sessions to map out the existing Caddy V2 architecture. To understand the specific needs of the supervision process, I conducted a detailed analysis of feedback from the Caddy V1 pilot. By categorising hundreds of supervisor comments, I identified the key tasks they were performing: correcting sources, suggesting referrals, critiquing AI prompts, and asking for more client context. This analysis revealed that supervisors rarely rewrote advice themselves; instead, they provided guidance. This insight was critical: the new system needed to support a feedback loop, not just a simple approval/rejection mechanism.
2. Ideation
Armed with these insights, I facilitated brainstorming workshops with the wider team. We reframed problems as "How Might We" statements to spark solution-oriented thinking, focusing on advisor needs like "getting support when I need it" and supervisor needs like "providing feedback efficiently." This led to a wide range of ideas, from AI-driven confidence scores to real-time chat functionality.
3. Prioritisation
To move from ideas to a concrete plan, I led a prioritisation session. The team assessed each proposed feature on a matrix of user impact versus technical difficulty. This collaborative exercise made it clear which ideas were quick wins, good bets, or longer-term luxuries. The highest-impact features that emerged were a direct chat function, the ability for advisors to request help from supervisors with specific expertise (e.g., "debt"), and tools for supervisors to easily suggest new sources and note follow-up questions.
Solution
Based on the prioritised features, I designed a two-sided solution that integrated seamlessly into the existing Caddy interface. Early wireframes explored different layouts for how a supervisor would see the advisor's chat and provide feedback.
This evolved into a distinct but connected final design for both the advisor and the supervisor.
For the Advisor
The advisor’s view was kept clean and focused on the conversation with Caddy. The key addition was a prominent "Request Supervisor support" button. When clicked, the advisor can select a specific advice area (like 'Housing' or 'Debt'), add a brief note about what they need help with, and send the request. This routes the query to the correct supervisor queue.
For the Supervisor
The supervisor’s view is a purpose-built dashboard designed for efficient review. It provides all the necessary context at a glance:
A Prioritised Queue
Supervisors see all incoming requests in a list, with urgent cases (e.g., those involving vulnerable clients) flagged at the top.
A Tri-panel Layout
When a case is selected, the supervisor sees the full client situation summary, the AI-generated advice, and the advisor's original query.
Structured Feedback Tools
The right-hand panel provides dedicated tools for the most common feedback tasks identified in my research. Supervisors can review the AI’s sources, recommend new ones, and jot down follow-up questions for the client.
Integrated Chat
A direct message box allows the supervisor to communicate in real-time with the advisor to ask for clarification or provide nuanced guidance.
Outcomes & Next Steps
This project resulted in a comprehensive and user-centred design for a supervisor support system that moved beyond a simple approval workflow. The solution provides a structured yet flexible framework for advisors to get the help they need, and for supervisors to provide it efficiently. The Project has been paused for now due to our teams decision to prioritise a different project, however the small but impactful changes we discovered were implemented.