Service Desk AI Quality Analyst

Daily Activities:

• Monitor AI-assisted Service Desk performance including deflection resolution escalation fallback and containment rates.

• Review AI interactions failed intents fallback cases and escalated conversations to identify quality issues and recurring patterns.

• Tag classify and analyze AI failures and user support scenarios.

• Identify gaps in knowledge articles intents automation coverage and AI responses.

• Analyze Service Desk tickets trends user feedback and operational data to identify opportunities for improvement.

• Prepare dashboards reports and insights on AI and Service Desk performance using tools such as Excel and Power BI.

• Provide structured recommendations and feedback to Knowledge Management Automation AI and Service Desk teams.

• Track improvement actions and assess whether implemented changes are delivering the expected outcomes.

• Support incident request and user inquiry management as part of normal Service Desk operations.

• Participate in Service Desk queue management and ensure appropriate handling and escalation of support cases.

• Collaborate with technical and business stakeholders to understand recurring user pain points and service experience gaps.

• Contribute to continual service improvement initiatives and the optimization of AI-assisted support processes.

Business Value:

• The Service Desk AI Quality Analyst helps maximize the business value generated from AI-enabled IT support by ensuring that AI capabilities are continuously measured improved and aligned with user needs. The role delivers value by:

• Improving AI effectiveness: Identifying and addressing failed intents inaccurate responses and ineffective automation.

• Increasing automation value: Identifying opportunities to increase AI deflection and reduce unnecessary human intervention

• Improving user experience: Using user feedback ticket data and interaction analysis to identify and resolve recurring experience gaps.

• Reducing support effort: Helping reduce repetitive Service Desk workload through improved knowledge automation and AI capabilities.

• Improving service quality: Ensuring AI-assisted support delivers consistent and reliable outcomes.

• Supporting data-driven decisions: Providing measurable insights into AI performance Service Desk trends and improvement opportunities.

• Driving continual improvement: Connecting operational Service Desk insights with AI automation and Knowledge Management improvement activities.

• Protecting business value from AI investment: Ensuring AI capabilities continue to deliver measurable improvements in efficiency quality and customer satisfaction.

The Role:

AI Quality & Performance Management

• Monitor and report on key AI performance indicators including deflection resolution escalation fallback and containment rates.
• Review AI interactions and identify quality issues failed intents and inappropriate or incomplete responses.
• Categorize and analyze AI failure scenarios to identify root causes and recurring patterns.
• Identify opportunities to improve AI intents conversational flows knowledge sources and automation.
• Validate the effectiveness of AI improvements and monitor performance following changes.
• Maintain accurate records of AI quality issues trends and improvement opportunities.

Data Analysis & Reporting

• Analyze Service Desk ticket data AI interaction data and user feedback to identify trends and service gaps.
• Develop regular AI quality and Service Desk performance reports.
• Create and maintain dashboards and analytical views using Excel Power BI or equivalent tools.
• Translate data into clear insights recommendations and actionable improvement initiatives.
• Track key performance indicators and identify areas requiring further investigation.
• Knowledge & Automation Improvement
• Identify knowledge gaps that contribute to AI failures or poor user outcomes.
• Recommend improvements to knowledge articles categorization intents and automation coverage.
• Provide structured feedback to Knowledge Management Automation AI and Service Desk teams.
• Support testing and validation of new or enhanced AI and automation capabilities.
• Help ensure knowledge and automation solutions reflect actual user needs and Service Desk demand patterns.

Service Desk Operations

• Handle incidents service requests and user support inquiries as required.
• Participate in Service Desk queue management and operational activities.
• Apply standard ITSM processes when handling incidents and requests.
• Escalate issues appropriately and maintain accurate ticket information.
• Use operational experience to identify recurring issues and opportunities for AI or process improvement.

Continual Service Improvement

• Identify recurring Service Desk and AI-related issues and recommend improvement actions.
• Participate in service improvement initiatives and working groups.
• Monitor the outcomes of improvement initiatives and provide evidence of their effectiveness.
• Promote a culture of data-driven user-focused continual improvement.
• Share insights and trends with relevant stakeholders to support informed decision-making.

-

The Requirements

Education & Experience

• At least 2 years experience working in an IT Service Desk IT operations or ITSM environment.
• Practical understanding of incident and service request management.
• Experience with ServiceNow or a comparable ITSM platform is preferred.
• Experience analyzing operational data ticket trends or service performance is preferred.
• Experience working with AI-assisted support chatbots automation or conversational AI is advantageous.

Technical & Analytical Skills

• Strong analytical and problem-solving capabilities.
• Good working knowledge of Microsoft Excel including data analysis and reporting.
• Experience with Power BI or similar business intelligence and visualization tools.
• Ability to interpret data identify patterns and translate findings into actionable recommendations.
• Basic understanding of conversational AI concepts including intents fallback scenarios escalation automation and knowledge-driven responses.
• Understanding of ITSM principles and Service Desk workflows.

Service & Business Skills

• Strong customer and user service orientation.
• Excellent attention to detail and ability to identify recurring patterns and anomalies.
• Ability to understand both technical and business perspectives.
• Strong written and verbal communication skills.
• Ability to present analytical findings clearly to both technical and non-technical stakeholders.
• Strong collaboration skills and ability to work across Service Desk AI Automation Knowledge Management and other technology teams.
• Ability to manage multiple priorities and work effectively in an operational environment.

WTW is an Equal Opportunity Employer