Leaderboard

The first and foremost traditional metric that maybe misleading in an AI enabled system is Efficiency (Process Cycle Time). Traditionally the way human performance was judged basis faster generation of output can not be used as-is to gauge the performance of an AI enabled system. This is because fast AI decisions can be inaccurate, hiding issues like poor model performance. Example - In HRO domain AI enabled systems might reduce time to hire but may overlook certain key organizational initiatives like diversity and inclusion and may source candidates faster but may result in poor job fit due to lack of analysis. Also Cost per hire can be an outdated metrics because AI enabled systems are reducing CPH in the short term but may fail to track Employee Churn Rate due to multiple factors initially ignored. New Metrics - Instead of these metrics we may prefer to use Candidate Fit Accuracy - Percentage of AI selected hires meeting the success criteria (performance, retention etc.) over a longer period of time. Whenever building a system, developers need to focus on striking the right balance between TAT and Accuracy and enable the model to achieve both.

Proposed Business Excellence Metrics for the AI Era - The use of AI in the core business processes is reshaping how value is created and delivered by organizations. Subsequently, the traditional KPI metrics we have used to measure performance in areas like quality, cost, and efficiency are becoming insufficient and redundant. Using these old metrics in an AI-driven environment can be misleading, causing organizations to optimize for the wrong behaviors and not reap ROI on their technology investments. Let us begin by assessing the Traditional metrics and their shortcomings in an AI driven environment. 1. Assessment of Traditional Metrics Metric 1: First Call Resolution (FCR) It has long been a primary KPI to monitor contact center efficiency and customer satisfaction, indicating a low effort experience for the customer and low cost for the business. In an AI-Driven Environment: Using AI-powered chatbots, IVRs, and self-service portals to manage simple, high-volume, transactional queries is an attempt to give the “Easy” Calls today to machines instead of humans. These were precisely the calls that used to be FCR wins for human agents. By filtering simple issues, AI is ensuring that the only calls reaching human agents are the complex, emotionally charged, or multi-faceted problems that the AI could not solve. And it turns out that these problems are more difficult to solve in one phone conversation. Following these developments, a high FCR rate might actually be a concern! It could potentially indicate that the AI is not being effectively used to screen issues, or human agents bring complex problems to a premature close just to attain an outdated target. While a lower FCR could signify that agents are appropriately handling the highly complex issues that require follow-up, research, and collaboration. Metric 2: Average Handle Time (AHT) AHT measures the average duration of customer interaction. It has been a pivotal metric in gauging operational efficiency, used for staffing models and cost control. The goal has always been to reduce AHT. In an AI-Driven Environment: Since the calls that are able to reach human agents as mentioned above are likely to be important ones. We shouldn't be obsessed with how soon the agent can get the customer of the phone but rather with what quality and value is one giving. A complex issue, high-value customer retention or an upsell opportunity might require a longer AHT. Stressing agents to cut AHT on complex calls can have detrimental effect not only with regards to poor outcomes, customer churn, and repeat calls (which negatively impact other metrics). The AHT metric also disregards entirely the time customers may have already spent interacting with an AI chatbot, rendering the “AHT” only a partial — and potentially misleading — view of the overall customer journey effort. 2. Proposed New Metrics In order to track performance in an AI-driven setting, we need new metrics capturing proactive problem-solving, and the utility of human-AI interaction. Proposed New Metric 1: Proactive Resolution Rate (PRR) PRR is the ratio of potential customer issues that are identified and resolved proactively by the AI system before the customer initiates contact. PRR Logic o The AI tracks customer journey data, usage patterns, and system logs for anomalies that indicate there is a problem in the process (e.g., missed payment, delayed delivery, odd user behavior in a software application). o The AI then initiates an automated resolution using the SOP’s, FAQ’s and KB updates to assist the customer (e.g., retries the missed payment, informs the logistics partner, proactively sends a "how-to" guide, or sends a system alert to the user). o PRR Calculation: (AI-initiated Proactive Resolutions / Total potential issues detected) x 100 · This metric, most importantly, switches the mindset away from reactive service and illustrates the value of preventative excellence. It captures a measure of the organization's ability to avoid problems, which is a far stronger indicator of operational excellence and customer-centricity than how effectively it cleans up messes. Proposed New Metric 2: Human-Assisted Value-Add (HAVA) · HAVA Score is a metric for evaluating the efficacy and efficiency of human agents involved in complex situations escalated by AI. The HAVA Score replaces the use of simplified metrics like AHT and FCR for these high-value encounters. · HAVA Logic: The HAVA Score is calculated after the interaction and based on a weighted calculation of the following: Problem Resolution Success (40%): Was the customer's issue ultimately resolved? (Binary: Yes/No, or a scaled rating). Customer Sentiment Analysis (30%): AI parses the text or audio of the communication to measure customer sentiment levels (i.e., measuring if the customer's levels of frustration decreased, positive language increased, etc.) Customer Lifetime Value (CLV) Impact (20%): Did the interaction led to customer retention, a new purchase, or an upgrade, this can be done by mapping the service interaction to CRM data. Knowledge Base Contribution (10%): Did the agent record the solution for this unique problem, so it could be used for training the AI in the future? (thus helping avoid similar escalations). · HAVA provides a path away from basic efficiency and instead reflects the true value of the human agent in the world of AI. HAVA rewards agents to be thorough and empathetic problem-solvers. HAVA also promotes a learning cycle in which the agent is incentivized to make the AI smarter through KB updates, contributing to the improvement of the system over time. 3. Linkage to Business Excellence These proposed metrics are directly aligned with the core principles of Business Excellence. Business Excellence Principle How Proposed Metrics Align Customer Centricity PRR is a measure of an organization’s ability to solve problems before the customer even knows about them, it is the most efficient form of customer-centricity and true commitment to an effortless experience. The HAVA Score ensures that when customers do need to talk to a human, the focus is all about solving their complex needs and maintaining the relationship that impacts their perception of value and care. Operational Excellence & Quality Improvement PRR actively measures the quality of operational processes. A high PRR means that the underlying systems and processes that are driving the standard approach we work towards, are efficient, intelligent and self-healing, which is an essential component of modern operational excellence. The HAVA Score assists and develops an environment for continuous improvement. Agents are rewarded for contributing to a knowledge base, ensuring human knowledge is captured, and then used to build up the overall human-ai capability to get smarter and smarter, and to be able to do more at scale over time. Employee Engagement & Empowerment HAVA, also enhances the human agent's role from "call handling" to "resolution expert or relationship builder." It enables and rewards them for spending time in solving complex issues whilst creating value - leading to higher job satisfaction and lower turnover. It recognizes and rewards the value of empathy, creativity and complex problem solving that are inherent to being human. Value-Driven Leadership With these metrics available to leaders, they can get a clearer and more informative view of their business performance. Instead of managing counterproductive metrics, they can focus on the real priorities: stopping customer issues before they occur, getting the most value for each human engagement, and designing a learning system that continuously improves with every transaction.