Leaderboard

Traditional KPI metrics such as productivity, quality, cost, delivery, efficiency, and many more should not leave management lenses, rather, targets associated with them should be adjusted accordingly. Customer and employee satisfaction surveys however can be done through AI, leveraging on its capability to detect emotion, interpret facial expression, body language, and many more which is difficult for human eye to decipher and prone to certain biases. To track AI’s real performance and value, I recommend Input Data Integrity, and Bias Detection as two additional KPI metrics that management should add under their lenses. These are crucial for AI’s model creation, accurate training and analysis, impacting AI’s recommendation for business decision-making process.

Proposed Business Excellence Metrics for the AI Era - The use of AI in the core business processes is reshaping how value is created and delivered by organizations. Subsequently, the traditional KPI metrics we have used to measure performance in areas like quality, cost, and efficiency are becoming insufficient and redundant. Using these old metrics in an AI-driven environment can be misleading, causing organizations to optimize for the wrong behaviors and not reap ROI on their technology investments. Let us begin by assessing the Traditional metrics and their shortcomings in an AI driven environment. 1. Assessment of Traditional Metrics Metric 1: First Call Resolution (FCR) It has long been a primary KPI to monitor contact center efficiency and customer satisfaction, indicating a low effort experience for the customer and low cost for the business. In an AI-Driven Environment: Using AI-powered chatbots, IVRs, and self-service portals to manage simple, high-volume, transactional queries is an attempt to give the “Easy” Calls today to machines instead of humans. These were precisely the calls that used to be FCR wins for human agents. By filtering simple issues, AI is ensuring that the only calls reaching human agents are the complex, emotionally charged, or multi-faceted problems that the AI could not solve. And it turns out that these problems are more difficult to solve in one phone conversation. Following these developments, a high FCR rate might actually be a concern! It could potentially indicate that the AI is not being effectively used to screen issues, or human agents bring complex problems to a premature close just to attain an outdated target. While a lower FCR could signify that agents are appropriately handling the highly complex issues that require follow-up, research, and collaboration. Metric 2: Average Handle Time (AHT) AHT measures the average duration of customer interaction. It has been a pivotal metric in gauging operational efficiency, used for staffing models and cost control. The goal has always been to reduce AHT. In an AI-Driven Environment: Since the calls that are able to reach human agents as mentioned above are likely to be important ones. We shouldn't be obsessed with how soon the agent can get the customer of the phone but rather with what quality and value is one giving. A complex issue, high-value customer retention or an upsell opportunity might require a longer AHT. Stressing agents to cut AHT on complex calls can have detrimental effect not only with regards to poor outcomes, customer churn, and repeat calls (which negatively impact other metrics). The AHT metric also disregards entirely the time customers may have already spent interacting with an AI chatbot, rendering the “AHT” only a partial — and potentially misleading — view of the overall customer journey effort. 2. Proposed New Metrics In order to track performance in an AI-driven setting, we need new metrics capturing proactive problem-solving, and the utility of human-AI interaction. Proposed New Metric 1: Proactive Resolution Rate (PRR) PRR is the ratio of potential customer issues that are identified and resolved proactively by the AI system before the customer initiates contact. PRR Logic o The AI tracks customer journey data, usage patterns, and system logs for anomalies that indicate there is a problem in the process (e.g., missed payment, delayed delivery, odd user behavior in a software application). o The AI then initiates an automated resolution using the SOP’s, FAQ’s and KB updates to assist the customer (e.g., retries the missed payment, informs the logistics partner, proactively sends a "how-to" guide, or sends a system alert to the user). o PRR Calculation: (AI-initiated Proactive Resolutions / Total potential issues detected) x 100 · This metric, most importantly, switches the mindset away from reactive service and illustrates the value of preventative excellence. It captures a measure of the organization's ability to avoid problems, which is a far stronger indicator of operational excellence and customer-centricity than how effectively it cleans up messes. Proposed New Metric 2: Human-Assisted Value-Add (HAVA) · HAVA Score is a metric for evaluating the efficacy and efficiency of human agents involved in complex situations escalated by AI. The HAVA Score replaces the use of simplified metrics like AHT and FCR for these high-value encounters. · HAVA Logic: The HAVA Score is calculated after the interaction and based on a weighted calculation of the following: Problem Resolution Success (40%): Was the customer's issue ultimately resolved? (Binary: Yes/No, or a scaled rating). Customer Sentiment Analysis (30%): AI parses the text or audio of the communication to measure customer sentiment levels (i.e., measuring if the customer's levels of frustration decreased, positive language increased, etc.) Customer Lifetime Value (CLV) Impact (20%): Did the interaction led to customer retention, a new purchase, or an upgrade, this can be done by mapping the service interaction to CRM data. Knowledge Base Contribution (10%): Did the agent record the solution for this unique problem, so it could be used for training the AI in the future? (thus helping avoid similar escalations). · HAVA provides a path away from basic efficiency and instead reflects the true value of the human agent in the world of AI. HAVA rewards agents to be thorough and empathetic problem-solvers. HAVA also promotes a learning cycle in which the agent is incentivized to make the AI smarter through KB updates, contributing to the improvement of the system over time. 3. Linkage to Business Excellence These proposed metrics are directly aligned with the core principles of Business Excellence. Business Excellence Principle How Proposed Metrics Align Customer Centricity PRR is a measure of an organization’s ability to solve problems before the customer even knows about them, it is the most efficient form of customer-centricity and true commitment to an effortless experience. The HAVA Score ensures that when customers do need to talk to a human, the focus is all about solving their complex needs and maintaining the relationship that impacts their perception of value and care. Operational Excellence & Quality Improvement PRR actively measures the quality of operational processes. A high PRR means that the underlying systems and processes that are driving the standard approach we work towards, are efficient, intelligent and self-healing, which is an essential component of modern operational excellence. The HAVA Score assists and develops an environment for continuous improvement. Agents are rewarded for contributing to a knowledge base, ensuring human knowledge is captured, and then used to build up the overall human-ai capability to get smarter and smarter, and to be able to do more at scale over time. Employee Engagement & Empowerment HAVA, also enhances the human agent's role from "call handling" to "resolution expert or relationship builder." It enables and rewards them for spending time in solving complex issues whilst creating value - leading to higher job satisfaction and lower turnover. It recognizes and rewards the value of empathy, creativity and complex problem solving that are inherent to being human. Value-Driven Leadership With these metrics available to leaders, they can get a clearer and more informative view of their business performance. Instead of managing counterproductive metrics, they can focus on the real priorities: stopping customer issues before they occur, getting the most value for each human engagement, and designing a learning system that continuously improves with every transaction.

No matter what metric we produced or how fast it gets produced, it provides zero value if it isn’t accurate, i.e., do we trust it to make critical decisions upon it? The further away we get from knowing how we produced a metric, the less we trust it. Because AI can “do” the formulas, calculations and algorithms for us – so quickly, and easily, we could mistakenly put more value on the speed of AI than the actual value we "think" it's producing. There are a couple of wise sayings that I’ve heard in my life. “Keep and honest man honest”. That idea has also been said, “Trust but validate”. Why? If an honest man is honest, then why do we have to make sure he’ll remain honest. Because everything has a bias and can make mistakes, including what metrics AI has created for us. So, for me, the most valuable metric is the one I can trust, whether created by AI or not. The balance is being able leverage the power and speed of AI as well as validating everything it generates. Ergo, Agentic AI. Outdated metric - “Average Employee Training Hours”. Investing in employee training is seen as a direct indicator of capability building, the “level-up” people skill-sets. Traditionally, we think, the more hours, the better. However, AI-driven learning that can hyper-target content to precisely educate where a particular skill is lacking. The hours spent now becomes less relevant but rather the efficacy and application of the learned skill become what is important. If companies rely on “average training hours”, they could easily over-invest in traditional training methods while missing the AI-enabled learning pathways. It shifts from input (hours) to outcome (applied skill). New metric – “Value Realization Velocity (VRV)”. This metric measures the speed at which AI-driven insights or recommendations are converted into tangible business value. Every business has struggled and failed to move an idea from “concept” to “production” to and to know it’s real “realized value”. VRV could track: - Time from AI model deployment to first measurable business impact (e.g., first dollar saved, first customer converted. - The percentage of AI-generated insights that lead to actionable changes within a given timeframe. - The monetary value generated per unit of time from AI-driven initiatives (e.g., incremental revenue per week from an AI-optimized marketing campaign). This metric directly ties AI initiatives to strategic business outcomes because it pushes beyond mere technical performance to demonstrate tangible ROI and agility. Because the AI-driven economy will, and is, so fast-moving, this will be paramount for Business Excellence.

Since each one of us works with data and metrics, plus given that AI is increasingly getting integrated in our processes and work, it will be a worth while investment to go through all the answers. You will get ideas on what you need to focus on and what you can let go. Best answer has been provided by Sargun Diwan. Well Done!!

The first and foremost traditional metric that maybe misleading in an AI enabled system is Efficiency (Process Cycle Time). Traditionally the way human performance was judged basis faster generation of output can not be used as-is to gauge the performance of an AI enabled system. This is because fast AI decisions can be inaccurate, hiding issues like poor model performance. Example - In HRO domain AI enabled systems might reduce time to hire but may overlook certain key organizational initiatives like diversity and inclusion and may source candidates faster but may result in poor job fit due to lack of analysis. Also Cost per hire can be an outdated metrics because AI enabled systems are reducing CPH in the short term but may fail to track Employee Churn Rate due to multiple factors initially ignored. New Metrics - Instead of these metrics we may prefer to use Candidate Fit Accuracy - Percentage of AI selected hires meeting the success criteria (performance, retention etc.) over a longer period of time. Whenever building a system, developers need to focus on striking the right balance between TAT and Accuracy and enable the model to achieve both.

Key performance indicator helps to evaluate business performance on various metric. We have been using Efficiency gain, cycle time reduction, first time right, average handling time, on time delivery, cost reduction and many more. With the evolving time and AI introduction to business processes, some of the old metric will loose its significance and so there is a need to introduce some metrics which can do justice to track the real performance and value delivered in the business KPI that should be excluded is average handling time as AI may reduce the average time per ticket which is time to close a ticket to user with response however this can lead to redispute from a customer if a complicated query is not answered correctly and will also lead to low customer satisfaction and will get some detractors and criticism KPI that should be included is the AI decision and confidence rate which will determine & measure the acceptance of AI solution by human users and indicate confidence level of AI solution this will indicate how much the AI solution and its response is reliable to users

Let us consider a scenario to describe the effect of capturing metrics in traditional Business Excellence way versus capturing the metrics in an AI based ecosystem Scenario 1: Consider an IT Development Project that uses niche skills (emerging technologies like React.js, Node.js, Mongo DB…). Let us try to put our metrics focus on, say, the following parameters - Cost, Quality, delivery, efficiency and customer satisfaction. Let us deep dive a bit into each of these metrics (with a sample, expected SLAs) Note: There are so many aspects to cost.. For simplicity sake, have put special labour cost that talks about niche skills (and therefore it does not consider the normal costs associated with the rest of the skilled workers). Now, let us reimagine this scenario with AI. So, what could be the new metrics, if any. Will there be any modification/deletion of any existing metrics? Lets see that now The Status column consists of a). ‘Removed’/’Not Applicable’ – which means the metric is not relevant for the AI based ecosystem b). ‘Retain’ – it means that the metric is retained from the traditional ecosystem (which means this can be used in AI system as well) c). ‘Improve’ – It means the metric has got more stringent SLA (as expected) d). ‘New’ – It means the metric is specific to the AI based ecosystem In the AI ecosystem (pls refer to the above table – table 2), let us ponder on few things: 1. The AI model can provide niche-skills specialization thereby eliminating the need for Special Labour Cost (it will loose its relevance, therefore). Note the fact that there could be initial setup cost for building AI infrastructure into the organization but that will be of an investment and may not be specific to any project. So, the Special Labour cost can go away. 2. Usually when it comes to traditional efficiency calculation, we will think of automation of testing tools such as Xunit (Developers’ testing tools) , Selenium(QA testing tools), Postman (API testing tool)…So in a way, the focus of automation was targeting towards the test cases, test scripts and stuffs. So what about the development life cycle where the developer is able to leverage AI based coding, reviewing, unit testing, building it and deploying in some staging environment for (functional/System) testing. This is where ‘AI development cycle time’ can be captured. One of the benefits of having an AI based system is, it increases the productivity of the developers and hence this metric is a key metric. We can also add metrics - say, “Ease of use” that should always be >= 7 (between range of 1 – 10), AI Ethical Concern Alert - ‘0’ (Zero) / release , just to name a few.. The metrics are always contextual and can be decided by the Organization/Project teams and the AI solution architect/expert (to ensure there is a feasibility of implementing the metrics) Conclusion: AI is a powerful technology that if leveraged properly can help us to reimagine our business excellence. AI can give an organisation a competitive edge and with the right AI model and KB powered Agents, we can focus on improving traditional metrics as stated over here and also to monitor how AI is improving the business excellence capability, we can also define our own metrics (relevant to what we are doing) and measure the outcome.. We can then use those outcome from the defined metrics and improve our AI system further.

As organizations would be more inclined towards AI enabled decision making or a AI driven organizations. Traditional Metrics measuring Quality. Cost and efficiency will be out dated and there will be metrics measuring quality and effectiveness of solutions rather than Time parameters. There will be more focus on predictability of problems or defects as organizations wants more insights on predictive analysis and forecasting accuracy. Here are some examples for Metrics that will be outdated and metrics that will be in place to be monitored: 1. AHT(Average Handling Time) - So when organizations would be AI enabled the measure will not be on the Time taken to resolve a problem as Bots and AI chatbots will be there to resolve it. The measure would be what is the effectiveness of the solution. This can be measured through CES(Customer Effort Score), it will measure How customers are satisfied with the solution and ease of implementation. AI Assisted Resolution rate, it measures how many problems are resolved by AI solutions. 2. Forecast Accuracy(MAPE) - It measures accuracy of forecast with real world. But it does not show volatility or drift. As in drift in data inputs with time. This can be replaced by Drift detection rate, it will measure and trustworthiness and stability of AI forecast.

While the traditional Business Outcome metrics still remain the same. The process metrics for AI enabled process need to be identified through a CTQ drill down exercise for all the areas mentioned Quality , Cost Delivery , efficiency and customer satisfaction . Here are my thoughts for each of these areas Quality: AI Decision Accuracy % : % of decision that matches human validated outcomes. Escalation Rate% : % of Al handled cases handover to humans Cost : Al Cost per Transaction : this includes the cost of licenses, regular updates and maintenance etc. of the AI infra . Human Effort Reduction : % Reduction in Manual effort due to AI Delivery and Efficiency : The traditional ,metrics will be same here but we can measure Average response time of AI to respond or complete a tasks can be the measure. Customer Satisfaction " CSat Score of AI interactions : These metrics are in addition to the fundamental process metrics and need to be identified , measured and tracked in additional to the regular metrics. Thank you

Yes, ready. There exists a relationship between Business Excellence (BE), AI transformation, and the measurement of organizational performance. Traditional metrics of Business Excellence were effective in systems driven by human input and repetitive transactions. AI is now responsible for decision-making, process optimization, and large-scale output personalization, some of these metrics may become misleading, incomplete, or even outdated. In medical coding field, the existing metric which is possible to be outdated/obsolete is the number of charts coded by coder in an hour. Earlier, the medical coder was able to code 25 charts per hour, but now they may only code 15 charts per hour because of complexity and AI codes 30 charts which are all easy and simple. Subsequently, the obsolete productivity metrics will show a decrease in trend. The new metric to introduce is Human-AI Collaboration Ratio (HACR) in Coding. It tracks how AI augments and not replaces workforce capacity The updated/revised business excellence scorecard for AI-enabled medical coding is for Productivity - The traditional metric is the number of charts coded per FTE. The updated AI-enabled metric is the total number of charts processed (AI + human) and HACR. How it is related to Business Excellence: These new metrics: • Connect AI performance with human empowerment • Assist leaders in proactively managing AI drift and coder disengagement How to Implement: • Test new metrics alongside existing ones to ensure continuity and facilitate comparison • Utilize heatmaps to illustrate performance by specialty or AI version, which is beneficial for CAC tuning • Ensure dashboards are aligned with compliance, revenue integrity, and training functions, not solely operations.

Below are two traditional metrics that would lose relevance in an AI-based setup 1. Processing Time for Manual tasks: As automation is reducing the time spend on some tasks and also increasing the accuracy of tasks. processes like entry of data, simple decision-making, simple critical thinking, and so on. 2. Manwork time saving: In an AI-based setup, manwork hours can be misleading, as the value of work done is not tied only to labor reduction. Therefore, AI has shifted the role a and added more value in such a way doesn't allow for manual work hours of work. Below are two new metrics that can be introduced to track real performance 1. Rate of AI Adoption in the organization: This metric will measure how widely the AI tools are used within organizations. 2. Accuracy of AI responses: This will measure how often the AI generates the responses that users want to see.