Q 781. Traditional Business Excellence metrics focus on quality, cost, delivery, efficiency, and customer satisfaction. But as AI begins to automate decisions, personalize experiences, and reshape workflows — what new metrics should emerge, and which old ones might lose relevance? Identify one or two existing metrics that may become outdated or misleading in an AI-enabled setup, and suggest one or two new metrics that should be introduced to track real performance and value.

 

The best answer will be selected on the basis of: 

• Clarity and insight in assessing traditional metrics  
• Relevance and originality of proposed new metrics  
• Linkage to Business Excellence and AI integration

 

Note for website visitors -

• This platform hosts two weekly questions, one on Monday and the other on Thursday. 
• All previous questions can be found here: https://www.benchmarksixsigma.com/forum/lean-six-sigma-business-excellence-questions/. 
• To participate in the current question, please visit the forum homepage at https://www.benchmarksixsigma.com/forum/.
• The question will be open until Monday or Thursday at 5 PM Indian Standard Time, depending on the launch day. 
• Responses will not be visible until they are reviewed, and only non-plagiarised answers with less than 5-10% plagiarism will be considered for winner selection. 
• If you are unsure about plagiarism, please check your answer using a plagiarism checker tool such as https://smallseotools.com/plagiarism-checker/ before submitting. 
• All correct answers shall be published, and the top-rated answer will be displayed first. The author will receive an honourable mention in our Business Excellence dictionary at https://www.benchmarksixsigma.com/forum/business-excellence-dictionary-glossary/ along with the related term.

• Some people seem to be using AI platforms to find forum answers. This is a risky approach as AI responses are error-prone because our questions are application-oriented (they are never straightforward). Have a look at this funny example - https://www.benchmarksixsigma.com/forum/topic/39458-using-ai-to-respond-to-forum-questions/

• We also use an AI content detector at https://quillbot.com/ai-content-detector. Only answers with less than 45-50% AI-generated content will be considered for winner selection.

Below are two traditional metrics that would lose relevance in an AI-based setup 
1. Processing Time for Manual tasks: As automation is reducing the time spend on some tasks and also increasing the accuracy of tasks. processes like entry of data, simple decision-making, simple critical thinking, and so on.
2. Manwork time saving: In an AI-based setup, manwork hours can be misleading, as the value of work done is not tied only to labor reduction. Therefore, AI has shifted the role a and added more value in such a way doesn't allow for manual work hours of work.

Below are two new metrics that can be introduced to track real performance
1. Rate of AI Adoption in the organization: This metric will measure how widely the AI tools are used within organizations.
2. Accuracy of AI responses: This will measure how often the AI generates the responses that users want to see.

Yes, ready.

There exists a relationship between Business Excellence (BE), AI transformation, and the measurement of organizational performance. Traditional metrics of Business Excellence were effective in systems driven by human input and repetitive transactions. AI is now responsible for decision-making, process optimization, and large-scale output personalization, some of these metrics may become misleading, incomplete, or even outdated. 

In medical coding field, the existing metric which is possible to be outdated/obsolete is the number of charts coded by coder in an hour.

Earlier, the medical coder was able to code 25 charts per hour, but now they may only code 15 charts per hour because of complexity and AI codes 30 charts which are all easy and simple. Subsequently, the obsolete productivity metrics will show a decrease in trend.

The new metric to introduce is Human-AI Collaboration Ratio (HACR) in Coding. It tracks how AI augments and not replaces workforce capacity

The updated/revised business excellence scorecard for AI-enabled medical coding is for Productivity - The traditional metric is the number of charts coded per FTE. The updated AI-enabled metric is the total number of charts processed (AI + human) and HACR.

How it is related to Business Excellence:

These new metrics:

• Connect AI performance with human empowerment

• Assist leaders in proactively managing AI drift and coder disengagement

How to Implement:

• Test new metrics alongside existing ones to ensure continuity and facilitate comparison

• Utilize heatmaps to illustrate performance by specialty or AI version, which is beneficial for CAC tuning

• Ensure dashboards are aligned with compliance, revenue integrity, and training functions, not solely operations.

 

No matter what metric we produced or how fast it gets produced, it provides zero value if it isn’t accurate, i.e., do we trust it to make critical decisions upon it?   The further away we get from knowing how we produced a metric, the less we trust it.  Because AI can “do” the formulas, calculations and algorithms for us – so quickly, and easily, we could mistakenly put more value on the speed of AI than the actual value we "think" it's producing.  There are a couple of wise sayings that I’ve heard in my life.  “Keep and honest man honest”.  That idea has also been said, “Trust but validate”.  Why?  If an honest man is honest, then why do we have to make sure he’ll remain honest.  Because everything has a bias and can make mistakes, including what metrics AI has created for us.  So, for me, the most valuable metric is the one I can trust, whether created by AI or not.  The balance is being able leverage the power and speed of AI as well as validating everything it generates.  Ergo, Agentic AI. 

 

Outdated metric - “Average Employee Training Hours”.  Investing in employee training is seen as a direct indicator of capability building, the “level-up” people skill-sets.  Traditionally, we think, the more hours, the better.  However, AI-driven learning that can hyper-target content to precisely educate where a particular skill is lacking. The hours spent now becomes less relevant but rather the efficacy and application of the learned skill become what is important.  If companies rely on “average training hours”, they could easily over-invest in traditional training methods while missing the AI-enabled learning pathways.  It shifts from input (hours) to outcome (applied skill).

 

New metric – “Value Realization Velocity (VRV)”.  This metric measures the speed at which AI-driven insights or recommendations are converted into tangible business value.  Every business has struggled and failed to move an idea from “concept” to “production” to and to know it’s real “realized value”.

VRV could track:

- Time from AI model deployment to first measurable business impact (e.g., first dollar saved, first customer converted.

-  The percentage of AI-generated insights that lead to actionable changes within a given timeframe.

-  The monetary value generated per unit of time from AI-driven initiatives (e.g., incremental revenue per week from an AI-optimized marketing campaign).

This metric directly ties AI initiatives to strategic business outcomes because it pushes beyond mere technical performance to demonstrate tangible ROI and agility.  Because the AI-driven economy will, and is, so fast-moving, this will be paramount for Business Excellence.

While the traditional Business Outcome metrics still remain the same. The process metrics for AI enabled process need to be identified through a CTQ drill down exercise for all the areas mentioned Quality , Cost Delivery , efficiency and customer satisfaction .

Here are my thoughts for each of these areas 

 

Quality: 

AI Decision Accuracy % :  % of decision that matches human validated outcomes. 

Escalation Rate% : % of Al handled cases handover to humans 

Cost :

Al Cost per Transaction : this  includes the cost of licenses, regular updates and maintenance etc. of the AI infra . 

Human Effort Reduction : % Reduction in Manual effort due to AI 

 

Delivery and Efficiency :

The traditional ,metrics will be same here but we can measure Average response time of AI to respond or complete a tasks can be the measure. 

 

Customer Satisfaction "

CSat Score of AI interactions : 

 

These metrics are in addition to the fundamental process metrics and need to be identified , measured and tracked in additional to the regular metrics. 

 

Thank you 

 

Scenario on the Downtime and maintenance

 

Traditional approach

Record of the downtime manually and longer analysis. 

Preventive Maintenance standard scheduling of the machinery based on the OEM recommendation. 

Both these metrics are outdated misleading due to human based errors, delays, biased decisions Preventive maintenance not done based actual machine conditions. 

 

Two approaches AI 

Predictive maintenance - Maintenance carried out based on the actual machine condition by monitor power, output etc. which provides machine maintenance accuracy and Reduces time and increase equipment life. 

Automated downtime report - Sensors based reporting on the machine provides real time data, flow or pattern of the downtime and classification of downtime which provides Real time data, No loss of data and identifying the root causes in patterns.

 

The same AI Approach can used quality control data, Regulatory compliance of fortification, generator data & maintenance etc.. 

 

Particulars                    Traditional & outdated                  AI 

Quality control data     Fixed quality control data       Real time product checks - Tracks flour quality 

fortification                  Manual measurement             Real time flow control- Tracks flour and fortifies with equivalent premix

Generator                    Preventative maintenance       Predictive maintenance - Engine oil conditions, temperatures 

Generator                    Log sheet data                         Real time data flow and checks and Predictive failure alarms &                                                                                                       stoppages  

As organizations would be more inclined towards AI enabled decision making or a AI driven organizations.

Traditional Metrics measuring Quality. Cost and efficiency will be out dated and there will be metrics measuring quality and effectiveness of solutions rather than Time parameters. There will be more focus on predictability of problems or defects as organizations wants more insights on predictive analysis and forecasting accuracy.

Here are some examples for Metrics that will be outdated and metrics that will be in place to be monitored:

1. AHT(Average Handling Time) - So when organizations would be AI enabled the measure will not be on the Time taken to resolve a problem as Bots and AI chatbots will be there to resolve it. The measure would be what is the effectiveness of the solution. This can be measured through CES(Customer Effort Score), it will measure How customers are satisfied with the solution and ease of implementation. AI Assisted Resolution rate, it measures how many problems are resolved by AI solutions.

2. Forecast Accuracy(MAPE) - It measures accuracy of forecast with real world. But it does not show volatility or drift. As in drift in data inputs with time. This can be replaced by Drift detection rate, it will measure and trustworthiness and stability of AI forecast.

Following metrics will become irrelevant when AI enabled solutions comes into practice:
• Productivity measurements involving human efforts alone
• Metrics involved in measuring Manual process effectiveness

Metrics recommendations relevant for AI solutions
• Percentage of AI Augmentation involved
• Measuring successful first time responses
• Percentage of human intervention

Answer: Business Excellence metrics plays crucial role in terms of continuous improvement, operational efficiency and foremost strategic goals, while AI has their own advantages which includes but not limited to, analysis of huge amount of data which helps to make informative decisions and with future prediction, reduce individual error and most important build attractive business models which can build trust and enhance customer satisfaction.

·         Followings are the new metrics that should emerge:

1.    Accuracy of decision: Teach the AI or feed the information in such a way that, the response should be correct first time and every time with zero error.

2.    Data quality index: The information from AI should be complete, consistent and precise.

3.    Effectiveness check of human and AI integration: This approach should be used to evaluate that how well human and AI work together.

 

·          Followings are the metrics that may become outdated or misleading in an AI-enabled setup:

1.    Average handling Time of customer.

2.     Customer services, personalized interactions may be outdated, as NLP and chatbots are generally customer focus by enabling effortless communications through AI agent.

3.    Business excellence contributes to process improvement, lean procedure and process optimization.

For example: If we start six sigma project considering all the above four factors, it takes significant resources, manpower, sometime increase cost and most important time, while enabling the AI integrated robotics process, improve productivity, reduce cost, time and manpower and most importantly will maintained the high quality standard.

 

·         Linkage to artificial intelligence integration and  Business Excellence

 

1.    This integration will provide highly accurate strategic decision, reduce human bias and enhance flexibility.

2.    The effective implementation of linkage ensures leadership development, skill enhancement and AI awareness programs will drive successful integration, which will enabling smarter strategic outcome, resilience, and long-term value creation.

 

Let us consider a scenario to describe the effect of capturing metrics in traditional Business Excellence way versus capturing the metrics in an AI based ecosystem

 

Scenario 1:

Consider an IT Development Project that uses niche skills (emerging technologies like React.js, Node.js, Mongo DB…).  Let us try to put our metrics focus on, say, the following parameters - Cost, Quality, delivery, efficiency and customer satisfaction. Let us deep dive a bit into each of these metrics (with a sample, expected SLAs)

 

Note:  There are so many aspects to cost.. For simplicity sake, have put special labour cost that talks about niche skills (and therefore it does not consider the normal costs associated with the rest of the skilled workers).

 

Now, let us reimagine this scenario with AI. So, what could be the new metrics, if any.  Will there be any modification/deletion of any existing metrics?  Lets see that now

 

 

The Status column consists of

a). ‘Removed’/’Not Applicable’ – which means the metric is not relevant for the AI based ecosystem

b). ‘Retain’ – it means that the metric is retained from the traditional ecosystem (which means this can be used in AI system as well)

c).  ‘Improve’ – It means the metric has got more stringent SLA (as expected)

d).  ‘New’ – It means the metric is specific to the AI based ecosystem

 

In the AI ecosystem (pls refer to the above table – table 2), let us ponder on few things:

1. The AI model can provide niche-skills specialization thereby eliminating the need for Special Labour Cost (it will loose its relevance, therefore). Note the fact that there could be initial setup cost for building AI infrastructure into the organization but that will be of an investment and may not be specific to any project.  So, the Special Labour cost can go away.

 

2. Usually when it comes to traditional efficiency calculation, we will think of automation of testing tools such as Xunit (Developers’ testing tools) , Selenium(QA testing tools), Postman (API testing tool)…So in a way, the focus of automation was targeting towards the test cases, test scripts and stuffs. 

 

So what about the development life cycle where the developer is able to leverage AI based coding, reviewing, unit testing, building it and deploying in some staging environment for (functional/System) testing.  This is where ‘AI development cycle time’ can be captured. One of the benefits of having an AI based system is, it increases the productivity of the developers and hence this metric is a key metric.

 

We can also add metrics - say, “Ease of use” that should always be  >= 7 (between range of 1 – 10), AI Ethical Concern Alert -  ‘0’ (Zero) / release , just to name a few.. The metrics are always contextual and can be decided by the Organization/Project teams and the AI solution architect/expert (to ensure there is a feasibility of implementing the metrics)

 

 

Conclusion:

AI is a powerful technology that if leveraged properly can help us to reimagine our business excellence.  AI can give an organisation a competitive edge and with the right AI model and KB powered Agents, we can focus on improving traditional metrics as stated over here and also to monitor how AI is improving the business excellence capability, we can also define our own metrics (relevant to what we are doing) and measure the outcome.. We can then use those outcome from the defined metrics and improve our AI system further.

With AI chatbots predominantly replacing human customer support agents in handling the first level of customer interactions, the existing customer center Customer Satisfaction Score (CSAT) may not be reflecting the customer's overall experience or their loyalty to the brand. A customer might be "satisfied" with a quick answer but not truly engaged or emotionally connected. Customers are increasingly concerned about the need to interact or speak directly with humans.

 

As AI begins to automate decisions, personalize experiences, and reshape workflows, new metrics should emerge that measure the efficiency and effectiveness of the AI models used. Say, for example, AI agents are used to predict the customer lifetime value for a telecom company—there should be right metrics factored in to continuously monitor the success of predictions and a feedback loop to train the models accordingly.

Proposed Business Excellence Metrics for the AI Era -

The use of AI in the core business processes is reshaping how value is created and delivered by organizations. Subsequently, the traditional KPI metrics we have used to measure performance in areas like quality, cost, and efficiency are becoming insufficient and redundant. Using these old metrics in an AI-driven environment can be misleading, causing organizations to optimize for the wrong behaviors and not reap ROI on their technology investments.

 

Let us begin by assessing the Traditional metrics and their shortcomings in an AI driven environment.

1.      Assessment of Traditional Metrics

Metric 1: First Call Resolution (FCR)

It has long been a primary KPI to monitor contact center efficiency and customer satisfaction, indicating a low effort experience for the customer and low cost for the business.

In an AI-Driven Environment:

• • Using AI-powered chatbots, IVRs, and self-service portals to manage simple, high-volume, transactional queries is an attempt to give the “Easy” Calls today to machines instead of humans. These were precisely the calls that used to be FCR wins for human agents.
  • By filtering simple issues, AI is ensuring that the only calls reaching human agents are the complex, emotionally charged, or multi-faceted problems that the AI could not solve. And it turns out that these problems are more difficult to solve in one phone conversation.
  • Following these developments, a high FCR rate might actually be a concern! It could potentially indicate that the AI is not being effectively used to screen issues, or human agents bring complex problems to a premature close just to attain an outdated target. While a lower FCR could signify that agents are appropriately handling the highly complex issues that require follow-up, research, and collaboration.

Metric 2: Average Handle Time (AHT)

AHT measures the average duration of customer interaction. It has been a pivotal metric in gauging operational efficiency, used for staffing models and cost control. The goal has always been to reduce AHT.

In an AI-Driven Environment:

• • Since the calls that are able to reach human agents as mentioned above are likely to be important ones. We shouldn't be obsessed with how soon the agent can get the customer of the phone but rather with what quality and value is one giving. A complex issue, high-value customer retention or an upsell opportunity might require a longer AHT.
  • Stressing agents to cut AHT on complex calls can have detrimental effect not only with regards to poor outcomes, customer churn, and repeat calls (which negatively impact other metrics).
  • The AHT metric also disregards entirely the time customers may have already spent interacting with an AI chatbot, rendering the “AHT” only a partial — and potentially misleading — view of the overall customer journey effort.

2.      Proposed New Metrics

In order to track performance in an AI-driven setting, we need new metrics capturing proactive problem-solving, and the utility of human-AI interaction.

 

 

Proposed New Metric 1: Proactive Resolution Rate (PRR)

• PRR is the ratio of potential customer issues that are identified and resolved proactively by the AI system before the customer initiates contact.
• PRR Logic

o The AI tracks customer journey data, usage patterns, and system logs for anomalies that indicate there is a    problem in the process (e.g., missed payment, delayed delivery, odd user behavior in a software application).

o The AI then initiates an automated resolution using the SOP’s, FAQ’s and KB updates to assist the customer (e.g., retries the missed payment, informs the logistics partner, proactively sends a "how-to" guide, or sends a system alert to the user).

o PRR Calculation: (AI-initiated Proactive Resolutions / Total potential issues detected) x 100

·         This metric, most importantly, switches the mindset away from reactive service and illustrates the value of preventative excellence. It captures a measure of the organization's ability to avoid problems, which is a far stronger indicator of operational excellence and customer-centricity than how effectively it cleans up messes.

Proposed New Metric 2: Human-Assisted Value-Add (HAVA)

·         HAVA Score is a metric for evaluating the efficacy and efficiency of human agents involved in complex situations escalated by AI. The HAVA Score replaces the use of simplified metrics like AHT and FCR for these high-value encounters.

·         HAVA Logic: The HAVA Score is calculated after the interaction and based on a weighted calculation of the following:

• • Problem Resolution Success (40%): Was the customer's issue ultimately resolved? (Binary: Yes/No, or a scaled rating).
  • Customer Sentiment Analysis (30%): AI parses the text or audio of the communication to measure customer sentiment levels (i.e., measuring if the customer's levels of frustration decreased, positive language increased, etc.)
  • Customer Lifetime Value (CLV) Impact (20%): Did the interaction led to customer retention, a new purchase, or an upgrade, this can be done by mapping the service interaction to CRM data.
  • Knowledge Base Contribution (10%): Did the agent record the solution for this unique problem, so it could be used for training the AI in the future? (thus helping avoid similar escalations).

·         HAVA provides a path away from basic efficiency and instead reflects the true value of the human agent in the world of AI. HAVA rewards agents to be thorough and empathetic problem-solvers. HAVA also promotes a learning cycle in which the agent is incentivized to make the AI smarter through KB updates, contributing to the improvement of the system over time.

3. Linkage to Business Excellence

These proposed metrics are directly aligned with the core principles of Business Excellence.

Business Excellence Principle	How Proposed Metrics Align
Customer Centricity	PRR is a measure of an organization’s ability to solve problems before the customer even knows about them, it is the most efficient form of customer-centricity and true commitment to an effortless experience. The HAVA Score ensures that when customers do need to talk to a human, the focus is all about solving their complex needs and maintaining the relationship that impacts their perception of value and care.
Operational Excellence & Quality Improvement	PRR actively measures the quality of operational processes. A high PRR means that the underlying systems and processes that are driving the standard approach we work towards, are efficient, intelligent and self-healing, which is an essential component of modern operational excellence. The HAVA Score assists and develops an environment for continuous improvement. Agents are rewarded for contributing to a knowledge base, ensuring human knowledge is captured, and then used to build up the overall human-ai capability to get smarter and smarter, and to be able to do more at scale over time.
Employee Engagement & Empowerment	HAVA, also enhances the human agent's role from "call handling" to "resolution expert or relationship builder." It enables and rewards them for spending time in solving complex issues whilst creating value - leading to higher job satisfaction and lower turnover. It recognizes and rewards the value of empathy, creativity and complex problem solving that are inherent to being human.
Value-Driven Leadership	With these metrics available to leaders, they can get a clearer and more informative view of their business performance. Instead of managing counterproductive metrics, they can focus on the real priorities: stopping customer issues before they occur, getting the most value for each human engagement, and designing a learning system that continuously improves with every transaction.

 

 

In today's AI-driven world, the traditional metrics like cost efficiency and average handling time are bit outdated. Many legacy metrics were created for manual workflows and don’t capture the evolving role of AI. Considering the capabilities of automation and its capacity to handle multiple tasks at a time, these old measures frequently miss the mark when it comes to identifying the strategic value that AI yield to business outcomes. Now it is time for organizations to adopt metrics that focus on trust, agility, and how well humans and AI work together in real time and accept indicators like AI Decision Accuracy and the Human-AI Collaboration Index which provide a better understanding of performance. We need to shift organizations performance metrics from just measuring how competent processes run to assessing intelligence, adaptability and innovation which is very much essential to truly be equipped for AI. Also, AI-enabled organizations should be cautious and watch over factors like model drift, ethical alignment, impact of decisions and the systems being evaluated on their ability to learn and adopt for creating excellence that is much required for success.

Key performance indicator helps to evaluate business performance on various metric. We have been using Efficiency gain, cycle time reduction, first time right, average handling time, on time delivery, cost reduction and many more.

 

With the evolving time and AI introduction to business processes, some of the old metric will loose its significance and so there is a need to introduce some metrics which can do justice to track the real performance and value delivered in the business

 

KPI that should be excluded is average handling time as AI may reduce the average time per ticket which is time to close a ticket to user with response however this can lead to redispute from a customer if a complicated query is not answered correctly and will also lead to low customer satisfaction and will get some detractors and criticism

 

KPI that should be included is the AI decision and confidence rate which will determine & measure the acceptance of AI solution by human users and indicate confidence level of AI solution this will indicate how much the AI solution and its response is reliable to users

Traditional KPI metrics such as productivity, quality, cost, delivery, efficiency, and many more should not leave management lenses, rather, targets associated with them should be adjusted accordingly.

Customer and employee satisfaction surveys however can be done through AI, leveraging on its capability to detect emotion, interpret facial expression, body language, and many more which is difficult for human eye to decipher and prone to certain biases.

To track AI’s real performance and value, I recommend Input Data Integrity, and Bias Detection as two additional KPI metrics that management should add under their lenses. These are crucial for AI’s model creation, accurate training and analysis, impacting AI’s recommendation for business decision-making process.

The first and foremost traditional metric that maybe misleading in an AI enabled system is Efficiency (Process Cycle Time).

 

Traditionally the way human performance was judged basis faster generation of output can not be used as-is to gauge the performance of an AI enabled system. This is because fast AI decisions can be inaccurate, hiding issues like poor model performance. Example - In HRO domain AI enabled systems might reduce time to hire but may overlook certain key organizational initiatives like diversity and inclusion and may source candidates faster but may result in poor job fit due to lack of analysis.

 

Also Cost per hire can be an outdated metrics because AI enabled systems are reducing CPH in the short term but may fail to track Employee Churn Rate due to multiple factors initially ignored.

 

New Metrics - 

 

Instead of these metrics we may prefer to use Candidate Fit Accuracy - Percentage of AI selected hires meeting the success criteria (performance, retention etc.) over a longer period of time.

 

Whenever building a system, developers need to focus on striking the right balance between TAT and Accuracy and enable the model to achieve both. 

 

 

Since each one of us works with data and metrics, plus given that AI is increasingly getting integrated in our processes and work, it will be a worth while investment to go through all the answers. You will get ideas on what you need to focus on and what you can let go.

 

Best answer has been provided by Sargun Diwan. Well Done!!