Leaderboard

At our e-commerce product company, we have an AI powered search and recommendation engine feature. It can be configured on each customer project to leverage multiple data sources (ERP, e-commerce, PIM, purchase history) to personalize search and product recommendations. Personalization features include adjusting results based on purchase history, brand preference, and customer profiles. Our learning has been The recommendation engine can personalize shop assortment for different customer segments. While designing customer flows for this feature, we must ensure that the engine does not unintentionally limit catalog visibility or surface exclusive categories disproportionately. If historical purchase data, browsing patterns, or segment profiles reflect societal biases (e.g., preferences along gender, age, ethnicity, or socioeconomic lines), the algorithms can and will replicate and propagate these biases—such as recommending certain products less to some demographic groups or showing limited assortments. Segment-based catalog restriction could reinforce silos and limit choices for certain customer groups, mirroring or reinforcing pre-existing marketplace or data biases. Customizing algorithmic weighting based on customer profiling without scrutiny could favor or disadvantage groups. We had a real example of a sports attire retailer using our product where we experienced that “Inclusive Sizing” (sizes beyond standard American XS–XL, such as plus sizes or petite/tall fit) appeared in only about 10% of products in a given search result. The dynamic facets logic tended to omit these size attribute from the filters entirely. As a result: Customers seeking inclusive sizes were unable to filter effectively. The represented bias favoured mainstream size ranges, thus marginalizing niche segments. The system then further skewed visibility toward products that align with majority sizing, and had potential to worsening representation over time. Some real world complains from users were - "I can never find anything smart with a good price in my size unless they are your top-of-the-line products" - "I see models wearing new designs in the ads but I can't find enough trendy but age-appropriate colours on the website" Additionally, one real risk that was evaluated was that our model/engine might consistently push popular products from high-traffic regions, while under-representing niche or emerging markets. This not only skews visibility but may also limit growth opportunities for less dominant segments. Some steps that we have attempted to apply Design Phase - Curate diverse and representative data inputs - Allow manual overrides for known critical attributes and for attributes deemed socially or commercially significant (e.g., inclusive sizing, accessibility features) were treated as “defined facets,” ensuring consistent visibility regardless of prevalence. - Ethical guardrails in personalization logic: Forbid certain features (like region or size) from driving recommendation weighting unless justified. Testing Phase - Synthetic Test Profiles across demographics - Manual Testing to find if the engine is developing such biases Monitor and Audit Facet Presentation - Track which facets are consistently hidden across queries and evaluate whether they represent systematically underrepresented groups or product lines - Before releasing compliance review is emphasized on Legal, Privacy(GDPR), Security & Accessibility These proactive steps are now taken on early and help ensure our AI serves all buyers fairly, avoiding the “bias in, bias out” trap in new implementation projects.

In 2022, Klarna launched a full-speed AI deployment automating most of its processes using AI solution and realized cost savings equivalent to 700 FTE. One of the processes they automated was their Customer Service Support. After a while, customer complaints and dissatisfaction ballooned. Customers claimed that AI responses were too generic and unhelpful when dealing with real-life problems. While AI solution like chatbots can handle simple and repetitive queries, emotions or complex issues were not addressed. Klarna realized that while AI solutions promise speed and cost savings, it can compromise service quality and customer satisfaction. Klarna decided to rehire employees to address poor service quality and customer complaints. This is a testament that AI solution isn’t about replacing humans, but rather, enhancing the human workforce with smarter tools and better support system. As an MBB, following were my recommendation: 1. Use VOC to identify critical customer requirements (CCR) where complex issues and customers needing to talk to human to solve their concerns will surface. 2. AI solution aims to enhance customer experience leveraging on personalized interaction for higher engagement. This was not apparent in case of Klarna. It is recommended to take advantage on Deep Learning capabilities of AI solution. Such model can identify complex patterns, making it suitable in image recognition, voice recognition, and natural language processing. 3. Lastly, while drawing the to-be process map, HILT (human-in-the loop) principle is recommended. In cases of complex customer concern, AI can escalate the concern to its human counterpart to further assess the given concern and provide necessary resolution.

According to what VK noted under his forum questions, “Some people seem to be using AI platforms to find forum answers. This is a risky approach as AI responses are error-prone..” AI is created by humans who are prone to error. We must always remember this and be diligent to make sure AI will make the best decisions. “Making sure” will ALWAYS to be and, I believe, will forever be, a human responsibility. I can’t ever imagine anyone shirking their responsibility and pointing at the AI solution and saying “It’s the AI’s fault that we lost revenue”. Yes. It might have been that we trusted the AI agent to make the decision but ONLY after we allowed it to make that decision. So, the real accountability still falls back to a human. Knowing that AI is prone to make errors, just as humans have done to mitigate making our own errors, we created guardrails to increase proper decision making and better outcomes – ergo, Business Excellence. Think of AI as another person. But now you are responsible for the decisions and actions of that person. It will need oversight, accountability, and transparency to make sure AI is making the right decisions on our behalf. Here are some of the elements that I think could be included in a governance framework to ensure responsible, high-impact use of AI in a process-driven organization. Creating a governance team or committee to oversee all AI solutions. This team would comprise people from IT, the business, legal, risk management and defining each role and responsibility throughout the AI development, deployment and maintenance. For transparency and accountability, conducting regular impact assessments to identify potential risks, biases and consequences of AI-driven decision. Also, implementing techniques that can provide insights into the how AI is making its decisions, such as feature attribution or model interpretability methods. Lastly, performing audit trails that let us see the data inputs, processing and outputs the AI used to make its decision. For agility and control, using agile development methodologies to allow for rapid iterations and deployment. Using change management to capture the all the changes made throughout the development which can easily be reviewed, tested, and validated. Lastly, establish access controls to prevent unauthorized changes to the AI system or data.

Proposed Business Excellence Metrics for the AI Era - The use of AI in the core business processes is reshaping how value is created and delivered by organizations. Subsequently, the traditional KPI metrics we have used to measure performance in areas like quality, cost, and efficiency are becoming insufficient and redundant. Using these old metrics in an AI-driven environment can be misleading, causing organizations to optimize for the wrong behaviors and not reap ROI on their technology investments. Let us begin by assessing the Traditional metrics and their shortcomings in an AI driven environment. 1. Assessment of Traditional Metrics Metric 1: First Call Resolution (FCR) It has long been a primary KPI to monitor contact center efficiency and customer satisfaction, indicating a low effort experience for the customer and low cost for the business. In an AI-Driven Environment: Using AI-powered chatbots, IVRs, and self-service portals to manage simple, high-volume, transactional queries is an attempt to give the “Easy” Calls today to machines instead of humans. These were precisely the calls that used to be FCR wins for human agents. By filtering simple issues, AI is ensuring that the only calls reaching human agents are the complex, emotionally charged, or multi-faceted problems that the AI could not solve. And it turns out that these problems are more difficult to solve in one phone conversation. Following these developments, a high FCR rate might actually be a concern! It could potentially indicate that the AI is not being effectively used to screen issues, or human agents bring complex problems to a premature close just to attain an outdated target. While a lower FCR could signify that agents are appropriately handling the highly complex issues that require follow-up, research, and collaboration. Metric 2: Average Handle Time (AHT) AHT measures the average duration of customer interaction. It has been a pivotal metric in gauging operational efficiency, used for staffing models and cost control. The goal has always been to reduce AHT. In an AI-Driven Environment: Since the calls that are able to reach human agents as mentioned above are likely to be important ones. We shouldn't be obsessed with how soon the agent can get the customer of the phone but rather with what quality and value is one giving. A complex issue, high-value customer retention or an upsell opportunity might require a longer AHT. Stressing agents to cut AHT on complex calls can have detrimental effect not only with regards to poor outcomes, customer churn, and repeat calls (which negatively impact other metrics). The AHT metric also disregards entirely the time customers may have already spent interacting with an AI chatbot, rendering the “AHT” only a partial — and potentially misleading — view of the overall customer journey effort. 2. Proposed New Metrics In order to track performance in an AI-driven setting, we need new metrics capturing proactive problem-solving, and the utility of human-AI interaction. Proposed New Metric 1: Proactive Resolution Rate (PRR) PRR is the ratio of potential customer issues that are identified and resolved proactively by the AI system before the customer initiates contact. PRR Logic o The AI tracks customer journey data, usage patterns, and system logs for anomalies that indicate there is a problem in the process (e.g., missed payment, delayed delivery, odd user behavior in a software application). o The AI then initiates an automated resolution using the SOP’s, FAQ’s and KB updates to assist the customer (e.g., retries the missed payment, informs the logistics partner, proactively sends a "how-to" guide, or sends a system alert to the user). o PRR Calculation: (AI-initiated Proactive Resolutions / Total potential issues detected) x 100 · This metric, most importantly, switches the mindset away from reactive service and illustrates the value of preventative excellence. It captures a measure of the organization's ability to avoid problems, which is a far stronger indicator of operational excellence and customer-centricity than how effectively it cleans up messes. Proposed New Metric 2: Human-Assisted Value-Add (HAVA) · HAVA Score is a metric for evaluating the efficacy and efficiency of human agents involved in complex situations escalated by AI. The HAVA Score replaces the use of simplified metrics like AHT and FCR for these high-value encounters. · HAVA Logic: The HAVA Score is calculated after the interaction and based on a weighted calculation of the following: Problem Resolution Success (40%): Was the customer's issue ultimately resolved? (Binary: Yes/No, or a scaled rating). Customer Sentiment Analysis (30%): AI parses the text or audio of the communication to measure customer sentiment levels (i.e., measuring if the customer's levels of frustration decreased, positive language increased, etc.) Customer Lifetime Value (CLV) Impact (20%): Did the interaction led to customer retention, a new purchase, or an upgrade, this can be done by mapping the service interaction to CRM data. Knowledge Base Contribution (10%): Did the agent record the solution for this unique problem, so it could be used for training the AI in the future? (thus helping avoid similar escalations). · HAVA provides a path away from basic efficiency and instead reflects the true value of the human agent in the world of AI. HAVA rewards agents to be thorough and empathetic problem-solvers. HAVA also promotes a learning cycle in which the agent is incentivized to make the AI smarter through KB updates, contributing to the improvement of the system over time. 3. Linkage to Business Excellence These proposed metrics are directly aligned with the core principles of Business Excellence. Business Excellence Principle How Proposed Metrics Align Customer Centricity PRR is a measure of an organization’s ability to solve problems before the customer even knows about them, it is the most efficient form of customer-centricity and true commitment to an effortless experience. The HAVA Score ensures that when customers do need to talk to a human, the focus is all about solving their complex needs and maintaining the relationship that impacts their perception of value and care. Operational Excellence & Quality Improvement PRR actively measures the quality of operational processes. A high PRR means that the underlying systems and processes that are driving the standard approach we work towards, are efficient, intelligent and self-healing, which is an essential component of modern operational excellence. The HAVA Score assists and develops an environment for continuous improvement. Agents are rewarded for contributing to a knowledge base, ensuring human knowledge is captured, and then used to build up the overall human-ai capability to get smarter and smarter, and to be able to do more at scale over time. Employee Engagement & Empowerment HAVA, also enhances the human agent's role from "call handling" to "resolution expert or relationship builder." It enables and rewards them for spending time in solving complex issues whilst creating value - leading to higher job satisfaction and lower turnover. It recognizes and rewards the value of empathy, creativity and complex problem solving that are inherent to being human. Value-Driven Leadership With these metrics available to leaders, they can get a clearer and more informative view of their business performance. Instead of managing counterproductive metrics, they can focus on the real priorities: stopping customer issues before they occur, getting the most value for each human engagement, and designing a learning system that continuously improves with every transaction.

Is AI solution biased? Well before asking this question, let us dwell more into human nature, is human response or process building biased, it has to be, it forms the basis of selecting criteria, a baseline on which the entire process is set or supposed to operate. Similarly, when we create an AI agent there will be a bias in AI-enabled customer service processes, especially in banking—can have serious consequences, from unfair treatment of customers to regulatory violations. Let’s break this down using your example of a third-party contact center handling banking queries, such as Annual Maintenance Charges (AMC) or unauthorized UPI transactions, and explore how bias can creep in and how to mitigate it. What Bias Can Appear in Banking Customer Service and Where? 1. Case Prioritization Risk of bias: AI may prioritize cases based on customer profile (e.g., high-value customers), potentially delaying resolution for others. E.g: AMC-related queries from senior citizens may be deprioritized if the model learns they are less likely to escalate. 2. Action Recommendations Bias possibility: AI may suggest refunds or escalations based on historical patterns that reflect biased decisions. Example: UPI fraud cases from Tier-2 cities may be less likely to get recommended for escalation due to historical underreporting. 3. Response Generation Bias Risk: Regional models may respond taking into consideration the tone of voice, choice of words, AI agent will respond differently given the tone, politeness and choice of words for customers based in northern part of India versus the same AI agent might find the customer’s similar language or choice of words as rude or condescending and might deny service in southern part of India. Language models may respond differently based on customer name, language, or tone. Example: A polite query may get a more helpful response than an agitated one, even if both are valid. 4. Billing Model Influence Bias Risk: If billing is based on connect minutes, agents may be incentivized to prolong calls. If based on call count, they may rush. Example: AMC queries may be wrapped up quickly without full resolution under a per-call billing model. So, what do we do to minimize bias in Design, Testing, and Monitoring A. Design Phase Diversify Training Data Be it low income customers or high rollers, you might want to include varied customer profiles, geographical regions of customers, languages, net worth of customers, and complaint types. Low amount frauds or frauds based on a certain amount should not matter when a customer is complaining of an unauthorized transaction by a merchant. There is a possibility of bias setting in based on a low or high amount transaction, AI might prioritize only high amount unauthorized transaction cases. We must ensure representation of certain vulnerable groups (e.g.,low income, senior citizens, rural customers). Provide clear objectives that kill bias Design AI models with fairness constraints (e.g., equal resolution rates across demographics). Avoid optimizing solely for efficiency metrics like AHT (Average Handling Time). Human-in-the-Loop Keep humans involved in sensitive decisions (e.g., refund approvals, fraud escalations). B. Testing Phase Inclusion of Bias Audits Test model outputs across different customer segments. Use synthetic data to simulate edge cases (e.g., same query from different regions). Scenario-Based Testing Create test cases for AMC and UPI queries with varying tones, languages, and urgency levels. Check for consistency in response quality and resolution. Metric Diversification Track fairness metrics alongside performance metrics (e.g., resolution equity, escalation parity). C. Monitoring Phase Set up real-time dashboards Monitor call outcomes by customer segment, query type, and agent behavior. Flag anomalies (e.g., unusually short calls for UPI fraud cases). VOC : Feedback Collect customer feedback post-call and correlate with AI decisions. Use feedback to retrain models and adjust flows. Billing Model Alignment Ensure billing models don’t incentivize biased behavior. Consider hybrid models (e.g., quality-adjusted call count) to balance efficiency and fairness. How do we break the “Bias In, Bias Out” Cycle Continuous Learning: Regularly update models with new, unbiased data and feedback. Make it transparent: Make AI decision-making explainable to agents and supervisors. Assign ownership: as a check mechanism, assign accountability for bias monitoring and remediation. Cross-Functional Collaboration: Involve friendly customer base, compliance team, QA team, and customer experience teams in AI governance.

Here's a methodical and useful way to keep track of versions, make sure performance is good, and produce clear documentation for AI processes and prompts that vary over time: 1. Make a formal versioning system Think about AI processes and prompts as code instead of making arbitrary changes: You can save your prompt and flow definitions as text files (JSON, YAML, Markdown) in Git or a program like it. Semantic Versioning makes it easy to communicate about changes: Major: A substantial alteration in the design's purpose or flow. Minor: New features or better prompts. Patch: Fixes or small modifications. Add commit messages that say what the change is meant to do and why it was made. Put both the prompt text and the evaluation/test cases in the same repository so that you can observe both the inputs and the outcomes over time. 2. Make a registry for Prompt and store information about it. Keep a well-organized register (this might be a spreadsheet, a Notion database, or an internal tool) that has: ID of the version Date of Release Writer/Owner Changes Explained Results of tests that are connected Cost, accuracy, latency, and satisfaction are measured/ indicates performance. Rollback Reference - to the previous version This registry is your traceability source to/whether you compare or go back. 3. Check Before You Start To make sure that upgrades are useful and not harmful: Use fake and real test cases from the past to execute the new flow/prompt in a sandbox environment. A/B Testing: Send a small quantity of traffic to the new version and see how it compares to the baseline version. Regression Checks—Check that crucial KPIs don't go down for scenarios that are known to be good. When you can, automate tests by generating a list of queries and expected outputs ahead of time and running them on both old and new versions. 4. Document errors/problems with corresponding causes If you change something, be sure to add: The problem statement, such - users didn't understand step 3 in the flow. The theory, like - making the language easier should lead to more people finishing. The proof after deployment, such as - the recall rate improved from 72% to 84%. You or another developer will be glad know what was wrong when you look at older versions again. 5. Be ready to go back Make sure that the last stable version is always straightforward to install. Make it easy to roll back your deployment process, ideally with only one click or command. Write down when and why rollbacks occurred. They can be just as useful as changes that happen in the future. 6. Find a way to blend stability with new ideas. The Innovation Track is an experimental branch, where you may test new techniques to get engineers to work without putting the stability of production at risk. Stable Track: Flows that are ready for use and only get revisions after a lot of testing. Changes from innovation should only be merged to stable when the metrics/performance are fine. This is basically a two-speed paradigm for development: fast testing and slow release. An example of a workflow Create a new prompt in any AI tool. Make your commitment clear: Make step 3 clearer to cut down on drop-offs. Do automated testing and have people look at old cases. Send 10% of traffic to A/B testing. If the metrics improve, merge into the main branch and change the version. Put notes and numbers in the Prompt Registry. Conclusion Managing different versions of AI flows and prompts requires the same amount of attention as building software. The best method to do this is to put together: Git and semantic versioning are examples of structured version control. Centralized Documentation (a registry with performance logs and other information that is easy to access) Strong testing and rollbacks, such sandboxing, A/B testing, and automated regression checks Two-speed development means having a solid track for production and an innovation track for testing. This makes sure that every change can be logged, tested, and undone, which helps teams come up with new ideas quickly while keeping things stable. In short, always have a way back, write down the why, and test the what.

When we first started using AI to track production downtime patterns, I built a simple flow that pulled operator inputs and generated quick insights for the shift leads. At one point, I decided to tweak the prompt that asked operators to describe the issue, just to make issues clearer and easy to understand by the technical team. I thought it was an improvement. A week later, my phone was buzzing during a site visit because the reports coming out of the system suddenly had big gaps. Turns out my “clarity” change made operators give shorter answers that didn’t have enough detail for the analysis to work. Since then, I’ve treated AI flows exactly the way I treat any process change in manufacturing: I save every version before I touch it. Not just the file but a quick note on what I changed and why. I run the new version in a controlled test with a small team, not the whole plant. If it performs better on the KPIs we care about like accuracy, speed, usability, then it graduates to live. If it doesn’t, I roll it back in minutes because the last good version is sitting in my folder. I also keep two environments: the stable one for what’s proven, and a “playground” for experiments. That way, I can test bold ideas without worrying about disrupting a live process. It’s the same mindset I use in CI projects: measure first, change deliberately, and always keep the option to go back. With AI flows, that discipline makes the difference between steady improvement and a messy guessing game.

How I Would Build a Feedback System for an AI Customer Service Agent? It’s like hiring a new customer service rep. - you would not throw them in front of customers on the first day and hope for the best, instead you would watch how they perform, collect feedback from customers and supervisors, and help them improve. An AI agent needs the same kind of ongoing training. Three Ways to Collect Feedback Ask Customers Directly but Keep It Simple: After the AI helps with a real question, show three quick buttons: thumbs up, neutral face, or thumbs down. Include a small text box so customers can add a quick note such as “Did not understand my mortgage question” or “Gave me the right answer but sounded robotic.” The key is to ask only after meaningful conversations, so customers are not continuously prompted after every single interaction. Have Human Experts Check the AI’s Work Once a week, experienced supervisors can review a sample of conversations, focusing on ones with poor ratings, long resolution times, or high-stakes topics like compliance. They will spot details that metrics miss, such as “The AI gave correct information but did not recognise that the customer was frustrated about a fee.” Reviewing a sample, rather than every conversation, keeps the process manageable. Track the Numbers Monitor essential metrics such as first-time resolution, the number of cases escalated to human agents, and average resolution time for each case. Occasionally, you may send test questions where you already know the correct answer to ensure the AI is still performing well. Making Sense of the Feedback Collecting feedback is easy, making it useful takes work. Start by grouping similar issues together, such as “Does not understand regional accents,” “Too formal when customers are upset,” or “Provides incorrect information.” Prioritise by severity. A calculation error is far more serious than sounding overly formal. Look for patterns, for example, whether accuracy drops on Mondays when there is a backlog from the weekend. Three Speeds of Improvement 1. Quick fixes can be made in a day or two, such as updating outdated information. 2. Regular updates can happen once a month, retraining the AI on the most common issues identified in the feedback. 3. Big changes, such as adding advanced document-reading capabilities such as OCR, will take longer and require more planning. Avoiding Feedback Overload Too much feedback can overwhelm the team; focus on the interactions that reveal the most. Address urgent issues immediately and save routine improvements for the monthly review. Once an issue has been resolved and stays fixed for a few months, stop monitoring it closely and turn your attention to new challenges. Keep People Involved Let customers and employees know their feedback matters. If you improve the AI’s ability to answer product questions based on someone’s suggestion, say so: “We have improved how our AI handles product inquiries based on your feedback.” When employees see that their input leads to real improvements, they will continue offering valuable suggestions. The Bottom Line Maintaining an AI agent is like maintaining a car. You make small adjustments as needed, schedule regular check-ups, and only conduct major repairs when something fundamental needs to change. The goal is steady improvement, so the AI gets better every week without frustrating customers or overwhelming the team.

Let's look at a real-world scenario to see how to construct a strong and valuable feedback loop for improving an AI agent after it has been put into operation. For instance, an AI customer service person that works for a company that provides financial services. This assistant helps people who have inquiries about how to manage their accounts, make purchases, and receive support with items. A Look at Feedback Loop Design There would be three stages of the feedback loop: Feedback that the user begins (Explicit) Feedback that the system gives you (implicit) Human (Supervisor or Lead) in the loop (HITL) should check it out. A centralized feedback processing pipeline receives feedback from each layer, sorts it, rates it, and sends it to either Automated learning modules for modifications that aren't too risky People look at significant or private issues in lineups Ways to collect feedbacks or comments 1. Clear feedback from users After each communication, you can give them a thumbs up or down or a star rating. Inline modifications or recommendations, like "That's not what I meant," start the process of capturing the intended revision. Short surveys after each session to get qualitative feedback Design tip: Keep it light and optional. Only ask for help after a big interaction or when a task is finished or not. 2. Implicit Feedback on Behavior: When a user quits a chat in the middle of it, they are giving feedback on their behavior. Asking the same inquiry over and over or getting a human agent involved Latency or hesitation (the user takes a long time to respond or suddenly changes the subject) To locate places where people are having problems interacting, these signals are marked and given a score. 3. Comments from the supervisor and the audit There are notes about human agent escalations, such as "AI got the request wrong." Random encounters are scored and grouped by quality during periodic audits (for example, tone mismatch or outdated information). Tagging for compliance, especially in sensitive areas like delivering financial advice Feedback that has been marked by a boss is more important. Getting criticism and learning from it Tiered Processing Pipeline: Automatically tagging and grouping similar problems, such "tone issues" and "entity mismatches," using heuristics and NLP classifiers. Making a decision based on risk assessment: Is it possible for the model to fix itself by retraining? Do you need to update the template or prompt? Or should this go to human developers? Routing Feedback: Adjusting the prompt or retraining on grouped samples automatically applies low-risk fixes. A person must look over and approve high-risk fixes before they may be added. How to Avoid Getting Too Much Feedback: Threshold-based Sampling: Only reveal feedback when there is a pattern, such when five or more people complain about the same item. A way to put feedback in order: Impact (frustration score) twice Frequency is the same as Priority Score Digest of the Day: Dashboards for teams that illustrate the most significant issues, possible solutions, and plans for putting them into action. Feedback Archiving Windows: Old feedback that has been dealt with is put away so it doesn't happen again. Finding Tone Mismatches: An Example in Action Users give the bot a "rude" rating in more than 10 sessions when it responds to late payments. A high pace of escalation in those negotiations is an implicit sign. The supervisor says that three interactions are "too formal." The system puts these together and offers a prompt modification to soften the tone: You haven't paid yet. Please repair this right now. To: "It looks like your payment is late. Let's work together to make it better! Used through A/B testing, watched, and proved that it got better Summary: Why This Works Practical: in the Real World Uses real signals (both implicit and explicit), automates low-risk tasks, and gets people involved when they need to be. Relevant: directly applicable to areas such as healthcare, HR support, financial services, and others. Balanced: teams are always getting better without too much stress, and there are built-in safety safeguards and human oversight.

Why do those wins sometimes slip away in the Control phase? There are various reasons why solution to improved process slip away in the Control Phase. Here are some: 1. Thinking that identifying a solution is enough. Oftentimes, organization and project team missed to establish controls to sustain and ensure consistency of the injected solution. Testing of impact and feasibility of the solution is missed, only to find out that local staff implementing the solution on their day-to-day task find it difficult to sustain. 2. Poor communication of intended changes. Formal handover of improved process to its process owner/local management is essential, more importantly to its local staffs who will deal with day-to-day work where improvement took place. A better understanding of what was the problem and why improvement was made must be clear and aligned with the local team. Buy-in and ownership of local team is vital to sustain the solution. This means making them involve from the start and all throughout the project’s phase. 3. Inadequate training in new condition. Lack of enablement of local management and staff who’ll be implementing the solution and who’ll be using it daily will surely make successful implementation fail. Conducting training and enabling local staff before full implementation of solution will provide knowledge, familiarity, build confidence, and likely ensure success of implementation. Adequate training puts the local staff in a controlled environment where learning curve is monitored and supervised. Under scrutiny, old behaviors will be guided and replaced with new intended behavior tied up with the change throughout the training process. Local management’s involvement in training brings alignment, trust, and confidence with local staffs. 4. New process not captured in written procedure. Procedure provides guide and clarity to the doers. It should contain the process details, working sequence broken down into elements, with hints and tips how to perform an activity or task. Updated written procedure of the improved process is an essential supplement to the enablement of local staff and management alike. Procedure should be crafted in such a way not bounded to misinterpretation in order to avoid human error. 5. No monitoring to check that solution is working as intended. Local management’s involvement in the handover of the improved process is essential. Management oversight alongside with tools and best practices such as Control Plan, Process Control Chart, Gemba Kaizen, and process audit should be understood and taught to them by the project team. Having a KPI metric that shows process performance where improvement was made is of equal importance for sustainable management oversight. What tools or techniques do you rely on to keep things on track and make sure the improvement sticks for good? Handover process management is crucial to sustain the successful process improvement in long term. Process documentation such as to-be process flow chart, training & enablement plan, process aids/visual references, control plan, process control chart, updated procedure, FMEA (whenever applicable), and implementation plan should form part of the handover process to local management. This is a structured way to provide clarity on how the improved process works and how it should be monitored to sustain the gain of its financial and non-financial benefits. Lastly, involvement of process improvement team or at least its leader in KPI monitoring and random process audit for the next three to six months upon handover is another key on sustainability of the improvement made.

When we talk about Continuous improvement, as a concept, we refer to a constant WIP mode of innovation, enhancement and incremental progress. Take all possible learnings and loop it back an input to further refine a product or process. VOC, VOB, error types, new data or new pattern or behavior of a certain process or a machine that can be studied, performance monitoring. Getting RCAs. Feeding it back into the system and closing the loop of a continuous self-learning improvement process with the help of Artificial Intelligence. Natural language programing Reinforcement Learning : AI agents can learn by interacting with end users and learning best practices from online forums and receiving feedback (rewards or penalties). Over a period of time, AI models or AI agents can improve their decision-making based on outcomes. Example: AI in customer service optimizing responses based on customer satisfaction scores. Also, in an AI agent environment if a customer rates a low score or not resolved on a survey. AI can ask customer if he or she wants to get redirected to additional support Utilize online libraries: System based or conventional training methods have a focused content and is periodically reviewed once or twice a year. Unlike traditional models trained once on a fixed dataset, online learning allows models to update continuously as new data arrives. Can prove to be extremely useful in dynamic environments like fraud detection system that can improve itself whenever a new fraud pattern emerges or email classification or query categorization. Optimized Human-in-the-Loop (HITL): AI backend can incorporate VOC of output or a human feedback to refine and improve performance. On a continuous basis which a key component of continuous improvement For example, customer service agents correcting AI-generated email drafts helps the model learn better phrasing, grammar, formatting, and tone. Use concepts like A/B Testing and Feedback Loops: A tried and tested AI system can test different strategies (e.g., email templates) and learn which ones perform better. Manual or online VOC and Feedback loops help the system adapt to changing customer behavior or business goals. e.g. In a Banking Email Customer Service Context: AI can learn from: VOC (NPS scores, complaints and RCA) Agent corrections to AI-suggested replies, check of all queries are answered in a multi query email. Frequency and Escalation patterns (e.g., which types of queries lead to dissatisfaction) Compliance checks (to avoid regulatory violations) Though there will be challenges to guarantee a continuous improvement on AI based models, if we study it enough it can be overcome. Challenges like Propagation of systematic Bias. For e.g. an AI model might be more favorable to a certain type of machine or high-performance shift timing, or certain region in terms of Sales etc. Distribution or pattern shift. Or drifting of parameters, Real world the situation changes dynamically so AI will have to be trained to Adapt. Failing which it will follow a fixed pattern and might not necessarily be effective. In manufacturing or Healthcare sector or if we speak from a Six sigma perspective AI can Conduct SPC if we feed it in initial stage. Analyze process deviations If we find some points or processes out of control, we will implement solutions to get the process in control, AI agent can learn from such corrective actions It can also Suggest process changes to reduce defects depending upon previous corrective actions taken by us or information available online. Would be better poised to predict process output or future failures or improvement opportunities

MBBs have deep expertise in process optimization and structured problem-solving, and their role in structured problem-framing approach is paramount, let’s understand this especially in AI solution implementations in high-touch environments like Contact Centers. The problems of Mis-framing leading to ineffective Solutions Contact Center Chatbot Deployment Scenario: A contact center faces long customer waiting times. To quickly reduce Average Speed of Answer (ASA), leadership launches an AI chatbot to handle frequently asked inquiries, aiming to ease pressure on live agents and thereby reducing ASA. Surface Level Problem statement: The project team stated the problem as “We have long wait times because our agents are overwhelmed. Let’s implement a chatbot to handle FAQ’s and reduce wait times by 50%.” While investing 100,000’s of dollars to develop an AI chatbot, train it on FAQ’s, and deploying it as a FPOC for all customer inquiries. The chatbot in itself was technically proficient, using NLP and ML algorithms to interpret customer requests. What was missed: The team did not perform a thorough root cause analysis. Key problems included understaffing staffing during peak hours (only 60% of required agents), inadequate training programs that left agents unprepared for complex product inquiries, fragmented knowledge management systems that forced agents to search multiple databases, and high employee churn (45% annually) from workplace stress and limited career advancement opportunities. The Effects of Mis-Framing on AI Performance Following the chatbot deployment, AI gave generic responses to complex customer issues, causing greater frustration among those needing detailed technical support. Instead of reducing call volume, the chatbot generated additional calls from customers seeking clarification on the AI’s responses or requested immediate escalation to human agents. Findings based on BSI Analysis: The pre-implementation baseline, calculated with the Bottleneck Severity Index formula (BSI = Volume × Cycle Time × (1 - First Time Right%) × Severity), showed: • Volume: 1,200 calls per day • Cycle Time: 8.5 minutes average handle time • First Time Right: 65% • Severity: 3.2 (scale of 1-5) • Baseline BSI: 11,424 Post-chatbot implementation revealed: • Volume: 1,350 calls per day (increased due to chatbot escalations) • Cycle Time: 11.2 minutes (longer due to frustrated customers) • First Time Right: 58% (decreased due to inadequate agent preparation) • Severity: 3.8 (higher customer frustration) • New BSI: 20,365 (78% increase) The AI solution made matters worse: with customer complaints increased, call deflection remained below 15%, and net promoter score (NPS) declined further, and the organization having to face increased operational costs due to higher call volumes and longer resolution times. In addition to the above consequences, wastage of resources and loss of stakeholder trusts add to the negative impact of mis-framing on AI effectiveness. Suggested Practical strategies for MBBs to improve problem framing in AI projects a. Engaging in structure problem statement development using LSS thinking and tools o Use SIPOC and VOC to clarify process boundaries and understand demand drivers o Defining CTQ’s and linking them to customer pain points rather than convenience metrics like ASA. b. Apply BSI for comprehensive bottleneck assessment o Train the project teams in evaluating each BSI component Component Key MBB Questions Volume Is the call volume avoidable or failure demand (e.g., repeat issues, unclear policies)? Cycle Time Are agents slowed down due to poor tools or unclear procedures? First Time Right % What’s the root cause of low FTR? Training, systems, or information gaps? Severity Are we prioritizing automation for high-impact or low-impact queries? o Trend Analysis: Ongoing BSI monitoring to spot patterns and predict bottlenecks before they become critical. This enables teams to address root causes proactively instead of reacting to symptoms. o Use Pareto analysis of BSI to identify Top drivers and guide the AI strategy accordingly. c. Facilitating structured problem definition workshops and fostering stakeholder engagement o Run problem framing workshops that bring together diverse perspectives and stakeholders (operations, IT, HR, training and customer experience.) o Use tools like affinity diagrams and root cause analysis techniques to identify underlying issues that may not be apparent to any single stakeholder group and before confirming the need for AI. o Translating insights into well-structured problem statements (what is wrong, where, when, to what extent and impact on CTQ.) o Making use of the RACI matrix to ensure comprehensive problem understanding. • Inform: Keep executive leadership aware of project progress and findings • Consult: Gather input from frontline agents, customers, and IT teams • Responsible: Include customer service managers, training coordinators along with operations teams and customer experience specialists in problem definition sessions • Accountable: Work closely with the project sponsor on the project approvals. d. Deploy Control Measures Before Automating o Test hypotheses through small-scale pilots that test technical functionality and business impact of the proposed solution before scaling AI. o The pilots need to monitor impact on Leading Indicators (FTR, Escalation Rate, Post-Chat Survey Scores) to validate alignment of proposed solution with identified root causes. Hence the mis-framing of problems in AI initiatives may lead to technically accurate but operationally ineffective solutions, wherein MBBs are mandated with the task of diagnosis with discipline. Using BSI as a key metric identifies real process friction points and thereby guiding the organization to ask the right questions before investing in AI, and ensuring the final solution addresses the true constraints, improve customer experience, and deliver sustainable business value.

I have not been trained or certified as an MBB but I can apply what I have learned in this course. Here's an example of an AI solution technically is working as it should but has become a part of the problem. Consider a business who has a customer service center and their customers are experiencing long wait times. In an effort to decrease the long wait times, they create a Chat Bot. After implementing this AI solution they certainly can see the the call wait times has significantly decreased because the Chat Bot can "answer" them quickly. So, technically, this AI solution is a success. Wait times have drastically decreased. But the company begins to hear from their customers how angry and frustrated they are, even more so than when they had to deal with long wait times. The business failed to understand that what they should have been really trying to solve was increasing customer satisfaction, not merely the symptom of addressing long call wait times. The Chat Bot caused greater unsatisfaction because customers now have to make repeated calls (even though they don't have to wait) because the many "simple" calls are often precursors to more complex issues and the Bot could not handle these, thus forcing customers to start over with an agent, which leads to more frustration. Also, agents may now have to deal with more calls from customers because the Chat Bot did not properly diagnose the underlying problem. This situation wasn't created by the Chat Bot, but by those who didn't have the foresight to really understand how they should have created the Chat Bot. At the end of the day, technology or technical solutions, such as AI, will not be blamed for these problems that arise. Those who created the AI solution will be. You don't want to be that person. Back to the original thought of creating an AI solution. The business thought it was to merely solve lowering long wait call times. But the real root of their issue was customer frustration and dissatisfaction. Their "AI solution" was focused on the wrong thing and it even caused a deeper problem for them How could this have been prevented? Using the FRT process and documentation which captures the Desired Effects (DE), the Undesired Effects (UDE), and the Negative Injections (NI) of any AI project and solution. FRTs can help to envision the ideal future state of an AI solution but also proactively identify negative consequences BEFORE a dime gets spent on creating the solution. The FRT would have captured the root cause by addressing and thinking through the UDEs and also creating NIs to create answers for these UDEs. Utilizing the FRT process and documentation, along with creating a very thorough and thoughtful BRD, would have greatly increased a proper AI solution that result not only in lowering call wait times, but mor importantly, raising customer satisfaction.

AI Governance Framework for Business Excellence AI integration is transforming how decision-making, and operations are performed in organizations. As AI automates more business functions, strong governance becomes essential for responsible deployment. An effective and well-structured governance framework builds trust, reduces risks, and aligns AI advancements with organizational goals, regulations, and stakeholder expectations while maintaining competitive agility. A. Proposed AI Governance Framework Elements - Ethical Guidelines o These are a set of clear, non-negotiable principles that guide all AI initiatives, translating company values into technical requirements. o Defining acceptable use cases and explicitly prohibiting any unethical or biased applications. o Using reference frameworks including – EU AI Act / NIST AI RMF etc to translate these principles into policies and decision logs to ensure how each AI solution meets the guidelines. - AI Governance Structure and Oversight committee o A council of senior executives with cross-functional representation responsible for strategic AI direction and policy approvals o The panel reviews AI projects not only for business objectives but also for ethical standards and societal impact o Conducts periodic audits and model validations including ad-hoc sessions for urgent issues - Data Management Guardrails o Its imperative to maintain an AI repository with the details of the AI models, training data sources and intended usage o Monitoring data quality, lineage and privacy controls to ensure compliance with the adopted guidelines frameworks and the existing data-governance policies - Risk assessment and mitigation o It covers categorizing potential risks into – Operational, reputational, legal and ethical headers with their respective mitigation strategies o A Tiered framework for risk assessment (Low, medium, high) allows for agility by matching the level of oversight to the potential impact of the AI projects, thereby, ensuring low-risk projects aren’t affected by unnecessary governance whereas the high-risk projects receive intense scrutiny o The protocol also covers the real-time tracking of AI performance metrics, bias emergence and unexpected outcomes with incident response procedures for addressing AI system issues - Stakeholder engagement and communication o It involves including the employees / end-users, customers and the external advisors in the loop during design and post deployment of AI projects, to ensure that development and deployment of AI are not done in silo o Comprehensive training for teams to understand AI capabilities, limitations and their role in governance o Publish the explanation of the AI models purpose, performance and disclosures to build trust with customers and partners - Performance and accountability mechanisms o Define AI performance metrics to measure accuracy, fairness, and business impact of AI systems o Recording of AI decision making processes, model changes and associated governance activities B. Stakeholders for AI Governance Stakeholder Role and Responsibilities Chief Ethics Officer / Governance Lead Manages the ethical application of AI and chairs the AI Governance Committee. IT / Data Science Teams Ensure models are technically robust, monitored, explainable, and secure. Business Process Owners To validate AI outputs against the business goals and customer outcomes. Legal & Compliance To ensure AI systems comply with regulations, data and privacy laws, and any ISO standards and AI frameworks, as applicable. HR & Change Management Conduct training, initiate communication, and change readiness for AI-impacted teams. Internal Audit Regularly review model performance, risk, and controls. C. Balancing Agility and Control - Real time monitoring and Alerts o Use of monitoring dashboards to track live model performance, flag issues and trigger alerts for intervention, thereby closing the gap between operations and governance. - Controlled Pilots and A/B testing o Iteratively test AI models in a secure environment before deployment to track issues during development itself. - Living document and Fact sheets o Document the assumptions, limitations, training and retraining cycles and model versions for transparency and control. - Continuous feedback loop o Use feedback from users and business scenarios into model retraining processes to support continuous improvement and ensure alignment with organizational objectives. Subsequently, we can conclude that an effective AI Governance framework anchors the principles of Transparency by laying down clear guidelines and documentation; Accountability by defining roles and responsibilities and putting in place the required controls and continuous improvement through real-time monitoring, feedback and evolution of the governance framework based on the best practices and stakeholder needs. By adopting globally established standards and frameworks in AI governance, organizations can harness the transformative power of AI without compromising ethical or operational integrity, while achieving its business excellence goals of quality, cost optimization and super customer satisfaction.

Since each one of us works with data and metrics, plus given that AI is increasingly getting integrated in our processes and work, it will be a worth while investment to go through all the answers. You will get ideas on what you need to focus on and what you can let go. Best answer has been provided by Sargun Diwan. Well Done!!

Below listed are the traits of a good AI Governance Model – 1.) Clear and Fair – Identify and reduce any biases in the model, provisioning options for human interventions for big decisions and ensuring POCs are defined and known to all stakeholders in case something goes wrong. 2.) Strategic Fit – Ensure every AI deployment directly supports the organization's strategic goals and delivers clear value 3.) Managed Lifecycles: AI systems should have a defined route from initial conception to continuing upkeep. This calls for extensive testing prior to deployment, ongoing performance monitoring, and a defined procedure for modifications or even retirement. We require accurate documentation of everything. 4.) Training – Ensuring that relevant teams are well trained to work effectively with the AI. Also, teams need to be aware of what the AI can and cannot do. Stakeholders of an ideal AI Governance Model – 1.) Leadership 2.) AI oversight group 3.) Ethics and compliance team 4.) Internal Auditors Mechanism to ensure both agility and control 1.) Smart Risk Assessment – Design and approval framework aligned with the risk quotient of the deployment. 2.) Use of standardized tools and reusable modules – Provision pre-approved tools and use of reusable building blocks to cut down redundant work. 3.) Build in governance from day 1. 4.) Centralized guidance from the AI oversight team. 5.) Clear and defined RACI. 6.) Provision Human-AI collaboration for high risk decisions.

Traditional KPI metrics such as productivity, quality, cost, delivery, efficiency, and many more should not leave management lenses, rather, targets associated with them should be adjusted accordingly. Customer and employee satisfaction surveys however can be done through AI, leveraging on its capability to detect emotion, interpret facial expression, body language, and many more which is difficult for human eye to decipher and prone to certain biases. To track AI’s real performance and value, I recommend Input Data Integrity, and Bias Detection as two additional KPI metrics that management should add under their lenses. These are crucial for AI’s model creation, accurate training and analysis, impacting AI’s recommendation for business decision-making process.

For years, the assumption was that “Chief AI Officer” meant a machine learning PhD, a data scientist, or a software engineer who could build models. That assumption is rapidly being dismantled. A clear trend is emerging across global enterprises, law firms, governments, and financial institutions: non-technical business leaders — lawyers, consultants, operations executives, economists, and brand strategists — are being appointed to the most senior AI leadership roles. And the pace is accelerating dramatically. The Numbers Don’t Lie Year New Non-Technical AI Chief Appointments Annualised Rate 2013 1 1 2019 2 2 2020 1 1 2023 3 3 2024 5 5 2025 8 8 2026 6 (Jan–Apr only) 14 (annualised) Rise of Non-Technical AI Chiefs (Annualised) In just the first four months of 2026, six non-technical leaders have already been appointed to Chief AI Officer or equivalent roles. Annualised, that projects to 14 appointments for the full year — nearly double the 2025 figure, and 14× the rate seen in 2019. This isn’t a blip. It’s a structural shift. Who Is Actually Getting These Roles?Below are documented appointments of non-technical leaders to Chief AI Officer and equivalent roles across major organisations: Organisation Role Background Date Herbert Smith Freehills Kramer (major international law firm) Global Chief AI Officer Lawyer (tech transactions, leveraged finance, legal innovation); former Senior Counsel at Mastercard, Global Head of Innovation at McKinsey Legal November 2025 WTW (Willis Towers Watson) Chief AI Officer Co-founder and former CEO of Newfront (AI-native insurance brokerage); MBA Stanford; finance and scaling expertise (non-coder) April 2026 Louisville Metro Government Chief AI Officer 25+ years in enterprise transformation and AI upskilling at Intel; former paralegal; English/paralegal degrees November 2025 Microsoft Chief Responsible AI Officer Law, Public Policy 2019 Goldman Sachs Chief Information Officer (AI-led transformation) Business + Tech Strategy (not pure coding role) 2019 / 2022 O.C. Tanner Chief Technology Officer (AI-led strategy) Business Strategy December 2013 Deloitte Global AI Institute Leader Business, Consulting ~2020 NTT DATA ($30B+ global tech services) CEO & Chief AI Officer Former McKinsey Senior Partner (TMT); MS Industrial Engineering (Stanford), B.Tech Mechanical Engineering (IIT Bombay); management consulting June 2024 / September 2025 Anthropic Chief AI Readiness Officer / COO Former founding COO of Google DeepMind; prior roles at Coursera (COO), Kleiner Perkins, Intel; engineering degree ~2026 IFS Nexus Black (industrial AI) CEO Former Chief Product Officer for LegalTech at Thomson Reuters; AI product strategy at GfK and Sage; founded AI for Good UK; MA Advanced Computer Science July 2025 HSBC Chief AI Officer COO of HSBC Corporate and Institutional Banking; nearly 20 years in operational and commercial banking roles April 2026 KPMG Vice Chair / Global Head, AI & Digital Innovation Former Head of KPMG US Consulting (15,000+ people); MBA and Master's in Professional Accounting October 2023 / August 2025 Littler Mendelson (employment law firm) Chief Artificial Intelligence Officer Nearly 15 years of employment law experience; led practice innovation at national employment law firm April 2026 Edelman UK Chief AI Officer, UK Communications and brand strategy executive; led integrated campaigns for global consumer and tech brands; Cannes Lions awards September 2024 LVMH Chief Data and AI Officer Director of Strategy and Innovation for EMEA at Nike; strategy and marketplace operations background March 2024 U.S. Department of Homeland Security Chief AI Officer & CIO Cyber and intelligence operations (U.S. Marine Corps); operational and intelligence background, not AI research March 2025 Wells Fargo Head of Artificial Intelligence (also Co-CEO, Consumer Banking & Lending) Former CEO of Consumer & Small Business Banking; former Head of Wells Fargo Technology; appointed from a business-leader seat November 2025 Mastercard Chief AI and Data Officer Former EVP of Corporate Strategy and M&A at Mastercard; corporate strategy and deals background, not engineering 2024 New York State (Office of Information Technology Services) Chief AI Officer Researcher at United Nations University; founded UN's first AI policy research lab; AI policy and governance background, not engineering January 2026 State of Oklahoma (OMES) Chief Artificial Intelligence and Technology Officer BBA in Management Information Systems; career in technology modernisation and business transformation across Fortune 500 and public-sector; business-and-operations rather than coding background November 2025 U.S. Department of Agriculture Chief AI Officer (also Chief Data Officer) Started in private-sector biotech; led data analytics team providing genomic services; data strategy and analytics leadership rather than ML/coding 2023 U.S. Department of Energy Acting Chief AI Officer Former Director for Technology and National Security at the White House NSC; policy and national security background, not engineering December 2023 U.S. Department of Labor Chief AI Officer Earlier Deputy CAIO at DOL; over a decade at the Bureau of Labor Statistics; operations and program management rather than AI research June 2025 U.S. Social Security Administration Chief AI Officer (also Deputy CIO) More than 20 years at SSA in IT operations and enterprise leadership; agency-veteran operational profile 2024 Morgan Lewis (global law firm) Chief AI & Knowledge Officer Former Chief Administrative Officer at a global law firm; business operations and process design (non-technical) 2025/2026 Generali Investments Chief AI Officer PhD/MSc in international macroeconomics; Professor of Economics; former Director of Research; senior roles at World Bank/UN PRI; economics/policy/research focus April 2026 Why Is This Happening?The role of a Chief AI Officer has evolved. In its earliest incarnation, it was about building — training models, architecting data pipelines, writing production code. Today, in most enterprises, the hard technical work is being done by vendors (OpenAI, Google, Anthropic, Microsoft) or by internal engineering teams. What organisations actually need at the C-suite level is someone who can: 1. Drive adoption — persuading reluctant stakeholders, managing change at scale 2. Govern responsibly — navigating legal, ethical, regulatory, and reputational risks 3. Connect AI to business outcomes — translating capability into commercial value 4. Work across functions — bridging legal, HR, finance, operations, and technology These are leadership and judgement skills. Not coding skills. The lawyers, consultants, and operators being appointed to these roles are not naive about AI. Many have deep domain expertise, years of AI-adjacent experience, and strong track records leading transformation. They simply did not build the models themselves. The Acceleration MattersThe annualised 2026 figure of 14 is not just a data point — it reflects a tipping point. Organisations that once waited for a “perfect” technical candidate are now actively choosing experienced business leaders and structuring the role around strategy, governance, and change management rather than engineering. If this trajectory holds, 2026 will see more non-technical AI Chief appointments than all years from 2013 to 2024 combined. The era of the non-technical AI Chief has arrived. What do you think is driving this shift? Are organisations right to prioritise business acumen over technical depth in these roles? Share your perspective below.

Process Context My team also manages the central master data management for 50+ plants today, and this will grow to 70+ plants by 2027. The entire fleet data management is handled by two people in my team. Every time we commission a new plant or acquire a new plant, we need to align it’s material master with our fleet database, to avoid duplication, planning errors, and wrong spares being introduced into SAP. In a typical post commissioning and acquisition, we review minimum of 5000+ incoming material records against an existing 345000 item fleet master. Practically for my team, each item review takes at least 6 minutes without AI. The AI-enabled Process To solve this, I built a Python + AI solution using a MiniLM semantic model, combined with rule based checks. The program setup classifies each incoming item into three categories, Auto, high confidence match to directly map & upload in SAP. Review, ambiguous match, reviewed by the master data team. Reject, no valid match, program generates a new master data creation template for my team, to directly load into SAP. You can clearly see, AI does not create master data blindly in this case, it recommends, and the team decides. When We Trust The AI I have defined clear rules after testing the model for almost 10 days with millions of lines, semantic similarity is high & critical identifiers (model number, size, rating). It checks if descriptions and attributes are complete and consistent. One more rule I have setup is to keep standard, low-risk categories, and excluding verified MRP items, and these items directly flow straight into Auto category & are uploaded without manual touch. When We Override The AI Team deliberately does the review when similarity scores are close across multiple candidates, technical digits conflict even if text similarity is high. Then we also look at if item is maintenance critical or safety critical. We jump to the poor descriptions as well. In all such cases, team’s priority is correctness, not the speed. Safeguards That Keep The Balance We have built simple controls to avoid blind trust or even excessive overrides, Strict thresholds for Auto classification, mandatory team’s review for all Review cases, spot audits of Auto mappings, tracking & analysis of override patterns to improve program, and we have clear ownership, AI suggests, Team decides. Impact In Real Numbers Now with this program, my team completes 5000 item migration in 10 days in total instead of 2 months. I have a clear breakdown of 10 days, Data setup + AI pre-load + first analysis is done in 0.5 day SAP mapping for Auto category takes 1 day Manual review is done for Review category in 7 days New MD setup for Reject category is done in 1.5 days This has really improved my team’s output and bandwidth, and also reduced the onboarding risk for new plants, and best part is, it is allowing two people to scale this work for our growing fleet. Bottom Line I trust AI where signals are strong & mistakes are low impact, I override it where ambiguity or risk is high. As you can see, we are improving the overall process, idea isn’t to remove people from the process, it’s to make sure people spend time only where judgement actually matters.

Domain: Aerospace MRO - Engine shop for CFM56/LEAP Turbofans for Performance Restoration (€ 220 Million yearly turnover , approx. 1,800 shop visits in a year, AI rolled over since late 2025 to predict HPT module rework needs based on borescope images, oil debris analysis, and in-service data) Specific AI-enabled process: Predictive HPT Blade Rework Forecasting The AI will recommend if the module needs full blade rework, partial (only the tips), or none, all with the goal of eliminating unnecessary shop time and expense without losing the zero escape target on critical parts. It went live on all CFM56/LEAP visits in Q1 2026 and initially deliver an average 18% reduction in TAT on HPT modules. How we ensure & monitor the process continues to deliver intended outcomes We are treating this AI-human decision loop as a live control system and continuing to develop it over time not like one tine install, the focus is on sustainable business value – TAT savings, cost per visit going down, safety and zero quality escapes. What we monitor (daily / weekly / monthly) 1. Leading indicators (daily dashboard – shop floor + engineering) · Prediction accuracy of AI vs. actual rework result (confusion matrix updated every 50 engines). · AI suggestion Override rate by technicians / engineers (accept, tweak, reject AI recommendation). · Confidence score variation (how often is the model <80% sure?) · Data drift indicators, distributional shift of input variables (eg iron particles in oil, borescope crack density, EGT margin so on) 2. Lagging business outcomes (weekly review – operations + finance) · HPT Module: Turn Around Time Variance (target < 35 days). · Rework cost per engine vs. Baseline · Escape rate / quality holds on HPT (target 0) · Spare Parts Consumption vs. Forecast (Over/Under-Stocking Signals) 3. Model health metrics (monthly deep dive – MBB + data team) · Population stability index (PSI) on key inputs (>0.25 = moderate drift, >0.5 = severe). · Calibration plot (predicted probability vs observed rework rate) · Feature importance drift (which inputs is most important to the model now vs at launch) How we react when the going starts getting tough We have a three-level escalation protocol: Level 1 – Minor Drift (Weekly Trigger) · Override rate >25% or confidence <75% on >20% of cases. Response: · Immediate feed back loop i.e. every override by enginers requires 1-click reason (dropdown + optional voice note). · Retrain model based on last 100 engines + overrides justificatipn. · Notify shop team lead, usually fixes within 1-2 weeks Level 2 – Business impact emerging (weekly trigger) · TAT +3 days or rework cost increased +8% vs rolling 4-week average · OR escape / hold on HPT (even one) Response: · Hold AI recommendations - return to manual disposition within 48 hours/ · Root Cause A3 with MBB: Data drift? New failure mode? Change in user behavior? · Temporary rule: AI confidence > 90% required for auto-accept · Full model retrain + validation on hold-out set before re-release Level 3 – Systemic failure (monthly or immediate on escape) · PSI >0.5 on critical inputs OR calibration slope deviates >15% · OR sustained TAT/cost > 15% Response: · Full pause of AI in production · Independent audit: data lineage, labeling drift, concept drift · Notification to the regulator of any escape which occurred · Re-baseline from scratch or switch to a fall-back approach (manual and old rules) · Shared across sites post-mortem – we’ve had one Level 3 (new low-sulfur fuel changed oil debris patterns in Q3 2026) Practical setup we use today · Automated alerts using Teams/Slack when threshold breaches · Monthly “AI Health Review” (30-min standing meeting: MBB, ops manager, data lead) · Quarterly external benchmark against OEM data (CFM/Pratt) · Annual review of AI usage (EASA Part-145 requirement) Bottom line from the teardown bay AI Drift isn’t an ‘if’ but a ‘when’ In MRO, the price of slow degradation can be a long turn-around time, excessive spares, or even a failure in service. The way we monitor our AI is how we would monitor an engine, performing routine checks every day, and only grounding it completely when we have to. The process remains alive since we do not assume model is “set and forget”.

Here are the winners of Q821. 🏆 Winner – Adil Khan (Aerospace Heat Treatment😞 outstanding NADCAP-compliant AI certification plan with controlled trials, Cpk > 1.67, and 4-tier sign-off governance. 🥈 Runner-up – Sattar Mohammad Imran (Complaint Chatbot😞 comprehensive multi-departmental certification and oversight framework ensuring fairness and compliance. 🥉 Special Mention – Arul Palani (AI Code Assistant😞 strong readiness and policy-based certification approach using ISO/IEC 42001 standards.

AI is very well capable of interacting with customer and make them feel personal in each interaction by merging data, context and empathy at scale without crossing boundaries of trust. Better customer interactions with less customer efforts and high CSAT scores will definitely help the business expand Customer’s expectation from any service are · Quick response on any queries raised · Receive accurate response on queries raised to avoid asking repeat questions · Faster query resolution with minimum interactions · Easy to interact with and have a sense of personal touch by letting them feel valued, heard and understood · Expect trust on data privacy AI helps in better interaction with customer by: · Collecting the context with consent by pulling required data keeping in mind not using hidden data · Knowing customer better and in depth by building customer profile using data from past purchases, past interactions or support tickets, browsing history, etc · Use smart prompts for Bots that works on prefilling summary and also acknowledge earlier context showing respect and care · Feel customer valued by predicting needs and suggesting next steps that will help query resolution prior customer asks the question · Switch tone basis customer tonality and empathize basis sentiment analysis that reflects customer sentiment For example, Account Payable Helpdesk is the function included in Procure to Pay process that handles queries from vendors, internal or external stakeholders related to invoices processed, payments done and PO created. Below are few AI capabilities mentioned that can be used in AP Helpdesk across PTP: · Build AI Chatbots for automated query handling with 24/7 support and less repetitive tickets · Routes the ticket to required department basis query raised for faster and convenient resolution · Captures invoice data using OCR and NLP from documents or emails · Proactively inform vendors or required stakeholders on invoice status thereby reducing escalations · Resolve complex queries by searching policies, SOPs, etc using intelligent search & knowledge base · Adjust the tone basis customer’s response and urgency thereby building trust · Flags invalid invoices, duplicate payments thereby preventing financial losses

Do you know how banks are getting smarter? It is by turning Knowledge as a Competitive Edge : The real game-changer is how they're using everything they know by decoding the customer brain through data by studying their patter 24/7/ 365 days. 1. They are ditching the old for the new Simplify and streamline operations: They are replacing legacy systems with modular, cloud-native architectures which was un-thinkable just a few years ago. Use continuous integration and delivery tools are used to reduce development time and improve agility. In action: Bank of America using Cloud Provider: Private cloud infrastructure. Saved approximately $2 billion 2. Investing big in data and AI They are Building a solid foundation: Banks are building unified data views from their scattered data sources. To connect all the dots which was their biggest pain. Use AI and Gen AI to generate insights, automate decision-making, and to give a personalize customer experiences. Example: Several Autonomous Decision Intelligence Platform is developed by banks to turn fragmented data into strategic assets for fraud detection, risk assessment, and marketing. 3. Shifting focus from maintenance to innovation: Redirecting funds: Tech budgets are being redirected from simple maintenance to things that where it is creating new value. Customer-first focus: The focus has shifted to improving the customer experience, personalizing everything, and getting to market faster. 4. Nurturing the right talent and culture Build a tech-savvy board and leadership team. Gen AI training are provided to the leaders along with it’s application. There is a significant Increase in the proportion of in-house engineers and reduce overhead roles. Foster a technology-first mindset across the organization. Example: Another US Bank Capital One, with it’s 12,000-strong tech team transitioned to a cloud-first model and began selling its own software products. 5. Use AI to Enhance Relationship Management AI isn't just for behind-the-scenes—it's also a powerful tool for customer-facing teams. Smarter advisors: Relationship managers are being armed with AI-generated client summaries, risk profiles, and insights. More meaningful conversations: They are spending more time on strategic, high-value conversations because gen AI solution is providing them with the necessary inputs. The payoff: This is leading to a stronger, more profitable customer relationships and faster decision-making.

AI-Powered Incident Management System At our e-commerce platform company (PAAS model), we need to support environments of all customers who rely on our system for their online ecommerce sales and we need to have continuous uptime and meet strict SLAs. We use NewRelic as our observability platform which collects massive volumes of logs, traces and metrics. After filtering we are analyzing over 60TB of telemetry data per month. We’re now building a system to query this data for AI model integration, enabling anomaly detection, incident correlation, and predictive analytics. Decision Making Scenario- Analyzing and classifying incidents in real time When an Alert is generated the Operations Manager need to make quick decisions during disruptions like outages, performance degradation, or critical bugs. These incidents needs to be evaluated quickly due to SLA requirements since they directly impact buyer satisfaction, conversion rates, and revenue for our clients. The challenge is to prioritize correctly, weighing severity, scope, resource constraints when choosing action steps. AI Agent Support We have been working on a design of an Incident Response AI Assistant with the final goal that it will - Aggregates and analyzes real-time data, such as error logs, user requests, system metrics, and incident tickets. - Scores incidents by combining: - Business impact (example: conversion drop, cart abandonment) - Customers or geographies affected - Urgency indicators like large number of error log entries, memory or cpu spike - Generates and recommends priority and type based on pre-learned categorization - Suggests response paths to help mitigate the problem - Quick mitigation (rollback the latest patch) - Escalation (pass to development team) - Monitoring (observe performance metrics) - Offers confidence scores on its solution For this we are creating solution with two key components to support the decision-making and the communication 1) Incident Analyzer Bot which will automatically detect system issues using AI/ML Learn from historical incidents to categorize alerts by severity and type and reduce false positives Correlate related events — for example, grouping three different alerts under a single outage — to help managers see the full picture quickly Find and provide reference of related incidents from our historical data to help provide the manager with information about the previous RCA and solution 2) Ops Chatbot is focused on communication (not customer-facing) If a critical issue is detected, it can notify customers automatically and proactively before they notice it themselves. It supports manual overrides, customizable communication methods (like email, message, chat) with pre-defined message templates. Makes message suggestions to the manager, who can review and approve them directly in tools like Microsoft Teams. Manages follow-ups automatically if the issue remains unresolved — for example, sending timed updates like “we are still analyzing your system” every 1, 6, 12 hrs depending on the case. This AI assistant will help to surface and prioritize incidents quickly but the final decision remains with the manager who reviews the categorization, solution recommendation and approves the communication suggested by AI. The manager will be have full discretion to override, approve, or modify the AI’s actions. We have planned to create feedback loop to help the system learn and improve over time and implement accuracy monitoring by comparing AI predictions to actual outcomes, review the AI's confidence scoring especially when AI uncertainty is high. This regular validation against historical incidents should help us ensure that the human/AI-assistant together work towards meeting customer's SLAs.

I recently worked on a project where we built what we called a CO-CEO AI agent for a marketing company. The idea was simple: instead of treating the AI as a background tool, we brought it into the decision-making process almost like another executive. Its main job was to help the CEO design marketing strategies for different clients. Now, here’s the twist. We didn’t just let the AI spit out strategies in isolation. It was invited to client meetings, not literally of course, but through structured prompts where the full context of the client’s business, challenges, and goals was fed in. That way, its recommendations weren’t generic “playbook strategies,” but tailored to the actual discussion the leadership team was having. That made it much more of a trusted advisor than a black-box machine. From this project, I learned that there are two layers to making AI advice truly useful and trustworthy for leaders: 1. Where It Should Assist Pattern spotting across campaigns: Leaders don’t always have the time to compare 50 different client reports. The AI could highlight trends (e.g., “clients in retail are seeing 20% higher engagement when campaigns run mid-week”). Scenario testing: Instead of one “best” strategy, the AI could lay out three options: low-risk, high-growth, or balanced. This gave the CEO choices rather than a single directive. Speed on background research: Before walking into a client strategy session, the AI could summarize competitor campaigns, past results, and market conditions in minutes. These are areas where AI’s scale and speed give leaders an advantage without replacing their judgment. 2. Checks to Keep It Reliable Context gatekeeping: The AI was only as good as the context it had. We made it a rule that client objectives and constraints must be captured first (almost like a briefing note) before the AI gave advice. No context, no strategy. Audit trail of reasoning: Every recommendation had to come with a short rationale, “this works because past campaigns in similar industries showed X, Y, Z.” This gave the CEO confidence in the “why,” not just the “what.” Version control for prompts: As we refined how we asked the AI questions, we tracked changes. For example, when we shifted from “generate campaign ideas” to “act as a CMO and propose three strategies with risks and trade-offs,” we documented it. That way, if a change caused worse outputs, we could roll back quickly. Human override always on: The AI was never treated as final authority. The CEO still made the call, but with stronger input in less time. Honestly, what made this whole setup work wasn’t the AI being “super smart.” It was the way we used it. We never treated it like it was going to run the company or make the final call. Instead, it was more like a second set of eyes, someone in the room who could throw out a few options, show the risks, and spot patterns the rest of us didn’t have time to see. The clever part wasn’t the output, it was the process: making sure it only answered once we’d given it the right context, tracking how we asked questions so we didn’t lose improvements, and always keeping a paper trail of its reasoning so it didn’t feel like magic. At the end of the day, the CEO still made the decisions. The AI just made those decisions faster and more informed. That’s really the trick to building trust. You let the AI contribute, but you don’t hand over the steering wheel. It’s not there to replace judgment, it’s there to make good judgment easier.

The roots of an organization lie in its culture. To stay competitive, be the innovator and the go-to market brand, organization transformation is needed at a cultural level. For an organization to have this transformation journey, AI can be one of the trusted partner only if used judiciously. Let us debate this with an example. To promote organization wide learning and development, upskilling and efficiency improvement, from a strategy point of view an organization invested on an AI solution. Every month AI report was shared which gave visibility on number of training hours spent at Department level. Say we have Department A with 40 hrs and Department B with 240 hrs. It was but obvious that Department B got recognized. This was misleading as Department A had 5 team members with an average of 8 hrs per spent on training per member whereas Department B had 120 team members with an average training hrs of 2 per member. The feedback was given to the involved stakeholders and right behaviors were promoted. AI workflows were rebuilt to trigger training modules as per Department needs and dashboard was re-aligned to report correct metrics. This was possible because leadership had the visibility to the dashboards and timely actions were taken to make the AI platform more interactive and enabled the organization to promote a learning culture. An average 2hrs of training spent per member moved to an average 8 hrs of training per member within 6months.This also contributed to employee engagement and continual improvement initiatives. For AI to be the trusted advisor for leaders, recommending following steps to be taken by leadership - Creation of Core team - Before onboarding the AI journey, it is crucial to have the buy in and alignment of all key stakeholders. Create core cross functional team having leaders representing domain, business and technical expertise. Strategic Initiative - Leaders are the influencers for setting the strategy. Leaders need to communicate the need and the objective for adoption and adaptation. Corporate to sponsor the strategic initiative. Go big bang with communication. Set the tone at the top. AI goals, KPIs , resources, etc. AI tools to be part of the short-term Annual Business Plan and long term 5-year strategy roadmap. AI awareness – Educate leaders on what AI is and make them aware of its limitation. Leaders to be mindful and cognitive to overcome biases related to cost, technology and speed. Message from leaders to team to maintain ethics and transparency. Building capability – One of the key reasons why AI can be the trusted advisor is for the speed of data availability, accuracy, reliability and faster decision making. Core team to explore and build AI solutions that would meet the business objectives. Core team to involve domain and technical expertise for bias in and bias out checks. Few key criteria’s for evaluation could be Cost, Quality, Risk of failures, Flexibility – Scalability, Processing Time, Implementation speed, data collection, processing, storage capability and ROI. Proof Of Concept - Create prototypes, use cases, assess opportunity, prioritize and validate, design, build, test and deliver. Course correct if needed. Calibrate, train and update knowledge repository. Roll out the solution. Communicate – Address myth that AI is people reduction method and promote it as value creation to customers and business. Send organization wide communication when milestones) are achieved. Share failure and success stories. Capture lesson learnt and submit solution in Knowledge repository. Governance and oversight – Management to have periodic governance. Reinforce ethical adoption of AI and compliance to data security and privacy and role based access to daily dashboards for better monitoring and oversight. Reward & Recognize – To promote and encourage continual improvement celebrate achievements and reward all those involved in making and leading the transformation. AI thus can be the trusted advisor for leadership if adopted in spirit and maintaining integrity and business ethics at the root level.

Q. Bais in, Bais Out: How to break the cycle? Answer - In a service delivery context given example of prioritizing cases, recommending actions and responding to customers, following steps can be take at various stages of solution development – Design stage – · Assess project objective, scope, metrics and success measures with timelines · Reach out to stakeholders incase of difference of opinion. · Interview and empathize the issues faced · Brainstorm and validate assessment criteria.Design FMEA · Course correct metrics and success measures if required · Involve Developers, testers in the kick off call Testing phase – · Develop use test cases. · Build Agentic AI workflow with what-if scenarios, And OR logic · Link knowledge base repository with correct calibrated clean database Monitoring phase – · Intelligent dashboards with powerapps workflow when any shift in data is observed · Calibrate and retraining AI for precision and accuracy. · Periodic governance This is how one can let the Bias IN and then Bias it Out through careful design, testing and monitoring to break the cycle.

I believe that show transparent the agent should be depends on the needs of the user and the context as well. If the context is pretty simple, like recommending content, drafting a message, then a short rationale should be enough because as mentioned "sometimes users just want the answer quickly". However, in more crucial cases, such as healthcare, finances, business, etc - the agent or system should provide more reasoning and detailed audit trail (if required) to secure trust.

How transparent ? I think is a grey area question. Can AI be transparent on logic used behind providing solutions ? The answer is Yes and No. The answer is as simple as the prompt provided and resources provided by user and as complex as are we providing a knowledge base and references to AI or sending AI on a goose chase on open internet. The transparency of AI agents depends upon what are we providing it as an input. At my place of work: banking customer service domain where decisions can significantly impact a customer's financial life, AI transparency is not just a nice-to-have — it's crucial. We can look at it from three different perspectives. Depending on the level of complexity we have build an AI agent in a customer service environment. If it is low risk and low stakes or high risk and high stakes. AI Transparency in Low Stake Transactions: · Short rationale: A brief explanation like “Based on your credit score and income, you're eligible for a lower interest rate.” · Confidence score: medium, but helpful to show how certain the AI is. Why: Customers want quick answers but appreciate knowing why they got a certain suggestion, so when we provide a rationale that because of your low credit score this is the best interest rate you can get. It satisfies the customer’s query. Why is this low stake? Because it is just an information and customer might not be loosing anything monetarily. AI Transparency in medium risk and Medium-Stakes Interactions (e.g., loan pre-approval, document verification) · Steps of rationale: what can be shared : Outline key factors considered (e.g., income, employment history, credit utilization). · Audit trail: Since this info is internally logged for compliance and review, not necessarily shown to the user. Why: Customers may want to contest or understand decisions, and regulators may require traceability. For e.g. if a home loan application gets rejected or rate of interest changes upon careful review of applicants credit history, customers will definitely seek explanations. The AI agent build might not provide the rationale behind the decision taken since it is based on a lot of internal criteria and due diligence by specific branch managers. Now if we consider a AI agent transparency in High Risk and High-Stakes Transactions or Interactions (e.g., loan rejection, fraud detection, dispute resolution) · A more detailed explanation is necessary: A clear, legible reasoning with references to policy or thresholds is necessary so that the customers get a complete picture of why a certain decision was taken, what is the basis. · Audit trail: It should be available for internal review & regulatory compliance. · Confidence score: Important to show uncertainty or borderline cases. Why: These decisions directly impact customer’s financial status, morale and can cause frustration or financial harm, so trust and fairness are critical. AI needs to be fair and transparent when the stakes are high. How to Balance Explanations and Simplicity Draw the line based on user intent and impact: If the customer is just browsing for options, keep it simple. If the customer is making a decision or facing a rejection, offer layered transparency — start simple, but allow deeper insights on request. We should lead with a progressive disclosure: Display short rationale first. Offer the customer, a “Why was this decision made?” button for more details. We can also give downloadable audit logs or summaries for compliance officers or advanced users. Golden Nugget Mining : Now what are some best Practices for AI Transparency in Banking We should use simple language: Avoid technical jargon when explaining decisions. Be open and consistent: If the customers with similar queries fall under same criteria, ensure similar cases have similar explanations. Opportunity of a VOC : Let customers contest, provide feedback or ask for clarification. Comply with regulations: Align with GDPR, RBI, or other local financial regulations on automated decision-making.

Below is how I will manage versions of AI flows and prompts in a claims processing scenario, where things are constantly evolving based on feedback from claim examiners, auditors, and compliance. 1. Keep Track of Changes While building claims-processing AI assistant, the prompt that guided the “claims eligibility check” step worked… but only for the first few weeks. Then, business rules changed, compliance flagged some outputs, and examiners started giving us feedback. Instead of editing the prompt and hoping for the best, I store every single version of my flows and prompts in a company GIT repository Each branch is new iteration — for example, feature-improve-prior-auth-check. I clearly document why I made the change: When I deploy a new version, I tag it in GIT and log that version ID in our monitoring dashboard, so when a claim examiner says, “The bot did not process a specific scenario,” I can instantly see which version they were using. 2. Documenting the Story Behind the Change Clearly document story behind the change in order to delineate why I made that particular change v2.1.2 — 2025-08-15 Change: Updated “denial reason explanation” prompt to include ICD-10 lookup when code not in local cache. Why: Several claim examiners escalated cases because the bot said “code not found,” even though it existed in the database. Expected Impact: Reduce “code not found” errors by 20%. This makes it easy for me to tell the story of the bot’s improvement over time 3. Testing Before I Roll Out I never just push changes live. In claims processing, one wrong rule application can delay thousands of claims. Below are few things I follow Shadow Testing: I run the old and new prompts side-by-side on 100 recent real claims (with PHI data masked). Regression Suite: I maintain a set of tricky test cases — like coordination-of-benefits disputes or secondary insurance retro adjustments — to make sure the new version doesn’t break things that used to work. SME Review: I share sample outputs with our senior claim SME for human- in loop- scoring. They tell me if the new explanation is actually clearer or just longer. 4. Metrics tracking and feedback from team After Deployment Once the new version goes live (usually to 10% of examiners first), I: Track auto-adjudication accuracy — if it dips, I know something’s off. Collect feedback tied to the exact version. Categorize any errors: prompt misunderstanding, missing data, or wrong business logic. This way, I don’t just hear “the bot is processing incorrectly” — I know why. 5. Protecting Against New Problems I’ve learned the hard way: never delete a working version. I keep the last stable prompt ready so if my experiment tanks, I can roll back in minutes. In claims processing world , the cost of a bad AI update is delayed payments, or regulatory fines or angry providers - un term seriously impact customer satisfaction By treating flows and prompts like living assets with a documented history, I never lose track of why something changed, and I can always prove whether the change actually helped. It’s not just version control — it’s trust control.

At the bank we have a clearly defined handing over process when a solution is deployed. From the development to the User testing and approval to deploy in Production. The Application team together with the Project Manager ensure that a proper handling over milestone as part of the project closure is done when the Application is handed over to business. In the project framework, when the project closure is done proper documentations are provided and most of the time, the application owner is the key stakeholder owning the application post-deployment. For example, we have recently deployed a self-onboarding application using AI. This allows the customer to initiate the process without having to go to a branch and wait in the queue to get served. Below process map depicts the different stages where AI facilitates the interaction with customer without the bank having to allocate additional resources to assist the customer. The following key components are documented as part of the hand over which happen when the solution is stabilised and working fine. 1. Technical Documentation 2. Operational Documentation 3. User Manual 4. Train the Trainer ( Champion ) 5. Performance and Monitoring 6. Governance and ownership 7. Change Request 8. Compliance and data protection policy 1. Technical Documentation The Technical team will prepare the release documents which will includes the rational behind the design and architecture of the application. Data sourcing , regular updates and what third party tools used to transform raw data will be included as part of the documentation Setting up of the different environments like the Production /Disaster Recovery/ User acceptance testing , all these environment will be needed to maintain a long term stability of the application. All these information will be useful for Audit review purpose. 2. Operational Documentation Deployment plan and rollout plan will give and overview of how it was deployed and how it can be roll back in case of malfunction of the system . Configuration set up and access rights given to which roles are important facts to allow for future references in the event of troubleshooting . 3. User Manual Both Technical and Functional documentations are required as part of the handover. This helps for a better understanding of the functionality of the system and limitations. The input required from business side in terms of mandatory fields and what should be the output are mapped in these manuals . Well defined guidelines are published in order to maximise on the usage and its potentials and just the solution can keep its effectiveness in the process. 4. Train the trainer ( Champion concept) With the deployment of the solution, it is important to have workshops and training done with both , the technical and functional users. This is done to showcast the capabilities of the system.Identifying champions in each key departments allow to form users as subject matter experts, these people will be the L1 support for assistance to queries from users and customers. Building up FAQs and feed the AI solution allow the customer to interact with a chatbot for basic level of request and queries 5. Performance Tracking and Monitoring Clearly defined Performance Metrics and Key performance indicators (KPIs) meticulously agreed upon during the development phase set as a baseline metrics for future performance evaluations. We do have Monitoring Dashboards which provide valuable insights into the system performance and complemented by Red Flag alert mechanisms in the event of significant performance degradation, data drift, or service interruptions. . 6. Governance and Ownership As part of the handling over of the solution , the different roles and responsibilities need to be properly defined which ensure continuity and scalability. The following key stakeholders need to fulfil their part of the process. • Product Owner that ensure overall business alignment and budget management with organizational goals. • Technical Owner that ensure regular ongoing maintenance of the infrastructure and implementation of critical updates. • Data Owner that ensure the accuracy, integrity, and availability of data necessary for the AI system's operations. • Support Team which is a dedicated group tasked with addressing user inquiries and providing solutions to minor issues, fostering a smooth user experience. • Escalation matrix which is important and clearly mapped out procedure for escalating issues, ranging from operational glitches to critical model performance challenges, along with designated contacts for first-line, second-line, and third-line support. 7. Change Management Every changes requested by business need to go for proper approval process in order to maintain a consistency in the modus operandi of the solution . The changes should go through the Change Management process which follows ITIL framework. 8. Compliance and Ethical Considerations Data Privacy Detailed documentation outlining the methods used to handle personal or sensitive data, including robust practices for anonymization or encryption to protect user privacy is critical to the success of the deployment . All the above checkpoints are done in order to maintain clarity and completeness of the solution deployed,

It is crucial to provide ongoing feedback to AI agents so that they can learn from the to keep providing updated information. Let us assume that we have an AI agent that converts legacy code to a cloud-native language during system migration. We would need a feedback loop to be as structured and domain-aware as the migration process itself. Below are some techniques to collect, interpret and act on real-world feedback (from users, supervisors, or performance data) to continuously improve the agent : - 1. Feedback Collection – a) Developers reviewing AI converted code can flag code blocks with recurring issues like syntax errors, performance issues or deviation from architectural guidelines. b) AI generated report that shows the confidence score for each converted code. c) Track time required for manual remediation of AI-converted code and post deployment deployment metrics like execution time, resource consumption of running migrated code in cloud environment. d) Testers and Migration leads can keep track of the recurring issues and statistics around it. 2. Feedback Interpretation – a. classify feedback into types — syntax/compilation, semantic mismatch, security compliance gap etc. b. Consolidate issues to identify patterns in migration c. Compare AI generated report ion confidence score vs the reviews conducted by developers, testers and migration leads 3. Act on the Feedback – a. Fine tune model based on frequently occurring error patterns b. Update prompt templates and transformation rules with explicit project-specific coding standards (naming, architecture patterns, security requirements While it is important to optimize the performance and outcome of the AI agent, we can prevent overloading by manual resolution of minor formatting issues , or instead of reviewing every conversion, we can prioritize low-confidence or high-complexity conversions. Thus the agent will not only convert code but also learn from every migration cycle with feedback loop designed to catch errors, preserve best practices to evolving cloud practices.

It's hard and sensitive for a financial services organization to deal with customer complaints, hence an AI agent is quite crucial. This is a very serious situation that needs to be dealt with in a careful, polite, and lawful way. This is a planned way to help an AI agent perform the appropriate thing in these kinds of situations: Chosen Process: How to Handle Customer Complaints in the Financial Services Sector Why it's hard and important: Customers have a lot of varied feelings. The SEC, FINRA, and GDPR are all laws and rules. Needs to know what's going on, such how the client has talked in the past. You need to do a couple things: find out what the problem is, talk about it, and then fix it. How to Keep AI Working and Running 1. Getting the prompt ready: How to Keep the Agent in Place Using the Role and Intent Method: At the start of the meeting, let the agent know what the tone and goal are. "You are an AI that helps people with their problems." Your main goals are to be clear, understand, be right, and, if you need to, move higher. Don't make any assumptions. Every time, read crucial items twice. Effect: This way of looking at things makes the agent immediately ready to be careful and pay attention to the user. 2. Flow Limits: How to Keep the Agent on the Right Path Divide the procedure into smaller steps, each with its own rules: Acknowledge: Be sure you understand what the issue is. Clarify: Use fixed dimensions like date, transaction ID, and client effect to get information. Putting things into groups, such urgent, legal, and technical, is called triage. Route: Either fix the problem or move it up. You can do this by utilizing logic flags and modifying the state between modules. Don't go forward until all of the important inputs are locked. For example, if the complaint isn't clear, stop what you're doing right now. 3. Checkpoints: Things to Do to Make Sure the Built-In Method Is Right: Add checkpoints before doing something important to make sure it's right. "To be clear, you're talking about a $1,200 charge that was questioned on June 3, 2025." Is that actually true? Effect: It makes it less likely that there will be a misunderstanding and makes sure that the AI and the user agree on the facts. 4. Questions to help you understand: Questions that are proactive and take the situation into account Instead of asking, "What went wrong?" try saying: "Please tell me what happened right before the problem." "Have you tried to fix it yet?" Use templates that match the type of complaint for follow-ups that are specific to the location. 5. Dealing with red flags: things that make you feel awful and make things worse. How to do it: Teach AI how to look for signs that things are becoming worse, like A lot of thoughts like "I'm so mad" and "This isn't right." There are words like "lawsuit" and "compliance" in the law. What to say: "I know this is really annoying," therefore you should know how people feel. The human escalation workflow should start on its own when certain conditions are met. 6. Things you can't know: How to Stop Giving Out Too Much Information: Use short response templates and seek for help if you need it. "I wrote this down for the people on our team who make sure we follow the rules." They will get back to you in a day. "This happened because it was hard to compare data from different countries..." Control: Based on how serious the complaint is, choose how many tokens and how much information to supply. 7. Things that help you recall and go over short sessions Check the facts every now and again to stay on track: "Here's what I've come up with so far: 1) They charged too much on June 3; 2) They haven't answered my support request since then; 3) I'm asking for a refund and an apology. Pro: It keeps both sides on the same page and makes it easier for conversations with more than one turn to go well. What happens in real life Checkpoints and modular flow make sure that things don't happen again or go in loops, which helps things run more smoothly. Boundaries help you stay on the right side of the law and make it easier to go forward. Using prompts and summaries that take tone into account indicates that you care about your users and know what you're doing. Conclusion AI agents can deal with tough situations rather effectively, but only if the interface is good. Usage of prompt-framing based on role-based, progressive flow control and empathy related checkpoints all together results in organized process but yet focused on customer/person. This enables the business to run smoothly with people on track with low risk and trust.

This response is a summary that fits the requirements: The ESG Risk Assessment of the Investment Analysis Process and Decision Making indicates that more and more organizations in the financial services industry are utilizing AI models to help them with challenges that have to do with the environment, society, and governance. They gather and organize a lot of unstructured data from other people, like news stories, company reports, and ESG evaluations. These AI-generated summaries could assist analysts figure out how to score companies on crucial ESG parameters and what risks they might face. They can also explain their clients where the data came from, how it wasBN was collected, and what it was used for. People often write down the ESG problems that a company has had to deal with in the past year. Next, consider about how these kinds of worries could make investing more dangerous. The way a prompt was made affects its reliability, tone, and accuracy. If you don't make a prompt appropriately, it could lead to: You are making a general statement when you say "Company X has environmental problems" without stating what or when. People say that a model is hallucinating when it thinks there are problems that aren't in the data. For instance, if the analyst hears too many accusations, it might influence how they think. You can't tell Company X to "write down all the ESG problems they've had to deal with lately." Things that are wrong: The term "all" makes it sound like everything is there, which the model needs to know. What do you mean when you say "problems"? Are you talking about threats to your reputation, claims, or problems that have been fixed? "From August 2024 to July 2025, write a summary of verifiable ESG controversies about Company X that were reported in credible news sources or regulatory filings." This could be a better suggestion. Please tell us what the claims are, what happened, and how the company responded. Use language that isn't biased to figure out what could go wrong. Why It's Better: If you write down what sort of source it is and when it happened, it will be easier to grasp. It asks for proof at different levels, which makes it more reliable. Shows how to change the tone of a summary to make it sound less terrifying or unfair. Being useful and imaginative This modification is in line with what people in the real world need since people who work in finance need to be able to make fair decisions based on facts. People won't have to guess or deceive if they know what the risks are. You can also use the same prompt templates for more than one ESG concern, such ignoring labor standards or scandals in governance, or for multiple types of inquiry, like looking at ESG ratings. This suggests that it can grow and compare information. You can also use the same prompt templates for more than one ESG issue, such scandals in governance or labor breaches, or for different sorts of research, like putting ESG ratings next to each other. This indicates it can get bigger.

Using myself as a practical example pre my knowledge of prompt engineering and my use of ChatGPT, I have used AI to write content scripts, respond to emails, draft the flow of strategy sessions for a business leader's workshop, design SOPs, training modules, and presentation scripts in a global manufacturing context. I can tell for a fact that the results I have gotten post my knowledge of prompt engineering and design shows how prompt design can elevate or limit the outcome. My initial prompts was something like "Create an SOP for the Logistics team" or "What caused Line 2's downtime yesterday" but has now evolved into "on yesterday’s downtime on Line 2, using available sensor logs, operator notes, and maintenance history, identify the most probable root causes and suggest low-cost, high-impact fixes relevant to skincare batch making process" or "Create a logistics SOP for a skincare manufacturing plant in Nigeria, including job grade responsibilities, escalation paths, tool access levels, and cross-functional dependencies.” I think the prompts given to an AI model is critical because it determines whether the output is surface-level or genuinely actionable and within the context of the problem you are trying to solve. The improvement didn’t just help me get better results; it helped the AI model understand my context more deeply. In other words, better prompts led to smarter AI support which in turn leads to better decision making.

The Bank has a Complaint Desk, comprising of a Complaint Officer and a Coordinator, working under Retail unit and reporting to Head of Retail. This is a unit that has lots of manual steps in its process and involvement of multiple staff within the bank whereby the collaboration of Human and AI is a must. The process starts from the different channels that the bank has , where it can register complaints or a request from the customer via 1. Face to face by customer 2. Through phone 3. Through email 4. Website 5. Letters / Suggestion Box The nature of the request can come in the form of a query or a complaint and this is where there is a filtering done by the person acknowledging the request in order to decide on its nature : 1. Inquire on type of complaint. 2. If query; 2.1 liaise with staff /department concerned. 2.2 To respond to client immediately and resolve issue. 3. If it is a complaint, 3.1 To ask client to sign a complaint form 3.2 Inquire on the issue, root cause, details from branch staff concerned or relevant department. 3.3 Escalate to Complaint desk by email under copy to Head of Retail. 4. Issue letter to client upon advice from Complaint desk 5. Log on LDC in case any operational loss 6. Prepare paper for approving authority in case of any refund to client. 7. Send letter to HOR for 2nd signature. Each department or unit has its own way of handling the request or complaint in the following way: 1.1 SME Level 1. If it is a complaint, 3.1 Received by RM, he/she shall handle the client & resolve the issue 3.2 If issue not resolved, log on to complaint ticketing system IN most of the cases, complaints are received through mails or website. Based on the fact it is an administrative activity, IT operator channels these request to Complaint desk, who in turn sends mail to SME unit for inquiry. 1.2 Corporate Level 1. Once a Query/complaint is received at the unit, same is sent to HOD. 2. HOD sends to the respective RM looking after the portfolio of clients for whom the query/complaint was received. 3. Matter is tackled at RM level after consultation either within the department or with any other department concerned with the case and HOD is copied/informed of all steps/progress. 4. Copies of the correspondences pertaining to the complaint are kept in file. 1.3 SAM Level 1. Most of the time Complaints desk channels complaints to SAM 2. HOD SAM refer the complaint to the RM in SAM handling the file. 3. RM calls client & tries to resolve the issue. 4. If unresolved, escalate to HOD 5. HOD tries to resolve the issue, talk to client 6. Letter is sent to client 7. Update Complaint desk Ultimately all request or complaints are channeled to the complaints unit and the internal process requires lots of follow up to ensure prompt response to the customer 1. Upon receipt of complaint, the details of the complaint is log on an excel format. 2. Then Identify whether it is a query or complaint. 3. Inquire & follow up with respective business unit 4. If matter resolved, update the excel 5. If unresolved, escalate to senior management 6. Prepare letter to customer 7. Prepare reports for Complaint forum (Compliance) 8. In case of any compensation, give assistance to Business unit to prepare paper for approval. 9. Prepare reply letters to regulator for cases reported by them 10. And report submission on a monthly or quarterly basis. In all the above units , either the RM/HOD/Complaints officer and Complaints coordinator are involved in handling the request of the customer. If AI has to be introduced to this process , then the following changes will happen in the process and certain activities within the roles of the front liners will be changed or eliminated. Nowadays the use of mobile and website are more adapted to the trend of live of people. And more and more people make use of these means to send their request With the introduction of AI, The activities at the initial stage which determine whether this is a complaint or a query can be handle by AI just by feeding in some parameters for the system to be able to channel the request to the appropriate stakeholder Also the frontliners like the branch staff will not have to attend to this tasks of sending this to the department or stakeholder concerned and do a follow up By providing some filters at time of input by the customer can already save the bank time and resources to complete this task. On top of it also , due to regulatory requirement, the bank has to acknowledge the request or complaint from the customer within a minimum delay. Which at time if done by human , this can be omitted or done outside the SLA The recording and tracking of the request or complaint will be done by the AI without anyone having to log the record on the system . This also reduced the tasks of both the officer and supervisor who are involved in the creation , verification and allocation of the ticket. On the reporting part , whereby the bank has to send a report to the regulator on a monthly or quarterly basis, this can be handle by the system through AI. In a nutshell the collaboration of Human and AI works smoothly in this scenario, since some optimization of the existing process was done to eliminate manual intervention as far as possible through AI , thus allowing the different players in the process to have more time to spend with the customers. For this process, the roles of the different stakeholders might not change drastically, it might be that some training to the OLD school ageing staff is needed, since they will have to adapt to the new process in collaboration with some of the steps being done by AI instead of Human.

As a part of the Business and process Excellence team’s across different organizations I have been involved in numerous projects involving process optimization and implementation of RPA’s. Having managed omni-channel contact centers, I have closely worked on implementations of ACD’s (automatic call distribution) and Autonomous IVR's for telephony set-ups. And have been fortunate to be a part of Live Chat implementation projects involving Chatbots in recent years. With the expansive capabilities of AI, what we had set out to implement was an interactive, self-learning, autonomous “Dynamic Packaging Module-DPM” for an OTA’s Holidays business. The idea was to integrate the DPM and Chatbot on the site, which would be able to handle Customer Inquiries related to the packages involving mostly customizations and any related questions to T&C’s, payments, offers, amendments and cancellations, upgrades, visa, insurance etc. immediately. Given, the fact that 60% of the inquiries were for “Customized Packages” it was important for our teams to be able to handle such customers by offering complete details “tailored” to their requirement in the First Interaction itself thereby improving conversions and customer experience, hence the decision to go ahead with the project. So, in addition, to building a Backend Product loading and managing tool involving the integrations of supplier API’s, Direct / Extranet contracts, GDS and Airline feeds, offline packages, ground transportation and ancillary services. We needed a Front end, which allowed both our teams and the End customers to be able to “Customize packages” as per their needs on real-time basis, while making use of the AI assistant. And finally, all interactions, requests, quotations to flow into the CRM and ERP tool for overall booking management, human agent and AI model training and retraining, Quality Audits using AI for all interaction summaries. The Front-end or the “DPM” was built using AI/ML algorithms, using on-premises LLM’s, which were trained on some 100,000+ itineraries for countries across the world. The AI QA tool had been trained on Sales scripts, product knowledge, soft skills and Quality monitoring sheets to score conversations. Both these tools had their respective KB’s. Let’s look at the project implementation in terms of restructuring of processes, new roles and responsibilities both for Human agents and the AI, handoff protocols laid down, Human-AI Collaboration framework and efforts, Phase-wise implementation plan, Training requirements for the team and Revised KPI’s. AI-Integrated Sales Process for Holiday Package OTA Revised Process Flow - Initial Customer Contact • The AI chatbot initiates the first interaction within 15 seconds of a customer visiting a Landing or Package page, offering assistance to “Customize a package” or offer the “off-the-shelf products”. • The AI qualifies the lead by gathering destination preference, departure city, travel dates, trip duration, no. of persons traveling, travel intent – couple, family, honeymoon etc. and budget, if any. • By using set parameters, the AI assesses the complexity of the inquiry. • Basis the complexity scores the system directs the next steps for the handling of the query. - Queries Handled by AI (70% of initial contacts) • Inquiries regarding package availability and pricing • Standard booking changes (dates, passenger details) • FAQ’s bout policies (cancellation, baggage, visa requirements) • Basic information on destinations and weather • Payment and confirmation support details • Post booking status updates and reminders - Human Agent Involvement (30% of initial contacts) • Multi-city itineraries that require customizations • Group bookings for 10 or more people • Special requests incl. accessibility, special medical needs, meals etc • Managing refunds that involve exceptional circumstances or exceed standard refund policies • Catering to High-value bookings above $X,XXX per person • Handling customer escalations or requests where customers explicitly ask for human assistance Evolution of Human Agents: From being Order Takers to Travel Consultants The agents are now responsible for offering strategic travel advice, instead of just processing standard transactions. While focusing on customer motivations, recommending upgrades, and creating WOW experiences, the agents now act as Travel Curators. - New Core Responsibilities of Human Agents • Curating complex itineraries involving multi-city routings • Developing long-lasting relationships with HNI and repeat customers • Promoting and upselling premium experiences and services • Handling sensitive customer service-related issues • Ensuring the quality of AI recommendations by regularly Auditing conversations • Training the AI system through feedback and updating information in AI KB for retraining AI – Human Handoff Protocols • Full conversation history and customer profile to be provided by AI to the agent with a ‘Warm transfer process’ to be followed. • The system marks the ‘Priority’ to define the urgency of the inquiry and provides the previous interaction sentiment. • AI provides recommendation to the agent on the suggested action items based on customer type. • AI to ensure customers don’t have to repeat themselves through smooth context transfer to the human agent. • Handoff scenarios include – Complex itinerary requests, emotional or sensitive situations, HNI customers, technical issues or complications. Collaboration Framework b/w AI-Human AI support by Human Agents - AI training and feedback • Agents flagging incorrect AI responses through live monitoring and post-chat audits. • Updating new product knowledge directly into AI KB. • Creating templates for recurring complex scenarios. - Quality Assurance and Risk management • Reviewing AI conversation transcripts on regular basis for accuracy. • Providing feedback for adjusting AI tone and persona. • Identifying and recommending cultural nuances for regional and international customers, which AI might miss. • Ensuring AI responses comply with industry regulations, company policies and meets customer privacy standards. - Data enhancement • Tagging successful booking and upselling conversations for AI learning. • Provide inputs on travel seasons (peak, off-season, shoulder) and booking patterns for AI to incorporate. • Highlighting objection handling patterns to assist with AI’s response improvement. • Providing SWOT analysis against competitors for improving AI pricing algorithms. • Regularly calibrating to improve AI response accuracy. Real-Time AI Support for Agents - Dynamic Sales Assistance • Recommending agents with relevant packages during live conversations. • Alerting agents of any ongoing flash sales or inventory updates mid-conversation. • Offering real-time access to competitor pricing during conversations. • Identifying and recommending potential upsell opportunities based on customer profile and sentiment analysis. • Offering alternatives in case the customer rejects the initial recommendation. • Providing real-time destination specific expertise when agents handle requests for unfamiliar destinations. - Quality and Administrative Task Automation • 100% QA of customer conversations. • Monitoring conversation sentiments and raising flags when necessary. • Identifying knowledge gaps and recommending relevant training’s. • Summarizing of customer conversations and updating CRM records. • Scheduling follow-up emails, callback reminders based on the conversation and commitments made by the agents. - Booking flow optimization • Automating mailers to customers with booking and document checklist. • Pre-filling of booking forms with customer data received during query and quotation stage. • Validating Customer’s travel documents for their visa, immigration and validity requirements, ticketing and booking confirmations etc. and alerting agents for any identified issues. • Generating the payment and cancellation schedules basis booking and travel dates • Populating instalment and discount offers recommendations. - Performance Optimization • Tracking conversion rates by agents and recommending relevant trainings for improvement. • Sharing insights into best practices and successful sales patterns. • Recommending staffing for aligning Work force according to peak intervals. • Tracking C-SAT Scores depending on the type of interaction and customer feedback. Gradual Phase-Wise Implementation Strategy • Phase 1: AI manages only basic FAQs and availability checks. • Phase 2: Introducing booking processing and payment handling. • Phase 3: Adding multi-city, complex package bundling and recommendation algorithms. • Phase 4: Achieving the complete integration with personalization features. Training Requirements for Agents • Tech skills for navigating through the AI systems. • Advanced consultative selling techniques training. • Training on cultural sensitivity for regional and international markets – New markets. • Advanced CX training for crisis management and de-escalation techniques. Introducing a Dual layer of KPI’s - AI KPI’s • Response time – Less than 5 seconds for 95%. • Deflection rate – 75% or more of routine inquiries handled by AI. • Accuracy rate – 95% correct information provided for each interaction. • Conversion% - Glide path targets for the first 06 months starting from 05% to 10%. • Availability – 99% uptime during business hours. • Handoff efficiency – 95% of AI to human transfers happen without customer repeating themselves or getting frustrated. • C-SAT scores - Above 4.25 out of 5. • Average order value – Increase in AOV’s by 15% MoM. - Agents KPI’s • Complex booking conversions% - Glide path targets for the first 06 months starting from 15% to 30%. • Up sales % - 30% of the bookings converted accept upgrades. • Customer Lifetime Value / Repeat customers - 30% increase in customer retention for agent handled customers. • Handoff efficiency – 90% of the AI transferred interactions to be resolved within first agent interaction. • Escalation resolution – 95% of the complaints to be resolved within 48-72 hours • AI training and feedback contribution – Achieving weekly and monthly contributions targets towards AI improvement and training. • C-SAT scores - Above 4.5 out of 5 Following an integrated approach, we were able to transition from a transactional to a consultative sales process. While AI handled the maximum volume and routine tasks, this allowed the human agents to focus on building relationships and solving complex problems. As a result, we were able to see faster response times, improved CX, and increased revenue per booking.

The Swiss Cheese model is a risk analysis and management model developed by James Reason. The model illustrates how accidents or failures can occur due to a combination of various factors. Think of a stack of Swiss cheese slices with the slices representing layers of defence, such as safety procedures, training or system design within a process or an organisation, and the holes representing the potential weaknesses or failure points in those defences such as human errors, system malfunction or a procedural flaw. In isolation, the holes may not create an issue; however, when they line up, they create a clear path for failure. Used in various industries such as healthcare, aviation, engineering, etc., the model helps 1. Analyse past accidents and identify areas for improvement. 2. Spot individual failures as well as systemic vulnerabilities 3. Understand that a single safeguard is never enough 4. Focus not only on adding additional layers, but also improving the quality of the existing ones (i.e. shrinking the holes) In my organizational processes, the cheese slices or the layers of defense are, 1. Process Documentation - Standard Operating Procedures for clarity on what needs to be done and how. 2. Technology and Tools - CRM, project trackers, automated reporting. 3. People Structures – Skilled team members and role clarity. 4. Audits & Reviews -Regular Check-ins, internal audits and client feedback loops 5. Training & Capability Building - Internal or external training programs, onboarding procedures, or process trainings. 6. Governance Frameworks - Approval systems, decision rights or escalation ladders And the holes i.e. the weak spots are, 1. Process Documentation: Outdated or poorly communicated procedures 2. Technology & Tools: Incorrectly implemented or data not updated 3. People: Poor Delegation, lack of ownership or unclear roles 4. Training: Theoretical but no practical implementation, old training systems or irrelevant modules 5. Governance - Micromanagement, or too much red tape, or unclear escalation If the holes line up, there can be client delivery issues, missed deadlines, and miscommunication, leading to business losses. However, by using Business Excellence principles, pitfalls can be avoided, as below, 1. Map your defences – Document all the defence mechanisms in the workflows by assessing where you rely on people, where on technology and where the decision-making is slow. 2. Identify and Prioritize risks – Run a FMEA analysis to spot where the holes might align, which layers are the weakest and which ones overlap. 3. Close the gaps – Use the PDCA, DMAIC to tighten each layer (shrinking the holes) such as, · Improve SOPs - standardise, train, test understanding, periodic reviews · Review CRM or reporting dashboards for data accuracy · Audit the impact of training, not just attendance 4. Design for resilience – Set up redundancy where needed by implementing backup approvers, escalation triggers, or multiple checkpoints so that even if one layer fails, another catches it. 5. Imbibe a culture of prevention – Encouraging teams to look beyond firefighting and ask what hole in our process allowed this to happen, and how can we patch the holes? For Eg: During any transition, · One layer is the communication plan. · Another is the handover process · Third is the data access and permissions · And fourth is internal task tracking If the comms aren’t clear or the handover isn’t fully documented or someone forgets to update access – that’s a failure chain. Final thoughts The Swiss Cheese model helps us see failure as a whole system and not just someone’s screw up. When combined with Business Excellence tools, you not only patch holes but build stronger and smarter slices.

Possible root causes for lapse in control phase Employee or workers resistance to Change: Lack of buy in from production team, Staff, if they find the new update or process difficult or confusing may revert to old habits, Insufficient periodic dip checks or audits: Without regular evaluation of a improved step or process, it is difficult to catch non adherence and the processes start reverting to previous, less efficient states. Inadequate Training: employees may not have had adequate training to adapt fully to new systems, which in turn might lead to improper use that negates expected improvement in the output Shift of business focus: Over time, changes in company strategy, customer expectations, or market conditions can render initial improvements less effective. Lack of executive level Support: Lack of technical or managerial support and disinterest can cause the momentum brought by the change to reduce drastically. Tools and Techniques to Maintain Improvements Smart Automated Tools: Implement tools that automate parts of the system to minimize human dependency and ensure consistency. For example, using AI-driven systems for intelligent call routing based on live data analytics. Recreate Framework and deploy updated SOPs: Documenting the new processes in detail, including any new workflows and best practices, helps ensure everyone is on the same page. Periodic Review and Audits: Regular management reviews like weekly business review and process audits can help maintain focus on the importance of the new process or method of working, ensuring it remains aligned with strategic objectives. Continuous Refresher Courses: Share the benefits and positive impact of the change on the business outcome Continuous training helps reinforce the importance of the new tools or processes and ensure staff remain proficient in its use. Motivates the staff to imbibe change VOB use to have a Balanced scorecard and revise Metrics and Dashboards: imbibe the efforts metrics of changed or updated process, Implement metrics to continuously measure performance against key performance indicators (KPIs) like average handle time and customer satisfaction rates. Dashboards can provide real-time insights to assist in making proactive adjustments. VOC and VOE: Once a change or improvement is implemented, we should regularly gather feedback from both customers and employees to identify areas for further improvement and address minor issues before they become major problems. Example Suppose a contact centre introduced a new call distribution system that reduced average call wait times from 10 minutes to 2 minutes. A smart use of AI agent project is run and implemented for automated categorization of query type by a Bot or IVR before the call reaches a live agent, Initially, this was a significant improvement, but over time, wait times began creeping back up. Upon analysis, it was discovered that: Impact from New Agents: Incoming staff were not being adequately trained on the new system due to high turnover rates and hurry in implementation. Lack of involvement of training department in live projects resulting in new agents being unaware of the changed or improved process System Updates: Regular updates that could enhance system efficiency were not being utilized due to a lack of technical support. To address these issues, the contact center: Implemented a mentorship program, pairing new employees with seasoned agents to ensure smooth transitions. Scheduled monthly system reviews to incorporate and evaluate new updates. Developed a dashboard allowing real-time tracking of call metrics, providing instant feedback to operators. In a Nutshell By implementing these strategies and tools, contact centres can better maintain improvements gained from new routing systems in Control Phase, adapt to changes proactively, and ensure customer satisfaction remains high. Keeping systems resilient against backsliding requires constant vigilance and adaptation to the changing environment.

Insightful answers to the question. The best answer is from Swapnil Madhav Chaukar. Well done! Answer from R Rajesh is also a recommended read. My 2 cents - The AI solution needs to be made a process document and like we revise process documents (e.g. - SIPOC, process maps, SOPs etc) after every process change, we need to revise the AI solution after every change!!

Let us answer this, with the help of an example. Cricket Ball Manufacturing process We will consider a Cricket Ball manufacturing process as a hypothetical example Background: Now, let us say we have 2 types of Cricket Ball manufacturers XYou Sports Inc and YMe Sports Inc. XYou Sports Inc produces ball in Country 1 where the ball will turn (Spin) more a YMe Sports Inc produces ball in Country 2 where there is more swing. Now the national team of Country 1 plays with the national team of Country 2 in one of its Cities… Therefore Country 2 feels uncomfortable playing to a different brand of Cricket ball (made by X) as it has got more spin. So, to counter that, Country 2’s Cricket board looks for XYou Sports Inc manufacturer to supply that brand to its national team. The steps mentioned above, depict the AS-IS process view AI-Enabled Process where human improvements were made As XYou Sports Inc manufacturer found out that, multiple countries may need multiple type of Cricket Balls, it decided to leverage AI in getting useful insights and ultimately getting things done quickly. As a first step, it ensured that AI process (and hence solution) was setup that addressed that the ball will be more conducive to Spin (turn) when implemented in Country 1. The AI was fed with data, and it was more like a supervised learning for the AI. The humans in the loop, were doing manual verification of key differentiating parameters/factors such as weight of the ball (in Ounces), size of the ball, bounce, seam, stitching, materials used … AI solution evolves with the Process Once XYou Sports Inc received the request from Country 2 as well, the team realized that they can still explore on AI technology and leverage it for bettering this process. It started to identify what kind of key differentiating factors can be included as part of the AI. The AI solution agent now has to look for an additional information(parameters) – how the prevailing weather in Country 2 impacts the cricket ball, how does the pitch/soil (in Country 2) supports this type of Cricket Ball (manufactured by XYou Sports Inc). How did the MBB help here for XYou Sports Inc? The MBB envisaged a plan. As this process is an ongoing journey and may have uncertainties and require constant feedback from the stakeholders (eg, National teams, Cricket Boards of respective countries..), he felt using Scrum as an agile framework might help the team to navigate through the uncertainties that might prop up.. He suggested the team to use this and also requested them to put a Kanban board for radiating information on a routine basis to all stakeholders He had put a high-level plan that stated as Planning Type Objective Remarks Vision Planning Developing a scalable tech excellence support Agent that supports the Cricket ball manufacturing process of XYou Sports Inc Release Planning Release Phase 1: Defining the AS-IS process to AI (Basic Knowledge Integration of the Cricket Ball manufacturing process) Release Phase 2: Re-imagining the defined process in phase 1 by considering all key parameters that is required to make the ball suitable in multiple countries (Country1, Country 2) Sprint Planning Release 1 Sprint 1 – Creating the KB agent and AI agent for defining few basic parameters (such as weight and shape) Sprint 2 - Defining the KB agent and AI agent for remaining parameters such as materials, bounce, seam, stitching Release 2 Sprint 3 – Defining the KB and AI agent for additional parameters such as pitch soil, weather conditions of a country (which can influence the ball behaviour) Duration: 2 weeks Note: Sprint 1 will have a minimum viable product (setting up the AI process and adding very limited functionality) Release 1 had 2 Sprints and Release 2 had 2 Sprints Sprint 2 had the feedback incorporated from the stakeholders that came from Sprint 1. Similarly for Sprint 3, 4 the team got feedback from the previous Sprints and incorporated the feedbacks which were relevant to them Subsequent releases had subsequent Sprints based on the emerging needs for the manufacturer. Integrating AI into improvement Cycles and AI process got adapted with proper feedback As we see from the above table, the team leveraged Scrum as a framework to build an iterative and incremental development of their AI based Cricket ball manufacturing process. What was a cumbersome exercise in getting huge amount of data across multiple parameters and few parameters which were changing based on the Country and its weather conditions (worst if there are multiple weather phases – summer, winter, autumn..in a country) , it became much more simpler when done with AI. Every Sprint produced some incremental values keeping the stakeholders happy. The AI system improved itself over a time period as it gathered more data (in the usage of how the cricket balls behaved in those countries across weather seasons) and had reinforced learning to improve itself With every Sprint, there was a Sprint Review that happened where the stakeholders were presented with the finished work (in the Sprint). Whatever feedback was given was taken into consideration and those were implemented. Strategic Role of MBBs in maintaining alignment The MBB was able to devise a strategy for AI solutioning of this Cricket Ball manufacturing process. He was able to setup a vision, release plan and then help the team to come up with Sprint goals for each of the Sprint making the team adjust to the uncertainties Conclusion: We saw here how AI enabled process with human efforts can navigate through complex scenarios/situations in an incremental manner addressing emerging needs with quick feedback cycles. The most important thing is quick release(deployment) of the value that you want your stakeholders to get and have continuous exploration of the market needs and adapt your system accordingly.

Example - Think of a situation in Human Resource Management domain where AI model is built to analyze and reduce employee turnover. In the absence of a MBB, automation/transformation team defines the problem as “predict which employees are likely to leave”. The model accurately predicted attrition rate basis the available data but failed to point out the root cause of the problem and reduce attrition. Only later by involving a Master Black Belt and post thorough root cause analysis it was tabled that the primary cause of attrition was lack of career growth opportunities within the organization which led to high attrition. The primary root cause was not a part of the HR data that the AI model was built on, hence incorrect problem statement will not result in the desired output. MBBs contribution to problem definition stage – 1. MBBs will ensure that the project goal aligns with the Business Case 2. CTQ drill down – Business Case will then be linked to operational objective and post baselining the current performance RCA will be conducted 3. RCA – Using techniques such as 5-WHY, Fishbone analysis , Affinity etc. root cause of the problem can be arrived at 4. Hypothesis Testing – Testing the hypothesis to identify the true trend before full scale deployment of the model is as imperative as the pre-work like clear problem definition Conclusion – To ensure successful deployment of AI model which gives desirable and effective output the pre-work done involving clearly defining the problem statement, it’s link to the strategic and operational objective, CTQ drill down, RCA etc. is imperative for MBBs to ensure the project is directed in the right direction.

Let us explain this question with an example. Example: In an IT environment, there were multiple teams situated across different locations. Most of the teams were struggling to deliver the goods (requirements) on time.. The development effort, were more due to several factors such as complexity involved, lack of coordination/alignment amongst the team members who are from different vendors. This means that every team had team members with different vendors. The Problem statement framed was to improve the development cycle time of a requirement, for these teams. Based on the problem statement, one team was piloted for understanding, the current ecosystem and was taken, for implementing the changes. But the solution provided by AI, did not provide the expectation that the business owner wanted. There were multiple teams, each team was unique – One was a component based team, another a cross-functional agile team, another distributed agile team (members from different places), another team was collocated… So with such diverse teams, the result was not having any improvement on the development cycle time for all the teams. The piloted team was a collocated team and the improvement shown was virtually nothing, as this was a well-knit team and the team operated at a better cycle time. This is where a MBB helped in rebranding the problem statement. With the help of a Project Charter, the MBB helped to write a proper business case and Problem Statement. The business case stated the impact of the cycle time taking long reasoning out the why part (as mentioned above) and which types of teams (distributed - cross-functional teams, component teams) are contributing (where it is happening) to this problem and how long it has been there. Then the Problem statement was defined as “Lot of Development effort goes on due to the complexity involved and lack of coordination amongst team members, in distributed component-based and cross-functional teams”. Then SMART Goal was developed stating 2 pilots would be done – one on Distributed Cross-functional team and one on Distributed Component-based team – within a month’s timeline . Out of Scope will be Collocated teams Once this clear-cut strategy was established, then it was clear to all the stakeholders as what to do and then the execution became laser-focused and the teams were able to improve (Reduce) their cycle teams Conclusion: It is essential to frame the problem statement right, irrespective of whether AI is used or not. AS an AI Solution Architect, we can encourage the person who provides the problem statement to use prompt engineering/fine tuning (depending upon how depth you want to explore something). We can also help them in using Chain of Thoughts, if they don’t know (assuming we also don’t know), how to structure our thoughts and leverage the problem statement further As a MBB, its important to ask insightful queries while going through a problem statement. The MBB has to know from the Business Owner/Problem Statement provider - Who are the impacted stakeholders - The impact/implications of the problem need to be understood - How long the problem exists - In-Scope/Out-of-scope - Tangible/in-tangible benefits need to be understood. Apart from this, the MBB need to know with the AI Solution Architect/AI teams - What is the AI solution trying to do - Will the AI solution be short term/long term Based on this response, the MBB will be able to make the AI Solution Architect design the solution catering to these needs. Thus you can see how with proper framing of the problem can yield the right results and how not doing the right framing of the problem, resulted in no improvement (in this case). But in general it can be like you may loose customer confidence, stakeholder dissatisfaction etc.. when you get a solution which does not provide value.. An analogy could be - while ordering food through online, it is like you get an eatery which you have not ordered, when you are hungry but looking for your ordered item!!

A robust control framework/guardrail is required in any process driven organization to sustain the process, create accountability for process owners, transparency, fairness and compliance with regulation. It integrates lawful and ethical considerations, ensure compliance, and operational guidelines to ensure that AI systems are developed and utilized with responsibility. Elements of AI governance framework · Align initiatives with organizational goals · Stakeholder engagement from the beginning and two way communication · There should be fairness, accountability and transparency establish · There should be Risk management strategies. that is, assess and mitigate risks for the process data including all scenarios and impact assessment before deployment · Embed AI taking overall process into consideration, ie core business workflows · SOP to be updated for the new process and procedure and all scenarios · Train team about the process changes and provide clear communication · Performance monitoring and evaluation Who should be included Role Responsibility Executive Sponsors/senior leadership They are there to set the vision and provide direction for AI initiatives. Culture embedment and collaboration are promoted by them Legal and compliance teams They establish the policy and ensure legal and compliance guidelines followed. They conduct the implemented AI monitoring. Business Process Owners Define SLA, test and validate the AI developed solution. Data Scientists / AI Engineers Build and maintain AI solution and ensure the best practices followed IT / Infrastructure Teams Ensure scalability and secure deployment of solution. MBBs / Process Excellence Leaders Ensure alignment with Lean/Six Sigma principles and eliminate the process waste In order to maintain both Agility and Control, we can integrate prioritization framework and for agility there can be cross functional squads working in sprints. we should define SLA for performance review and monitoring and light weight governance for minimum viable product. there should be performance dashboard as control and for agility there should be feedback loops from users

The first and foremost traditional metric that maybe misleading in an AI enabled system is Efficiency (Process Cycle Time). Traditionally the way human performance was judged basis faster generation of output can not be used as-is to gauge the performance of an AI enabled system. This is because fast AI decisions can be inaccurate, hiding issues like poor model performance. Example - In HRO domain AI enabled systems might reduce time to hire but may overlook certain key organizational initiatives like diversity and inclusion and may source candidates faster but may result in poor job fit due to lack of analysis. Also Cost per hire can be an outdated metrics because AI enabled systems are reducing CPH in the short term but may fail to track Employee Churn Rate due to multiple factors initially ignored. New Metrics - Instead of these metrics we may prefer to use Candidate Fit Accuracy - Percentage of AI selected hires meeting the success criteria (performance, retention etc.) over a longer period of time. Whenever building a system, developers need to focus on striking the right balance between TAT and Accuracy and enable the model to achieve both.

No matter what metric we produced or how fast it gets produced, it provides zero value if it isn’t accurate, i.e., do we trust it to make critical decisions upon it? The further away we get from knowing how we produced a metric, the less we trust it. Because AI can “do” the formulas, calculations and algorithms for us – so quickly, and easily, we could mistakenly put more value on the speed of AI than the actual value we "think" it's producing. There are a couple of wise sayings that I’ve heard in my life. “Keep and honest man honest”. That idea has also been said, “Trust but validate”. Why? If an honest man is honest, then why do we have to make sure he’ll remain honest. Because everything has a bias and can make mistakes, including what metrics AI has created for us. So, for me, the most valuable metric is the one I can trust, whether created by AI or not. The balance is being able leverage the power and speed of AI as well as validating everything it generates. Ergo, Agentic AI. Outdated metric - “Average Employee Training Hours”. Investing in employee training is seen as a direct indicator of capability building, the “level-up” people skill-sets. Traditionally, we think, the more hours, the better. However, AI-driven learning that can hyper-target content to precisely educate where a particular skill is lacking. The hours spent now becomes less relevant but rather the efficacy and application of the learned skill become what is important. If companies rely on “average training hours”, they could easily over-invest in traditional training methods while missing the AI-enabled learning pathways. It shifts from input (hours) to outcome (applied skill). New metric – “Value Realization Velocity (VRV)”. This metric measures the speed at which AI-driven insights or recommendations are converted into tangible business value. Every business has struggled and failed to move an idea from “concept” to “production” to and to know it’s real “realized value”. VRV could track: - Time from AI model deployment to first measurable business impact (e.g., first dollar saved, first customer converted. - The percentage of AI-generated insights that lead to actionable changes within a given timeframe. - The monetary value generated per unit of time from AI-driven initiatives (e.g., incremental revenue per week from an AI-optimized marketing campaign). This metric directly ties AI initiatives to strategic business outcomes because it pushes beyond mere technical performance to demonstrate tangible ROI and agility. Because the AI-driven economy will, and is, so fast-moving, this will be paramount for Business Excellence.

Well in my opinion, every organization want to function more efficiently, effectively, and successfully. At times, this involves modifying and improving an existing procedure. In other cases, it entails taking a step back and radically rethinking how the job is done, particularly in light of the latest developments in AI, which provide new approaches to activities that previously required human intervention. When to Improve the Existing Process: Start by determining whether this procedure still accomplishes its goals but simply needs to be more effective. If so, conventional techniques of improvement, such as eliminating pointless procedures, cutting down on wait times, and standardizing activities, might be beneficial. Such modifications are beneficial when: -Although there are delays or inefficiencies, the process is steady. -Employees are following the procedures, yet the results differ. -Though the experience might be more seamless, customers are typically happy. -Though insights are difficult to act upon, data is already being used. In these situations, we enhance what is currently in place by streamlining operations, educating employees, modifying schedules, or streamlining chores. When to rethink the Process Using AI: But sometime, the procedure is outdates, many things are manual, or designed to solve non-existent problems. Simple improvement won't result in significant change in certain situations. We then pose the question: Can AI enable us to radically rethink the way this task is carried out? Consider rethinking with AI when: -People devote too much time to commonplace, repetitive jobs. -Despite producing a large amount of data, the process is not being used efficiently. -Consumers anticipate quicker, more individualized service that is impossible for people to provide alone. -The present procedure wasn't created to meet the demands of the modern world years ago. AI can anticipate human needs, make judgements in real time, automate replies, and recognize patterns that humans would overlook. This has the power to change a process in a way that adds more value as well as making it faster. AI is capable of making judgements in real time, automating replies, anticipating human needs, and seeing patterns that humans would overlook. In addition to making a process faster, this can fundamentally change it in a way that adds value. An Example: A hospital could wish to shorten patient wait times. How? Option-1: Process enhancement might entail quicker check-ins, clearer communication, or improved staff scheduling. Option-2: Conversely, AI-driven redesign may incorporate systems that prioritize patients according to urgency and historical health data, predictive models to identify busy periods, or virtual assistants to respond to enquiries instantly. Though they meet distinct needs, both strategies are beneficial. One last observation: Before taking any action, need apply critical thinking: -Is this procedure still appropriate? -Will it make enough of a difference to be improved? -Or is it time to start over, using AI to help us accomplish things in a more intelligent, contemporary manner? Enhancing and rethinking are methods for different contexts and are not mutually exclusive. Making the correct choice is essential for sustained success.

In order to pick the best suited solution to fix a problem, generate ideas targeting the root cause of the problem which is identified at the analyze phase. Start by developing potential solutions for the validated root causes by using techniques such as: a) Round Robin Style of Brainstorming b) Critical Thinking via Six Thinking Hats Evaluate solutions using: a) Pugh Matrix b) Delphi Technique c) Multi Voting d) Nominal Group Technique or/and FMEA. These tools will help prioritize one solution over the other considering multiple criteria such as cost and potential benefits. For e.g. Pugh Matrix may be used to draw comparison between two alternative solutions in a software company tackling slow bug fixes, if they should add more coders or enhance their testing tools. Both the solutions can be scored using multiple criteria such as a) Cost b) Data Privacy c) Accessibility d) Training Requirements e) Time to deploy etc. Changes can be first tested using DOE (Design of Experiments to evaluate process inputs generating desired output. Post which the selected solution can be implemented (pilot run) in a smaller sub-section of the process before full scale implementation.

So by using structured root cause analysis and hypothesis testing techniques, we can validate the findings and propose targeted corrective strategies. In addition, we need to start by analyzing different sets of data to understand trends, timelines, forecast accuracy etc. Data sources may include: Sales Data (12 months): Seeking trends to indicate a mismatch between high-demand periods and stock availability leading to Lost Sales impact. Inventory Records: For stock-out frequency of high-volume SKUs across the above period. Forecast Accuracy Reports: To check for variance from actual demand. Delivery Timeline Logs: For Supplier delivery windows. On-time delivery performance >95%, suggesting minimal impact from logistics delays. Key Insights identified : Forecast Error Correlation: High forecast inaccuracies coincide with stockout events, especially for promotions and new product launches. Replenishment Lag: Time between forecast input and stock arrival often spans 10–14 days, reducing responsiveness. Demand Volatility Ignored: Forecasts do not factor in localized demand surges, social trends, or weather-related events. Inventory Turnover Analysis: Low turnover rates in some categories suggest misallocation of inventory resources. Analysis Techniques Used 1. Process Mapping (SIPOC Analysis) 2. Root Cause Analysis - 5 Whys Technique Problem: Products are frequently out of stock 3. Fishbone Diagram (Cause & Effect Analysis) 4. Hypothesis Testing Test 2 primary hypotheses using statistical analysis: Hypothesis 1: Delivery delays cause stockouts. We found out that on-time delivery has remained stable at > 95%, so we can confidently disprove this hypothesis. It's noise, not the cause. Hypothesis 2: Poor forecasting drives stockouts. By mapping “Forecast accuracy %” against “stockout incidents” we can find out the correlation between them. With a significant p-value we have statistical proof that this is the primary driver. How to Prevent Chasing the Wrong Cause - Define the Problem with Data - Begin with a problem statement from the Measure phase (e.g., "From Q2 to Q4, stockouts on A-list items increased by 40%, contributing to an estimated $1.2M in lost sales."). This is our ultimate benchmark. If an identified "cause" doesn't statistically impact this metric, it's not the right one! Gemba walk – Speaking with the store managers, the inventory planners, and the logistics coordinators is important as they have qualitative insights that can help with data analysis. Use a combination of Tools: Using a combination of tools like the Fishbone Diagram provides the structure, the Regression Analysis provides the statistical proof, and the 5 Whys provide the deep dive.

How to Pick the Right Numbers: When I measure patient wait time, the temptation is to just pull what is easily available (example: Time from check-in to doctor consult). But that may miss important context or root causes. To ensure my metrics reflect the real process: Start with the SIPOC & Process Map This helps to visualize where delays happen, so I don’t just rely on one data point. May be the wait starts before check-in. Use the Voice of Customer (VOC) If patients complain “I waited for long time before anyone response me in helpdesk”, then I need to measure time from entry to first contact, not just from check-in. Focus on CTQs (Critical to Quality elements) Understand “What makes or breaks the experience?” A single long wait might impact satisfaction more than average wait times. Break Down the Wait Time into Key Components: Instead of measuring one overall figure, I would track: Time at registration Time waiting for nurse Time waiting for doctor This breakdown helps identify specific bottlenecks in the process. Balance Leading and Lagging Indicators Lagging: total wait time Leading: number of staff on shift, patient volume/hour This gives early warning signs. Tricks to Catch Bad or Incorrect Data Gemba Walk or Time Study I will observe the actual process for a few patients. Compare this to what is recorded in the system. Discrepancies are red flags. Check for Missing or Impossible Values Negative wait times? 0 minutes for a complex step? We could spot these using basic filters in Excel. Run a Control Chart or Histogram Early Abnormal spikes, flat lines or bimodal distributions might mean inconsistent measurement or data entry errors. Logical validate Compare time stamps from two sources. For example: EHR vs manual logs Front-desk data vs patient-reported times Use Operational Definitions Everyone must record data the same way. “Check-in time” must be clearly defined. When patients enter the door, or when they are logged into the system? Pilot Before Full Measurement I will try data collection method on a small sample. Fix issues before scaling. Quick Example: If we track only “average wait time” from EHR timestamps, but the nurse doesn't log start time until after triage, we miss the true wait. Solution: Add a “patient greeted” timestamp by the front desk as a new data point based on what we learned from process observation and VOC.