Leaderboard

In 2022, Klarna launched a full-speed AI deployment automating most of its processes using AI solution and realized cost savings equivalent to 700 FTE. One of the processes they automated was their Customer Service Support. After a while, customer complaints and dissatisfaction ballooned. Customers claimed that AI responses were too generic and unhelpful when dealing with real-life problems. While AI solution like chatbots can handle simple and repetitive queries, emotions or complex issues were not addressed. Klarna realized that while AI solutions promise speed and cost savings, it can compromise service quality and customer satisfaction. Klarna decided to rehire employees to address poor service quality and customer complaints. This is a testament that AI solution isn’t about replacing humans, but rather, enhancing the human workforce with smarter tools and better support system. As an MBB, following were my recommendation: 1. Use VOC to identify critical customer requirements (CCR) where complex issues and customers needing to talk to human to solve their concerns will surface. 2. AI solution aims to enhance customer experience leveraging on personalized interaction for higher engagement. This was not apparent in case of Klarna. It is recommended to take advantage on Deep Learning capabilities of AI solution. Such model can identify complex patterns, making it suitable in image recognition, voice recognition, and natural language processing. 3. Lastly, while drawing the to-be process map, HILT (human-in-the loop) principle is recommended. In cases of complex customer concern, AI can escalate the concern to its human counterpart to further assess the given concern and provide necessary resolution.

Let us explain this question with an example. Example: In an IT environment, there were multiple teams situated across different locations. Most of the teams were struggling to deliver the goods (requirements) on time.. The development effort, were more due to several factors such as complexity involved, lack of coordination/alignment amongst the team members who are from different vendors. This means that every team had team members with different vendors. The Problem statement framed was to improve the development cycle time of a requirement, for these teams. Based on the problem statement, one team was piloted for understanding, the current ecosystem and was taken, for implementing the changes. But the solution provided by AI, did not provide the expectation that the business owner wanted. There were multiple teams, each team was unique – One was a component based team, another a cross-functional agile team, another distributed agile team (members from different places), another team was collocated… So with such diverse teams, the result was not having any improvement on the development cycle time for all the teams. The piloted team was a collocated team and the improvement shown was virtually nothing, as this was a well-knit team and the team operated at a better cycle time. This is where a MBB helped in rebranding the problem statement. With the help of a Project Charter, the MBB helped to write a proper business case and Problem Statement. The business case stated the impact of the cycle time taking long reasoning out the why part (as mentioned above) and which types of teams (distributed - cross-functional teams, component teams) are contributing (where it is happening) to this problem and how long it has been there. Then the Problem statement was defined as “Lot of Development effort goes on due to the complexity involved and lack of coordination amongst team members, in distributed component-based and cross-functional teams”. Then SMART Goal was developed stating 2 pilots would be done – one on Distributed Cross-functional team and one on Distributed Component-based team – within a month’s timeline . Out of Scope will be Collocated teams Once this clear-cut strategy was established, then it was clear to all the stakeholders as what to do and then the execution became laser-focused and the teams were able to improve (Reduce) their cycle teams Conclusion: It is essential to frame the problem statement right, irrespective of whether AI is used or not. AS an AI Solution Architect, we can encourage the person who provides the problem statement to use prompt engineering/fine tuning (depending upon how depth you want to explore something). We can also help them in using Chain of Thoughts, if they don’t know (assuming we also don’t know), how to structure our thoughts and leverage the problem statement further As a MBB, its important to ask insightful queries while going through a problem statement. The MBB has to know from the Business Owner/Problem Statement provider - Who are the impacted stakeholders - The impact/implications of the problem need to be understood - How long the problem exists - In-Scope/Out-of-scope - Tangible/in-tangible benefits need to be understood. Apart from this, the MBB need to know with the AI Solution Architect/AI teams - What is the AI solution trying to do - Will the AI solution be short term/long term Based on this response, the MBB will be able to make the AI Solution Architect design the solution catering to these needs. Thus you can see how with proper framing of the problem can yield the right results and how not doing the right framing of the problem, resulted in no improvement (in this case). But in general it can be like you may loose customer confidence, stakeholder dissatisfaction etc.. when you get a solution which does not provide value.. An analogy could be - while ordering food through online, it is like you get an eatery which you have not ordered, when you are hungry but looking for your ordered item!!

MBBs have deep expertise in process optimization and structured problem-solving, and their role in structured problem-framing approach is paramount, let’s understand this especially in AI solution implementations in high-touch environments like Contact Centers. The problems of Mis-framing leading to ineffective Solutions Contact Center Chatbot Deployment Scenario: A contact center faces long customer waiting times. To quickly reduce Average Speed of Answer (ASA), leadership launches an AI chatbot to handle frequently asked inquiries, aiming to ease pressure on live agents and thereby reducing ASA. Surface Level Problem statement: The project team stated the problem as “We have long wait times because our agents are overwhelmed. Let’s implement a chatbot to handle FAQ’s and reduce wait times by 50%.” While investing 100,000’s of dollars to develop an AI chatbot, train it on FAQ’s, and deploying it as a FPOC for all customer inquiries. The chatbot in itself was technically proficient, using NLP and ML algorithms to interpret customer requests. What was missed: The team did not perform a thorough root cause analysis. Key problems included understaffing staffing during peak hours (only 60% of required agents), inadequate training programs that left agents unprepared for complex product inquiries, fragmented knowledge management systems that forced agents to search multiple databases, and high employee churn (45% annually) from workplace stress and limited career advancement opportunities. The Effects of Mis-Framing on AI Performance Following the chatbot deployment, AI gave generic responses to complex customer issues, causing greater frustration among those needing detailed technical support. Instead of reducing call volume, the chatbot generated additional calls from customers seeking clarification on the AI’s responses or requested immediate escalation to human agents. Findings based on BSI Analysis: The pre-implementation baseline, calculated with the Bottleneck Severity Index formula (BSI = Volume × Cycle Time × (1 - First Time Right%) × Severity), showed: • Volume: 1,200 calls per day • Cycle Time: 8.5 minutes average handle time • First Time Right: 65% • Severity: 3.2 (scale of 1-5) • Baseline BSI: 11,424 Post-chatbot implementation revealed: • Volume: 1,350 calls per day (increased due to chatbot escalations) • Cycle Time: 11.2 minutes (longer due to frustrated customers) • First Time Right: 58% (decreased due to inadequate agent preparation) • Severity: 3.8 (higher customer frustration) • New BSI: 20,365 (78% increase) The AI solution made matters worse: with customer complaints increased, call deflection remained below 15%, and net promoter score (NPS) declined further, and the organization having to face increased operational costs due to higher call volumes and longer resolution times. In addition to the above consequences, wastage of resources and loss of stakeholder trusts add to the negative impact of mis-framing on AI effectiveness. Suggested Practical strategies for MBBs to improve problem framing in AI projects a. Engaging in structure problem statement development using LSS thinking and tools o Use SIPOC and VOC to clarify process boundaries and understand demand drivers o Defining CTQ’s and linking them to customer pain points rather than convenience metrics like ASA. b. Apply BSI for comprehensive bottleneck assessment o Train the project teams in evaluating each BSI component Component Key MBB Questions Volume Is the call volume avoidable or failure demand (e.g., repeat issues, unclear policies)? Cycle Time Are agents slowed down due to poor tools or unclear procedures? First Time Right % What’s the root cause of low FTR? Training, systems, or information gaps? Severity Are we prioritizing automation for high-impact or low-impact queries? o Trend Analysis: Ongoing BSI monitoring to spot patterns and predict bottlenecks before they become critical. This enables teams to address root causes proactively instead of reacting to symptoms. o Use Pareto analysis of BSI to identify Top drivers and guide the AI strategy accordingly. c. Facilitating structured problem definition workshops and fostering stakeholder engagement o Run problem framing workshops that bring together diverse perspectives and stakeholders (operations, IT, HR, training and customer experience.) o Use tools like affinity diagrams and root cause analysis techniques to identify underlying issues that may not be apparent to any single stakeholder group and before confirming the need for AI. o Translating insights into well-structured problem statements (what is wrong, where, when, to what extent and impact on CTQ.) o Making use of the RACI matrix to ensure comprehensive problem understanding. • Inform: Keep executive leadership aware of project progress and findings • Consult: Gather input from frontline agents, customers, and IT teams • Responsible: Include customer service managers, training coordinators along with operations teams and customer experience specialists in problem definition sessions • Accountable: Work closely with the project sponsor on the project approvals. d. Deploy Control Measures Before Automating o Test hypotheses through small-scale pilots that test technical functionality and business impact of the proposed solution before scaling AI. o The pilots need to monitor impact on Leading Indicators (FTR, Escalation Rate, Post-Chat Survey Scores) to validate alignment of proposed solution with identified root causes. Hence the mis-framing of problems in AI initiatives may lead to technically accurate but operationally ineffective solutions, wherein MBBs are mandated with the task of diagnosis with discipline. Using BSI as a key metric identifies real process friction points and thereby guiding the organization to ask the right questions before investing in AI, and ensuring the final solution addresses the true constraints, improve customer experience, and deliver sustainable business value.

Following are the problems that are surfaced when AI solution solves a wrong problem. 1. Misleading results - When AI solutions are not aligned with the real or exact problems, They will give results bit will not solve the actual problem,. 2. Inefficient use of resources - Wastage of efforts invested in building, designing and architecting the solution. As the solution is inefficient in solving the real issue all these efforts are a waste which results efforts invested in terms of Man Months, which can directly impact cost. 3. Loss of Trust - As Solutions addressed wrong issue's stakeholders start to loose confidence on AI solutions that can start losing trust and affect sponsorship for budget allocation and project execution. 4. Consequential Harm - If AI solutions solves wrong issues in critical areas like healthcare, Legal & compliance it can lead to serious consequences that can harm the reputation of the organization and will result in negative impact on brand. Following are the some of the ways in MBB's can help in identifying correct or real problem statements for deploying AI solutions. 1. In Depth Analysis - MBB's go exactly deep for finding out the actual problem with relevant data analysis for backing the problem statement. 2. Stakeholder engagement - MBB acts as facilitator for driving engagement 3. Data driven insights - MBB uses data to drive statistical inferences for validating results and decisions taken during execution of the AI solutions. 4. Benefit Analysis - MBB helps in visualizing the benefits in terms of value, cost and efforts invested.

Example - Think of a situation in Human Resource Management domain where AI model is built to analyze and reduce employee turnover. In the absence of a MBB, automation/transformation team defines the problem as “predict which employees are likely to leave”. The model accurately predicted attrition rate basis the available data but failed to point out the root cause of the problem and reduce attrition. Only later by involving a Master Black Belt and post thorough root cause analysis it was tabled that the primary cause of attrition was lack of career growth opportunities within the organization which led to high attrition. The primary root cause was not a part of the HR data that the AI model was built on, hence incorrect problem statement will not result in the desired output. MBBs contribution to problem definition stage – 1. MBBs will ensure that the project goal aligns with the Business Case 2. CTQ drill down – Business Case will then be linked to operational objective and post baselining the current performance RCA will be conducted 3. RCA – Using techniques such as 5-WHY, Fishbone analysis , Affinity etc. root cause of the problem can be arrived at 4. Hypothesis Testing – Testing the hypothesis to identify the true trend before full scale deployment of the model is as imperative as the pre-work like clear problem definition Conclusion – To ensure successful deployment of AI model which gives desirable and effective output the pre-work done involving clearly defining the problem statement, it’s link to the strategic and operational objective, CTQ drill down, RCA etc. is imperative for MBBs to ensure the project is directed in the right direction.

AI has been trained to resolve the issues basis the historical data and symptoms which is solving the issue at the surface level however not solving at the root cause level. Well, it optimized prediction but not prevention. A best example can be forecasting/ predicting employee attrition in an organization AI tool can flag the early leaver basis the historical data and the anticipation basis the tenure, performance rating and engagement level, their survey results and the salary. however no one deep dives into the reason for disengagement and does not solve the actual issues solving the culture and other professional concern related to growth, employers, role etc MBB analyse and help in framing the right problem statement. they use SIPOC, Voice of employee and critical to quality tools to bring into the relevant KPI to solve the problem basis employees’ view MBB utilizes LSS tools like fishbone and 5 whys to get to the root cause MBB can question the past data and ensures the validity of the data and no seasonal data included to influence or provide bias MBB indicate how the process and its achievable result looks like before the AI model is deployed in solution. MBB define the actionable and intervention needed at al level and not just retention tacts only MBB bring in all the cross functional teams and stakeholders to avoid the silo approach and work collaboratively to resolve the problem with engagement Difference AI MBB AI detects and solves issues at surface level MBB deep dive and solves the problem by identifying the underneath root cause of the issue AI provides prediction accuracy basis the historical data & trends MBB ensures the business outcomes meeting the ned user/customer requirement and creating positive impact within customer’s life Generates speedy results via automation Generates values and take risk into account while delivering the outcome Biases basis the historical data Applies accuracy basis the current situation and fairness MBB act as a eyes and ears of the process and ensures that the solution deployed by AI fits in well and resolves the root cause and eliminate the process waste to optimize the process efficiently and effectively

I have not been trained or certified as an MBB but I can apply what I have learned in this course. Here's an example of an AI solution technically is working as it should but has become a part of the problem. Consider a business who has a customer service center and their customers are experiencing long wait times. In an effort to decrease the long wait times, they create a Chat Bot. After implementing this AI solution they certainly can see the the call wait times has significantly decreased because the Chat Bot can "answer" them quickly. So, technically, this AI solution is a success. Wait times have drastically decreased. But the company begins to hear from their customers how angry and frustrated they are, even more so than when they had to deal with long wait times. The business failed to understand that what they should have been really trying to solve was increasing customer satisfaction, not merely the symptom of addressing long call wait times. The Chat Bot caused greater unsatisfaction because customers now have to make repeated calls (even though they don't have to wait) because the many "simple" calls are often precursors to more complex issues and the Bot could not handle these, thus forcing customers to start over with an agent, which leads to more frustration. Also, agents may now have to deal with more calls from customers because the Chat Bot did not properly diagnose the underlying problem. This situation wasn't created by the Chat Bot, but by those who didn't have the foresight to really understand how they should have created the Chat Bot. At the end of the day, technology or technical solutions, such as AI, will not be blamed for these problems that arise. Those who created the AI solution will be. You don't want to be that person. Back to the original thought of creating an AI solution. The business thought it was to merely solve lowering long wait call times. But the real root of their issue was customer frustration and dissatisfaction. Their "AI solution" was focused on the wrong thing and it even caused a deeper problem for them How could this have been prevented? Using the FRT process and documentation which captures the Desired Effects (DE), the Undesired Effects (UDE), and the Negative Injections (NI) of any AI project and solution. FRTs can help to envision the ideal future state of an AI solution but also proactively identify negative consequences BEFORE a dime gets spent on creating the solution. The FRT would have captured the root cause by addressing and thinking through the UDEs and also creating NIs to create answers for these UDEs. Utilizing the FRT process and documentation, along with creating a very thorough and thoughtful BRD, would have greatly increased a proper AI solution that result not only in lowering call wait times, but mor importantly, raising customer satisfaction.

According to what VK noted under his forum questions, “Some people seem to be using AI platforms to find forum answers. This is a risky approach as AI responses are error-prone..” AI is created by humans who are prone to error. We must always remember this and be diligent to make sure AI will make the best decisions. “Making sure” will ALWAYS to be and, I believe, will forever be, a human responsibility. I can’t ever imagine anyone shirking their responsibility and pointing at the AI solution and saying “It’s the AI’s fault that we lost revenue”. Yes. It might have been that we trusted the AI agent to make the decision but ONLY after we allowed it to make that decision. So, the real accountability still falls back to a human. Knowing that AI is prone to make errors, just as humans have done to mitigate making our own errors, we created guardrails to increase proper decision making and better outcomes – ergo, Business Excellence. Think of AI as another person. But now you are responsible for the decisions and actions of that person. It will need oversight, accountability, and transparency to make sure AI is making the right decisions on our behalf. Here are some of the elements that I think could be included in a governance framework to ensure responsible, high-impact use of AI in a process-driven organization. Creating a governance team or committee to oversee all AI solutions. This team would comprise people from IT, the business, legal, risk management and defining each role and responsibility throughout the AI development, deployment and maintenance. For transparency and accountability, conducting regular impact assessments to identify potential risks, biases and consequences of AI-driven decision. Also, implementing techniques that can provide insights into the how AI is making its decisions, such as feature attribution or model interpretability methods. Lastly, performing audit trails that let us see the data inputs, processing and outputs the AI used to make its decision. For agility and control, using agile development methodologies to allow for rapid iterations and deployment. Using change management to capture the all the changes made throughout the development which can easily be reviewed, tested, and validated. Lastly, establish access controls to prevent unauthorized changes to the AI system or data.