Leaderboard

Domain & Project Context This case comes from a lean six sigma project I led in one of my previous company (Medical Equipment Manufacturing), focused on a critical sub-assembly line of Mobile Viewing Station (MVS), which is critical part of all C-arms medical machines we produced. This supported the Yearly revenue of 85 Million Euros, Monthly output/demand of MVS was 150, each MVS required one complete set of LCD Carriers. To give better context, MVS displays live the images from C-arms of patients for the surgery and doctor’s operates C-arms by MVS controls. Since MVS was the mandatory piece of final machine, if this line stops, C-arms assembly lines stops as well, thus impacting revenue. What Actually Happened We started seeing the severe rejections of LCD carriers at the MVS line. We observed minor shape distortion, visually parts looked fine, but during the assembly LCD monitors would not seat properly. At our end, rejection rate climbed to 85%. We looked at few process details, injection molding time 7.5 Minutes, followed by 10 mins in cooling fixture. We observed on paper, everything looked stable, however in reality, something in that window was introducing deformation. We attempted the rework to keep the production alive, with success rate of 35%, rework was clearly a containment action, not a solution for us. If I had AI Available then & How I Would Use It Today If I had to deal with this problem today, I would bring AI early. AI could help me quickly correlate: Rejection rates against injection molding cycle times Cooling fixture duration vs shape deviations Differences across mold cavities & production shifts Batchwise behaviour at the supplier Measurement variation between supplier & our plant Raw material behaviour (33% Glass-Fibre Nylon From BASF) Assembly fixture tolerance stake-up at our line It would have helped me to form hypothesis faster & narrow the investigation space. However, in my view that’s where AI’s role ends, these correlations are signals not evidence. It could not tell us which one was the true root cause. As LSS expert, it I treat AI output as a hypothesis accelerator not a validator. Statistical Confidence (What was good enough & What wasn’t) This project reinforced something I strongly believe, The level of statistical confidence required depends on how irreversible the decision is. We needed speed and for short actions like rework to protect the output, directional confidence was acceptable. But for irreversible decisions, statistical rigor was non-negotiable for us: Redesigning the molds Changing the cooling fixtures geometry Locking supplier process parameters Declaring the permanent fix of the issue With 85% rejection, limited rework success & 85 Million Euro revenue at risk, we relied on below: MSA at supplier Independent MSA at our plant Controlled trials on molding & cooling parameters Physical verification of carrier geometry at assembly No AI could replace the level of proof. Data Quality & Credibility (Still LSS Responsibility) One of the early discoveries we made was that measurement variation itself was part of the confusion, the same carrier set measure differently at the supplier & at our plant. Just imagine, if I had trained AI on that data without first fixing the measurement system, it would have produced confident but misleading insights. This reinforced a core LSS principle for me, before trusting any analysis by human or AI, validate the measurement system. In my view, AI processes bad data faster, it does not make it more credible. Correlation, Prediction & Causation This project clearly force us to respect the difference in between all three: Correlation: Certain batches & cooling windows showed higher rejection Prediction: AI could likely flag high-risk batches Causation: Only controlled testing proved that cooling fixture design and post-molding deformation were the real drivers Causation was confirmed only after we: Stabilizing the measurement systems, validating the material behaviour, redesigning cooling fixtures, updating the assembly fixture geometry. That sequence mattered for us. Bottom line- Where AI should accelerate & Where it must not In my view, AI should accelerate inEarly Pattern detection,Hypothesis prioritization, & Faster narrowing of investigation scope. However, traditional statistical validation must remain non-negotiable for us when: Root causes are declared, Tooling/mold changes are approved, supplier processes are modified, production release decisions affect the final product. On conclusion note, AI would help me move much faster at the front end, however when decisions affect 85M Euro revenue, medical equipment (patient safety) and irreversible tooling changes, classical lean six sigma discipline still defines what counts as proof.

Project : Reduce cycle time and improve quality of Narratives draft for property offering final memos. Problem statement : Current average time to produce a first draft Narrative ( Property overview, Market Overview, Borrower Sponsor, Maps & Aerials, Demographics, Crime Reports, Strenghths & Weakness) is 5hours per deal, with 16% of drafts requiring major rework due to : Missing or inconsistent data Grammar, UK/US English used interchangeably Misalignment across sections(e.g strenghths not matching the market facts) Goal : Reduce average drafting time by 30 - 40% and reduce work by 20 - 30% while maintaining or improving Narratives quality( rated by originators & underwriters) Where AI Fits : Auto generating data from known sources ( internal systems & vetted external websites) Gnerating a first draft of : Property overview, Market Overview, Borrower Sponsor, Maps & Aerials, Demographics, Crime Reports, Strenghths & Weakness Improving grammar, UK/US English usage, Overall stylistic alignment Analyst still : confirms source credibility, checks factual accuracy, adds nuance specific to the deal, approves/ edits strengths & weakness. Hypothesis Formation : In this project, AI outputs are best treated as structured hypotheses, for example: The property's location near XYZ business park is a key strength due to strong employment growth Crime rates in the submarket have declined over the five years, supporting stability of the assest Market rent growth has moderated but remains above long term averages. From an MBB standpoint: These are not facts, they are claims to be tested AI's role in Define/Measure is propose: Here is what might be true, here are patterns implied by the data I see. Guidance to analysts: Treat AI text as hypothesis rather than validated conclusions Use prompts that frame AI output as provisional. For example: "List 5 potential strengths of this property based on the data below, and mark each as High confidence or Needs verification with a short reason. Hypothesis testing : AI can help structure tests, but LSS will be led by humans. For Narratives quality, Hypothesis will be like H1 : Using AI to draft market overview reduces average drafting time by 30% without lowering quality scores H2 : AI generated strengths/weaknesses will be atleast 80% aligned with analysts final conclusion AI can help design experiments or sampling plans( eg: How many deals do we need to test to detect a 20% reduction in cycle time at 95% confidence level) and Suggest such metrics to track ( Time saved, number of factual errors, number of grammar corrections, etc) But the actual test( data collection, calculations, significance tests) should be performed in transparent tools like Excel, Minitab, etc. where we can see the formulas and logic. AI derived statistics ( eg: This is 90% likely) should not be treated as valid inferential statistics unless we have a clearly documented model and method behind it. Hence MBB needs to have rule of thumb that : AI is acceptable in forming hypotheses and in helping us set up tests AI is not acceptable as the only source of evidence that a hypothesis is confirmed. Statistical validation must be done through transparent methods. Statistical Confidence: We are building marketplace with in Berkie (In house AI) for all the prompts to be used for Narrative drafting In the marketplace initiative, there are two layers of confidence Confidence about workflow change (Process Improvement) : Does using AI actually improve the quality/productivity? Confidence about individual deals output ( deal level narrative accuracy) : Is the AI generated narrative for this deal accurate enough to use? Confidence in workflow change: Here as an MBB, we should use classic LSS experiments. Define Metrics : Drafting cycle time per deal, Number of factual corrections per Narrative, Number of grammar/style fixes, and Originator & analysts satisfaction score Design : Pre/Post study like N deals before AI, N deals after AI or Some analysts use AI, some dont, over the same period of time Analyze: Use standard tools (T tests, control charts) to confirm 1. Is there a statistically significant decrease in drafting time 2. Is quality maintained or improved Document : Effect sizes, confidence intervals and any risks observed(e.g common type of AI errors) Confidence about individual deals output: Here instead of 95% confidence in a strict statistical sense, we define operational acceptance criteria. For example : For an AI assisted Narrative to be acceptable as a draft, we require 1. 0 critical factual errors (data that could mislead credit decision) 2. less than or 2 minor factual discrepancies ( currently aiming for 97% accuracy, acceptable error is three low critical errors) 3. Grammar and usage of UK english errors below X per 1000 words 4. Analyst can review and finalize within 30mins for 90% of the deals. For an MBB perspective, we can: Use sampling and inspection like 1. Randomly sample AI drafts 2. Score them against a checklist 3. Track defects per unit and apply control charts Overtime we can characterize AI as a process with a certain defect rate, Just like any human process. Assessing data quality and credibility when AI is in the loop This is critical as the data for Narrative is pulled from multiple sources (Internal systems, public websites, vetted websites, etc) Separate source credibility from AI credibility : A) As MBB, We should enforce a clear hierarchy Source credibility rules(upstream of AI) : Internal : Salesforce/Omniview, Berkadia internal property/loan systems, internal market research has highest trust, Trusted external: Offical census data, reputable third party market reports, public crime data portal has highest trust, Open web or generic searches (google search) has low trust unless specifically validated. AI Usage rules: AI is allowed to Summarize and rephrase known, structured inputs from trusted sources and Highlight patterns or inconsistencies across those sources. However, AI is not allowed to freely invent data not present in the supplied sources and acras the primary source of record for quantitaive facts. B) Data lineage and traceability: To keep Lean Six Sigma like rigor, we need traceability For each factual statement in the AI narrative, analysts should be able to trace source and transformation We should ask AI to annonate paragraphs with references e.g(Source : Crime Report 2024 Q2) or (Source: Internal rent roll 2025-01) or generate a separate evidence log for the narrative From an MBB angle, this is like keep data collection forms and measurements system documentation for our Y's and X's. C) Measurement system analysis for AI and analysts : We will take AI and analysts as measurement devices for Narrative content. We wil run Gage R&R style excerise : we will select a set of deals we will have a) AI only draft(first pass) b) Analyst only draft(without AI) c) AI+Analyst (AI draft, then analyst edit) We will have independent reviewers ( seasoned and quality specialist) rate accuracy, clarity, grammar/UK US english usage, alignment between sections We will assess variations attributing to AI v.s Human v.s Combination. Where AI improves repeatbility/consistency(e.g style/alignment) v.s where it adds risk ( e.g subtle factual errors) This quantifies data quality and helps set where AI is safe v.s risky Where AI should accelerate decisions, and where traditional validation is non - negotiable A) Areas where AI can safely accelerate and automate: In the Narrative AI marketplace, AI is well suited for: Drafting and rephrasing : a) Converting bullet research into fluent US english paragraphs b)Enforcing consistent style and structure Summarization: Summarizing long third party reports like Crime Reports, Market Reports, Sponsor histories, etc Formatting and aligning : a) Aligning terminology across sections(e.g consistently referring to submarket names, asset class labels b)Ensuring internal consistency between section (e.g strength and weaknesses must refer to facts already stated in the narrative. Highlighting potential strengths and weaknesses : a) Suggesting deal strengths/risks based on the data provided b) Tagging statements as "requires analyst verification" where sources are weaker or ambigous Quality checks : a) Flagging likely grammar issues b)Enforcing US spelling conventions c)checking for obvious contradictions (We said crime is low in one section and high in another section of Narrative) Here an MBB can reasonably reduce manual efforts while keeping risk low, especially with defined guardrails and human review. B) Areas where traditional validation is non negotiable From a Lean Six Sigma and risk perspective the following should not be delegated solely to AI Quantitative facts that influence credit/investment decisions e.g Occupancy rates, rent levels, reports fulled from trusted source Casual statments and forward looking claims e.g Because of X, We expect Y, Crime reductions mean lower risk for the asset. AI may propose such statements however Analyst must evaluate casuality using domain knowledge and data. Use traditional reasoning and where relevant, simple statistical checks(e.g trend analysis) rather than trusting AI's intuition Compliance, regulatory and reputation risk : Any statement that touch on fair housing, sensitive demographic descriptions, legal regulatory interpretations must be reviewed and where necessary, crafted and reviwed by analysts. AI may help draft neutral language but it should not be the final authority. Final sign off and accountability: The accountable owner of the narrative is the analyst ( and ultimately the deal team) not the AI. MBB guidance should codify that a) AI is a tool in the Measure/Analyze steps b)Approve/Reject decisions remain with analysts. In conclusion, AI should treated as a powerful assistant, not a replacement for Lean Six Sigma rigor. It can speed up research and drafting the Narrative, but the standards for data validity, statistical confidence, and final judgement must remain with the analysts. Where the impact is high, traditional verification is still essential and where risk is low, AI can safely help us work faster and more consistently.

In Lean Six Sigma, the Black Belts have training to believe only that which is statistically verifiable: p-values, confidence intervals, root-cause confirmation by designed experiment. But AI changes the game. It does not always describe a why, but tends to indicate what has been connected or predictive, and in any case on a magnitude and at a rate beyond human ability. And then, when AI crops up an insight, such as, collaboration between transactional processes in a BPO violation of SLA, or non-obvious drivers of handling time, how should an MBB treat it? So how do you recognize when it is an evidence or a pattern that is yet to be accomplished? We can base this on a real life BPO situation. Domain: BPO - Large-Scale Finance and Accounting (Order-to-Cash) operations. Problem Statement: Within 6 months, decrease the Turnaround Time to less than 12 days in Technical Invoice dispute resolution against the current invoice dispute resolution of 18-22 days and still not increase write-offs and customer dissatisfaction. How AI helps here: An educated artificial intelligence model that uses more than three years of transactions, emails, tickets, and CRM data, focused on... Those invoices that are likely to become disputes are to be anticipated. Best anticipated resolution effort. Adding routing, prioritization and root-cause tags before the actual occurrence of disputing. The Core Tension for MBBs Lean Six Sigma was developed on an solid concept: Were we not unable to show its validity statistically, then we should not act upon it. AI takes a different approach by offering: High accurate prediction. Identification of patterns on thousands of variables. No classical hypothesis stating recommendations or p values. It is not the question of the usefulness of AI in the MBB challenge, but the question of what the insights provided by it have to do to gain the right to leadership. Hypothesis Formulation and Testing in an AI-Facilitated DMAIC: The classical hypothesis in our example on Order-to-Cash could be: H0: The TAT of the resolution of invoice is not related to the root cause of the dispute. H1: There are root causes that contribute to an increase in TAT of resolution to a significant extent. This would generally be confronted with: ANOVA / regression Clearly defined variables Controlled samples The manipulation of Hypothesis formation (Not elimination) by AI. In this instance, AI had led to the appearance of something other: The likelihood of becoming long-tail disputes is 2.4x higher with invoices where pricing inconsistency is partial and when emails are received by offshore customers when out of business hours. This observation was not determined by a pre-deemed hypothesis. It was based on pattern recognition in thousands of features. How an MBB Should Treat This: AI must be discussed as a generator of hypotheses and not validate one. For this project: Insights of AI were rendered into falsifiable hypotheses e.g. Proactive intervention on invoices with such features will decrease the TAT of disputes by 25 percent. Classical DMAIC discipline continued to be applied: No blurry definition (end-to-end TAG) Control vs. pilot groups Before/after comparison Key Principle: The AI cannot provide the answer to the question of why, but still MBBs need to create the evidence. Setting Statistic Confidence when AI is involved: The Temptation Our BPO case presented the AI model that demonstrated: The ability to predict the likelihood of disputes with 87 percent accuracy. High lifts curve and ROC. The temptation is to say: The model is correct - hence we take action. That does not qualify as Six Sigma thinking. What Confidence Means Now In the case of MBBs, the confidence should come to: Do we care whether the coefficient is significant? to Does this insight reliably respond to the improvement of the CTQ? What We Did in This Scenario: Invoices prioritization was done based on AI predictions in just one pilot region. We measured: Lessening the mean dispute TAT. Early resolution percentage percentage increase. None of write-offs or customer escalations. The results were statistically validated, rather than the AI model internals. Non-Negotiable Rule Process confidence is not similar to AI accuracy. It still requires confidence to be won via: Controlled pilots Measured deltas Stability over time Measuring Data Quality and Credibility in AI-Based Analysis: In BPO, AI Can Be very powerful. In this project, AI: Digested 1.2M invoices, messages and tickets in days. Patterns that had been identified had not been phrased by any SME. Normalized data quality problems that had been flagged by humans. Where MBB Judgment Matters. In spite of volume and sophistication: Artifacts of some "strong predictors" were: Legacy process exceptions Regional policy dissimilarities. Ineffective root-cause tagging. Historical inefficiency was the first aspect which AI was exposed to, and not the intended way to act. MBB's Existence of credibility. MBBs apply: What systems, what time period, what definitions Data lineage checks Before, during, after data definitions. Discrimination (regions, customers, type of disputes) SMEs validation- Does it sense operationally? Rule: AI can scale data. Only humans can assign trust. When AI Should Be Making Decisions faster vs When validation is Non-Negotiable: AI should speed up decision making when: It is possible to reverse the decision (routing, prioritization, alerts). The cost of being wrong is low The use of the AI recommendation is decision support rather than automation. Example from the Scenario: Using AI to address high-risk invoices to the initial stages and fast.. Traditional Validation cannot Be Compromised When: The decision modifies policy, controls or customer commitments. The effect is on compliance, revenue recognition or terms of the contract. The machine wisdom goes against familiar logic of processes. Example from the Scenario: Still needed: redesigning dispute ownership models, or modifying definitions of SLA. Statistical validation Pilot control groups Leadership sign-off The Bigger Shift MBBs Need To Make. AI does not remove Lean Six Sigma rigor. Where rigor is used, it re-positions. Hypotheses are transferred to after-insight. Median changes place an emphasis on statistical models to statistics on the process. The credibility of data is not a technical presumption, but a specific leadership role. Concluding Lesson of the BPO Engagement. AI accelerates insight. Lean Six Sigma has the right to take action. MBBs who treat AI as: A quick-fix solution to problems will not be trusted. A hypothesis engine in DMAIC will be much faster and no less credible. Those who will win will not be substituting statistical reasoning with AI, but those who make AI work within trained finesse logic. Only optimization matters because it optimizes faster, when you are still maximizing the right things.