Q 758. In many processes, exceptions are treated as noise — but some exceptions may hold hidden patterns worth learning from. Think of a real or hypothetical scenario in your domain where repeated exceptions handled by humans could teach an AI system to improve. What would you track, and how would you design the AI to decide whether and what to learn?

 

🏆 The best answer will be selected based on:

• Relevance of the chosen scenario

• The practicality of the proposed learning approach

• Clarity & Creativity in identifying what and how the AI should learn

 

Note for website visitors -

• This platform hosts two weekly questions, one on Monday and the other on Thursday. 
• All previous questions can be found here: https://www.benchmarksixsigma.com/forum/lean-six-sigma-business-excellence-questions/. 
• To participate in the current question, please visit the forum homepage at https://www.benchmarksixsigma.com/forum/.
• The question will be open until Monday or Thursday at 5 PM Indian Standard Time, depending on the launch day. 
• Responses will not be visible until they are reviewed, and only non-plagiarised answers with less than 5-10% plagiarism will be considered for winner selection. 
• If you are unsure about plagiarism, please check your answer using a plagiarism checker tool such as https://smallseotools.com/plagiarism-checker/ before submitting. 
• All correct answers shall be published, and the top-rated answer will be displayed first. The author will receive an honourable mention in our Business Excellence dictionary at https://www.benchmarksixsigma.com/forum/business-excellence-dictionary-glossary/ along with the related term.

• Some people seem to be using AI platforms to find forum answers. This is a risky approach as AI responses are error-prone because our questions are application-oriented (they are never straightforward). Have a look at this funny example - https://www.benchmarksixsigma.com/forum/topic/39458-using-ai-to-respond-to-forum-questions/

• We also use an AI content detector at https://quillbot.com/ai-content-detector. Only answers with less than 45-50% AI-generated content will be considered for winner selection.

Let’s say you work in a customer support team at a company that processes thousands of return requests daily. Most returns follow a standard path — approved automatically if they meet certain criteria (within 30 days, unused, receipt provided, etc.). But every day, there are a handful of exceptions that don’t fit the rulebook. These are flagged and handled by human agents.

Now, here’s the thing — if we just treat those exceptions as noise and move on, we miss out on why they happened and whether there's a pattern. Maybe customers are returning items that arrive late even if it's just outside the return window. Or perhaps the system misreads handwritten receipts, and that’s causing valid returns to get stuck.

These human-handled exceptions are exactly what an AI system could learn from.

What to track?

• The reason the return was flagged as an exception

• The final decision made by the human (approved/denied)

• The rationale or notes written by the agent

• Patterns in product type, location, customer feedback, or even time of year

Our team performs organization wide internal audits, the scoping of accounts, processes etc for the various business units (BU's) happens in the beginning of the financial year.  At times there are exceptions requested by BU's to postpone the audits, or an account sunset's and no longer active, In such scenarios the new planned audit numbers are required to be changed manually. AI that operates in changing environments should learn from exceptions as part of its ability to adapt to new patterns over time. The AI model should be designed to differentiate between genuinely valuable exceptions and noise

The one situation that I can think of is a credit card Process.

While a customer calls the customer service for certain information regarding the credit card most common questions like validity, limit, usage, billed and unbilled amount can be answered by AI bot , even some amendment requests can be take over by AI. However, at certain point on dispute management there is Human intervention required.

 

The signals or the criteria that AI can follow are some key words like not satisfied with the answer, unacceptable, further escalate to higher management, report on social media or legal website etc. 

 

With such key words AI should escalate this further for human intervention and validation of response before communicating the customer

AI system can enhance exception monitoring by learning from data, automating detection and predicting potential process or product issues learning and improving efficiency and accuracy in identifying and addressing exceptions. 

 

Example:

In a process were refunds are common procedure, refund may impact the client owned business through financial cost, product returns and delivery charges. AI system can identify the anomalies out of the data and identify patterns from users and user approvers by identifying exception approval patterns and may even identify problematic procedures and processes that may require improvement. This may result in fewer exceptions of the same kind.

From US Healthcare - Claim submitted are denied for various reasons from insurance companies. these denials are corrected by human, who then resubmit the claim with corrected information. Identify a pattern from combination of Insurance, procedure and diagnosis. using this trend AI can learn and track from particular insurance with combination of particular procedure will always get denied. Design a proactive denial predication and alert the respective department to solve the denial even before it could happen. This way AI can reduces denials and improve quick turnaround time on provider collection. 

So I am taking the example of Health care domain:

Background:

In Health insurance companies the claim processing systems often denies certain Medical procedures due to missing or unclear prior authorization details. These denials triggers the Manual Appeals process, where the Appeal resolving analyst handles  review and either uphold or overturn the denials based upon the additional medical documents submitted by providers or members.

 

Repeated Human exceptions:

1. Analysts notice that a particular type of CPT code for spinal injections (ex -99213) was being frequently denied due to missing prior authorization, , but during the Appeals :

• The provider is categorically indicating that is a urgent Need and submit in the clinical notes as well
• Authorization had been requested but delayed
• The procedure was performed in an emergency setting, exempting it from pre-authorization.

 

Impact - More than 80 % of the cases were reversed the denial on Appeal after reviewing the additional notes/data.

 

Human in the loop learning loop:

• Initial AI performance
  •  The ML/AI based machine will flag the appeals with missing Prior authorizations and valid denials, offering no recommendation for reversal
• Human interventions to modify the AI generated outcomes:
  • Analysts consistently Overturned(Approved) those cases which contains  structured Comments such as mentioned below:

" Emergency setting, Prior Auth not required policy X"

"Provide er submitted auth request but payer systems delayed it"

• Pattern Identification:
  • The AI system uses NLP(natural language processing) to read clinical notes and Appeal attachments/documents. Its resembles the keywords such as Clinical urgency phrases, emergency settings , or Auth request timestamps with Overturned decisions
• Model Re-training:

 AI model is required to retrained periodically with these type of examples  which eventually results in:

• Achieve the  probabilities of that a denial will be overturned
• Suggest to Auto escalate for Appeal or even can Pre-approve under certain conditions

Impact:

•  Reduction in unnecessary manual Appeals by 30 % Min.
•  Faster resolution times for Providers and patients
• Improved trust from Providers due to fewer incorrect denials

Example from the Audit Industry

 

Scenario – Auditing Expense Reports:

 

Imagine an AI-driven audit system that reviews employee expense reports to detect possible fraud or irregularities. The system is initially trained on historical expense data, learning the normal spending patterns for various categories (e.g., travel, meals, lodging). Over time, it establishes thresholds that typically flag deviations such as unusually high travel expenses or an abnormal number of expense claims submitted in a short period.

 

Exception Identification and Human Validation:

During a routine audit, the AI system flags a set of expense reports that contain a significantly higher than usual frequency of luxury hotel stays—expenses that far exceed the standard policy limits. Initially, the AI might classify these as potential fraudulent or erroneous claims. However, upon review, a human auditor discovers that these reports all pertain to a newly approved company initiative—a temporary policy change that allowed for upgraded accommodations during an international conference.

 

Learning from the Exception:

Once verified, the following actions occur:

• Feedback Loop: The auditor’s findings are used to label this group of expenses as legitimate despite their deviation from historical norms.

• Model Adjustment: The AI system incorporates this new information into its training dataset. It learns to adjust its threshold for flagging expense report anomalies during similar events, ensuring that future genuine exceptions (like policy-driven changes) aren’t misclassified as fraud.

• Updating Detection Logic: The audit system refines its decision logic to differentiate between fraud and policy-driven changes. It now considers context—such as company-wide policy adjustments or external events—when determining if an expense should be flagged.

 

By learning from this exception, the AI system becomes more resilient and capable of distinguishing between fraudulent activities and legitimate operational changes. This reduces the workload on human auditors by minimizing false alarms and improving the precision of anomaly detection.

Let's take the example of E2E process going from a mixing of a set of ingredients to a Pressing and Wrapping machine.

 

To press the mixt you have key parameters like the moisture and the temperature anabling the activation of the binder the mixt. to predict the behavior of the mixt during pressing and wrapping we need to monitor those parameter but it's lagging.

What is more mindfull is to predict the moisture which can be use directly without waiting time for Moisture and temperature drop.

For that the main ingredient carring moisture in the mixt are monitored manually by running analysis each hour and from there define the amount of water to be added in the mixt  in order to get the desired moisture able to run perfectly in the pressing and wrapping line with higher performance and quality.

AI could be trainned to control automatically critical ingredients moisture with close loop for the added water.

 

 

In the context of Six Sigma a data driven approach to quality mgmt , AI can learn from the exceptions to improve process in below scenarios

Anomalies, Process Deviations and Customer feedback, VOC 

Anomalies : AI Should learn from exceptions when anomalies /outliers are detected as these are detected as these can indicate underlying issues or opportunities for improvement 

Process Deviations - AI can learn to identify RCA and implement CAPA

Customer feedback/VOC - customer feedback/pain area and or complaints and satisfaction to improve overall CSAT & Opportunity of growth.

org can drive CI and achieve operational excellence 

E.g

consider a customer service chatbot in an e-commerce domain :

exceptions worth learning from 

Unusual product complaints -issues  may report with products that are not typical such as damage during shipping or unexpected side effects 

Complex order issues 

Edge cases 

AI can learn from these exceptions they can track the data : exception log , Feedback (Customer ) Human support interactions 

use below approach to lean from exceptions 

• Anomaly detection 
• Clustering
• Natural language processing for analyze the customer feedback ,human support interactions extracting insights and sentiment 
• rule based system
• Reinforcement learning 

 

 

As an audit function, we often encounter numerous exceptions from accounts that require validation to determine their legitimacy. AI should be designed to learn from these exceptions when they reveal significant patterns, such as control weaknesses, emerging risks, or recurring issues, which are validated by auditors. These insights enable AI models to prioritize critical anomalies in future audits, thereby enhancing risk detection and efficiency. It is crucial for AI to avoid learning from isolated, immaterial, or unverified exceptions, as these can introduce noise and reduce accuracy. When properly governed, learning from relevant exceptions allows for smarter audits, focused reviews, and better resource allocation. This approach supports the audit team's judgment while improving overall effectiveness.

We conduct an annual scoping of clients based on revenue priority and headcount to determine which clients will be included in our audits. Throughout the year, some clients may fall out of scope due to a decrease in revenue or a reduction in headcount. These clients, now considered out-of-scope, are treated as exceptions and require manual handling after obtaining approval from the relevant stakeholders.

To streamline this process, we propose utilizing AI to identify these exceptions. The AI will analyze input files that are regularly updated with client information. By doing so, it can promptly detect clients that have moved out of scope and need to be addressed. This approach eliminates the need for manual checks each time, ensuring a more efficient and accurate identification of exceptions.

The AI system will provide alerts or prompts for these exceptions, allowing stakeholders to review and approve them without the repetitive manual effort. This automation not only saves time but also reduces the risk of oversight, ensuring that all exceptions are handled appropriately and in a timely manner.

We can take example of Document Verification Process for employee onboarding. Candidates have to submit various documents, like, resumes, certificates, IDs, marksheets, etc.

• Company has implement AI-enabled document verification system using Computer vision, which includes OCR and Image Processing, and Natural Language Processing (NLP).
• Any mismatches in the documents, formats, etc, are called out as exceptions by tagging them as Incomplete document, Missing Document, Invalid Format, etc, and sent for human intervention.
• Designated HR official goes through the exceptions, captures the reasons and the required solution, which is again fed in the system for further learning.
• AI continually learns from the responses given by the designated HR and starts handling exceptions over a period of time on its own.

The above set-up is an example of Supervised Learning as the documents are labelled and the exceptions are also categorized and labelled.

Exception handling in invoice processing, AI can improve exception handling by analyzing patterns in frequent exceptions, such as mismatched invoice amounts or missing POs, and learning from human interventions. By tracking key data like exception types, resolution methods, and time taken to resolve, the system can identify the most effective solutions and apply them to similar cases. Continuous feedback from users ensures that the AI adapts and refines its decision-making process over time, ultimately reducing manual effort, improving accuracy, and enhancing processing efficiency. The AI should prioritize learning from frequent, high-impact exceptions by tracking resolution success and user actions, focusing on patterns that lead to effective outcomes.

Scenario: Handling Customer Support Tickets

In customer support, agents deal with tickets (requests or complaints) from customers. Sometimes, the issues are more complicated than usual. These are called "exceptions." Currently, these exceptions are treated as one-off cases, but they can teach the system how to improve over time.

What to Track:

1. Ticket Categories: Keep track of the types of issues agents mark as exceptions (e.g., bugs, service outages).

2. Response Time and Outcomes: Note how long it takes to fix the issue and if the resolution was successful.

3. Customer Feedback: Track how satisfied customers are with how their issue was resolved.

4. Agent Behavior: Record what steps agents take to solve issues and any special procedures or workarounds.

5. Patterns in Exceptions: Identify what causes these complex issues (e.g., certain actions by the customer or specific errors).

How the AI Decides What to Learn:

1. Detecting Exceptions: The AI needs to spot when something unusual happens, like:

   • Too many tickets in an uncommon category.

   • Tickets that get escalated in strange ways by agents.

2. Recognizing Patterns: Once the AI spots an exception, it should:

   • Look for similar issues or solutions in past tickets.

   • Check if resolving the issue quickly leads to higher customer satisfaction.

3. Learning and Improving: When the AI finds patterns, it can:

   • Suggest better ways for agents to handle similar issues.

   • Use the new information to improve how it automatically handles tickets in the future, without needing much human help.

4. Updating Models: Based on the patterns, the AI could create new rules or systems to handle these exceptions more efficiently.

5. Human Oversight: Even though the AI is learning, humans should still validate the AI's decisions to ensure it’s improving in the right way.

Scenario: Resume Screening Process in Talent Acquisition

In a traditional resume screening procedure, an AI-driven system arranges resumes according to job descriptions, searching for keyword matches, essential skills, years of experience, and educational qualifications. However, there are frequently resumes classified as "exceptions," determined to be inappropriate by the AI but subsequently referred to human recruiters, who often approve them upon examination. These continual human interventions are frequently perceived as trivial background noise; nevertheless, they genuinely embody substantial contextual intelligence.

What to Track:
1. Human Overrides: Monitor each instance in which a human overturns an AI rejection.
2. Resume Attributes: Gather metadata from resumes that qualify as exceptions — including transferable skills, domain transitions (e. g. , finance → fintech), career gaps, or distinctive combinations of soft skills.
3. Job Context: Record job-specific context, such as the hiring manager's flexibility, alignment with team culture, or shifting role expectations.
4. Outcome Tracking: Assess whether the candidates who received approval were interviewed, selected, and if they performed successfully.

How the AI Should Learn:
1. Exception Clustering Engine:
- Develop a module to categorize overridden resumes based on common characteristics. For example, it may reveal that career changers possessing online certifications and project portfolios are frequently overridden and subsequently employed.
- Employ NLP to extract contextually insightful data from cover letters, LinkedIn profiles, or interview notes when available.
2. Human-in-the-Loop Learning:
- Establish feedback mechanisms that enable recruiters to provide rationales for overriding the system (e. g. , "strong soft skills," "relevant side projects," "culture fit").
- Utilize these tags as labeled training data for the improvement of future models.
3. Confidence Adjustment Module:
- Instead of issuing complete rejections, the AI detects resumes with low confidence for assessment by humans.
- As patterns are discerned, the model flexibly adjusts thresholds for comparable future instances, shifting from inflexible filters to responsive, context-aware evaluations.
4. Reinforcement Learning Loop:
- Instruct the AI to assign "reward scores" based on the long-term accomplishments of candidates (e. g. , hiring quality metrics, retention).
- Enable the system to refine decisions that reflect past human-approved exceptions with high success rates.

Why This Matters:
What may appear to be an exception frequently indicates a nuance that inflexible AI rules are unable to capture in comparison with human judgment. By learning from these exceptions:
• The AI advances into a more insightful entity aligned with business requirements.
• The hiring process transforms to be more inclusive, accommodating non-linear talent trajectories.
• Recruiters can save time by minimizing false negatives and clarifying the true definition of "qualified. "

North America Group Health Claim Processing - Requires 3 documents, Employee claim Statement, Employer claim Statement & Attending Physician diagnosis Statement (APS)

Exception Scenario - Missing documents/info

Description - In as claim operations office claims are generated with either of these docs received by Ins. company. However, SOPs advise to gather all three documents before processing the claim. 

In several scenarios, Claim Examiners take/provide exception approval, to proceed with claims in lack of all the information. For ex – APS can be skipped for Maternity claims, etc.

 

Why - AI can be leveraged to filter claims for Examiners to proceed or send back to claimants, basis previous decision patterns.

How - After every claim decision, force examiners to input/tract below details.

What - Data to track > Missing Info: Y/N, Type of Info missing: <text>, Decision: <Proceed/Send Back>, Rational for Decision: <text>, etc

Our internal team has developed an AI-based platform to manage the internal queries raised by team members as part of managing their activities. The knowledge base (KB) is collated from training material, standard operating processes, client feedback, and other relevant sources. However, the AI may need to escalate scenario-based questions that are not part of the KB to humans for resolution.

In such cases, the AI can learn from these provided solutions by:

• Analyzing the given responses.
• Identifying processes where such exceptions occur frequently.
• Measuring the time taken for responses.
• Detecting any patterns or rules in the responses collected through such exceptions.
• Tracking the frequency of these exceptions.
• Assessing the need to update the KB based on the frequency and nature of exceptions.
• Monitoring further queries that arise post such exception cases.

This approach ensures continuous improvement of the AI system, enhancing its ability to handle a wider range of queries autonomously over time.

A real scenario in this process is where we have a gray area for moderating features like city, mountains, monuments, historical places etc. These cannot be moderated by bot and are currently moved for Operator validation. 

 

If we can track these exceptions efficiently, AI can be trained to improve moderation accuracy & efficiency. 

 

I would track the below:

Applicable Unique identifiers, Users reporting the issue, Details of the place, Count of reports, Resolution, Category of resolution, reason for resolution. 

 

Post capturing this information for a sample set of data then identify the reasons, resources and policy considered for taking the decision, clusterise it and feed it back to AI for learning

While designing an audit plan for my span, we always end up doing fewer audits than agreed at the beginning of the year. The reduction is due to some exceptions taken by the account owners for exemption from audits as their respective processes don't deal with sensitive information of customers and are inherently low in risk. Some exceptions are due to higher call volumes and band width issues of the spocs involved in the audit. Currently when these exceptions are generated, we have to manually check for the validity of the reasons considering the risk involved, the headcount and revenue of the process etc and approve the exceptions. Considering the last two to three years worth of data along with the latest risk report, AI should be able to track common patterns like exceptions being taken due to low risk, low revenue generating accounts, call volumes as per seasonality etc and decide to learn common practices. We can have some guidelines to ignore exceptions due to band width as this may not always be a factor. However, AI can learn the risk patterns and implement  while creation of future audit plans.

It is great to see so many respondents. Almost each one has a different example. I would also like to link the question to the Kano Model where some of the issues, concerns related to delighters might be treated as exceptions but over a period of time, the same exceptions will become the norm and AI should be retrained to handle them. In that sense, the learning will never stop for AI

 

The best answer to this question has provided by Puneet Vohra. Well done!

 

Answers from Satheesh, Amit Suri, Smita Vaval are also an interesting read.