Q842

When AI becomes part of a business process, it doesn’t remain static.
Inputs change, user behaviour evolves, and the AI’s responses may slowly drift — even if nothing “breaks” outright. Think of a specific process in your domain where AI is/ could be involved in making decisions or recommendations.
How would you ensure that the overall process continues to deliver the intended outcomes over time?
What would you monitor, and how would you respond when performance starts to slip?

⚠️ Any answer that is generic or does not connect with a specific process will not be approved.

🏆 The best answer will be selected on the basis of:

• Relevance of the chosen process

• Practicality of the monitoring and response approach

• Thoughtfulness in linking AI behavior to business outcomes

• 
  

Note for website visitors—

• This platform hosts two weekly questions, one on Monday and the other on Thursday. 

• All previous questions can be found here: https://www.benchmarksixsigma.com/forum/lean-six-sigma-business-excellence-questions/. 

• To participate in the current question, please visit the forum homepage at https://www.benchmarksixsigma.com/forum/.

• The question will be open until Tuesday or Friday at 9:00 AM Indian Standard Time, depending on the launch day. 

• Responses will not be visible until they are reviewed, and only non-plagiarised answers with less than 5-10% plagiarism will be considered for winner selection. 

• If you are unsure about plagiarism, please verify your answer using a plagiarism checker tool such as https://smallseotools.com/plagiarism-checker/ before submitting. 

• All correct answers shall be published, and the top-rated answer will be displayed first. The author will receive an honourable mention in our Business Excellence dictionary at https://www.benchmarksixsigma.com/forum/business-excellence-dictionary-glossary/ along with the related term.

• Some people seem to be using AI platforms to find forum answers. This is a risky approach, as AI responses are error-prone because our questions are application-oriented (they are never straightforward). Please take a moment to review this amusing example—https://www.benchmarksixsigma.com/forum/topic/39458-using-ai-to-respond-to-forum-questions/

• We also use an AI content detector at https://app.originality.ai/. Only answers with less than 45-50% AI-generated content will be considered for approval.

Domain: Solar Cell & Module Manufacturing Sector

( Capacity: 1.3 GW Solar Cell & 1.6 GW Solar Module Production and Supply per annum).

When AI becomes part of a business process, it doesn’t remain static.
Inputs change, user behaviour evolves, and the AI’s responses may slowly drift — even if nothing “breaks” outright. Think of a specific process in your domain where AI is/ could be involved in making decisions or recommendations.
How would you ensure that the overall process continues to deliver the intended outcomes over time?
What would you monitor, and how would you respond when performance starts to slip?

Improvement Initiative: Build an AI enabled model for scheduling Preventive Maintenance to improve the following metrics in module manufacturing floor:

1. Pre Lamination FPY%.

2. Cell yield

3. Stringer Uptime.

4. Line Capacity Utilization.

5. Final Quality Yield %.

Why is this so important?

It has been observed in almost all solar Module Manufacturing lines that the preventive maintenance has been always granted the least priority over continuous running line. This affects the line's Output quality degrading till the start of the PM and the recovery of the line capacity also gets delayed post preventive maintenance hand over.

This has led to pre to post PM degradation of :

1. Pre Lamination FPY% by 8-10%

2. Cell yield by 30-50%

3. Stringer Uptime by 20-25%

4. Line Capacity Utilization 20-30%

5. Final Quality Yield % 30 -50%.

So instead of improving or maintaining the metrics, PMs often had degraded results for some durations and the business faced losses. These losses made the habit of PM so unpopular in the shop floor that higher management becomes reluctant to activate the PM schedule.

If AI can be strengthened with all the historical data and learnings, it can optimize and generate a significant model for Preventive maintenance and the schedule for it.

Monitor the Indicators for Positive delivery:

Control charts: Though AI claims that the model of the schedule is still going good, the SPC of the above mentioned metrics can tell us the truth.

Monthly Management review driven by MBB: As per ISO 9001, AI generated models should be reviewed thoroughly. if the metrics are drifting then the model is definitely failing whether the model accuracy is good or not.

Learning correct reality: AI learning reality need to be kept in check. If the engineers acceptance rate is getting lower day by day, it may indicate that the model is not in the correct reality. It can affect catastrophically in near future.

Parameter changes: if post PM , the parameters are still inside the recommended window then the model's health is good. Otherwise the model needs to be reverified.

Control Strategy when performance slips:

Some ground rules need to be confirmed before applying any AI model:

1. There should be one golden PM schedule that is made with experts before starting the implementation, so that if any slip is identified one can revert back to the golden schedule.

2. There should not be any sudden conclusion to scrap the model, first the root cause of incorrect reality needs to be identified and then rectify that.

   ( Example: Suppose a scheduling of PM for Ribbon Traction area in a stringer within an interval of 2 weeks has degraded the quality of ribbon alignment with cell busbar drastically. So primarily it may be misunderstood that the high frequency of the PM has affected the slot quality of that section and for that the degradation happened.

   But the reality may be that the frequency of the PM has nothing to do with that. It may be due to:

1. Wrong selection of Technician for such an important section, or

2. Wrong selection of the tools those have been used to do the PM. )

Hence to keep the model healthy and keep away from wrong decisions or reality , a well trained MBB needs to govern the model properly.

What makes the model successful:

1. Pro-activeness of triggering PM before the machine's performance starts degrading.

2. Post PM the recovery should be minimized and reduced after every iteration.

3. High positive Business Impact.

Process Context

In our fleet size of more than 50 plants, we use SAP IBP as an AI-enabled planning engine for forward spare parts demand planning, replenishment strategies and inventory strategy for all operations and maintenance. We manage roughly 200 Million Euro, 78000 SKUs in inventory.

Let me explain what IBP does, It autonomously calculates, recommends replenishment strategies (VB, PD, ND), and planning paraments such as safety stock, Reorder point, and Max stock, using consumption history, PM demand, variability and lead times. Currently, for us, the process already runs with minimal manual interventions.

So the real challenge for us is not making AI work, it is ensuring it continues to work as intended over time.

How We Prevent Drift While Keeping the Process Autonomous

We follow the simple principle, Automation with explicit guardrails.

Any IBP recommendation that changes planning parameters for items by more than +/- 30% from the current setting automatically triggers a workflow approval in SAP.

Our workflow goes to Planning manager, inventory controller, Final approval from PGM & Myself.

We have build a clear summary in approval process, along with raw numbers, it shows number of SKUs impacted, Net Inventory Impact on Max stock (Positive/Negative), Risks.

This gives completely details to reviewers, to approve, reject, fine tune recommendations based on the actual business context, In our process AI proposes, humans decide only when risk crosses a defined threshold.

 

What We Monitor To Ensure The Process Stays Healthy

We simply monitor the business behaviour & outcomes. If I say very specifically, we look at frequency of parameter changes exceeding the 30% threshold, Repeated SKUs entering the workflow approval, Manual override trend, Inventory vs Service levels, Gap between IBP recommendation & executed parameters.

 

How We Respond When Performance Starts To Slip

When we see the drift, we don’t fix the AI engine, we review the decision logic, PFEP.

Typically we revalidate the demand classification, Lead time assumptions, ABC calculations. We do the corrections at the policy & logic level, not through the repeated manual intervention.

 

Evidence That This Approach Works

Our this structure helped us to optimize around 12000 SKUs, delivering 3 M Euro Inventory reduction, 10 Million Euro Cost Avoidance in a year. Review cycle time has been reduced by 80%. Now process auto approves 70% of recommendations.

 

Bottom Line

In my view, AI enabled process stays effective over time not because it is constantly supervised, but because decision rights, thresholds, and escalation rules are designed upfront.

We let SAP IBP run high volume, repeatable decisions, while we intervene only when changes signal the real business risk, and then we judge system health by outcomes, not by how often people touch it, this shows the clear example of how autonomy & control coexist with AI delivering the value not drifting the process.

 

How to assure that an AI-Enabled Process lives to Work as Intended - Over Time.

The actual problem is when AI is integrated into a business process, go-live is not the beginning. It actually starts after. Input evolves, vendors become erratic, the exceptions shift, and quietly the AI can be drifting, its output may be as valiant as before, but it is not giving the results the AI was trained to provide.

Let take a real scenario of the BPO Domain:

AI-enhanced invoice processing of an international retail customer, managing millions of invoices every year across geographies, tax regimes, currencies and types of suppliers. It makes decisions here: Which types of invoices to classify, how to diverge on exceptions, automating the posting, letting it go, or failing on its own judgment and request human intervention. It is not whether the AI will operate in the world on Day 1, but how we make sure that the AI will continue working in the business on Day 500.

The relevance of invoice processing as one of the most involved AI applications.

Cost, control, cash flow Invoice processing is at the crossroads:

• Any 1-2% decrease in accuracy may be, at scale, a material financial leakage.

• Increased delays have a direct effect on Days Payable Outstanding (DPO) and relations with vendors.

• Mistakes might lead to audit results, tax liability or over payments.

In the case of a global retailer, it is also non-stationary by its structure:

• Every month onboard new vendors.

• Forms of invoice vary without warning.

• More advantages, returns and interchange between country distort normal trends.

This renders the invoice processing a perfect test in how to maintain AI performance in the long run.

What Good Looks Like - Aligning AI to Business Results.

Human-owned intent definition always comes first since it is the most important step before discussing monitoring.

In this venture, we at least spelled out business success:

• STP >=85% Straight-Through Processing.

• <=0.5% of financial material error.

• Average reduction in invoice cycle time of 20 to 25 percent.

• Duplicates and fraudulent payments are never accepted.

AI may be expected to optimise within such limits- but not reinvent them.

What's There Is to Keep Track of - and What It Matters.

Observation is not concerned with a single score of accuracy. It is related to linking AI practice to process health and financial performance.

1. Input Shift (Early Indicators) & Model Shift (Early Indicators).

We continuously monitor:

• Layout, language, tax field Modifications in Vendor formats.

• Emergence of new SKU or charge-type.

• Quantity increase due to promotion or seasonal changes.

Why it matters: When there are invoice mix changes before the model adapts STP rates decline silently - long before mistakes are even noticed in the audits.

What to do: Do not completely rebuild a model, impose trigger targeted retraining or rule overlay of the targeted clusters of vendors.

2. Confidence of decision vs. human Overrides.

We track:

• Posting time confidence score of AI.

• Occurrence, causes of human overrides.

• Difference in AI suggestion and end post.

Why it matters: Increasing overrides are an initial indication that the mental model of invoices of the AI is not tracking with operational reality.

What to do: Review of systemic cause overrides to weekly (e.g., has tax misclassification been done to one country rather than to a single processor).

3. Straight through Processing Verses Exception Aging.

STP alone can be misleading. We pair it with:

• Exception backlog aging

• Rework percentages on billed out exceptions.

Why it matters: An AI that is aggressive in auto-posting has the effect of increasing STP, as well as pushing more complicated invoices into an already extended exception queue-damaging the cycle time and cash flow.

What to do: Rebalancing AI targets the optimization of end-to-end cycle time, and not local efficiency.

4. Financial Leakage & Control Metrics (Lagging however critical).

Monthly controls focus on:

• Incidences of duplication of payments.

• Tax posting errors

• Post-audit adjustments

Why it matters: These are the results that CFO and the auditors actually are interested in. There is no sense in having a model with no financial integrity.

What to do: Root-cause analysis provides feedback to the feature engineering and decision rules, and any change that is control-relevant has to be signed by humans.

Our Reactions to the Decline in Performance.

The most important principle: it is not AI that corrects itself during a business process, but people.

When metrics start to shift or change:

Evaluate the layer: Is it misuse of change of input, misuse of decision logic, misuse of confidence calibration or misuse of process?

Step in proportionally for:

• Vendor specific retraining.

• Threshold tuning

• Short term human-in-the-loop enlargement.

Getting stuck in and locked on: Not model weights, but updated SOPs, exception playbooks and retraining measures.

Ownership remains clear:

• AI recommends and predicts

• Human beings make choices, rule and are responsible.

Control Model: AI: Ensuring it Sticks in the Real World.

The steady state represents a combination control loop:

• AI checks on itself on a regular basis.

• Performance is checked on a weekly basis by process owners.

• Outcomes are monthly checked by endorsing finance and compliance.

• Models are re trained not responsively.

Practically, it will result in less surprises and quicker invoices and the confidence in the system rather than blindly relying on the system.

Practical Results to continue AI-powered Invoice Processing Success:

• Maintain >=85% Straight-Through Processing (STP) through active checking of changes in inputs (e.g., new vendor formats, change in tax) and in response to those changes, active retraining of the models in order to avoid silent drift.

• Reduce financial errors to <=0.5% by bridging the performance of AI and what is performed by core financial control (duplicate payments, tax errors) and by human signoff of control-critical changes.

• Attain 20-25% decrease in invoice cycle time with an optimal focus on end-to-end effectiveness, rather than local AI precision, and exception queue backups.

• Root-cause analysis of leaks and feeding the findings into AI decision rules and feature engineering can eliminate duplicate/fraudulent payments.

• Human override trends serve as an early warning of AI model drift, where weekly reviews should address the underlying causes (e.g. misclassification of tax regionally).

• Establish a feedback and control mechanism with multiple levels. Incorporate inbuilt AI internal controls, weekly owner review, monthly finance and compliance checks, as and when necessary; adjust levels or re-train some vendors. Create a defined ownership through making AI give recommendations and having humans make decisions, create rules, and be responsible to each other. Refresh SOPs and exception playbooks to learn and apply new understanding. Develop financial functions that can provide predictable outcomes and manage growing needs since artificial intelligence will enhance the security and maintenance functions of our financial systems.

The Bottom Line here is:

The invoices processing process using AI does not collapse when models decide to stop. It collapses when AI behaviour moves outside of the business purpose- silently and subtly.

More automation is not the answer, but improved ownership with:

• Human-defined results or outcomes

• Business-linked observations

• Strict intervention upon warnings.

When used properly, AI does not simply render invoices faster to process, but it also makes the entire financial process more predictable, lessening the control, and scalable over time. And that is what counts as the success.

 

There was an escalation which was coming from client for increase in CTQ Verification

This process involves calling and non-calling cases to get verification; by conducting brainstorming and analyzing the past data it has been observed in most of the cases where verification is required from vendor, those vendors remain same.

 

Here, the problem was agent has to spend a time daily research task for calling and non-calling element which was impacting their productivity and unable to get successful verification.

 

How would you ensure that the overall process continues to deliver the intended outcomes over time?

 

So, with the help of Data Analytics team a AI model was built which could gather last 2 years of calling and non-calling cases and mapped all the vendor from where we get verification

 

We were getting good result data verification of 30% for non-calling cases and 40% on calling and other cases resulting in 70% mapping and verification improvement.

 

 

What would you monitor, and how would you respond when performance starts to slip?

Post 6 months verification no. again was leading to issue, also mapping % got reduced, so a brainstorming was conducted with agents to understand the issue and also data was analyzed, the key cause to the problem was new clients gets added quarter on quarter for which AI model needs to upgrade.

Solution Planned: Process has been set for upgradation of AI model every quarter.

 

Hence, the control plan was created a checkpoint was added in Monthly Audit checklist to ensure the updated data base to be appended in the AI model for new client to get better mapping

 

 

 

In a banking world, particularly payment operations (SWIFT MT / ISO 20022 flows) AI is increasingly used to flag payments, recommend repair, and adjudicate risk at the first level for sanctions false positives. Now since this is a high-risk and a high-volume process where a minor drift directly impacts business outcomes such as SLA breaches, regulatory exposure, customer dissatisfaction, and operational cost it is of utmost importance to monitor the outcome of AI ensuring that that the output is in line with the rules embedded.

AI cannot be treated as a “set-and-forget” capability. Instead, we would need a closed-loop operational governance model with clear ownership and actions keeping the objectives simple but non-negotiable. Some of the key indicators that we I would keep a tack of to ensure if they slip and AI is not doing its job regardless of accuracy scores.

Some of the key parameters that I would look at are:

·         Payment repair rate

·         Hit rate

·         Customer query rate

·         Customer satisfaction score

If these metrics degrade, AI drift is already hurting the business even if model precision appears stable.

Domain : Manufacturing-Oils and Gases

Problem statement : Air separation manual product process line up after start of plant is taking 32 hours instead of 18 hours, leading to production loss by 8 T with an wastage of electrical consumption of 405 kwh/T, which corresponds to 3240 KWh loss.

This subject is been a pain point for on-site customer supply and delay in supply due to late product delivery would impact the on-site customer and leads to penalty.

The intent was to optimize the manual product line up with all the expertization and data mining to build a AI model with predictive control which smoothen and increase the efficiency of the product line up.

AI Enabled Predictive Model for Process start up and product line up :

Risk & Bias : The whole process was studied for Severity of failure, occurrence and detection, especially on detection failure at each step of the process and instruments while incorporating the logics for predictive model, bias limits were identified, precision modelling were used to arrive at the accuracy of the flow meters and control valves tuning for ramp up and ramp down.

 

Model Definition : AI predictive Model was developed by considering historical manual operations data, Risk thresholds, keeping continuous sustainable focus with Model metrics, logical validation, establishing stability and calibration of all instruments and analysers.

Simulation & Dry Run : The developed model was tested on dry run simulation to see the model is a best fit and need of correction and fine tuning for desired output, simulation run was executed to identify all the predictive proposal based on assumption for the defined out put of each process parameters, Valves, sensors with the reduced bias.

Cycle time reduction at each process and instrumentation output was measured and noted, temperature and pressure requirements at each process/instrument outputs, logics were corrected, replaced, fine tuned for desired output.

Commissioning and Go-Live & Intended sustenance overtime : The plant was taken in line with AI Model predictive system, observed the simulation nodes and possible deviation would lead to failure and the bais, taken control to verify the observed deviation optimized further for consistent out put and supply to customer.

Positve Business outcome :

• Cycle time efficiency was reduced by 3.8 hours on the first opportunity and with further optimisation in successive start ups cycle time reduced to 6.5 hours consistently.

• Cusromer staisfaction due on on-time delivery

• Savings in Power consumption

• Savings idle running of equipment and detiriration.

• Realization of product soon after few hours of Factory start up

Now for the above practical case, the question is how do we sustain this AI enabled process Intended to work overtime :

Though the AI model matrix is designed and developed on previous manual vast data and live realistic data extracted from ASPEN system, FOXBORO to meet the real auto situation, still it may miss the sustenance or consistency without inclusion of real scenarios in physical process, also user settings given by operator or team on live process, how AI Predictive model consider’s that manual live commands correction with real time predictive correlation/correction, sometimes user corrections and sudden real time process prediction/alarms from the downstream process would generate low confidence output from AI Model.

Here is an example, which is even out of human control : AI Model can’t accommodate the dynamic process variation of downstream or upstream caused due to sudden power fluctuations. During these times It’s necessary to take the process out of AI Predictive contol and operate manually to bring the process under control and then after manual stabilisation give back the control to AI Model.

With evolution, AI model should be incorporated with highlights that immediate correction of response for high risk and also ensures high detection to keep the severity of failure as low as possible.  

Regular Audits & AI performance review Documentation until complete sustenance

• System alarm tracking/decision lists of shifts

• MOC, Management of Change method implementation

• Clear accountability (list of monitors, approver, in charges, especially on Logical changes)

• Review for bias, fairness, and Legal & regulatory compliance

• Maintain deviation logbooks for each parameter, failures against the target values

AI Model performance review, correction, maintenance  and development  :

Review the performace of the AI system on continuous basis with team and adapt the necessary change to be brought in to the model new dynamics. Take all the learnings from the feedback given by the alarms of system and people.

Training and communication

Ensure each and every change in the model is communicated to the end to end team members and provide full necessary training on the actions to take. Example, if AI Model is to be over rided by Manual control or visa versa from manual control to AI Model, Ensure to provide hands on experience.

Conclusion :

AI Enabled process is not one time installed or invested and forgetton, it’s to be tracked and trated as element of evolution and a process of Continuous improvement and evolution really holds good for this to assure system should not fall behind when the AI solution keeps emerging with new horizons.

Lets take example of AI based predictive maintenance for CNC machines in manufacturing plant. the AI system uses ML model like anomaly detection on vibration sensors, temp sensors etc to predict when tool/machine is likely to fail or when process parameters are heading towards out of spec parts/products. Then the AI based system makes recommendations to change tool, adjust parameters etc, be it automatically or manually. there are KPIs associated with the same, i.e. downtime, rejection, Yield, OEE etc.

How to ensure that the overall process continues to deliver the intended outcomes over time

Because of dynamic environment of manufacturing process, i.e. changing raw material specs, ambient conditions, govt rules, human involvement, it all leads to drifting of outcomes. there can be data drift (calibration issues, etc) or correlation shift (same vibrations may not mean early or late failure like it used to in past). then the AI system can predict wrong output, it can increase maintenance costs due to unnecessary spare changes or increase the breakdowns because of non-recommendations of maintenance when there was a need.

so to keep the system relevant, we need to have

a) continuous monitoring of KPIs, SPC charts (Breakdowns, Yield, OEE etc)

b) AI system monitoring metrics like AI recommended breakdown windows vs actual breakdown, calibration error patterns

How to response when performance starts to slip

a) if detected early and KPI is still ok - investigate route cause and take action

b) if detected late and KPIs are not ok - Then switch to time-based/stroke-based model, Human oversite for critical processes. Retrain the system again with newfound data/history, the redeploy with oversite.

Case Study: Amazon Online Sales for Mobiles (Electronics) post implementing Machine Learning

Category	2020	2021	2022	2023	2024
Electronics	15%	10%	5%	8%	17%

 

During 2020, the COVID-19 lockdowns restrictions forced online shopping with store closures, and no offline alternatives were available. The demand for mobiles increased due to Work from home implications. During 2021, we saw a continued growth through retaining customers with sustained online habits by offering more product choices and improved logistics with faster delivery.

In 2022, we identified the pattern of reduced spending due to high inflation, product availability due to supply chain issues, slowly resuming to offline shopping (return to stores). In 2023, we identified moderate growth due to international markets expansion, new product launches with stable prices.

During 2024 we identified steady growth based on AI recommendation post studying the customers pattern in online shopping better cross-selling AI recommended smart bundle recommendations such as phone case, wireless earbuds, fast charger, screen protector with a bundle discount of 15%.

1.       Based on user data, Machine learning was embedded to personalize recommendations based on product purchase history, online browsing history, search queries, session duration.

 

2.       Post data collection, the next step is embedded with AI model processing through customer segmentation, product affinity analysis, demand forecasting and price optimization.

 

3.       Based on the above model, AI or machine learning helped in designing Cross-selling execution i.e. product page recommendations, carte page suggestions, email campaigns and push notifications

 

4.       The main step in continuous improvement is through tracking conversion rates, measuring revenue impact, gathering feedback and retrain models.

Machine learning helped to analyze customer patterns based on model selection, phone features, performance monitoring, customer reviews, battery, price comparison, brand priority.  Based on demand, machine learning helped to design dynamic pricing concept through discounted prices i.e. 20-40% off (discount range) on lightening deals, which attracted a greater number of buyers.

This machine learning also helped to understand the customer payment methods\options that helped to tie up with different banks with multi payment options. AI helped to study the customer choices of best-selling mobiles like I phones (22-24%), Samsung Galaxy Series (19-21%), One-Plus devices (2-3%) and Xiaomi & Redmi phones (12-13%). Overall, Machine Learning helped Amazon in identifying the Smartphone Market study with customer behavior insights. It also helped with ‘Better recommendations’ for Cross-selling conversion rates, ‘Smart bundling’ through Average Order Values, ‘Personalization’ through customer lifetime value, and finally ‘Relevance improvement’ through ‘Recommendation Click Rate’

Type of industry - Scientific publishing company.

The AI project I am going to discuss about is ensuring AI enabled auto type setting process which continues to work as intended.

Type setting process in any publishing industry is very critical process because that ensures the correctness and completeness of the publication. Small wreor in the typesetting process will have direct and huge impact on brand image as well as the research results from the manuscripts. But automating such critical process have become the need of the time because the increased volumes of the submissions made it very difficult to manage the type setting process manually.

How we ensure AI enabled auto type setting process which works without a drift :- In our company we are using AI to support the auto type setting in which the AI is performing checks and making changes in the content. But the type setting itself is an critical task if an AI fail to identify any symbol or a I just put a single decimal value here and there, this will going to have a huge impact in the entire scientific research, so it becomes very important for us to track and monitor the performance of the AI that we have deployed and for that we have defined taken measures.

1.Defined criterias and metrics - With this define criterias we are continuously tracking this AI performance to identify any kind of signs of the performance drift. We are frequently tracking and monitoring the outcomes of AI by sampling and checking typesetted articles.

2.Keeping human touch for complex tasks - while AI is handling the significant proportion of the type setting work we have kept production editors and type setters actively involved in working on the complex layouts and highly technical content which is not expected or let's say it is impractical to automate using AI. For this specialised work the highly skilled and experience humans are allotted and AI model is trained in away that it identifies such scenarios then that particular article will directly routed to this skilled people.

E.g. Article with Latex is a case of comolex typesetting. Hence whenever this scenario occurs, the articles are removed from the automation que and they are sent to the typesetters and content checkers for the manual work.

3.Final stage- We are continuously monitoring the versions of AI models . We monitor the layouts and templates provided and each update is tested against the historical as well as the ongoing articles and manuscripts to make sure that the outcomes are improved and better than the previous versions,like wise we are also continuously training and monitoring the AI models.

Key learning - AI is capable of handling such a critical task like Typesetting in scientific research publication. But this involves high risk because every researcher, student, company or any consumer of article demands high accuracy and have zero tolerance towards mistakes. Hence it becomes very important to keep updating and monitoring the AI model so that smallest drift or change could be caltured and resolved promptly.

In my opinion there are processes where there is drift in decision making. The larger the data set is, baseline changes or drifts from point A to Point B and sometimes skewed.

I had a process AI model to detect false positives and we monitored it through tableau. It worked on 95pc confidence and 90pc recall rate i.e. possibility pf outcome being true.

One day the file or the batch job failed which had around circa 1m records and model operated as usual and that it dis not met the percent of recall rate and also the confidence level. This was the biggest file which got missed and was specific to UK and all other rest of the world records were low in number.

In order to manage this, a self check algorithm was developed and ensured that model runs only after the files are received and was governed by model

Monitorig process

Domain: Aerospace MRO - Engine shop for CFM56/LEAP Turbofans for Performance Restoration

(€ 220 Million yearly turnover , approx. 1,800 shop visits in a year, AI rolled over since late 2025 to predict HPT module rework needs based on borescope images, oil debris analysis, and in-service data)

Specific AI-enabled process: Predictive HPT Blade Rework Forecasting

The AI will recommend if the module needs full blade rework, partial (only the tips), or none, all with the goal of eliminating unnecessary shop time and expense without losing the zero escape target on critical parts. It went live on all CFM56/LEAP visits in Q1 2026 and initially deliver an average 18% reduction in TAT on HPT modules.

How we ensure & monitor the process continues to deliver intended outcomes

We are treating this AI-human decision loop as a live control system and continuing to develop it over time not like one tine install, the focus is on sustainable business value – TAT savings, cost per visit going down, safety and zero quality escapes.

What we monitor (daily / weekly / monthly)

1.      Leading indicators (daily dashboard – shop floor + engineering)

·       Prediction accuracy of AI vs. actual rework result (confusion matrix updated every 50 engines).

·       AI suggestion Override rate by technicians / engineers (accept, tweak, reject AI recommendation).

·       Confidence score variation (how often is the model <80% sure?)

·       Data drift indicators, distributional shift of input variables (eg iron particles in oil, borescope crack density, EGT margin so on)

2.      Lagging business outcomes (weekly review – operations + finance)

·       HPT Module: Turn Around Time Variance (target < 35 days).

·       Rework cost per engine vs. Baseline

·       Escape rate / quality holds on HPT (target 0)

·       Spare Parts Consumption vs. Forecast (Over/Under-Stocking Signals)

3.      Model health metrics (monthly deep dive – MBB + data team)

·       Population stability index (PSI) on key inputs (>0.25 = moderate drift, >0.5 = severe).

·       Calibration plot (predicted probability vs observed rework rate)

·       Feature importance drift (which inputs is most important to the model now vs at launch)

How we react when the going starts getting tough

We have a three-level escalation protocol:

Level 1 – Minor Drift (Weekly Trigger)

·       Override rate >25% or confidence <75% on >20% of cases.

Response:

·       Immediate feed back loop i.e. every override by enginers requires 1-click reason (dropdown + optional voice note).

·       Retrain model based on last 100 engines + overrides justificatipn.

·       Notify shop team lead, usually fixes within 1-2 weeks

Level 2 – Business impact emerging (weekly trigger)

·       TAT +3 days or rework cost increased +8% vs rolling 4-week average

·       OR escape / hold on HPT (even one)

Response:

·       Hold AI recommendations - return to manual disposition within 48 hours/

·       Root Cause A3 with MBB: Data drift? New failure mode? Change in user behavior?

·       Temporary rule: AI confidence > 90% required for auto-accept

·       Full model retrain + validation on hold-out set before re-release

Level 3 – Systemic failure (monthly or immediate on escape)

·       PSI >0.5 on critical inputs OR calibration slope deviates >15%

·       OR sustained TAT/cost > 15%

Response:

·       Full pause of AI in production

·       Independent audit: data lineage, labeling drift, concept drift

·       Notification to the regulator of any escape which occurred

·       Re-baseline from scratch or switch to a fall-back approach (manual and old rules)

·       Shared across sites post-mortem – we’ve had one Level 3 (new low-sulfur fuel changed oil debris patterns in Q3 2026)

Practical setup we use today

·       Automated alerts using Teams/Slack when threshold breaches

·       Monthly “AI Health Review” (30-min standing meeting: MBB, ops manager, data lead)

·       Quarterly external benchmark against OEM data (CFM/Pratt)

·       Annual review of AI usage (EASA Part-145 requirement)

Bottom line from the teardown bay

AI Drift isn’t an ‘if’ but a ‘when’

In MRO, the price of slow degradation can be a long turn-around time, excessive spares, or even a failure in service. The way we monitor our AI is how we would monitor an engine, performing routine checks every day, and only grounding it completely when we have to.

The process remains alive since we do not assume model is “set and forget”.

Domain: Polymer Manufacturing Facility

Project: Reduction of Quality inspection time by eliminating inprocess checks

Process: Once the material has been produced in the reactor, it is transferred to the next vessel known as blowdown vessel. In this vessel, basically chemical addition is done to remove the volatile organic component (VOC) of the batch. Next the same material is transferred to a blender vessel. In this vessel, pH, Viscosity and Solid content is adjusted by adding water and some chemicals. In both vessels we can track the Agitator Amperes which correlates to viscosity, the weight of the material in the vessel which can be correlated with solid content.

Present Scenario: We have successfully achieved the results by making models and implementing the same using a tool known as TrendMiner. It is an advanced AI analytics tools which is linked to DCS to directly take the data and predict the future outcomes based on past data. It has also been mapped batch wise. Using this tool, continuous SPC are plotted and in real time we have a visibility on how the data will vary.

Challenges: The challenge here is over the period of time

• The data will start to map based on present trends and information of the past is lost.

• There are some physical challenges where for example weight of the blender and blowdown increases due to accumulation over time.

• This will alter the data trend. Which will lead to wrong correlation impacting the results ultimately resulting in failure of the project.

Action to be taken: When using AI, it is not a one time approach.

• A continuous monitoring of the process and models have to be done.

• The equations have to be tested to see if they represent the real world scenario.

• Accordingly inputs have to be given inorder to modify the models.

• Role of the process owner becomes critical. In the Control Plan of the project this has to be taken into account.

• New parameters have to be taken into account to adjust the models.

Evaluation Summary: Sustaining AI-Enabled Processes Over Time

🏆 Best Answer

Adil Khan
Exceptional depth and rigor. Clearly treats AI as a live control system with leading/lagging indicators, explicit drift thresholds, tiered escalation, fallback mechanisms, and regulatory alignment. Strong linkage between AI behavior, process health, and business risk. This is a benchmark-level response.

---

✅ Approved

Ankit Kulkarni
Strong real-world SAP IBP use case. Clear guardrails, outcome-based monitoring, and policy-level corrections instead of manual firefighting. Very practical and well anchored in business results.

Taby Sheikh
Thoughtful and well-structured BPO invoice processing example. Strong linkage between AI behavior, financial controls, STP, overrides, and business outcomes. Good balance of monitoring, human ownership, and corrective actions.

Bharath CN
Solid manufacturing case with predictive control. Good articulation of when AI must step back and manual control must take over. Monitoring, MOC, audits, and operator training are clearly connected to sustaining outcomes.

Abhinandan Kunder
Relevant chemical manufacturing example. Clearly recognizes model drift, physical system changes, and the need for continuous validation and ownership in Control. Grounded and practical.

---

🟡 Conditionally Approved

(Good intent and relevance, but future responses should consistently anchor on a clearly defined process, explicit metrics, and business impact.)

Rabiya Bronekar
Relevant process and corrective action identified. Monitoring logic is present, but future responses should strengthen outcome metrics and escalation logic.

Kush Singh
Strong domain relevance (banking/sanctions). Good identification of outcome-level KPIs. Future responses should expand on response mechanisms and governance actions when drift appears.

Abhishek Chaudhary
Clear predictive maintenance example. Monitoring is appropriate; future responses should go deeper into decision thresholds and structured escalation.

Vijay Yivaturi
Rich business context and AI usage. Future responses should tighten the linkage between AI drift signals, control actions, and sustained performance mechanisms.

Aditya Bhavsar
Good critical-risk use case (scientific publishing). Future responses should explicitly define drift indicators and response cadence.

Dhruva Kapur
Interesting failure scenario. Future responses should clearly frame the end-to-end process, monitoring framework, and preventive controls upfront.
Suman Acharjee
The preventive maintenance scheduling use case in solar module manufacturing is well chosen, clearly business-critical, and supported with concrete impact on FPY, yield, uptime, and utilization. For future responses, make the trigger-to-action loop more explicit—clearly stating what signals prompt retraining, rule changes, or temporary rollback—so governance and response logic are unambiguous.

---

Not Evaluated

Dipali Yadav – AI content: 100%
Vijay Gonsalves – AI content: 100%
Smitha Muralidharan – AI content: 100%
Arun Bhatia – AI content: 100%

---

Not Approved

Anil Kumar – Generic description; lacks a concrete process, monitoring framework, and response logic.