Q 822.

 

Every AI system has a lifecycle. Over time, its data may become outdated, its logic less relevant, or its cost of maintenance too high compared to new alternatives. Yet organizations often struggle to decide when to retire, redesign, or replace an AI system — especially if it still “mostly works.”
 

Think of one process in your domain where an AI solution has been (or could be) running for a long time.
What indicators would tell you it’s time to sunset or replace the system?
 

How would you manage the transition to ensure continuity, learning, and risk control?

⚠️ Note: Any answer that is generic or does not connect with a specific, relevant process will not be approved.

🏆 The best answer will be selected on the basis of:

• Relevance and clarity of the chosen AI lifecycle scenario

• Practicality of the retirement or replacement criteria

• Insight into ensuring smooth and responsible transitions

 

Note for website visitors -

• This platform hosts two weekly questions, one on Monday and the other on Thursday. 
• All previous questions can be found here: https://www.benchmarksixsigma.com/forum/lean-six-sigma-business-excellence-questions/. 
• To participate in the current question, please visit the forum homepage at https://www.benchmarksixsigma.com/forum/.
• The question will be open until Tuesday or Friday at 9:30 AM Indian Standard Time, depending on the launch day. 
• Responses will not be visible until they are reviewed, and only non-plagiarised answers with less than 5-10% plagiarism will be considered for winner selection. 
• If you are unsure about plagiarism, please check your answer using a plagiarism checker tool such as https://smallseotools.com/plagiarism-checker/ before submitting. 
• All correct answers shall be published, and the top-rated answer will be displayed first. The author will receive an honourable mention in our Business Excellence dictionary at https://www.benchmarksixsigma.com/forum/business-excellence-dictionary-glossary/ along with the related term.

• Some people seem to be using AI platforms to find forum answers. This is a risky approach as AI responses are error-prone because our questions are application-oriented (they are never straightforward). Have a look at this funny example - https://www.benchmarksixsigma.com/forum/topic/39458-using-ai-to-respond-to-forum-questions/

• We also use an AI content detector at https://quillbot.com/ai-content-detector. Only answers with less than 45-50% AI-generated content will be considered for winner selection.

Technology and processes evolve over time and eventually become outdated. Replacing a technology comes from the decision of when that tool no longer brings value to the businesse and customer experience. At this moment, I have two business cases in my project which are to replace a Rule-Based AI for Customer Support and the company’s website ChatBot. We detected that many customers start the chatbot and give up, either because they did not get the expected support (they want a human agent or a more intelligent virtual assistant) or because the chatbot really did not deliver the solution and the customer had to contact support by email. The value that these tools once delivered to the business is no longer there when there was an increase in the number of cases being created in the system by customers. The KPI of cases resolved by the chatbot has decreased, feedback is negative, and the “next best action” tool has become outdated and not always the “next best action” corresponded to the real scenario, even if most of the criteria pointed to the suggested action. To replace this Rule-Based AI, we are implementing an AI Agent that will understand the context, capture customer sentiment, and assist the support agent in resolving the case. For the website, we will implement an AI Chat so that customers have the experience of talking to a human, and this AI Agent has the autonomy to solve more complex problems, with escalation to a human support agent in cases of greater complexity.
The plan is to keep the existing tools running in production until the development of the AIs is ready in the UAT environment for rigorous testing and feedback for adjustments and improvements. After approval, the AIs will be implemented in production and will have continuous monitoring through KPIs and sentiment data to maintain quality and compliance.

Rule Based chat bots are getting obsolete with newer and modern AI solutions beckoning a change.

 

Every technology has a lifecycle. This statement is true and not limited to physical products, but we have seen that it extends to software solutions too. In my domain, a Chat Bot was introduced to assist Customer Support in 2022. The Rule-Based Bot was developed using early AI systems that had predefined decision structures and responded to matched key words. In today’s world of NLP models these technologies stand obsolete in the world of AI.

 

Key signals to Bot's obsolescence

 

1. Limited Understanding: The Bot failed to handle unstructured language or queries that were complex.

2. Unsatisfactory User Experience: The users gave feedback that the Bot response is not as expected, and chat was closed abruptly due to limited options ending up speaking with Human agent.

3. Scalability constraints: Upgrading the system to accommodate requirements as per Business Growth was limited. This forced continuation of legacy system left little scope for system improvement leading to progressive performance issues.

4. No Learning Capability: The critical factor noted was that the system had no scope of learning or training itself towards improvement from the interactions.

 

The Customer Care design team gathered to analyze certain key metrics to convince the Leadership into the need for replacement of this Chat Bot. Some important metrics that were leveraged to indicate Obsolescence. Additionally, Industry standards were compared to benchmark against them. They were as follows:

 

Customer Satisfaction Score (CSAT):

  Legacy bot: 60–70%

  Modern AI (LLMs): ~85–90%

First Contact Resolution (FCR):

  Legacy bot: ~40–50%

  Modern AI: ~75–80%

Average Handling Time (AHT):

  Legacy bot: Higher due to comprehending capabilities, further escalations and Human touchpoints.

  Modern AI: Significantly lower due to higher FCR

Maintenance Cost:

  Legacy bot: High due to manual rule updates. Cost-benefit imbalance

Lifecycle:

  Legacy bot: 2-4 years before major redesign

  Modern AI:  6–12 months as the model versions evolve rapidly

Learning Ability:

  Legacy bot: System remains Static over time

  Modern AI: Keeps improving based on interactions

 

Other factors that trigger Sunset or Replacement

1.      Performance Degradation-Drop in accuracy and increase in false positives

2.      Regulatory Changes: New compliance requirements (e.g., updation of KYC norms) that the current system cannot accommodate without major rework.

3.      Technology limitations to integrate to new APIs for real-time data access.

 

How to manage this transition?

Post Leadership alignment on the need to replace the ChatBot, the next step was to have a Management of Change (MOC) framework in place. Few steps to manage this transition were -

1. Creating the MOC board with representation from IT, Compliance, Business Process Owners, and User Experience teams
2. Define scope, objectives, and success metrics for the new system with expected accuracy, resolution rate, tone, response speed (where possible with metrics)
3. Parallel deployment of both systems: Deploy the new system alongside the old one for a defined period while analyzing and validating accuracy and reliability of the new system in comparison to the old.
4. Knowledge transfer: Extract FAQ databases, conversation flows, intents, and historical user queries from the old system. Documenting the historical failure patterns.
5. Assess impact and risks: Maintain rollback capability to the old system during initial phases.
6. User engagement and communication: Early announcement of new system rollout explaining the necessity of upgrade and benefits of the new system.

 

Effective lifecycle management of AI systems includes recognizing early indicators of obsolescence. A proactive mindset, monitoring of performance of existing systems, staying ahead of the curve through timely identification, quick solutioning, timely execution and responsible transition keeps the business at a competitive advantage ensuring business continuity and institutional learning.

Domain: Banking and Financial Services — Tax & Compliance Automation

A few years back, several banks deployed an AI system that handled most of our indirect tax work.
It was trained to classify transactions, map them to the right tax slabs, and prepare filings under the old VAT and Service-Tax framework.
At the time, it worked almost flawlessly — fast, accurate, and reliable.

Then the world around it changed.
When GST came in, the tax structure itself was rewritten.
New commodity codes replaced old ones, input-credit rules changed, and exemptions started coming from central as well as state authorities.
Now with GIFT City offering 10-year tax holidays and a 5 % concessional rate for certain branches, the system’s logic simply doesn’t fit anymore.
The AI still runs — but it’s working from a rulebook that no longer exists.

---

When It’s Time to Retire or Replace the AI

We didn’t rush the decision; we looked at hard signs that told us the system had reached the end of its useful life:

1. Regulatory drift – More than a quarter of the old rules it uses are obsolete, and the team spends most of its time overriding its suggestions manually. That’s a red flag.

2. Data irrelevance – The historical VAT-era data doesn’t align with the GST framework, so the AI ends up learning from outdated patterns.

3. Logic fragility – Each new policy requires yet another patch. What once delivered 95 % accuracy now struggles to hit 80 %. Maintaining it costs more than replacing it.

4. Compliance risk – Even one wrong classification for a GIFT-City transaction can cause a penalty. When the risk of keeping it is higher than the cost of change, it’s time to move on.

---

How We Managed the Transition

Step 1 – Run both systems together.
We launched the new GST-compliant AI in shadow mode for a few months. Every report was compared side-by-side with the old system to check for consistency.

Step 2 – Map and retrain.
All the old tax codes were mapped to new HSN/SAC classifications, and the new AI was retrained on three years of post-GST data so it understood the current reality.

Step 3 – Dual review.
For the first two filing cycles, finance and compliance teams reviewed both outputs manually to make sure there were no surprises.

Step 4 – Keep the knowledge.
Before shutting down the old AI, we archived its logic and exception rules. That archive now serves as a reference for audits and for the new model’s design notes.

Step 5 – Official sunset.
Once the new system met all accuracy and compliance targets for two quarters, the old AI was formally retired under IT governance procedures.

---

A Real Example

When GIFT-City rules introduced the new 5 % tax rate, our old AI didn’t even recognize the exemption codes.
It flagged every single transaction as an error, and the manual correction list kept growing.
That’s when we knew patching wouldn’t cut it.
The replacement model was trained on updated GST and GIFT-City data, tested for six months, and finally approved after hitting 98 % accuracy.
The legacy system was then archived, but its logic notes were saved for reference.

---

In Summary

Sometimes an AI doesn’t fail technically — it just gets left behind by reality.
When the laws, data, and business model evolve faster than updates can keep up, retiring the system isn’t wasteful; it’s smart governance.
By running both systems in parallel, retraining carefully, and documenting every step, we managed a clean handover that kept compliance solid and business uninterrupted.

 

In this ever changing and dynamic world it is very important to keep AI solutions up to date. The more relevant the data the more precise the results.

When AI model’s accuracy, compliance alignment, or cost-effectiveness consistently deteriorate - or when newer, smarter automation approaches can deliver better governance value, we should sunset the AI model.

 

Project- Rogers Communication

Goal- Classify the sensitive data, assign tags and masking policies to it, dyanmically

Tools and Technology- Collibra for data governance, Snowflake for integrating python with SQL to achieve the results

Approach- We designed and built data governance framework which was an AI-based governance model that continuously learns from Collibra metadata and applies data classification and masking policies to sensitive classified data assets dynamically. How? -

 

Once the metadata is ingested and standardized:

• The AI model analyzes the patterns in column names, data types, and Collibra classifications.
• It predicts appropriate tags (e.g., PII, Confidential, Sensitive).
• Based on classification, it dynamically applies Snowflake masking policies - without manual intervention.
• The AI model works in a self-correcting feedback loop:
  • It monitors policy success/failure,
  • Learns from exceptions,
  • And adjusts tagging accuracy over time.

This model ensures that any new column added in Collibra automatically reflects in Snowflake - eliminating the need for traditional rule-based programming.

 

When any of the following conditions start to appear, we retired and decommissioned our AI model-

 

1.     Decline in Model Accuracy-

a.       Our data governance model started misclassifying sensitive data and was failing to detect new classifications correctly which caused us decline in quality of the model

b.       Example: Masking policy was not correctly applied to newly added columns and incorrectly applied to non-sensitive ones

c.       What triggered us to change the Model: After 2–3 continuous monitoring cycles we observed accuracy dropped below a defined threshold (95%)

 

2.     Business Logic Drift or change in Policy-

a.       New regulatory requirements (revised GDPR, HIPAA updates) or internal policy frameworks was evolving faster than our older model could learn, which significantly reduced the output accuracy

b.       What triggered us to change the Model: Model went inefficient to adapt to change in new schema or new policy structure that broke the model’s functionality.

 

3.     System or Architecture changes-

a.       The underlying platform (e.g., Snowflake version, Collibra API schema, or data pipeline architecture) was changing but the model was not able to adapt to new changes

b.       Our existing AI model no longer integrated smoothly and efficiently with new systems

c.       What triggered us to change the Model: Model dependencies were deprecated and performance was degrading due to infrastructure changes

 

4.     Model Maintenance Overhead exceeds Business Value

a.       The cost of retraining, monitoring, and maintaining our older AI system started exceeding the savings and efficiency it provided which causes cost overhead to us to maintain such a model

b.       What triggered us to change the Model: Frequent manual interventions, debugging, or re-deployments were required which caused more operational overheads

 

5.     Security and Compliance standards-

a.       Our model’s design or learning data introduced risk of exposure or non-compliance with governance standards, which causes failure in adhering to compliance standards by our older model

b.       Example: Audit logs show us the inconsistent masking on classified columns which lead to poor function

c.       What triggered us to change the Model: Compliance violations when detected during internal/external audits.

 

6.     Better Alternative Emerges-

a.       Newer models (e.g., LLM-driven policy orchestration engines or AutoML-based governance tools) went outperforming the current approach in scalability and interpretability which made the new model very much fit and necessary

b.       What triggered us to change the older Model: ROI analysis or pilot comparison showed us > 50% performance improvement on newer retained model, so the older model has to sunset

 

AI Model Lifecycle Best Practices-

To manage this proactively we established a Model Lifecycle Governance Framework-

 

1. Monitoring phase- Here we monitored continuous drift, accuracy, and track the performance.

 

2. Review phase- We continuously checked for quarterly, semi-annual performance & compliance audit.

 

3. Retraining phase- We periodically retrain the model using new metadata for better adapting and sync.

 

4. Versioning phase- Here we maintained the version history of AI models and data sets used for understanding model efficiency and adaptability for future use cases.

 

Sunset Plan- Here we defined success criteria and process for decommissioning or upgrading at the very start of creating AI models so that we are releasing new and added features to our product in a phased manner making efficient use of AI models and sunsetting the older versions which no longer serves the business.

When Should an AI System Be Retired or Replaced?

 

An AI life cycle scenario is describes how an AI System is developed, deployed, used, maintained, and eventually retired or replaced over time in a real business process.

Just like a product or a machine, AI systems also have a life cycle — from birth (development) to end of life (retirement).

 

Indicators for the replaced or retirement

 

1.Outdated Data

Over the time the data set used to train the AI model might not represent the current product mix ,machine changes, style changes and labor performance. It will not identified new product changes .and AI system will be new to the system .The dataset used for training no longer represents and  new product range cannot be integrated with model.

 

2.Data Accuracy

AI is initially trained to identify defect in limited range .but Over time, the AI model’s accuracy drops —  false negatives increase as new fabric types, dye shades, or patterns are introduced. When such errors comes it will be aa problem to quality control decisions and customer satisfaction. This indicates AI should be redesigned or replaced. 

 

3.High Maintenance cost

  Maintenance cost will be high over the period .sometimes buying a replacement will be more cost effective

 

4.Business Goals change

Goals might changed over the period .if its designed for specifically to detect defect for woven fabric so now the problem is for some other fabric type so it wont give better results .

 

5.Tecnolgy advancement 

if new machines has inbuilt defect sensors ,might no need AI system to detect fabric defcts.so AI add no more value to that .This is where retirement of AI is indicating 

 

so to ensure smooth transition we required a risk control Plan.

• Parelle run Phase
  • To identify gaps, deploy the new AI mode in shadow mode
• Knowledge retention.
  • Document the old system logics, reports and findings specially false negative and false positive patter
• Operator training
  • Conduct hands-on sessions for quality teams and maintenance engineers to understand how the new AI model behaves and how to interpret its dashboard. Governance and Monitoring
  • Define new KPI and need to create a monitoring power BI dashboard for continuous learning. set Reviews to asses post deployment Stability

apparel industry evolve very fast so the  AI model implemented to identify fabric defects models accuracy ,relevance and cost effective factors can be declined over the time  so evaluating performance with new AI technologies and plan for a replacement while marinating smooth production flow is very important.

 

Unlike traditional application lifecycle, an AI system or AI agent will also degrade over time due to data drift, concept drift and environmental or infrastructure changes. When setting up AI system/agent as an DevOps CI/CD agent, we will need to define criteria as to what is acceptable and what is unacceptable or obsolete.

Define few criteria to consider to what is unacceptable:

 

1.       Performance: if the performance is consistently below acceptable SLA measured through key metrics like accuracy, resource usage, precision, performance, data retrieval, latency, etc. Takes lot of time to build and deploy built application to the environments.

2.       Data consistency:  if training data is no longer consistent with the production data, Application builds starts giving errors like no library found, the features we are trying to test is not supported.

3.       Ops efficiency: if pipelines consistently fails to build, takes longer time to build and test the application, does not log reasons for failures, metrics does not match or missing.

4.       Compliance and security:  if built application fail to pass security tests, flagged for non-compliance with regulations or data handling. Exposes vulnerability by downloading libraries without testing or intimation to the team.

5.       Maintainability:  if AI agent is no longer able to support the QA tools, testing tools and programming languages due to deprecated APIs or not supporting latest application architectures required by these tools (example: supporting only 32 bit systems instead of both 32 and 64 bit machines).

6.       Relevance: if DevOps agent no longer align with the business policy or process. There is a deviation every time an application is built. Does not allow modifications that needs to be done to adapt to new or updated policies.

 

Monitoring DevOps CI/CD AI Agent: 

A continuous monitoring process needs to be in place. 

1.       Performance dashboards which will track key metrics against the baseline metrics. 

2.       Setup tools like EvidentlyAI or WhyLabs (open source) which will track data drift, model performance degradation, model bias, diagnose issues and do predictive quality. Give summaries and plots at specific periods on the performance of ML. Detect, alert, and control against common LLM issues including toxicity, prompt injections, malicious activities.

3.       AI Agent’s event audits to track deployment failures, rollbacks , false triggers for not finding resources like library files and modules, compatibility issues in building applications for a particular architecture or OS, etc.

4.       Incident logs track which issues are getting logged more and regularly in spite of fixing issues in AI Agent.

 5.       User feedbacks. 

 

Perform end of life assessment:

 

1.       Check model health and performance review via MLOps governance process (if one is adopted).

 

2.       Perform a root cause analysis as to why the Agent/Model is failing consistently whether it is due to data drift, code regression, infra shortcomings or environmental change. 

3.       Do a cost-benefit run to check whether doing a retraining or upgrading is cheaper than replacing with a new model/agent. 

4.       Check for version conflicts, un-supported dependencies or any tech debt. 

5.       Evaluate risks and document the same if you plan to continue or retiring the agent.

 

 AI Lifecycle governance policy: 

Will need to adopt AI lifecycle governance policy or framework which can guide in taking a well informed decision on when and how to retire or continue with DevOps CI/CD agent apart from helping in assessment of risk, performance and other stages in the AI life cycle. Few frameworks that can be considered are NIST - AI RMF 2023 or ISO/IEC 42001:2023 which helps in having governance frame work in place for the DevOps CI/CD AI Agent or for any AI adopting in the organization.

 

 Retirement Setups: 

Once the decision is made to retire or decommission the DevOps CI/CD AI agent, we need to take following steps

 

1.       Have the new DevOps AI agent ready. Trained with new set of data and tested parallelly in lower environments for accuracy of built applications before going into production with new agent.

 2.       Stop re-training the old model

 3.       Store all the data related to old model including training data, artifacts, metadata in a secure storage as per the compliance rules for auditing.

 4.       Record impact analysis, metrics which was involved and why the decision was made to retire the old model/agent.

 5.       Disable pipelines, remove access, disconnect tools attached to the old model.

 6.       Have a transition plan as to what will be temporary solution post retiring the agent for building applications and new framework to do ground work on the replacement AI Agent or automation process.

 

 

The objective of any technology is to meet business goals in an efficient and secured manner. When it comes to AI systems, the accuracy of outcomes, timeliness and the relevancy of models behind the scenes to the business needs at that point of time. If accuracy keeps dropping despite fine tuning, or if data and concept drift make predictions unreliable, it’s time to take a hard look. High maintenance costs compared to the value delivered are another red flag. Sometimes, the issue is simply technological obsolescence; newer models may offer better performance and easier integration. If the system no longer supports current objectives or creates friction with integrating with cross platforms, that’s a sign too. Bias or lack of transparency uncovered during audits is a serious concern, and losing user trust is the ultimate deal-breaker.

The AI system is built on assumptions, architecture and data patterns which might eventually become outdated as the industry or business grows. In certain cases, the fundamental changes (process, business goals, hardware or regulations) in the working environment create a void as the AI’s logic does not match with the reality or intended use. Hence, even with continuous learning capability, the system would still retire or be replaced as it cannot overcome the obsolete model design, structural changes in business goals or incompatibility with new technology.

 

An example of this will be the retiring of legacy slotting optimization AI when the warehouse transitions from manual picking to autonomous mobile robot (AMR) operations. The legacy system used to recommend storage locations for SKUs in a warehouse, based on order frequency, item velocity and was designed for human pickers operations in the warehouse.  Now the AMRs introduce entirely new constraints around robot turning radii, safety zones, charging oaths and robot specific aisle rules which need to be incorporated with the current asset entitlements to churn out optimal task sequence involving AMRs and human labor. But the legacy AI system cannot incorporate the robot specific physical parameters even with continuous learning capability. To operate the AMRs efficiently a new model built on robotic kinematics and real-time fleet coordination must replace the old one.

To manage the transition, the warehouse and IT team will have to:

1. Capture lessons from the legacy system around what worked well and what instances operators frequently override during daily operations
2. Test the new AI model in shadow, alongside the old one, comparing their recommendations without changing daily warehouse operations
3. Gradually roll out the new system by introducing the new AI model in a single zone (say returns area) with a smaller set of SKUs (during the low volume window in the day)
4. Monitor core KPIs (pick rate, travel distance etc) along with operator override instances to ensure the effectiveness of the new system

After 3 – 4 months of close monitoring by the warehouse team, confirm that the new AI model successfully outperforms the legacy system while working seamlessly with the real time fleet coordination. Then the team can retire the legacy system and fully migrate to the new system.

🏆 Q 822 – Final Results

 

🥇 Shashank – Warehouse Slotting AI → AMR Transition (Clear lifecycle, strong indicators, excellent transition plan)

 

🥈 Adil Khan – Banking Tax & Compliance AI (VAT → GST → GIFT City) (Excellent regulatory drift example + clean transition approach)

 

🥉 Arul Palani – DevOps CI/CD AI Agent Lifecycle (Very detailed lifecycle governance + practical retirement criteria)

 

✅

Approved (Scenario-specific & meets all criteria)

 

• Manisha Boolchandani

• KP Bijesh

• Barbara Feijen

• Asangi Pathirana

• Santosh Rachamalla

 

Not approved (Generic responses without a specific AI process)

 

• SS – Provided only general statements, no domain or concrete example

• VK – Listed high-level factors but no real process scenario

• NM – Too generic; no industry or system context

• JJ – Broad principles; lacked any actual AI lifecycle scenario