Najmuddoja

Unlike traditional systems, which often exhibit signs of failure in a specific pattern, AI systems fail quietly and silently, not displaying obvious signs of failure. To detect AI failures, proactive monitoring is required to identify when inconsistent behavior began, when results deviated, and when it started to deviate from compliance, as well as when corruption of intent started. Example: A multi-model AI chatbot handling critical application information. Signs of Warning of AI failure - The model starts giving wrong information about applications - Model starts giving low confidence to queries that it previously did well - Ingesting of non-compliant data after initial load. - Low confidence indicates that the prompt is becoming misaligned. - Started to see escalations or users asking for human intervention. - Fail to handle queries effectively. - Hallucination increases - Poor sentiment analysis - Answer to queries degraded due to the user starting to ingest data in a format other than the initial training data. - The user is not ingesting new data when the system's changes occur. - New policies are not ingested into the system. To mitigate the above signs of failure, we need to implement a monitoring strategy. - The most important thing to monitor is the system's performance, checking the accuracy of answers and setting thresholds to track performance. - Always have a base KPI to measure the performance. - Check the escalations of users. Again, review the number of escalations occurring daily, weekly, and monthly, and aim for a percentage increase of 10-15% over the baseline. - Keep track of how often human intervention occurs. - Monitor the time it takes to obtain accurate data about applications and compare it to the baseline. - How is the quality of the result? - Prompt: Check how many tokens it's using now and what the latency is. Risks of AI Automation. AI poses unique risks in automation. - AI will fail without any notice. - Trust without validation leads to team failure. - An AI system will lose its learning ability when the feedback loop breaks

That’s true today; traditional auditing methods primarily focus on static inputs and outputs, as well as human decision points. However, when we discuss incorporating AI into business, we need to think beyond traditional methods. When we introduce AI into the mix, we need to consider its impact on business processes. We need to shift from static checklists to dynamic checklists, as the very nature of an AI-infused system is to evolve; so, our auditing should also evolve. The following criteria should be considered when auditing the AI process. 1- The quality of data is crucial, as it should be dynamic, and we continually ingest high-quality data. 2- The performance of the process is increasing or decreasing. 3- The process is handling the intent. 4- Provide improved prompt templates and regularly test them. 5- Continue testing new real-world cases, looking for new edge cases. 6- Keep on updating documentation and SOPs. 7- Revisit training data sources for accuracy and completeness. 8- Evaluate prompt engineering regularly for safety. 9- Keep feedback in the loop when AI outputs hallucinate. 10- Audit of change logs and tracking of model updates. 11- Does AI generate outputs following a logical path or in response to a prompt? 12- Is the system compliant with regulatory requirements, including HIPAA and GDPR, as well as ethical standards? 13- Does the system support business goals, e.g., customer satisfaction and revenue growth? Practical Implementation of Controls to Ensure Sustainability. 1- Firstly, testing should be comprehensive, using both scenarios and real-world examples. 2- Continue to monitor the quality of outputs, user feedback, and hallucinations. 3- Implement CI/CD for proper checkpoints and change management. 4- Always have up-to-date documentation, version control, and traceability of prompts. 5- Review of ethics and compliance regularly. 6- Include SMEs from cross-functional teams, e.g., data scientists, legal teams, and ethical champions. Due to the changing or dynamic nature of AI systems, it poses unique challenges and risks. 1- Decision outcomes are often unclear, and unintended consequences can lead to discriminatory practices. 2- AI systems are overly reliant on LLMs without a human in the loop. 3- Low-quality LLM prompts can result in malicious manipulation. 4- As it relies on the quality ingestion of data to LLM, performance will degrade if retraining is irregular. Conclusion. As mentioned earlier, an AI system audit should be dynamic and introspective, examining why AI did X instead of Y. If we want our AI system to evolve with new real-world challenges while being safe, ethically aligned, and aligned with business goals, we should keep humans in the loop, maintain transparency, and closely monitor the system from day one.

Manifestations of fragility in AI solutions Diminished performance, an increased error rate, and inconsistent outputs arising from changes in user behavior, variable data, or evolving knowledge bases. Indicators of AI Fragility. 1- Performance Decline A notable and lasting decrease in performance is observed, characterized by heightened fallback rates, reduced confidence thresholds, slower response times, and diminished accuracy. 2- Human Overrides and Interventions Consistent participation from humans, such as support from agents, manual fixes, and behind-the-scenes adjustments, reflects the system's vulnerability. When the team often needs to bridge gaps, correct mistakes, or ensure proper operation, it reveals flaws in the design and diminishes the AI's independence. 3- Elevated User Frustration We are noticing misclassification are increasing, from false positives and miss detections. Rising user dissatisfaction is a result of a widening gap with the context. These elements suggest that the AI system is not progressing in tandem with the needs of its users. When we increase in users report errors or stop engaging, it becomes clear that system is drifting from clear intents and is not fulfilling expectations. 4- Obsolete Knowledge Base When AI agents depend on fragmented, outdated, or inconsistent knowledge sources—such as old KB articles or isolated repositories—they become susceptible to retrieval mistakes, hallucinations, and fragile performance. 5- Dealing with Exceptions When we not able to take real-world scenarios and hard-coded logic to manage outputs , it represents a breakdown in the learning cycle. Achieving Sustainability - Governance Knowledge base should be updated regularly, regular review of Model drifting and keep on checking Prompt performance - Define structured protocols for updating data, models, and content to ensure continued relevance and stability.. - Business priorities of AI Evolution should be aligned with Governance, while continuously monitoring customer experience, ethics, and compliance. Feedback Loops Continuous improvement can be achieved through leveraging feedback loops from end-users, human agents, and system logs. Analyze ratings, flagged failures, unrecognized intents, and behavioral shift signals via click patterns and sentiment shifts. Inform retraining, update prompts, refreshing of training data, and fine-tuning system architecture. Resiliency Our focus should be on testing real world cases, modifying inputs which are outdated, and testing those queries which are ambiguous to validate rebustness. Adaptability Utilize modular architectures (e.g., Agentic RAG, plug-in tools) to enable upgrading individual components (e.g., LLM version, KB connector) without requiring a complete stack overhaul. Summary In short, you know when an AI system begins to show signs of fragility and when its drifting from relevant real-world user behavior, shifts in business logic, or domain knowledge. Real sustainability comes from embedding alignment and adaptability into the architecture itself.

Framework AI Prioritization: Blueprint for Business Readiness Any project begins with a concept, followed by goals and objectives, a discovery phase, a design phase, a development phase, a testing phase, and a deployment phase, among others. And it will be no rocket science when you start an AI project. COPQ refers to the cost of producing a product of poor quality, which fails to serve its purpose, resulting in not only financial loss but also a poor reputation for the company. Let's explore how to analyze the production of an AI product using the Business Impact vs. Feasibility matrix through the lens of Cost of Poor Quality (COPQ). We will consider a few strategic imperatives and tie them to the AI Initiative. 1- Map AI on a 2X2 Matrix Y-axis: Business impact à Profitability, Revenue growth, Risk reduction X-axis: Feasibility à Technical readiness, data availability, maturity of model. 2- Apply COPQ metrics: a. Prioritize what matters and will reduce rework, minimize defects, reduce red tape, and be very careful with compliance risks b. Prioritize building a process that reduces the delay in claim processing, rather than creating a new claim system. 3- Strategic Fit a. Every initiative should have some strategic goals instead of "good to have" Y-Axis Business Impact Business Impact Strategic Goal Sample Cost effective Operational efficiency COPQ reduction Revenue Customer Lifetime Value Sales conversion Risk Reduction Reputation, Fraud prevention Fraud cases prevented X-Axis Business Impact Feasibility Strategic Focus Sample Data Availability Good quality Data-driven decision making Accessibility of clean, structured, labeled data Model Maturity AI readiness Are we using a proven model or a POC? Implementation Readiness Time to market, tech alignment Do we have the required tools and skills to deploy now The Decision-Making Test Is the solution we are building novelty, or will it serve a strategic goal? Novelty: If the answer is 'good to have,' vague, or everybody is doing AI, etc. Strategic: Eliminate the measurable pain, e.g., reduce the delay of claims processing by 40% Real Life Example: In a large IT company, a critical business failure can make retrieving essential information related to applications, servers, and certifications a time-consuming task. The complexity of applications, multiple inventories, and artifacts will exacerbate the situation, resulting in prolonged application downtime. This delay in bringing back applications impacts companies, membership, customer satisfaction, and revenue. Building a custom multi-model AI and machine learning Chat solution that can quickly identify the context and intent of users and provide the necessary information within seconds, rather than 30 or 40 minutes. DMAIC for AI-Driven Knowledge Retrieval in IT Incident Management Phase Application to the Real-Life Scenario Define Identify the core problem: delays in retrieving critical data during business failures lead to extended downtime, poor customer experience, and revenue loss. Define project goal: implement an AI-powered chat solution to reduce response time from 30–40 minutes to mere seconds. Measure Average time to locate information Number of systems and artifacts accessed per incident Downtime impact on revenue and customer satisfaction Analyze Fragmented inventories and repositories Lack of a unified access layer Manual search and contextual interpretation delays Improve Build and train a multi-model AI/ML chat system Integrate structured and unstructured data sources Build contextual retrieval and recommendation layers Control Build contextual retrieval and recommendation layers Monitor retrieval precision Impact: Building such a strategic AI system becomes the lever for Resilience, efficiency, and excellence in experience across enterprise support workflows.

Swiss Cheese Model Every complex system has weaknesses and vulnerabilities that can lead to its failure. A complex system has multiple layers of defense, and weaknesses or failures can occur if those weaknesses or holes align and allow hazardous materials to pass through. This concept of multiple defense layers and holes is widely known as the Swiss Cheese Model. "Slices of Cheese" is called the Defense layer Holes in cheese are called vulnerabilities. Application of the Swiss Cheese Model to My Project In my current Digital Transformation initiative, I lead the Cloud Migration process from on-prem to Cloud Slices of Cheese or Defense Layers - Solution Design Reviews o The Architecture Review Board (ARB), in collaboration with Security Champions, evaluates architectural designs to ensure they meet compliance standards while remaining scalable, resilient, and secure. - SRE Reviews o The Site Reliability team, with a strong focus on Reliability, observability, and operational efficiency, reviews cloud architecture. - CI/CD Pipelines o Automating development, testing, deployment, and validation using IAC (Infrastructure as code) to ensure consistency. - Segregation o Segregation of data across environments - Change Management and Approval Process o Stakeholders should approve the implementation of any change. Every change to production should have a Change Control approved ticket. - UAT (User Acceptance Testing). o Business users should test every significant change to validate functionality against real-world workflows. - Certificate management o An automated system should manage the certificate during the migration from on-premises to the Cloud. - Monitoring & Observability Tools o Using tools to monitor real-time metrics, logs, and alerts to catch any anomalies early - Business Continuity o Implement DR strategies to ensure business continuity. - Documentation o Create documentation of workflows, best practices, etc - Post-Deployment Reviews & Feedback Loops o The team continuously captures insights in a shared knowledge base, fostering an environment where Architects and engineers engage in ongoing training and learning. - Segregation o Segregation of data across environments Holes in Cheese or Weaknesses (Vulnerabilities) - Design Reviews o Business requirements Misalignment may lead to the omission of edge cases. - Roles o Teams often neglect to enforce the least privilege access, increasing the risk of unnecessary exposure and security vulnerabilities. - Gaps in Automation o Not version control, outdated scripts, and environmental variables not correctly set up - Approvals Slips o Change management rubber-stamping without fully understanding the change or without complete risk analysis - Monitoring o Teams frequently misconfigure alerts or apply inappropriate threshold settings, leading to ineffective monitoring of critical processes. - Post-Deployment o Lack of motivation in the lesson learned. - Audit o The team rarely reviews audit trails. Using Business Excellence to Strengthen Reliability. Using DMAIC, we can strengthen reliability. - Define o Align the project's goals with strategic priorities, identify stakeholders who will be impacted by the project, and define roles and responsibilities using a RACI framework. - Measure o Identify knowledge gaps and tailor training to meet the needs of individual staff members through targeted feedback. Involvement of cross-functional and all stakeholders - Analyze o Lead the actionable task through workshops, build a pilot, and let stakeholders stress-test and make improvements on small tasks. Additionally, educate stakeholders on how to navigate difficult conversations. - Improve o Encourage full participation in training and simulation exercises to enhance team preparedness and reinforce best security practices. Guide pilot efforts and build team capability through iterative learning. - Control o Embed changes to documents. Strong emphasis on conformance audit & compliance checks and strategic direction, or a roadmap of what to monitor, how to monitor, and how often. Who is responsible? Also includes plans in case metrics exceed targets. A personalized and focused approach to skills development drives continuous growth and improvement. Summary Examining the cloud migration process through the lens of the Swiss Cheese Model enables us to mitigate risks proactively. DMAIC tools help close holes, vulnerabilities, and weaknesses and align slices.

DMAIC is the acronym for a problem-solving approach. Define, Measure, Analyze, Improve, Control How change management can impact the success or failure of a project: the absence of change management can lead to even the most robust statistical solution stalling at adoption or slipping back into old habits. Define: Focus: Before starting any project, the vision should be clear, and "why" needs to be defined to gain the alignment of sponsors and stakeholders. Charter: Align the project's goals with strategic priorities, identify stakeholders impacted by the project, and define roles and responsibilities using a RACI framework. Example: In a project of building a bot that can use the documentation from applications, we define which documents we can ingest, so there is no issue with compliance. Measure: Focus: CM focuses on building credibility and mapping how baseline metrics affect stakeholders. Scope: Identify knowledge gaps and tailor training through targeted staff feedback. Involvement of cross-functional and all stakeholders. Example: In a bot project, involve stakeholders from each application early on to define how the data will be captured and used clearly. Analyze: Focus: The change management focus should be on translating gathered data into actionable design and changing existing thinking with actionable data-driven insights. Scope: Lead the actionable task through workshops, build a pilot, and let stakeholders stress-test and make improvements on small tasks, and educate on difficult conversations. Example: In a bot project, educate stakeholders (application teams) on how to format their documents, how large documents should be, and what can and cannot be included in documents during the early stages. Improve Focus: Change management ensures that new SOPs are followed, users adopt new ways of working, and manages resistance. Scope: Make it compulsory for the team to complete the training and simulation. Guide pilot efforts and build team capability through iterative learning. Example: In a bot project, we create templates on how to create documents and how to enter data into different tabs, as well as what types of data need to be captured. Control Focus: Change management has to focus on how to sustain improvement and gains and embed the changes Scope: Embed changes to documents. Strong emphasis on conformance audit & compliance checks and strategic direction, or a roadmap of what to monitor, how to monitor, and how often. Who is responsible? Also includes plans in case metrics exceed targets. A personalized and focused approach to skills development drives continuous growth and improvement. Example: In a bot project, we make sure application teams adhere to SOPs, a standard template, no PII or PHI data, and automated ingestion of updated documents

First, we need to understand why improvements lead to slips. • Non-Standardization: Undocumented applied procedures, workarounds applied, band-aids not documented, and worked solutions not embedded into SOPs. • Limited Training: Non-standard training, insufficient training, no formal training or checks, and cultural mindsets led employees to revert to the "old school of doing things." • Non-Proactive Monitoring or Surveillance: There is no proactive monitoring in place to detect undocumented changes or outdated practices, leaving them unable to identify when issues arise. • Drifting leadership: Leadership misses undocumented changes, does not track the training and behavior of people, and accountability fades. • Scope Unchecked: Small, undocumented changes reintroduce variation. Due to not keeping "scope" in check, uncontrolled growth happens, and complexity increases Tools and Techniques to sustain improvements. • Plan of Control: Strategic direction or roadmap of what to monitor, how to monitor, and how often. Who is responsible? Also includes plans if metrics go outside targets. • Documentation: Update the training manual to include new work instructions and changes. • Ownership & Accountability: Assign ownership using a RACI matrix. Review performance metrics regularly. • Conformance Audit & Compliance Checks: Regular audits can help catch early signs of slippage. • Training: A personalized and focused approach to skills development drives continuous growth and improvement.

Most AI models, after deployment, are static, and without continuous improvement, they are vulnerable to becoming outdated. AI can be trained for constant improvement with a feedback loop intentionally embedded. The system must be designed for adaptability to ensure AI solutions evolve with continuous improvement efforts. Steps for Continuous Improvements Ingestion on a regular cadence, Updated Process Data, and Human feedback Automating feedback Ingestion: Feeding monitoring data and data pipelines back to the model Retraining: Automating the retraining of the model based on KPI Framing the Problem: AI should address the problem within its proper context. Governance: Aligning AI outputs with current process goals and standards. Decision Makers: Keep humans in the process of validating, optimizing, and overriding for better learning. Auditing: Versioning for clear visibility on when, how, and why AI models are updated with business alignment. AI model improvement is not a “one-size-fits-all” nor a “one-time” solution, as with every other model in the world; things keep changing based on new data available, new processes, new ideas, and new and improved technologies. Therefore, it should be a living component of the process.

If the question isn't framed with clarity, the AI will solve the wrong problem; for example, a doctor's office builds an AI model to predict which patients are likely to miss appointments, aiming to improve clinic efficiency. The model accurately displays the results. However, at the same time, the AI model may book double bookings, leading to patient dissatisfaction. As it misses the root cause of missed appointments, it fails to understand why patients miss them (e.g., poor communication, transportation issues, etc.). This issue can be dealt with by asking subsequent questions, e.g., What are we trying to achieve? Factors contributing to the issue? Is this a cause or a symptom To address these situations, MBB can conduct interviews. This approach enables teams to move beyond symptoms and frame the problem precisely. The result will be, instead of solving the problem wrongly, which works well, but is not of real use

When should a process be improved? Improving the process when the process is working but inefficient due to being outdated. Deliver meaningful value with incremental gains Optimize a process through automation Regulatory requirements need to be improved Example: Voice-controlled phone in cars Reimagine a Process with AI. If a process does not meet the current fast-paced environment and heavily relies on humans, it is error-prone. To achieve results and performance which was not attained previously, e.g, Predictive analysis, personalization, and decision making. User experience needs an overhaul, and 24/7 availability to improve user experience New way of doing business. Example: Using AR Smart Glasses for Maintenance and Operations