Mark Wexelberg

1.  The obvious is that if humans are having to correct or being more involved in the solution than prier, it's a good indicator
2. The Agent starts to provide nonsensical answers, or generating irrelevant outputs
3. Models can output confidence levels. If they drop, it means the model will be less certain of it's answer or predictions
4. Seeing performance drop, latency issues 
5. Requiring more compute resources such as CPU, RAM, I/O speeds 
6. Data "corruption". Anything that happens to the data that changes from it's original state that the model was trained on.  As they say.  GIGO
7. How we define and use words today may change and mean different in the future but the model still thinks the "old" way.  Or customer preferences change, making recommendations on old preferences will not be good
 
These are just some examples of how an AI model can become outdated and not useful.
 
To help the models remain robust, stable and sustainable, here are some ways.
 
1. Have a way and always monitor the performance of the model, looking latency, confidence scores, accuracy, etc..
2. Know what can impact or change the data the model is using.  This can happen very easily
3. Use statistical methods (e.g., PSI, A/B testing, A/B/n testing with challenger models) to detect shifts in the relationship between inputs and outputs.
4. Have automated pipelines in place to retrain the model on fresh, representative data.
5. Maintain clear documentation of data sources, transformations, and usage to understand the provenance of training and inference data.
6. Regularly review data for biases and ensure it aligns with evolving ethical guidelines.
7. Keeping Humans-in-the-Loop.  Human intelligence layer.  Thank goodness for this
8. Clear AI governance and responsibilities.
 - Who is responsible and has ownership. 
 - Establish a review cadence.  
 -Foster strong communication and collaboration between data science, MLOps, business teams, legal, and compliance
 - Thoroughly document model architecture, training data, deployment processes, and monitoring strategies to ensure continuity and enable future improvements.
 
  

I have not been trained or certified as an MBB but I can apply what I have learned in this course.
 
Here's an example of an AI solution technically is working as it should but has become a part of the problem.
 
Consider a business who has a customer service center and their customers are experiencing long wait times.  In an effort to decrease the long wait times, they create a Chat Bot.  After implementing this AI solution they certainly can see the the call wait times has significantly decreased because the Chat Bot can "answer" them quickly.  So, technically, this AI solution is a success.   Wait times have drastically decreased.  But the company begins to hear from their customers how angry and frustrated they are, even more so than when they had to deal with long wait times.  The business failed to understand that what they should have been really trying to solve was increasing customer satisfaction, not merely the symptom of addressing long call wait times.  The Chat Bot caused greater unsatisfaction because customers now have to make repeated calls (even though they don't have to wait) because the many "simple" calls are often precursors to more complex issues and the Bot could not handle these, thus forcing customers to start over with an agent, which leads to more frustration.  Also, agents may now have to deal with more calls from customers because the Chat Bot did not properly diagnose the underlying problem.  This situation wasn't created by the Chat Bot, but by those who didn't have the foresight to really understand how they should have created the Chat Bot.  At the end of the day, technology or technical solutions, such as AI, will not be blamed for these problems that arise.  Those who created the AI solution will be.  You don't want to be that person.     
 
Back to the original thought of creating an AI solution.  The business thought it was to merely solve lowering long wait call times.  But the real root of their issue was customer frustration and dissatisfaction.  Their "AI solution" was focused on the wrong thing and it even caused a deeper problem for them 
 
How could this have been prevented?
Using the FRT process and documentation which captures the Desired Effects (DE), the Undesired Effects (UDE), and the Negative Injections (NI) of any AI project and solution.  FRTs can help to envision the ideal future state of an AI solution but also proactively identify negative consequences BEFORE a dime gets spent on creating the solution.  The FRT would have captured the root cause by addressing and thinking through the UDEs and also creating NIs to create answers for these UDEs.  Utilizing the FRT process and documentation, along with creating a very thorough and thoughtful BRD, would have greatly increased a proper AI solution that result not only in lowering call wait times, but mor importantly, raising customer satisfaction.