Mark Wexelberg

Proper prompt Engineering to LLMs is equivalent to having someone speaking properly to me in my native language. The better someone can speak my native language and with proper inflections in their enunciations and tone, the better I will understand exactly what they mean to say. If they can't, it leaves me more up to trying to interpret what they are trying to say. We've all been in that situation where we weren't clear what someone was trying say and we completely get them wrong. It's not necessarily our fault its just that we aren't getting clear communication. No difference when we can't speak (prompts) the native way LLMs can understand. If we speak poorly, the LLM has to more interpreting. So, It is up to us if we want the LLM to completely understand what we are asking for. If we don't, we have to expect that the LLM will easily misinterpret, be confused, think that we may be bias one way or another, and making it unclear for the LLM to answer with a precise response. So yes, expect the output of an LLM to never be reliable, concise or predictable if we don't know how to speak (prompt) in such a way where there is no ambiguity between what I am asking for and what the LLM understands. It's not difficult for us to imagine any use case where AI is used that things can go horribly wrong very easily and quickly.

With AI, doing more and more of our human tasks through automation, there are several things that come to mind. 1. Since AI will be replacing much of our coders today or anything that we tactically do within the data realm, this is going to fundamentally shift the responsibilities of the coders if they want to remain marketable. They must allow themselves to change with the technology and wrestle with where and how they may work within AI. Their responsibilities will change. They will be expected to change. 2. One of the people I respect said that we, the technologist, should not look at ourselves as "doers" of technology any longer, but as "directors". I think this is absolutely right on. Our roles are going well we can create and directly AI to create the solutions. AI is going to fundamentally change the way we think about what is productive. How we once defined professional "success" will also fundamentally change because we no longer have to "do" the things or "perform" the things that we once measured our success upon. We must lift our eyes and imagination higher now, out of the "doer" mentality to a "what can we imagine", "what are the possibilities", "how can I use AI technology be harnessed to serve humanity in ways", that were just a few short years ago, unimaginable. 3. Designing a human-AI collaboration process, I think, is a huge question that we are all scratching our heads about,, largely because this is all very new to the masses. But like any good design or solution, we must always be mindful who AI is serving. Implementing AI for the purpose of AI, should NEVER be the goal. As AI Solution Architects, we must always make sure that whatever is designed by us, created by AI, that it comes back to the question, "does it make a difference in making the life of our customers, or humanity, richer and better?" I think keeping "US" central to the goal of anything we design with AI, it will become more apparent where it makes sense to implement human-AI collaboration.

From what I have read on this topic, most companies are going to more interested in how we implement AI governance (understand regulatory frameworks), spot and manage AI risks, ensure ethical AI behavior, build explainability and transparency into our models, and the being able to track the AI compliance with precision. Regulatory Framework: No what policies that any AI solution must abide. This mean, in the USA, Federal, State, and of course, corporate policy. “Operationalizing” these laws means conducting privacy impact assessments, documenting AI lifecycle processes, ensuring human oversight, and keeping up to date as regulations evolve. Some ways in which we can develop and enforce these policies into our organizations would be: 1) building cross-functional AI governance committee, 2) define ethical principles and acceptable AI use cases, 3) establish an internal review process for approving model, 4) integrating checkpoints (e.g., bias testing, data review, etc..) into the development process or workflow, and 5) provide organizational-wide training on AI compliance. Risk management needs to always be in the forefront of any AI solution we create. Creating the solution is not the difficult part. Managing the risks will be and it takes a lot of due diligence, hard work and strict compliance. Some of the tools we can use are 1) regular audits and compliance checkpoints, 2) thorough documentation of how the model was trained and how it performed, 3) implementing some 3 party monitoring tools to detect drift and anomalies, 4) embedding risk reviews into the agile development workflows, 5) collaborating early with the legal and compliance teams, and 6) designing a sandbox environment for ethical experimentation. Ethical considerations need to be evaluated along with addressing potential ethical conflicts. Some of the key considerations are 1) avoiding harm to both the company and the customer, 2) promoting fairness, 3) preventing the misuse or overreach of the AI solution. Some of the documentation and reporting that can be produced to demonstrate AI compliance are 1) model cards (describing model intent, limitations, and metrics), 2) datasheets for datasets (providing detail such as origin, composition and bias review)

1. The obvious is that if humans are having to correct or being more involved in the solution than prier, it's a good indicator 2. The Agent starts to provide nonsensical answers, or generating irrelevant outputs 3. Models can output confidence levels. If they drop, it means the model will be less certain of it's answer or predictions 4. Seeing performance drop, latency issues 5. Requiring more compute resources such as CPU, RAM, I/O speeds 6. Data "corruption". Anything that happens to the data that changes from it's original state that the model was trained on. As they say. GIGO 7. How we define and use words today may change and mean different in the future but the model still thinks the "old" way. Or customer preferences change, making recommendations on old preferences will not be good These are just some examples of how an AI model can become outdated and not useful. To help the models remain robust, stable and sustainable, here are some ways. 1. Have a way and always monitor the performance of the model, looking latency, confidence scores, accuracy, etc.. 2. Know what can impact or change the data the model is using. This can happen very easily 3. Use statistical methods (e.g., PSI, A/B testing, A/B/n testing with challenger models) to detect shifts in the relationship between inputs and outputs. 4. Have automated pipelines in place to retrain the model on fresh, representative data. 5. Maintain clear documentation of data sources, transformations, and usage to understand the provenance of training and inference data. 6. Regularly review data for biases and ensure it aligns with evolving ethical guidelines. 7. Keeping Humans-in-the-Loop. Human intelligence layer. Thank goodness for this 8. Clear AI governance and responsibilities. - Who is responsible and has ownership. - Establish a review cadence. -Foster strong communication and collaboration between data science, MLOps, business teams, legal, and compliance - Thoroughly document model architecture, training data, deployment processes, and monitoring strategies to ensure continuity and enable future improvements.

I think this question is quite similar to the question previously asked and that is "how do you keep an AI agent output relevant and up-to-date while depending on having humans-in-the-loop as a significant part of keeping continuous improvements information up-to-date?" I'm paraphrasing what I remember that question to be. The importance of an AI Solutions Architect isn't that we merely create targeted, robust, reliable, predictable, sustainable, scalable, etc... AI solutions and that have been designed through a rigorous, thoughtful, intentional, and successfully proven design process, but we are also, and I think most importantly, stewards and overseers that AI isn't running so autonomously that we have released our responsibility over what AI "does for us". In other words, we are trusting and thinking AI doesn't need human intervention because we say it's smarter, faster, better as what humans can do. This is a completely and wholly improper perspective to have of AI. Therefore, the ultimate "tool" is really us staying engaged and by implementing ways to help US monitor that the AI solution stays true to it's design intent. I'm not an expert at knowing or leveraging these tools because I'm not formally trained in them, but they certainly make sense to me as I read about them, in keeping AI solutions "on track". They make sense because I have implemented these tools in overseeing projects and business intelligence platforms, just not knowing their professional labels. Here are some tools and techniques I have found, without their explanations, because I know you are the MBB expert that will recognize and understand what they are and their applications. Control Plans, Statistical Process Control (SPC), Poka-Yoke (mistake proofing), standard operating procedures (SOPs), training and continuous education, visual management tools, audits and reviews, response and reaction plans, process ownership and accountability, leader standard work, regular communication, and continuous improvement culture (Kaizen).

May I respectfully suggest these questions be answered from an "AI Solution Architect" persona rather than a MBB Here are the AI/ML techniques I would employ into the Agent, the implementation steps, evaluation strategy and ethical considerations AI|ML Techniques: Reinforcement Learning: The Agent can learn actions through trial and error, guided by human feedback as reward or penalties Active Learning: This involves selecting the most informative data points for human feedback, which will minimize the amount of labeled data needed while also maximizing learning efficiency. Human-in-the-loop (HITL): This integrates human feedback directly into the learning process, allowing for real-time adjustments and improvements. The Agent can capture all the human feedbacks and add it to the KB data, keeping it up to date and relevant. Transfer Learning: This technique adapts to changes without having to retrain from LLM scratch. It leverages the pre-trained models but by also fine-tuning them with new data and feedback. An interface, like a web browser, where people can provide their updates and the Agent will append the updated data into the KB file or database. The Agent will always pull from that updated KB, thus not becoming obsolete and misaligned.

I have not been trained or certified as an MBB but I can apply what I have learned in this course. Here's an example of an AI solution technically is working as it should but has become a part of the problem. Consider a business who has a customer service center and their customers are experiencing long wait times. In an effort to decrease the long wait times, they create a Chat Bot. After implementing this AI solution they certainly can see the the call wait times has significantly decreased because the Chat Bot can "answer" them quickly. So, technically, this AI solution is a success. Wait times have drastically decreased. But the company begins to hear from their customers how angry and frustrated they are, even more so than when they had to deal with long wait times. The business failed to understand that what they should have been really trying to solve was increasing customer satisfaction, not merely the symptom of addressing long call wait times. The Chat Bot caused greater unsatisfaction because customers now have to make repeated calls (even though they don't have to wait) because the many "simple" calls are often precursors to more complex issues and the Bot could not handle these, thus forcing customers to start over with an agent, which leads to more frustration. Also, agents may now have to deal with more calls from customers because the Chat Bot did not properly diagnose the underlying problem. This situation wasn't created by the Chat Bot, but by those who didn't have the foresight to really understand how they should have created the Chat Bot. At the end of the day, technology or technical solutions, such as AI, will not be blamed for these problems that arise. Those who created the AI solution will be. You don't want to be that person. Back to the original thought of creating an AI solution. The business thought it was to merely solve lowering long wait call times. But the real root of their issue was customer frustration and dissatisfaction. Their "AI solution" was focused on the wrong thing and it even caused a deeper problem for them How could this have been prevented? Using the FRT process and documentation which captures the Desired Effects (DE), the Undesired Effects (UDE), and the Negative Injections (NI) of any AI project and solution. FRTs can help to envision the ideal future state of an AI solution but also proactively identify negative consequences BEFORE a dime gets spent on creating the solution. The FRT would have captured the root cause by addressing and thinking through the UDEs and also creating NIs to create answers for these UDEs. Utilizing the FRT process and documentation, along with creating a very thorough and thoughtful BRD, would have greatly increased a proper AI solution that result not only in lowering call wait times, but mor importantly, raising customer satisfaction.

According to what VK noted under his forum questions, “Some people seem to be using AI platforms to find forum answers. This is a risky approach as AI responses are error-prone..” AI is created by humans who are prone to error. We must always remember this and be diligent to make sure AI will make the best decisions. “Making sure” will ALWAYS to be and, I believe, will forever be, a human responsibility. I can’t ever imagine anyone shirking their responsibility and pointing at the AI solution and saying “It’s the AI’s fault that we lost revenue”. Yes. It might have been that we trusted the AI agent to make the decision but ONLY after we allowed it to make that decision. So, the real accountability still falls back to a human. Knowing that AI is prone to make errors, just as humans have done to mitigate making our own errors, we created guardrails to increase proper decision making and better outcomes – ergo, Business Excellence. Think of AI as another person. But now you are responsible for the decisions and actions of that person. It will need oversight, accountability, and transparency to make sure AI is making the right decisions on our behalf. Here are some of the elements that I think could be included in a governance framework to ensure responsible, high-impact use of AI in a process-driven organization. Creating a governance team or committee to oversee all AI solutions. This team would comprise people from IT, the business, legal, risk management and defining each role and responsibility throughout the AI development, deployment and maintenance. For transparency and accountability, conducting regular impact assessments to identify potential risks, biases and consequences of AI-driven decision. Also, implementing techniques that can provide insights into the how AI is making its decisions, such as feature attribution or model interpretability methods. Lastly, performing audit trails that let us see the data inputs, processing and outputs the AI used to make its decision. For agility and control, using agile development methodologies to allow for rapid iterations and deployment. Using change management to capture the all the changes made throughout the development which can easily be reviewed, tested, and validated. Lastly, establish access controls to prevent unauthorized changes to the AI system or data.

No matter what metric we produced or how fast it gets produced, it provides zero value if it isn’t accurate, i.e., do we trust it to make critical decisions upon it? The further away we get from knowing how we produced a metric, the less we trust it. Because AI can “do” the formulas, calculations and algorithms for us – so quickly, and easily, we could mistakenly put more value on the speed of AI than the actual value we "think" it's producing. There are a couple of wise sayings that I’ve heard in my life. “Keep and honest man honest”. That idea has also been said, “Trust but validate”. Why? If an honest man is honest, then why do we have to make sure he’ll remain honest. Because everything has a bias and can make mistakes, including what metrics AI has created for us. So, for me, the most valuable metric is the one I can trust, whether created by AI or not. The balance is being able leverage the power and speed of AI as well as validating everything it generates. Ergo, Agentic AI. Outdated metric - “Average Employee Training Hours”. Investing in employee training is seen as a direct indicator of capability building, the “level-up” people skill-sets. Traditionally, we think, the more hours, the better. However, AI-driven learning that can hyper-target content to precisely educate where a particular skill is lacking. The hours spent now becomes less relevant but rather the efficacy and application of the learned skill become what is important. If companies rely on “average training hours”, they could easily over-invest in traditional training methods while missing the AI-enabled learning pathways. It shifts from input (hours) to outcome (applied skill). New metric – “Value Realization Velocity (VRV)”. This metric measures the speed at which AI-driven insights or recommendations are converted into tangible business value. Every business has struggled and failed to move an idea from “concept” to “production” to and to know it’s real “realized value”. VRV could track: - Time from AI model deployment to first measurable business impact (e.g., first dollar saved, first customer converted. - The percentage of AI-generated insights that lead to actionable changes within a given timeframe. - The monetary value generated per unit of time from AI-driven initiatives (e.g., incremental revenue per week from an AI-optimized marketing campaign). This metric directly ties AI initiatives to strategic business outcomes because it pushes beyond mere technical performance to demonstrate tangible ROI and agility. Because the AI-driven economy will, and is, so fast-moving, this will be paramount for Business Excellence.

I am not an MMB, so obviously I cannot speak with specificity or be a voice of authority of one who is. But I would imagine this would be no different than an MMB being involved with other project or solution they would find themselves in. There's always been language barriers between those who work in the front of the business and those who work in the back of the business. But they should both be driving and measured by the same objective and that is, at all times, to be customer-centric. To try to answer this question, I had to do a reading on what an MMB role is. I find that I functioned in that role, to a large degree, all my career and I attribute it as been one of the key factors in my success. So, here is how I would answer the two part question. 1. MMB role in AI projects: Pre-Implementation: Define current process, uncover the pain-points, map the value streams, translate business problems into AI solvable problem, keeping the project always customer or business-centric, project requirements gathering, defining the metrics where the AI initiative can be measured as a success, identifying operational risks, ethical considerations, business continuity issues relate to the AI solution Implementation: Change management, preparing the organization for the new AI-powered process, identifying operational risks, ethical considerations, business continuity issues relate to the AI solution Post-Implementation: Helping to track and quantify the business value delivered by the AI solution against the initial objectives 2. Aligning AI initiatives with process excellence, customer value and organizational priorities Communication, communication, communication: - Translating between the two domains. Being the "Rosetta Stone" between the business and technical teams - Holding joint workshops and helping each domain understand each others terminologies and objectives - Use frameworks that integrate both business process mapping and AI solution design - MBBs should be involved from the very inception of AI initiatives, co-creating with AI Architects, rather than handing off requirements - Keep the initiative focused on providing the solution for the business problem, not just the technology and developing business cases that clearly articulate the ROI of AI solutions - Implementing governance structure by implementing steering committees from both business and technical sides - Create a structured decision-making process for AI projects - Use Lean Six Sigma tools to manage the AI projects to keep things structured and towards continuous improvement - Conducting process mapping before AI implementation to identify optimization opportunities - ADVOCATING FOR CUSTOMER AND END-USER.