yuvaraj krishnan

Any AI design should incorporate a feedback mechanism to ensure the AI is on track consistetly and serves the purpose of its design. This is an important design aspect, as any system design will always possess an element of uncertainty which can come in a form of extreme use cases, extreme users and other environment conditions for which th AI would not have been trained. The feedback mechanisms can be incorporated by the following ways Direct feedback mechanism loop : This kind of feedback is very essential in a conversational AI agent. We can build in a thumbs up or a thumbs down icon for very response that AI agent gives. If the user gives thumbs down, the design can prompt to elaborate why the response is not as per the users expectation. We can gather periodically the thumbs down and analyse and make improvements. The most important thing is to improve incrementally and not radical overhaul. By this way we can ensure that the system and the users are not overwhelmed by the change. Also there a lot of AI agents having an option of reporting and issue. Building this option can be very helpfull if the AI agents response is inappropriate and not related to the subject question it is designed to solve. This can be analysed and AI can be retrained to fix the concerned use case issue. Performance Monitoring: This is another method to elicit feedback. For example in a IT ticketing support agent, we can measure the AI system performance by gathering the number of accepted resolution or okayed and the number of unresolved tickets. This will directly help us to check the use cases where the AI agent is not able to resolve. The AI agent can then be trained separately to address these uses cases so it learns and then monitor again to see if these use cases are resolved in real time scenarios during ticketing process. Continuos monitoring and incremental improvement: A team comprised of developers, functional consultants and internal IT users amcan be formed to monitor the AI agents performance and effectiveness of functioning in the real world, simulate it in a test environment with extreme uses cases and conditions so as to proactively identify issues and retrain the AI and keep it on track consistently. In summary, feedback mechanisms is an integral part of the AI agent design which helps the AI agent to be ontrack and serves the purpose for which it was designed.

Any AI agent is adept in solving objective questions and is very simple. But when an AI agent is used for continuous interaction and workflows to solve a problem, then the AI design will have to consider factors such as clarification of the issue , summarize the questions at appropriate stages, managing the memory from previous interactions, focus on the goal which is the very purpose of the interaction, more importantly empathizing with the user etc to ensure the AI agent is helpful and focused and efficient in addressing the problem. I can think of a process which I was planning to implement in my company, and this question really picks my mind to consider many things though my plan is to start small and focus to a specific area. Let’s say, I have developed an AI agent to address user issues related to invoicing. Going by the ticketing history, most of the invoice related issues will have the ticket description as “Not able to print invoice” So, my AI agent will have to ask user to do some checks, based on the result of those checks, there will be some other checks and based on that some other action will be directed to the user. Sometimes it can go to 5 or 6 checks before identifying the root cause. This is basically emulating the manual trouble shooting by the support personnel. Let’s say a POC is as below. AI Agent – Hello, how can I help you with your invoice related issue? User – I cannot print invoice for an order AI Agent – Do you intend to run invoices for all the lines in the order or specific lines? User – Only one line in the order which must be urgently dispatched AI Agent – Is the status of the order line at “58” User – Yes AI Agent – Do you see in the application any hold code for the order line you intend to invoice User – No AI Agent – Is there a price for the line you intend to invoice? User – Yes AI Agent – Can you give me the version of the invoice you ran? User – INVSO3 AI Agent – Sorry, no version exists, please use version INVSO3I User – it was a typo; I ran the correct version INVSO3I AI Agent – will you be able to check the scheduled invoice date if it is today or lesser date? User – I don’t know AI Agent – Can you open application PSINV25U? User – Sorry I don’t have access AI Agent – Should we first create request for you to have access? User – I need this to be resolved urgently AI Agent – Sorry, please raise an IT ticket, so a support staff can assist you better. From the above POC, what I see is the design of the AI agent is vey direct and straight forward, but after reading about this topic I need to tinker a bit the design so the AI Agent will also exhibit the following: Being Empathetic, understanding user sentiments and respond accordingly: The user is in a hurry to take invoice print for the goods that must be dispatched from the warehouse. The trucks are waiting. So instead of “Do you intend to run invoices for all the lines in the order or specific lines”, I would recommend “Sorry to hear that, don’t worry we can resolve the issue together quickly”, would be a better response from AI that gives an assurance to the user and calms the user. Summarize the issue for better user engagement and focus: Instead of asking for repeated checks in isolation, I think after couple of checks there could be a summarization like” the status is correct, there is price, there are no holds, still not able to invoice, interesting, bear with me for few more checks”. So, the user would be engaged and not frustrated, and the user will know the AI agent is analyzing and really trying to troubleshoot with the inputs of the user Focus on goal though there are several checks and misleading inputs: Instead of asking user to solve the access issue, which was not the very purpose of the interaction , it could have said “ No problem, can I connect you with a support agent to resolve the invoice prints, so the issue will be resolved in the next 15 minutes, is it ok?” something like this I am wondering. Also, I am thinking of making communication very efficient, which would require more iterations and use cases. Overall, I feel using the above steps or techniques a conversational AI can navigate multiple step processes/workflows and emotionally charged interactions, effectively ensuring that the user feels engaged and the interaction is productive

Prompt design basically defines the character of the AI system. If we take movie making as an analogy, Prompt desiging is like a director of a movie who defines the characters of the actors as per the story. Without proper definition of a character the role of that actor would be flat and look out of place .The character will not display appropriate emotions, dialogue delivery, subtle changes in body language etc., for different scenes, ultimately resulting in the failure of the movie due to a big disconnect between the audience and the characters. Hence the quality of a prompt design will directly impact the quality of the output of AI. When we consider a conversational BOT, though it can understand Natural language and process it, how it responds as per the defined role will play a major role in the AI's usability and experience that user gains. Let's say for example a 10 year old is eager to know the Theory of relativity. If the AI responds too technically that not even a single word makes sense to that 10 year old, then in the eyes of the 10 year old AI is not of any help to him. Instead, if the prompt design had included to check the age of the requestor or use appropriate analogy to explain concepts as per age of the requestor, then it would have been more useful. Another example, say a shop floor manufacturing supervisor who enters the production data into the ERP systems, finds few keys on his key pad non functional. So he chats with company's IT support BOT and types " some keys not working in the keyboard", the BOT responds by asking what is the OS version, what is the system Tag, when was the last software patch done etc..The user just shuts the BOT, picks up the phone and talk directly to IT support personnel..The very purpose of the IT support chat BOT failed. Here, the BOT should know that it is interacting with a non technical employee and has to converse in his language, ask information which he is capable of giving. There is a famous quote in Marketing, 'always talk the language of the customer ". When computers were boom in the business world, it started with personal computer, slowly it evolved to personalized computing. The same way AI is just entering to business world, there is no perfect system, but there must be a robust process put in place to monitor, gather feedback and improve the systems, in case of conversational BOTs , improve the prompt. Again back to movie analogy, the result of a movie be it success or failure depends on how well the audience recieve the product. Successful or not, by having a feedback mechanism it helps the director to learn and make improvements in his next movie. Again it is a cycle of iterations that is applicable in every aspects of life. The same is applicable to AI system as well. Like how a director cannot ask an aged character to dance difficult steps in a movie, the prompts too must be specific, formatted, be on the context and talk the language of the user. In my view, the generalised AI will slowly be evolved into personalized AI in the coming years. The AI will develop capabilities to understand about the different users and serve them as per their personalized requirement. In summary, The prompt design (continuously evolving and improving) will make the AI descisions more relevant, useful and will serve the purpose for which it was designed.

A dialogue from a yesteryear movie goes like this " Take your greatest fear and multiply it by X". I wonder if this is how today's workforce feel!! While it is true the whole business world is fascinated by the development and prowess of AI, the global workforce at least from low to mid-level feels a strong sense of fear of job security and panic. Especially people above 40 years. Some say AI can automate repeated, mundane tasks so as to free some bandwidth for the employee to focus more on creative, enriching and fulfilling tasks in the current or new role. While others say it replaces many jobs which are labor intrinsic. We know both are true though skewed towards later. In my view, AI system is and has to be used as a tool to empower workforce rather than be seen as a threat to the workforce. AI system by itself is not responsible for impact on the workforce with respect to layoff, but very poor Strategic Planning by the leadership of the companies Is. Now looking at the topic, what happens if AI becomes part of the team, how would the roles change, well I can say ideally the following needs to happen even before companies plans to implement AI system to ensure there is a smooth Human –AI collaboration. Objective of the implementation of AI system and its Impact: What is the business problem that AI is going to solve? Is it to improve efficiency, improve customer satisfaction, increase top line or reduce bottom line (most relevant eye opener for business) etc., what is the impact that it is going to bring to specific tasks, processes and roles? An assessment needs to be done and quantify the impact clearly. Is it going to automate the tasks done by a role if yes by what percentage? This is the basic activity leadership team has to do, as this will impact the work force. Communication, Transition plan, feedback mechanism: Once impact assessment is done, there must be a clear, honest, transparent communication from the leadership team to the employees. Leadership should very clearly state the reason for the implementation and how it will directly or indirectly affect the workforce. Leadership should be absolutely clear that AI is to empower employees and not replace them. It could be an opportunity for the employees to re-skill, upskill wherever there needs to be a transition in roles. Also allow employees to give honest feedback about how they feel, what their fears are, any inhibitions etc. This way leadership can be empathetic and can understand and act on important pointers. After all technology serves and empowers its most valuable resource: its people Analyzing an example where AI can be a part of the team and how the roles can change. Let’s say a company is planning to automate the routing of a IT support ticket. The current process is; IT support tickets are raised from across the region on a centralized IT support tool. There is a resource in IT support whose major role is to monitor the queue and allocate it to a support resource who has the least open tickets in his/her queue. IT support team receives on an average 35 tickets a day, approx. 4 to 5 ticket allocations to be done every hour manually. It takes around 5-7 minutes to allocate a ticket to appropriate resource based on the least ticket logic. Though the ticket queue monitoring resource will have approximately 20-30 min an hour to work on support tickets, in reality the ticket frequency is dynamic, so the resource needs to break frequently between resolving the tickets and to monitoring and allocating the tickets to appropriate resource. Major problems here are ticket queue monitoring and allocation of tickets is not considered to be an intellectual task to be done by a skilled resource and due to this the ticket resolution by the same resource is becoming in efficient. Let’s consider that due process as mentioned above is followed before implementing the AI system and inducting it into the IT support team, The AI will now monitor and automatically assign the IT support ticket to appropriate resource based on the least ticket queue logic within few seconds. The improvement from 5-7 minutes to assign a ticket will now be changed to few seconds. Moreover, the resource who handled this task is now free to resolve more tickets than before and there is an opportunity for that resource to upskill to not only giving inputs, testing the AI system, but also can be a part of improvement teams for future AI projects as this experience becomes invaluable for the company. So this AI system implementation has relieved the resource form a repetitive, mundane task but also has allowed to learn and up skill and open a door of opportunities for being more creative in problem solving across the company. Hence I see AI becoming part of the team will improve team by empowering them, provide opportunity to upskill and transition to new roles. If companies say that they have to layoff due to automation or optimization then there is one classic example, Microsoft laid off around 2000 employees in 2021 citing various reasons, two years later there were close to 2000 new employees recruited by Microsoft with new age tech skills with more pay out! Ultimately there is no reduction in headcount but one set of people replaced by another. I believe this trend will prevail for few years, until companies realize that the value of a resource is much greater than life cycle cost of AI.

While the business today, wants to be agile and implement several AI systems into their processes, it is also important that due diligence must be put into how and when can an AI system fail, how to mitigate the risks of an AI system failure and how quickly or agile the business can come back up after the failure with lessons learned. This is a key topic as the AI's failure unlike other systems is silent and confident. It doesn't raise any alarm, it simply provides a wrong assessment. In my view the failure of an AI starts with the designing of an AI system in the first place. Though no system is perfect by itself, a well designed system will not only be high in cognitive prowess but also highly accurate and reliable. The AI systems starts to fail when the accuracy and its reliability goes down. That is why a big chunk of time is spent on testing of AI models before implementation. The system goes through rigorous testing with high quality data and robust algorithms that minimizes errors and biases. Though all business does it (at least on paper sometimes) still there are chances of failure. The common cause of an AI failure is the drift. It is nothing but the performance degrade of AI system over time. This happens when the data with which the AI system was trained is different to the real world data when it is operational. Drift is of two types , Data drift and Concept drift Data Drift happens when the statistical properties of the input data changes to that of the machine trained data sets. Concept drift happens when the relationship between the input data and the output variable changes. The key is to have sufficient monitoring systems in place, identify the drifts, analyze , retrain and re-induct. This should be like a never ending cycle as the business environment is never static. I can think of an example where the AI system can undergo drift, and how to device a strategy to monitor and catch the drifts early and mitigate the risks. Example: Many companies used AI to integrate with ERP systems and make use of its predictive analysis prowess. Lets say a company used AI to integrate with its ERP for demand forecasting so as to optimize the inventory for the predicted sales. This system has to analyze the historical sales data, seasonality's, marketing promotions, supplier lead times for different supplies etc and predict future demand for all products , automatically create purchase orders for raw materials that is required to manufacture those products. All is good until a new regulation from government saying that import of specialty raw materials will attract more tax . Due to this the business decides to source some of the key raw materials locally for which the lead time is low, instead f the usual sister company in Germany. Now the AI system still looks into the previous Bill of Materials required to manufacture the same product, calculates the demand and places order to the previous supplier overseas with high lead time. This not only results in wrong supplier getting the order to manufacture the raw material, but also the lead time is very high where as in reality the lead time should be less also with significant lesser cost as the raw material will be locally sourced. This is a concept drift and external factor omission as the new supplier and local source decision was not included or not accounted by the AI system. This is one small example showing how an AI system can be drifted. In such cases it is always important to have a human element overseeing the approval of the purchase order and cancelling or redirecting the the same to appropriate supplier. Number of such manual overrides by itself will be a good indicator of how well the AI is functioning or drifting. Based on this , business can take appropriate steps to put definitive measures in place to monitor and catch the drift early and take necessary steps to prevent it for further scenarios. Business also uses Mean Absolute Percentage error which will measure AI's Forecast vs Actual. Its stability over a period of time is a direct measure of AI's stability. To sum up, though AI systems are becoming more powerful and autonomous, I still see Human to be the master of the AI system to Design, Train, Correct and Improve it at all times, without which it will be chaos.

Thanks for picking the brain on this very important, relevant and much need of the hour topic. Auditing an AI system is very different from the traditional systems audit on various fronts. Traditional systems audit has a standardized framework widely accepted by various governing bodies across the world. By and large the process knowledge, the skill required , the checks and validations are mostly well defined and standardized across the globe. But for the AI systems audit, the above is not well defined and does not have an unified globally accepted or mandated requirements governed by a single entity, as it is very dynamic and ever evolving. I did some research and found that the below are some of the institutes trying to develop frameworks and standards for the AI landscape. 1. National Institute Of Standards and Technology (NIST)- America 2. Institute of Internal Auditors ( IIA)- America 3. International Organisation for standardization ( ISO)IEC 42001) - Independent 4. BSI (British Standards Institute) - UK 5. European Commission - Europe Union From the above it is evident that not a single institute governs the entirety of AI Auditing framework. Each country (US, UK and Europe Union) has separate institutes to standardise the auditing of AI systems. In my view, though most of the steps in traditional systems audit is still applicable to AI system audit, the main challenges will lie in the following 1. Quality of Data : How does one determine if there is bias in the data or not? How does one test the data on bias with which the Machine Learned? As we know the LLM model can use internet for specific searches and collate info based on models , what if the information that is available in the net by itself is bias?? Who knows the truth? This validation can be done only by qualified person who knows the unbiased factual truth. So error is inherent in AI..But what percentage of it is acceptable? Will it be measured by the impact that it creates in the business or the margin of error standardization irrespective of the impact? This is very challenging to decide. So I believe the companies and the qualified auditors have to reach a common point here which is easy said than done. 2. Reviewing a model: This is where the qualifications of the auditors come into picture. To review the AI model one must be an expert in statistics to point out of the results by AI is a flawed one. There are very complex algorithms involved and the time and effort required to decode and test the results will also be a key factor. What kind of sample would you choose ? Is sample relevant for AI auditing ?? How many kinds of adverse questions that can be framed for an AI to answer ?? What key elements should one focus on the adverse or trick questions to assess??.These are few questions I believe are very challenging to answer. 3. Data Governance and Security: AI can integrate with multiple systems through various interfaces, I even learn that AI can integrate with another AI agent form a different platform, while we can appreciate the capabilities one should also think how safe and secure the data is?. Can it be prone to hack, manipulation etc..I have no answer but I am sure the governing bodies will very soon. Even then it will still be an evolving one. 4. Cost of Audit: Any traditional system audit will not be more than couple of days or utmost a week. But auditing an AI system, since there is no standards the time spent on auditing will remain a debatable topic.More the time, more the money spent by the business on Auditing. I will not be surprised if Companies apportion significant amount of money in the budget just for auditing. 5. Ever Evolving Standards: As AI byi itself is evolving so will be the standards that govern them. This means the AI auditing institutes must also have resources who literally continuously research on AI systems and it's capabilities to set Standards. This is still theory not sure how it would be materialized. To conclude, I am also intrigued on how the global players are going to come out with an uniform standards that govern AI systems to address the concerns that I mentioned above effectively. Only time will tell.

AI solution looks to be the buzz word in today’s business world. Though AI has immense power to transform the business, one should always understand that any project or program designed to bring about improvement whether incremental or radical will have its own entropy. (Rate of disintegration of a system) That is where a culture of continuous improvement comes into picture. The core ideology of continuous improvement is that there has nothing best done yet. Meaning, whatever best that was done yesterday is not considered to be the best today. So there is always something to do to be the best. There is nothing perfect in this world, but by pursuing excellence we shift the goal post of perfection and redefine it every time. This is done by continuously reviewing the improvements, identifying areas within areas for improvement and act upon it. This will be a never ending cycle. Just has to be a method to it and the culture to pursue it. When we apply the same ideology on AI projects, Review, identify areas of improvement, act, implement, feedback, review etc., We create a continuous evolving AI which will deliver results consistently and be future ready delivering long term sustainability. When we talk about Chat BOT or conversational AI, we need to ensure the KB it refers to is updated and not out dated. Need to review the conversation and make it more engaging. Can rethink the application of conversation agents not to merely answer queries but also gain valuable insights from the users and create a demand document for future projects. The main indicators that the AI solution is becoming fragile or outdated as as follows: 1. Declining accuracy 2. Increased latency 3. Lack of reproducibility 4. Main is decline of user satisfaction There are multiple review mechanisms to monitor few of the above indicators. This needs to be done on a regular basis and proper action taken to keep the AI solution on track.

Any Business improvement programs or projects will start with company’s objective which stems from the Vision - Mission statement and percolates across various functions at various levels within the organization. Many companies use Policy deployment as a mean to transfer the objectives to action items across levels in various functions, some companies use Future Reality Tree (FRT) which would serve the same purpose. For example, if we chose FRT to get to the actionable items across various functions, we can analyze each of the actionable items across functions, its cost benefit and then prioritize. It is rationale to select one or more actionable items from the FRT or the Policy deployment and use the best fit AI approach as injectors to execute the project. By this we align the AI projects with the company’s overall objective and also we can directly align the value add with respect to company’s vision. Hence sticking to FRT or Policy Deployment as a basis for any AI project will be the way to go for businesses pursuing transformation

In my view The Swiss Cheese Model Frame work will be a perfect ally for companies pursuing Business Excellence. In today’s dynamically changing world, business focuses beyond usual cost reduction Programs or marketing strategies, instead they have started to transform digitally, creating rich user experience on using the goods and services. Companies pursuing business excellence have started to focus more on digital transformation not only on processes, products or services but the very design of itself They make use of the technological advancement and leverage them for more versatile, sustainable, scalable business models. This can only be achieved with a continuous improvement culture and the leaderships focus and drive on it, which is the fundamental fabric of business excellence. This is where the Swiss Cheese model framework plays a pivotal role. Traditionally this model focuses only on the product or service failure and diagnose by identifying the active and latent failures, different layers of defence and the holes (flaws) in the process. Nowadays, this framework is not only used as a diagnostic tool helping companies overcome product or service failures and prevent accidents, but also revamp or revisit the entire design of the product/service and associated human interference elements. Example: I can think of one clear example where the Swiss Cheese Model can be used not only to identify root cause or underlying causes but also help the company redesign it One of the problem of a product failure is the car engine seize. When it is analysed using the Swiss cheese model , there are multiple chain of failures that cause this. Engine over heat -> no coolant circulation -> Coolant motor non function-> no charge from battery to run the motor-> Alternator non- function -> Alternator belt snap. When we see the above, it is very evident that a 1000 RS belt snap causes an engine to seize resulting in product failure. There are multiple latent failures resulting in flaws and ultimately the product failure. There are guidance on maintenance and advise on the alternator belt, but not all the time the failure of the same is due to wear and tear. Even then, the price the customer has to pay is huge compared to the actual price of the alternator belt. The customer will start to question the reliability of the product which is a key component of the quality of the product. The reliability is the bond between the customer and the brand. Brand is a promise by the company to the customer which earns the trust of the customer resulting in brand loyalty. Business simply cannot afford this. So coming back to the example, the Swiss-Cheese model in this case can be used to redesign the entire system of alternator and the dependencies of it, so as to ensure there is no product failure or there is sufficient time for the customer to mitigate the risk when such things happen. So I leave it to you folks, the Swiss-Cheese model in my view can be a powerful tool to redesign the very core of the process / Products/Service to ensure reliability of the product so as to meet the promise to the customers that the brand represents. Hence Business excellence adopting Swiss Cheese model framework in any of its improvement programs will produce transforming results for the business

When a process is not well defined, inconsistent and not standardized, then process improvement must be done. Use of AI will not be effective in such cases. AI can be used on standardized, consistent processes to optimize, automate and transform When we understand that a standardized process is saturated in terms of capability, capacity, efficiency and effectiveness and can no longer transform from its existing state, then we can reimagine new ways of working to transform the process using AI.. In my organisation, when an issue arises in ERP, the end user captures the error details and raise a ticket manually. The ticket automatically goes to support team and will be assigned to someone. The support executive will analyse the prob and fix it, record the resolution in known error data base(KEDB)then asks confirmation from the end user and closes the ticket manually. This will take approximately on an average 4 hours per ticket,. If that issue is fixed but root cause is still not found, then it will be parked as a CI initiative and will be worked upon by a different team, who will do RCA and fix it permanently.This will take approx a month. This process of raising a issue , resolution ,RCA and closure of ticket is a well defined process. To bring about improvement in this process is possible by automating the resolution of known issues by designing few automated orchestrations that help support executive ..but it will not result in radical transformation of the process. Ideal process is that which has no errors which is not probable in real world. So to transform the process of issue resolution the following can be done Integrate an AI agent with the ERP which has cognitive ability to analyse and fix the issue permanently, without having to go through the current laborious process This AI agent will be trained with the KEDB ( Known error data base for the past 5 years) so that it can analyse and identify the root cause and have the power to correct. So the imagined process would be like the following When an user gets an error msg, automatically the AI agent window pops up and asks if this issue needs fixing. If the response is Yes, AI does the analysis and communicates the root cause and asks for a confirmation to proceed to fix.If yes the issue will be fixed based on the KEDB and informs the user to proceed. This way, the user need not spend time to capture the error msg , raise a ticket and wait untill it gets resolved..Every known errors can be fixed with couple of minutes..This makes the process more productive, clears up time for the user to focus on his job and the support function need not have too many people to do breakdown maintenance, instead they can spend time to design and develop more intelligence to train the AI agent to make it more robust..