Debanjana Basu

Let us consider a large Application Maintenance Program for a Data Analytics Program running for a huge bank. We will explore how integrating an AI solution can help flag potential risks and what mechanisms can be used to take corrective actions based on these alerts. 1. Predict job failures - Use AI to analyze historical run logs, error codes, dependency chains, and infrastructure metrics to predict which ETL processes or batch jobs or reports are at risk of failing. 2. Detect the pattern of unusual spikes/drops in data volumes that will impact the batch completion and hence delay the downstream processes like reports to be generated for business users 3. Repeated failures in a batch process can help to identify a potential bug and hence prevent recurring job failures 4. Analysis of user tickets by priority, enhancement backlog and runtimes can be used to generate predictive analytics to identify hot spots and take necessary measures to prevent them. Some of the mechanisms that can be implemented so that corrective actions are taken based on these alerts are – 1. Create Alert Tiers based on the alert generated so that high probability alerts are not ignored. 2. Provide context to the alert as how many business users or how many critical business users will be impacted if a certain alert occurs. 3. Add a human loop to verify the alerts time-to-time and provide feedback on false alarms.

AI agents can be a great addition to schedule and allocate resources in BPO or service industries. While these agents can scan through rules/past data and help balance demand, skills , preferences and compliance in no time, but we should also bake in few checks in the process to make it more effective like – write in other words 1. Skills and experience – Maintain updated records of employee skills and relevant experience so that the AI agents can choose the most eligible person 2. Fair distribution of shifts – Enable rules that ensures late night shifts or shifts during weekends are equally distributed amongst the team members 3. Compliance to labour law – Ensure the rules set by the labour laws in terms of hours of working , overtime, local holidays etc are taken care of 4. Personal preferences – Maintain a log of approved employee preferences arising from – health issue/ personal time-offs/ preferred shifts due to any personal situation etc are maintained and the AI agent scans through the log before planning the roster. 5. Business Continuity – The AI agent should also be able to identify backups in case of unplanned leaves 6. Training opportunities – the AI agent should be able to identify the gap between required skills and the available skills and recommend training opportunities to the LnD department. In order to ensure the above checks and balances are in place, supervisors should review the recommendations of AI-agents and be able to override the AI agent recommendation if there is a need. Also, there should be a system to capture feedback from supervisors and employees and can be used to train the AI agent. In this fashion, AI becomes a tool to balance business efficiency with employee satisfaction — not just a scheduling engine but a partner in creating a fair and sustainable work environment.

Let us consider the example of a healthcare customer care support system built to answer queries from patients regarding healthcare plans, check claims status , provide guidance on medical services or book appointments. An AI agent powered by prompts and flows can deliver the work but need to follow certain guidelines to deliver personalization and staying within the boundaries of privacy & trust. To personalize the experience of the customers, the AI agent can – Provide response referring to the healthcare plan opted by the customer and thus providing personalized answers instead of generic ones. Provide relevant information along with specifics. Eg; if a patient enquires about the amount of medicine to be picked up , AI can scan the details of medicines delivered to the patient in the recent past and provide relevant details Provide discreet details without making any medical assumptions; eg; if there was a annual health-check-up scheduled last week but not completed, then provide a friendly reminder to the patient. To be trustworthy, the AI agent can follow – Access only those information which the patient has consented to access, beyond which any PHI/PII data will not be accessed. Provide reference of the documents, plans etc which the AI agent is accessing to retrieve the results. This maintains transparency and trustworthiness. Thus a customer after interacting with the AI agent will have a better experience as they will receive more contextual information along with sources of information from where the Agent has pulled the data from.

Let us consider a software engineering program intended to migrate applications from legacy technologies to cloud technology. Usually, legacy programs running for decades have fragmented knowledge base resulting from poor knowledge management, inconsistent updates to the knowledge base by different stakeholders , multiple documents maintaining the same details etc. This results in efforts to search the details from the available documents, duplicate analysis effort that causes re-work. A prompt + flow-based AI knowledge solution can transform this challenge into a competitive advantage by the following means- · Maintaining a central repository of documents, following a standard template and automating the process of adding the technical documents, important meeting notes in a structured format. · Enable contextual discovery so that data engineers can query in natural language and get response in precise manner with links to the reference documents · Integrate the AI workflow with project management and version control tools to regularly update the knowledge base documents like – runbooks, SOPs, Technical design documents etc. · Before production deployment add a step in deployment runbook to trigger workflows like opening a Jira ticket to update documents and notify leads So, the prompt+flow based AI knowledge solution help the program gain by providing faster, reliable and consistent knowledge base, thus reducing the risk to the current program and a dependable knowledge base for future programs.

In today’s fast paced world where data is in abundance, AI can help scan through the data and provide insights to leaders for effective decision making. However, this approach can be flawed and the recommendations provided by AI agents can be questionable if the underlying data is incomplete or biased or if the recommendations are conflicting. In such cases the AI recommendations must be compared against the prevalent domain context and based on experience decisions need to be taken. Below are a few examples from the software services industry where AI can support decision-making:– 1. Employee Upskilling - AI can analyze employee skill data and project demands to identify gaps and provide recommendations to design learning and development programs. 2. Contracting – AI agents can help to draft contracts and later reviewed by experts, thus reducing the turnaround time 3. Program Implementation – For greenfield projects, AI can help to design the data model, create source to target mappings (STTM), generate code according to the STTM document , generate test data etc. For migration projects as well, AI can help to convert legacy code to cloud native codes, help to assess the legacy landscape, scan through code lineage and help determine a migration plan etc. 4. Data Management – AI can assess source data quality, help in master data management and data governance. 5. Performance Management – AI can help to provide insights about an employee performance by analyzing inputs from status updates in Jira, certificates completed , learning programs attended by the employee , tracking the allocation % in a year and also seeking insights from client feedback. Though AI can provide good insights, but final decisions should be reviewed and approved by Delivery/Account leadership to ensure alignment with organization goals.

It is crucial to provide ongoing feedback to AI agents so that they can learn from the to keep providing updated information. Let us assume that we have an AI agent that converts legacy code to a cloud-native language during system migration. We would need a feedback loop to be as structured and domain-aware as the migration process itself. Below are some techniques to collect, interpret and act on real-world feedback (from users, supervisors, or performance data) to continuously improve the agent : - 1. Feedback Collection – a) Developers reviewing AI converted code can flag code blocks with recurring issues like syntax errors, performance issues or deviation from architectural guidelines. b) AI generated report that shows the confidence score for each converted code. c) Track time required for manual remediation of AI-converted code and post deployment deployment metrics like execution time, resource consumption of running migrated code in cloud environment. d) Testers and Migration leads can keep track of the recurring issues and statistics around it. 2. Feedback Interpretation – a. classify feedback into types — syntax/compilation, semantic mismatch, security compliance gap etc. b. Consolidate issues to identify patterns in migration c. Compare AI generated report ion confidence score vs the reviews conducted by developers, testers and migration leads 3. Act on the Feedback – a. Fine tune model based on frequently occurring error patterns b. Update prompt templates and transformation rules with explicit project-specific coding standards (naming, architecture patterns, security requirements While it is important to optimize the performance and outcome of the AI agent, we can prevent overloading by manual resolution of minor formatting issues , or instead of reviewing every conversion, we can prioritize low-confidence or high-complexity conversions. Thus the agent will not only convert code but also learn from every migration cycle with feedback loop designed to catch errors, preserve best practices to evolving cloud practices.

During multi-step or emotionally charged interactions like guiding a customer to raise a complaint in a Telecom company, AI agents can sometimes go off-topic, providing unnecessary or wrong details or misinterpreting the user’s response. It is important to keep improving or applying adequate measures to keep improving User experience. Here are some of the techniques that can be applied to keep the interaction meaningful, focused and efficient – 1. Use brief and clear messages describing the purpose of the agent. This helps to establish the purpose and clarifies the boundaries of the agent. 2. Breaking down the interaction into logical steps to be able to gather information in a concise manner – eg ; understanding the nature of problem (network issue, billing problem, etc.) , gathering relevant details like ( date of issue , phone number, effected service etc.) . It may take more than one question to determine the problem. 3. Summarize the problem statement and ask for user confirmation before moving ahead with the next step like raising a complaint. 4. In case of uncertainty or unable to determine the nature of complaint due to vague or phased answers from the user, offer fallback mechanisms like encouraging the user to rephrase their statement or offering the user to be able to speak with a customer care executive 5. Also, it is important to use polite expressions and add empathy as appropriate.

While a well designed prompt helps to improve the accuracy, tone or trustworthiness of the outcome of the workflow, a poorly designed prompt may result in inaccurate outputs or biased replies even if the underlying model is very powerful. Use Case - Build an AI assistant that can review the logs and summarize potential risks in a data warehouse migration program between Oracle and Databricks. A poorly framed prompt like " Consolidate the migration process" will generate outcomes like - listing out the various processes like schema migration, code conversion, data migration , validation etc. But the outcome will not help us to summarize the potential risks. It will ignore the errors, will not capture any actionable insights and will fail to support decision making process. But upon improving the prompt and calling out explicitly - "List the potential risks with regards to schema mismatches,data integrity issues, performance issues " the AI assistant will scan through the logs and provide relevant details for schema mismatch like - loss of precision between number datatypes; missing timestamp value in Databricks 'date' column vs 'date' in Oracle. These details help to improve the accuracy, trustworthiness of the AI assistant, thus helping to deliver better results.

The Swiss Cheese Model, introduced by James Reason in 1991 in Human Error, explains that failures typically occur not because of a single mistake, but due to a series of gaps in multiple layers of defense. When these gaps align, they allow errors to pass through and lead to a major failure. For example while executing a greenfield project including building a data lake in cloud would involve the following slices of cheese - Project Governance, Requirement Gathering, Change Control Board, Scrum , Design principles, Maintaining code development standards, Efficient code reviews , thorough SIT and UAT testing, track risk and mitigation planning and providing post deployment monitoring and warranty support. The holes in the cheese slices could be - poorly defined scope, inadequate documentation of requirement or misunderstood requirements, inefficient management of the scope changes, Infrequent sprint reviews or incomplete retrospectives, inefficient code reviews , limited testing of the features , inadequate documentation, no rollback plan or insufficient testing of the rollback plan in case of a failure. So upon aligning the multiple holes - a project may fail due to unclear scope > incomplete requirements > limited testing leading to production defect and client dissatisfaction. Applying the following checks or procedures can help improve the process - 1. Regular steerco reviews to review the scope, progress and risks 2. Maintain a comprehensive project checklist 3. Extensive testing of the use cases and seek business approval 4. Use retrospectives for continuous improvement

A process can be improved when incremental changes help to optimize the existing processes. The changes involved have low business disruption and are mainly low risk changes.eg; automated data validation checks in existing clinical trial data-entry system to reduce errors. Whereas re-imagine involves implementations of data transformation like creating data meshes or data products which often require changes in business model to redefine how work is done. eg; creating virtual replicas(digital twins) of processes to simulate and optimize real world operations in a factory.