Jess Balmaceda

AI Management System is a structured framework used to govern the development, deployment, operation, monitoring, and continual improvement of artificial intelligence systems in an ethical, safe, and efficient manner. It ensure alignment on organization’s goal, regulatory requirements, and social values. Similar with other management systems, one of its key elements is Policies & Standards. This element pertains to documentation of existing AI workflow, prompt improvement, and version control for any changes made. It is strongly recommended that any organization engaged in AI solutions be certified in AI Management System.

As a former Quality Management System auditor and QMR (Quality Management Representative), I would still rely on ISO standard to facilitate audit on AI-infused processes. ISO provides internationally and consensus-driven management system guideline from strategic down to operational level that span to multiple sectors, not just industry. Introducing ISO 42001:2023, this new standard focuses on building trustworthy, transparent, and risk-aware AI system supported by good governance, responsible leadership, and continual improvement. This is also known as Artificial Intelligence Management System (AIMS). Like any other ISO standard, AIMS is structured to facilitate thought-provoking questions such as but not limited to: “How to monitor performance and ensure AI behavior aligns with intended purpose?” “What are the requirements in conducting internal audit of AI Management System?” “When and how to address non-conformities and take corrective actions?” “What are the AI Lifecycle stages?” These guide questions were embedded in ISO 42001:2023’s clauses to steer AI solution experts and management alike in establishing a fit-for-purpose management system manual. In addition to ISO, organizations can also refer to NIST AI Risk Framework. NIST is a U.S. government agency that develops standards, guidelines, best practices, and measurement techniques with primary focus on Cybersecurity, IT, and metrology. AI professionals and management alike can refer and use this framework which was designed to help organizations across industries and domains to manage risks of AI systems and improve their trustworthiness.

AI system can quietly degrade overtime if left unmonitored. It can compromise integrity, accuracy, and confidence. AI solution models were developed with an objective to aide human in the process, increase accuracy, and improve throughput. If these objectives are eventually not observed, AI models require scrutiny. The increasing customer complaints, increased ticket escalation, internal users stop trusting AI solution, and observable wrong or irrelevant decision are some signs of AI models’ degradation. To ensure long-tern sustainability, following measures should be established: 1. From its inception, establish parameters or an AI-model KPI that would indicate performance, drifts, and signs of degradation. These parameters should form part of management KPI monitored and evaluated regularly (i.e. weekly, monthly) to recognize out of control markers. In the absence of these markers, corresponding corrective action will not be taken to address early signs of deterioration. 2. Random testing of AI outcomes to determine its validity and relevance to current reality. This is more important on volatile industries, environments, and behaviors. 3. Timely update of knowledge base. Align models with evolving business goals. 4. Keep human-in-the-loop (HILT) especially on high-risk decision-making process. Timely feedback improves the model, keeps it up to dated, and relevant.

The objective of any business is to create and sustain long-term value. While AI is beautiful and popular, transformation professionals should not be lured by fad. Choosing the right project to prioritize is vital in achieving such objective. AI solution applied in all other areas of business wherever possible yet not aligned with its strategic objective is detrimental to the organization’s growth and bottom line. It can flood the AI engineers’ workflow to extent of choking its way to success. Stressful and costly system would likely to emerge. More or less, due to long queue of proposed AI solution projects, the more important one can either be missed out or deprioritized. Therefore, it is critical to the management specially to transformation professionals to establish a due diligence framework in project prioritization such as business value analysis – regardless of whether its AI solution-based or not. Business priorities can be deciphered, understood, and better aligned through collaboration with the CFO or Finance Controller. This should be the first step when it comes to project selection and prioritization. Transformation professionals should serve as the bridge between top management, operations (marketing, sales, production, supply chain, etc.), and finance. As Lean Six Sigma or Transformation Professional focuses on financial benefits of every project, aligning first with Finance is paramount. Taking this step early on would yield on project results aligned with business priorities. This method would prevent stressful situation where competing priorities prevented, and lead to real financial benefit appreciated by top management and the business as a whole. While profitability, operating expense, and cash flow are essential financial metrics, throughput projects necessary to increase profitability should take on the priority seat. This is where AI can add strategic value to the business not by fad or chance, but by focused intent.

THE DEFENSE LAYER Screening operation is an integral part of KYC’s end-to-end value chain. It is consisting of four lines of defense. The First Line-of-Defense (1LoD) comprised of screening investigator professionals who facilitates the name screening, transaction screening, and adverse media screening. They follow stringent rule-based approach to distinguish clean alerts from false-positives, unable-to-eliminate (UTE), and true hits. The Second Line-of-Defense (2LoD) is the Money Laundering Reporting Officer (MLRO). These are money laundering experts following risk-based approach of decision-making and address escalations from 1LoD like UTE and true hits. The Third Line-of-defense (3LoD) is the Quality Assurance (QA). These are group of subject matter experts (SME) who strategically facilitates sampling check of all screening alerts. Despite only doing sampling, these group of SMEs are still able to detect gaps from both 1LoD and 2LoD. The Fourth and last Line-of-Defense is CAS Audit. CAS Auditors facilitates periodic process and systems audit to ensure that all procedures, policies, regulatory requirements, and commitment to regulators were uphold and complied. LAYERS OF DEFENSE POTENTIAL WEAKNESS The 1LoD inherent risk can be attributed to its investigator’s level of competency and razor-sharp focus. Undetected true hits and UTE is risky and compromises banks’ KYC objectives and potentially cause integrity issue in the eyes of the clients and regulators alike. Both 2LoD and QA process poses a risk of wrong or misled decision-making due to marginality of the case. Breach within KYC system is very costly through regulatory penalties and compromises banks’ integrity as a whole. IMPROVEMENT BASED ON BUSINESS EXCELLENCE Rigorous training is provided to the first three layers of defense, and they must pass all examinations before going-live in operations, MLRO, and QA alike. They maintain a common repository of all procedures and policies in a Confluence page. Regular alignment call (i.e. Bi-weekly, monthly, etc.) were facilitated by MLRO and QA in different occasions to discuss quality and process related topics. In this forum, root-cause analysis (RCA) and corrective actions were tackled to prevent potential misses, go into best practice discussion, and promote continuous improvement. Future Reality Tree can also be a good introductory tool to use to improve the Screening end-to-end process. A good AI solution is a future-tilt breakthrough to mistake proof the entire value chain.

Top management recognizes that company’s failures are systemic. They then apply the levers of change on many fronts simultaneously because the dysfunctions cannot be viewed and corrected in isolation. It demands deep top management involvement and exposure. Change whether perceived as good or bad has always been faced with resistance. People often resist change for various reasons like moving them out of their comfort zone, out of fear, ambiguity, etc. Every change then requires management in order to mitigate potential resistance within the organization, making sure that everyone is aligned, agreeable, and supportive on the change implementation. The most significant cost of change implementation failure is indirect and long term, where confidence in leadership decreases, resistance to change increases, and future change/s more likely to fail. MBB together with the management team (MT) must manage resistance to achieve aggressive timeframes for implementation. If they do not manage resistance to this change, it slows down the implementation of future changes or worst they're bound to fail. Ultimately, resistance from employees will impact the customer. MBB should systematically analyze and identify significant personal and organizational barriers to change and generate tactics to increase readiness and decrease the time and resources required to achieve business results. MBB should help build a compelling business case for action translated to stakeholders’ - that may include sponsors, GB/YB/BB change agents, employees – “frame of reference.” This is the foundation of defining the project. There must be a clear and commonly held definition of the project by everyone impacted by the change, where the present state (is now) and desired state (will be) is aligned and understood for successful change implementation. Stakeholders' buy-in is the key. The business case for action must address three basic questions:  What are we changing?  Why are we changing it?  What are the consequences of not changing? In 2007, an Electronics Manufacturing Services company in the Philippines was engaged in manufacturing for its cash cow client’s autonomous Wi-Fi network solution. The production method was a low-volume high-mix, wherein high variety of product model were produced in small volume. Frequent transitioning from one variety of product model to another requires long accumulated changeover time which means idle time for production employees, overtime required to meet production quota, and higher opportunity loss for sales. As a Lean Six Sigma Black Belt and Operations Manager in-charge for manufacturing that time, I found this case as an opportunity to improve the process. I was equally aware that there will be strong resistance to change for two reasons – First, everyone in the company got used to such condition (long changeover time, idle, overtime, etc.). Secondly, I’m a newcomer in the organization, got no connection, and got a lot to prove. Anticipating the potential resistance to change, a compelling business case for action was presented to the company’s management team including its CEO addressing the following questions:  What are we changing?  Reducing changeover time from X to Y.  Why are we changing it?  The company incurs an opportunity loss in sales to the tune of $xxxx per annum attributed to prolonged downtime caused by changeovers, plus higher cost of manufacturing heavily due to overtime, reworks, and late delivery.  What are the consequences of not changing?  Potentially losing business with the cash cow client for not fulfilling its demand on time, higher price, and low quality. The objective of presenting a compelling business case for action was to reveal what the problem was, acquire a good sense of how important the case was to the management team, obtain their agreement, commitment, and strong sponsorship from the top. Furthermore, building credibility to the management team was also vital to getting their trust and confidence. Done that by introducing myself as a Lean Six Sigma Black Belt in conjunction of my operations role, provided them brief walkthrough of DMAIC framework to educate them, manage their expectations, and emphasizing their vital role as sponsor/s. The next step was to select team members from production and different engineering disciplines, recognizing and leveraging on their respective expertise, present the business case, obtain their agreement and commitment to support the change. More importantly, making them realize what’s in it for them once change implementation succeeds. Educating them on DMAIC framework, introducing necessary LSS tools essential to the project and helping them understand how to use them established trust and confidence in the process of change implementation. Listening to everyone’s perspective (like operators, inspectors, and technicians) with respect to said process improvement project developed overall buy-in and good sense of group ownership, which made the change implementation successful. The motivation to implement (both individual and organizational) must be stronger than the motivation to stay the same.

Why do those wins sometimes slip away in the Control phase? There are various reasons why solution to improved process slip away in the Control Phase. Here are some: 1. Thinking that identifying a solution is enough. Oftentimes, organization and project team missed to establish controls to sustain and ensure consistency of the injected solution. Testing of impact and feasibility of the solution is missed, only to find out that local staff implementing the solution on their day-to-day task find it difficult to sustain. 2. Poor communication of intended changes. Formal handover of improved process to its process owner/local management is essential, more importantly to its local staffs who will deal with day-to-day work where improvement took place. A better understanding of what was the problem and why improvement was made must be clear and aligned with the local team. Buy-in and ownership of local team is vital to sustain the solution. This means making them involve from the start and all throughout the project’s phase. 3. Inadequate training in new condition. Lack of enablement of local management and staff who’ll be implementing the solution and who’ll be using it daily will surely make successful implementation fail. Conducting training and enabling local staff before full implementation of solution will provide knowledge, familiarity, build confidence, and likely ensure success of implementation. Adequate training puts the local staff in a controlled environment where learning curve is monitored and supervised. Under scrutiny, old behaviors will be guided and replaced with new intended behavior tied up with the change throughout the training process. Local management’s involvement in training brings alignment, trust, and confidence with local staffs. 4. New process not captured in written procedure. Procedure provides guide and clarity to the doers. It should contain the process details, working sequence broken down into elements, with hints and tips how to perform an activity or task. Updated written procedure of the improved process is an essential supplement to the enablement of local staff and management alike. Procedure should be crafted in such a way not bounded to misinterpretation in order to avoid human error. 5. No monitoring to check that solution is working as intended. Local management’s involvement in the handover of the improved process is essential. Management oversight alongside with tools and best practices such as Control Plan, Process Control Chart, Gemba Kaizen, and process audit should be understood and taught to them by the project team. Having a KPI metric that shows process performance where improvement was made is of equal importance for sustainable management oversight. What tools or techniques do you rely on to keep things on track and make sure the improvement sticks for good? Handover process management is crucial to sustain the successful process improvement in long term. Process documentation such as to-be process flow chart, training & enablement plan, process aids/visual references, control plan, process control chart, updated procedure, FMEA (whenever applicable), and implementation plan should form part of the handover process to local management. This is a structured way to provide clarity on how the improved process works and how it should be monitored to sustain the gain of its financial and non-financial benefits. Lastly, involvement of process improvement team or at least its leader in KPI monitoring and random process audit for the next three to six months upon handover is another key on sustainability of the improvement made.

In 2022, Klarna launched a full-speed AI deployment automating most of its processes using AI solution and realized cost savings equivalent to 700 FTE. One of the processes they automated was their Customer Service Support. After a while, customer complaints and dissatisfaction ballooned. Customers claimed that AI responses were too generic and unhelpful when dealing with real-life problems. While AI solution like chatbots can handle simple and repetitive queries, emotions or complex issues were not addressed. Klarna realized that while AI solutions promise speed and cost savings, it can compromise service quality and customer satisfaction. Klarna decided to rehire employees to address poor service quality and customer complaints. This is a testament that AI solution isn’t about replacing humans, but rather, enhancing the human workforce with smarter tools and better support system. As an MBB, following were my recommendation: 1. Use VOC to identify critical customer requirements (CCR) where complex issues and customers needing to talk to human to solve their concerns will surface. 2. AI solution aims to enhance customer experience leveraging on personalized interaction for higher engagement. This was not apparent in case of Klarna. It is recommended to take advantage on Deep Learning capabilities of AI solution. Such model can identify complex patterns, making it suitable in image recognition, voice recognition, and natural language processing. 3. Lastly, while drawing the to-be process map, HILT (human-in-the loop) principle is recommended. In cases of complex customer concern, AI can escalate the concern to its human counterpart to further assess the given concern and provide necessary resolution.

Traditional KPI metrics such as productivity, quality, cost, delivery, efficiency, and many more should not leave management lenses, rather, targets associated with them should be adjusted accordingly. Customer and employee satisfaction surveys however can be done through AI, leveraging on its capability to detect emotion, interpret facial expression, body language, and many more which is difficult for human eye to decipher and prone to certain biases. To track AI’s real performance and value, I recommend Input Data Integrity, and Bias Detection as two additional KPI metrics that management should add under their lenses. These are crucial for AI’s model creation, accurate training and analysis, impacting AI’s recommendation for business decision-making process.

KYC domain is composed of different process such as CDD (Customer Due Diligence), Transaction Monitoring, FATCA & CRS, and various Screening processes. All of these processes were aimed in combatting money laundering, terrorist financing, and other financial economic crimes. I would like to focus on CDD process which is more complex of all other processes that requires comprehensive review of both onboarding and existing client profile, mostly private corporations and financial institutions. CDD review require investigation skills and technical mastery on Office Due Diligence, Name and Adverse Media Screening, Syndicated Lending, Corresponding Baking, etc. These reviews were rule-based in approach. At the end of this arduous process, the CDD Investigator need to make a decision whether to accept and/or continue doing business with the client – so long as the client is clean and is not potential FEC violator. Overtime, Operational Excellence folks made various study to improve CDD’s processing time and they’d been quite successful on this journey. Meanwhile, understanding some AI solutions, applying the power of Expert Systems in AI will create a revolutionized solution to enhance the speed of processing, improve quality of reviews, while utilizing human resources on other valuable tasks - particularly risk-based decision-making approach. Expert Systems in AI was designed to emulate human decision-making while leveraging on its components such as Knowledge Base, Interference Engine, and User Interface. CDD process have a wide range of repository of facts, policy, procedure, and rules as an input to Knowledge Base and through the Interface Engine logical rules will be applied suggesting whether the client under review is FEC compliant or violator. The User Interface is where the CDD Investigator gets the advice and will make a decision using risk-based approach.

How can Business Excellence leverage Voice of Employee? Business Excellence has a framework that serve as tool to provide holistic organizational diagnosis to identify its key strengths and opportunities for improvement in order to achieve the organization’s desired outcome. Business Excellence framework is commonly composed of 7 drivers namely - leadership, customer, strategy, people, process, knowledge, and results. One of the attributes of an excellent organization is Valuing People and Partners. Valuing people and partners means to create a culture of empowerment where employees are highly skilled and deliver high performing result. Organizations must build strong partnership for shared ownership and achievement of its strategic goals. Cultivating a culture of employee empowerment demands attentively hearing employees through their collected voice or Voice of the Employee (VoE). The VoE grants an invaluable understanding of the workforce’s travails, cares, and ambitions. Tailoring improvements in response to VoE discussions cultivates psychological safety and care. A supported staff translates to optimized operations and outcomes. What challenges might hinder gathering actionable employee insights? - People agenda is not part of organization’s strategic focus. - Lack of channel/s to capture Voice of Employee. - Lack of trust by employee to organization, where people are disengaged. - Voice of Employee was collected in the past but there was no analysis and action taken to address concerns. - Leadership’s lack of political will and genuine thrust to ensure that all employees respond to pulse surveys, focus group discussions, townhalls, etc. Discuss strategies to overcome these challenges. Elaborate with an example where a project was identified using Voice of Employee. The tone from the top is an important factor to keep employee engagement active. People agenda must be part of the organization’s overall strategic focus. An organization’s People Strategy may be in the tune of Home of Personal Growth, Hire and Admire, Employer of Choice, and World-Class Talent Magnet. In order to fulfill People Strategy, it should be espoused with appropriate channels to capture Voice of Employee such as annual/bi-annual pulse surveys where leaders actively encourage employee to response. Survey results should be bounded by anonymity to build employee trust and result has to published and actioned. Furthermore, to put structure and accountability, Pulse Survey Response rate should form part of leadership’s KPI. In 2018, a department who does customer due diligence experienced an attrition rate of 23% YoY, where employee expertise on process and global regulatory requirements were crucial. Onboarding of new employees took 9 weeks of classroom training and 2 more months practical training. This lengthy capability building plus high attrition rate has been department’s pain point. There were focus group discussions (FGD) and Exit Interview at that time, unfortunately, no one did deliberate action to study the information and data available out of these channels. A project was initiated to reduce the attrition rate. Root cause analysis revealed that employees were dissatisfied by way their performance was subjectively rated, while promotions were on the basis of favoritism. The solution made this subjective performance appraisal and promotion via favoritism reduced the attrition rate from 23% YoY to 14% YoY. A Scorecard system was developed by the project team with selected employees consulted to capture their perspective and foster co-ownership.

1.⁠ ⁠Can Key Risk Indicators (KRIs) be used to manage a process? KRI or Key Risk Indicators provides early signals of potential risks that could impact the achievement the operational objectives. KPI or Key Performance Indicator on the other hand, assess performance outcomes against operational objectives tied up on strategic goals. While KPIs can help understand how well the company is doing in relation to its strategic plans, KRIs can help pinpoint and prepare for potential risks to minimize the chances of unfavorable outcomes. KPI can be considered as lagging indicator, while KRI as leading indicator. KPI presents the outcome while KRI help identify potential risks and impact to not achieve KPI. Therefore, KRI is very useful in managing a process to augment KPI. 2.⁠ ⁠Why do companies tend to prioritize Key Performance Indicators (KPIs) over KRIs for process monitoring? Most of the company fell in love on the idea of performance indicators since KPI were tied up on company’s strategic objectives. All along, everyone was focused on achieving KPI targets without systematically understanding nor putting emphasis on potential risk factors that may hamper the achievement of these KPI targets. Furthermore, back in the late 90’s - early 20’s, KRI was not known (popular) to most of the companies which is why KPI was widely used in measuring performance in most of the industries. 3.⁠ ⁠Discuss the benefits and limitations of using KRIs with examples. Having KRI along with KPI establishes a good foundation to identify and mitigate potential risk factors that directly impact the achievement of organization’s performance target. Let’s take Attrition Rate as KPI with a target of 13%. KRI: High People Engagement – At least 1x/month one-on-one with line manager, and Quarterly Rewards & Recognition. YoY Career development & growth – Staff movement either lateral or vertical the organization Compa Ratio – Organization’s compensation compared with market/same industry to remain competitive.