Q 761. In an organizational process, imagine an AI agent makes a decision or takes an action that leads to an error — maybe it affects service quality, violates a policy, or escalates unnecessarily. Describe a realistic scenario where AI accountability becomes a concern. How would you assign responsibility — to the AI, the designer, the human reviewer, or someone else? What design safeguards would you include to ensure transparency and traceability?

 

🏆 The best answer will be selected on the basis of:

• Relevance of the trust challenge

• Practicality of the trust-building approach

• Creativity in human-AI interaction design

 

Note for website visitors -

• This platform hosts two weekly questions, one on Monday and the other on Thursday. 
• All previous questions can be found here: https://www.benchmarksixsigma.com/forum/lean-six-sigma-business-excellence-questions/. 
• To participate in the current question, please visit the forum homepage at https://www.benchmarksixsigma.com/forum/.
• The question will be open until Monday or Thursday at 5 PM Indian Standard Time, depending on the launch day. 
• Responses will not be visible until they are reviewed, and only non-plagiarised answers with less than 5-10% plagiarism will be considered for winner selection. 
• If you are unsure about plagiarism, please check your answer using a plagiarism checker tool such as https://smallseotools.com/plagiarism-checker/ before submitting. 
• All correct answers shall be published, and the top-rated answer will be displayed first. The author will receive an honourable mention in our Business Excellence dictionary at https://www.benchmarksixsigma.com/forum/business-excellence-dictionary-glossary/ along with the related term.

• Some people seem to be using AI platforms to find forum answers. This is a risky approach as AI responses are error-prone because our questions are application-oriented (they are never straightforward). Have a look at this funny example - https://www.benchmarksixsigma.com/forum/topic/39458-using-ai-to-respond-to-forum-questions/

• We also use an AI content detector at https://quillbot.com/ai-content-detector. Only answers with less than 45-50% AI-generated content will be considered for winner selection.

Accountability Framework for AI Agent Decisions in Enterprise Audits

Scenario:

An AI agent is designed, developed, and deployed to automate access reviews for enterprise users. It executes approval and rejection decisions based on predefined business rules, confidence factors, and system prompts. During a business process audit, an action taken by the AI is questioned by the auditor, requiring justification and accountability, possible some corrective measure/control implementation.

Task:

The core accountability concerns include:

1. Who is responsible for addressing auditor queries regarding AI decisions?

2. Who is accountable if an AI decision is deemed incorrect during the audit process?

Action:

AI agent deployment should follow established governance frameworks similar to traditional product deployment, ensuring clear ownership across business and technical domains:

• Agent Business Owner: Responsible for defining the AI’s operational objectives, business rules, and system prompts. Ensures alignment with enterprise policies and regulatory requirements.

• Agent Technical Owner: Accountable for the technical deployment, access governance, and security controls surrounding AI decision-making. Oversees model behaviour, validation mechanisms, and anomaly detection to mitigate risk.

To maintain auditability and explainability, the AI agent must incorporate:

1. Logging and Traceability – Each decision must be recorded with metadata (e.g., rule applied, confidence factor, rationale) to facilitate retrospective analysis.

2. Decision Transparency – AI-generated approvals/rejections should be structured in a way that enables easy reference during audits.

3. Human Intervention Protocols – In cases of ambiguous(gray-area) or high-impact decisions, escalation pathways should allow human oversight before finalising outcomes.

Result:

The AI agent must ensure robust explainability for its actions, enabling auditors to trace and validate decisions efficiently. While AI operates autonomously, human ownership remains central, with business and technical stakeholders accountable for the agent’s actions. They are responsible for responding to auditor inquiries, implementing corrective measures, and refining AI governance frameworks based on audit findings.

Scenario: Error in AI-driven R&D Formulation Development

Scenario: A Pharma R&D organization uses an AI system to generate formulations for a new Anti-cancer medication. The AI suggests a formulation with an upper limit of excipient, provided by the USFDA, which claims that it improves efficacy and stability. The formulation clears automatic system checks but is later found to cause mild adverse reactions (e.g., vomiting and acidity) in a few volunteers during trials. The investigation confirms that the AI was trained mostly on data from healthy young populations, but it was missing sensitivity in paediatric and geriatric populations with different metabolisms.

This is a critical matter due to the involvement of patient safety, regulatory risk, and reputation damage.

 

Assigning Responsibility

(1)   AI Engineer – Prime responsible - Due to not including diverse population data e.g., paediatric and geriatric patients. Also, not setting stringent standards for the safety of excipients.

(2)   R&D Scientists – Scientists have conducted trials without validating facts regarding excipients.

(3)   Complinace Teams – The Team has skipped review due to confidence on AI system.

 

Design Safeguards for Transparency & Traceability

1. Training Data
   • Data should include all patient demographics (age, gender, comorbidities) and include treatment-specific data for edge cases (e.g., patients with gastric disorders).
2. Confidence score for Formulation Decisions
   • The AI should output not just a recommended formulation, but also include confidence scores (e.g., "90% as highly recommended, but only 70% recommended for elderly patients").
3. Human interference for Critical Decisions
   • Suggested formulation is near safety thresholds (e.g., excipient limits) and must require the scientist's sign-off.
4. Real-time Feedback Integration
   • Model also updates data simultaneously, including all adverse events.

This approach helps AI-driven formulation development without compromising safety, keeping accountability on the humans who design, develop, validate, and use the system.

 

Mentioned below is the break-up of the question for our discussion.

1.       Describe a scenario where AI accountability can be a concern.

2.       What responsibilities can be assigned to AI agent and stakeholders.

3.       Design safeguard mechanisms to ensure transparency and traceability.

 

1.      Scenario where AI accountability can be a concern:

A Community Management company has developed and deployed an AI agent to manage their incoming maintenance requests from tenants across different sites by asking & recording required details, prioritizing requests, creating work orders and assigning available technician. The agent uses historical training data to identify the impact and priority.

On a typical day, a tenant encounters electrical sparks in the electrical panel due to water dripping from the AC. The tenant follows the protocol, contacts and provides all details asked for by the AI agent. This is obviously a matter of danger.

AI processes the recorded details and classifies this as “low priority” and schedules the workorder for the next day. Here the agent failed to sense the urgency of matter and act accordingly, which is a matter of grave concern. Who shall be blamed for this failure? How can be build responsibilities?

 

2.      Responsibilities that can be assigned to AI agent and stakeholders:

 

a.       AI agent: It is not a legal being and hence cannot be made accountable for any of its outputs or results.

b.       Data scientist: Ensure to provide clean, contextual, clean, unbiased and adequate data for training the model. This is crucial to determine the accuracy of the model. Accountable for model performance.

c.        AI developer: Identify the right technology, platform, build risk-aware models and integrate fail-safe, traceability and explainability features. Assess the relevant model performance assessment metric such as sensitivity / specificity etc. Ultimately responsible for logic / code related flaws and failures.

d.       Operations team: Continuously monitor the system’s performance until satisfactory levels of performance are achieved. Intervene and override model outputs when the model provides incorrect outputs. Provide feedback to finetune the system to continuously improve its performance.

e.       Compliance & Audit team: Responsible for assessing all risks with its counteractions. Define acceptable levels of performance. Periodically audit to identify bias, flaws and security breaches.

f.         Core committee (Governance Team): Provide strategic directions and take ownership of AI initiatives and outcomes. Manage escalations and legal conflicts arising due to AI related crisis.

 

3.      Design safeguard mechanisms to ensure transparency and traceability:

 

Deploying AI agents replacing humans could affect quality and compliance of service. How can we build transparent and traceable systems?  

 

a.       Transparency:

                                 i.            Build Explainable AI: The AI agent must have the capability to explain the rationale behind the decision using simple language.

                               ii.            Realtime risk scoring: Display the risk score in real-time to the tenant and technician that AI computes based on the keywords in the problem description. This will help them refine their words to increase the risk score.

                             iii.            Safety layer & Human-in-the-loop (HITL): Add a safety layer to the model defining key words such as leak, spark, fire etc. where the AI agent can bypass and notify a human when the defined keywords are detected.

 

 

b.       Traceability:

                                 i.            Detailed Decision logs: Maintain details logs of all version changes, access, feedback loops, communications, actions, decisions and decision overrides.

                               ii.            End to end tracking: Create unique records for each cycle, starting from tenant inputs, AI actions / decision, technician assignment and workorder closure with CSAT scores. This will help trace the workflow and perform forensics.

 

Conclusion:

The above case demonstrates the necessity of establishing clear responsibilities for stakeholders involved in the AI deployment. By building robust traceability and transparency in AI systems, organizations can ensure that the outcomes are both understood and accounted for. Ultimately, the Governance Team shall take responsibility for the outcomes and take appropriate measures to build a robust and successful journey.

 

 

 

Scenario: Human Error — QB Validation Report Failure

For instance, an HR helpdesk-based AI assistant automatically escalates a report of harassment to the manager of the accused party, violating privacy and causing distress.

Trust Challenge: Transparency
• The resolution was concluded through AI, but it lacks a specific aim.
• The designer failed to incorporate role-check safeguards.
• Human Resources Policy The owner did not delineate escalation rules in a machine-readable format.
• Sensitive situations are not subjected to human review.
 

Primary: Policy Owner and Designer
Secondary: Governance team for insufficient oversight

Trust and Traceability Safety Measures
1. Escalation filters specific to roles
2. Logs of decisions that are transparent
3. A requirement for human review concerning significant topics
4. Identification of conflicts that are flagged as critical
5. Dashboards designed for Explainable AI
6. A feedback loop aimed at continuous learning

Innovative Human-AI Collaboration
In sensitive processes, utilize shadow mode (AI proposes, human validates) and confidence sliders to achieve a balance between automation and oversight.
In sensitive processes, utilize shadow mode (AI proposes, human validates) and confidence sliders to achieve a balance between automation and oversight.

In a Finance Operations scenario, wherein for payment of a completed order, AI has been enabled to auto populate the bank account details of the Customer for payment to be made. In case of any error OR multiple account details updated in the database, AI may consider the incorrect details and payment may be pushed to a wrong account. The financial impact caused due to this error may vary with variable impact

 

In the above mentioned scenario, human reviewer and database creator will be assigned responsibility. One of the safeguards that can be enabled in such a situation - Strict governance around the data updates along with regular audits and continuous enhancements and updation. 

Accountability Assignment

• AI System: The AI's role as an autonomous decision-maker makes it partly accountable for the specific error. However, it lacks consciousness and intent.
• Designer/Developer: The individuals or teams responsible for training the AI and creating its decision-making model bear accountability for inadequate programming or data biases.
• Human Reviewer: If the organizational process included human oversight (e.g., verifying flagged cases) and the reviewer missed correcting the error, they share responsibility.
• Company Leadership: If organizational policies failed to set sufficient safeguards around AI usage, the leadership carries overarching accountability.

The degree of responsibility assigned to each party depends on the design of the AI system, the clarity of the roles, and the transparency of the protocols.

Design Safeguards

To avoid such scenarios:

1. Traceability: Ensure decisions made by the AI are logged in detail, recording factors considered, probability scores, and reasoning pathways. This creates a "decision trail" to audit errors.
2. Explainability: Deploy mechanisms that make AI decisions interpretable for users and reviewers. For example, a dashboard showing why the AI flagged a case.
3. Human-in-the-Loop: Require human reviewers to validate high-impact or complex decisions before execution.
4. Feedback Loops: Include mechanisms for customers or human agents to report errors directly to the AI, which can be used to refine its future decision-making.
5. Ethics & Compliance Monitoring: Establish oversight committees to ensure AI decisions align with regulatory, ethical, and organizational standards.

I thinks humans are responsible when AI goes wrong because humans are responsible for deployment of AI systems.

1)The reason could be due to inadequate testing and lack of proper oversight and insufficient training of the AI model
2)AI systems using machine learning take time to develop their understanding and deploying them without sufficient knowledge pool could prove hazardous 
3)AI tools should be used by humans keeping in mind their limitations, and not blindly depend on them for critical decisions. eg: completely depending on AI for medical diagnosis  could prove risky

AI Accountability in Organisational Processes — A Statutory Audit Perspective

 

Scenario: Misclassification by AI During Financial Audit (FY 2023–24)

 

A statutory audit team used an AI-powered anomaly detection tool during the financial audit of a large manufacturing company. The AI agent was configured to flag unusual revenue transactions for deeper scrutiny, based on historical data and predefined thresholds.

 

During the FY 2023–24 audit cycle, the AI flagged numerous deferred revenue entries — all of which were compliant with Ind AS 115 — as anomalies. These false positives diverted the audit team’s attention, consuming valuable time and effort. Meanwhile, a genuine material misstatement in inventory valuation went undetected due to lack of review in non-flagged areas.

 

This resulted in an incorrect clean audit opinion, which only came to light during a subsequent regulatory review.

 

Accountability Assessment:

 

Assigning responsibility in this context requires layered consideration:

• Auditors (Human Reviewers):
  Ultimately accountable for the audit opinion. Over-reliance on AI and lack of judgment-based review in unflagged areas indicate a lapse in professional skepticism.
• Audit Firm (Implementers):
  Accountable for selecting and deploying the AI system. There was a failure to validate whether the tool’s training data and logic were suitable for a manufacturing context.
• AI Designers/Developers:
  Partially accountable — but only to the extent of the design and training scope shared with them. The tool operated within the technical limits of what it was built for.
• AI Agent:
  Not independently accountable. AI is a tool and cannot bear responsibility in a legal or ethical sense.

Design Safeguards for Transparency & Traceability:

 

To prevent such failures and promote responsible AI adoption in audits and other sensitive domains:

1. Explainable AI (XAI):
   Ensure that the AI provides clear reasons for flagging entries — not just binary outputs — so auditors can judge the context.
2. Audit Trails:
   Log every AI decision and its rationale, enabling after-the-fact traceability for internal reviews or external regulators.
3. Contextual Calibration:
   Regularly recalibrate models using industry-specific data to avoid overfitting or misclassifications across different sectors.
4. Human Override Protocols:
   Build workflows that allow auditors to challenge or bypass AI decisions based on contextual knowledge and experience.
5. Dual-Review System:
   Flag both anomalies and “blind spots” — areas the AI deems low-risk — to ensure balanced scrutiny.

Conclusion:

 

When AI makes a flawed decision in an organizational process like a statutory audit, responsibility primarily lies with the human reviewers and the organization deploying the AI, not the tool or its creators in isolation. A robust ecosystem of design safeguards, traceability protocols, and human oversight is essential to ensure AI remains a reliable assistant — not an unchecked decision-maker.

It's safe to say that AI can't be inherently accountable, as it's a tool which is making decisions based on parameters set by humans.

 

AI Designer/Developer can be largely responsible. The AI may have been trained improperly, using biased data or necessary safeguards were lacking, the designer should hold accountability.

 

If the organization has human oversight mechanisms in place, however the reviewer misses to catch the error, the reviewer bears responsibility.

 

If the leadership did not create a thorough BRD, prioritized automation speed over accuracy or failed to properly audit the AI, responsibility should be extended to them.

 

To ensure the design has enough transparency and traceability, I would implement the following safeguards:

• Explainability Features,
• Human-In-The-Loop Mechanisms,
• Maintain detailed logs of interactions, decisions and rationales, so that errors can be traced and accountability can be assigned.
• Update training data regularly, which will help in reduction of biases and errors that accumulate over time.

Real-life scenario: Dutch Childcare Benefits Scandal

 

Dutch government, in 2013, used an AI driven algorithm to detect potential fraud in childcare benefits. The system flagged thousands of families (mostly with dual nationality or lower incomes) as fraudulent claimants. As a result, many faced debts (few exceeding 100,000 Euros) and more than 1000 kids were placed in foster care as a result of financial distress. The scandal was exposed in 2019, which led to resignation of the then Dutch government in 2021.

 

The AI accountability in this case was a concern, considering the following:

• Bias in Decision-Making: The AI model disproportionately targeted minority groups, which was a systemic discrimination.
• Lack of Human-oversight: The AI's decisions were not properly reviewed, which allowed the errors in decision making to escalate.
• Human life disruption: These wrongful accusations shattered families due to financial issues and emotional distress.

A new algorithm regulator was proposed (after the scandal was exposed) ensuring AI systems were transparent and fair.

 

This algorithm regulator established various aspects, I would mention few of them below:

• Algorithm Register: Government introduced a public register which documented the details of govt-used algorithms.
• Bias & Risk Auditing/Assessments: Regular audit mechanisms were implemented in AI models (specially for the ones used for public services) to detect biases and ensure compliance with fairness standards.
• Human oversight requirements: Human review mechanisms were introduced to all AI driven decisions affecting financial or legal matters.
• Explainability Standards: Government was liable to provide clear explanations for AI decisions.

The AI agent in itself cannot be blamed the same way a human can be as it is not a cerebral or moral agent. In the example of an AI chatbot for a healthcare provider a query goes wrong - the AI agent says yes to a facility not covered by the patient's insurance - things can go awry. The solutions agent responsible for creating the agent and those who overlook the development of it can be held accountable. To ensure transparency I would make sure that proper communication among teams and developers is being done and regular testing is done to ensure accuracy. 

Suppose AI wrongly flags a normal transaction as fraud and locks the customer’s account. 
The customer’s upset. The truth is, the AI didn’t mess up on its own, people built it, trained it, and let it run without enough checks. 
So the blame falls on the designers, the testers, and the company that rolled it out. You can’t just blame the technology. To avoid this, there should be humans reviewing key decisions, clear records of what the AI does, and regular system checks.

The person accountable in the RACI  who is oftem a leader or  manager

I can think of an example where in A company uses an AI tool to help with ISO 9001 audits, which check if businesses meet certain quality standards. The AI looks at data, finds problems, and writes reports. During an audit, the AI mistakenly says a process that is actually fine is not meeting the standards. This mistake causes the company to get a bad report, hurting their certification and reputation.

 

In this scenario, accountability can be distributed to

1)      AI Agent: The AI made a mistake, but can't blame the AI alone. The AI's actions depend on how it was programmed and the data it was trained on

2)      Developer: The designers and developers of the AI system are mainly responsible. They need to make sure the AI is properly programmed and tested to handle different situations. In this case, the AI's mistake shows there was a problem with how it was designed and tested

3)      Human Reviewer: If a human was supposed to check the AI's actions, they would also be partly responsible. The human should have systems to spot and fix mistakes before they become bigger problems.

4)      Management: The management team that decided to deploy the AI without sufficient safeguards or oversight also bears responsibility. They should ensure that proper protocols and fallback mechanisms are in place.

 

For transparency and traceability:

1)      Make sure a human checks and approves important decisions made by the AI, like flagging problems. This helps prevent mistakes from happening without someone noticing.

2)      Regularly check the AI system to make sure it's working correctly. This means looking at its decision records and updating its training with new data to make it more accurate.

3)      Keep detailed records of everything the AI does. This helps track down where and why a mistake happened

4)      Give clear reports to stakeholders that explain how the AI makes decisions, how often it makes mistakes, and what is done to fix those mistakes. This helps build trust and accountability.

5)      Create systems that automatically stop or highlight potentially harmful actions for review before they happen.

6)      Set up a system where companies can report problems caused by AI decisions. Use this feedback to keep making the AI better.

The example here we want to consider is for healthcare industry and in the medical field where AI agent helps with the diagnosing medical conditions based on patient data. On one instance AI incorrectly diagnoses a patient with a different illness then actual one leading to incorrect treatment.

 

Responsibilities to be assigned as below :

 

To the AI Agent : Evaluate that data base and give the right set of information, if not found the relevant information then to escalate to Human intervention.

 

To The AI Designer : Check the AI codes and algorithm at a regular frequency and ensure to the update the trained models.

 

To The Human Reviewer: Timely Audit of the AI agent, ensuring all the compliance and security policies are abide to in order to avoid any unnecessary escalations.  

Scenario:

An AI agent is deployed in a telecom company’s customer service system. Its role is to analyze incoming customer support messages and triage them: auto-resolving basic queries, routing technical issues to relevant specialists, and flagging high-risk or potentially angry customers for escalation to a supervisor.

One day, the AI mistakenly escalates a minor billing question to the company’s crisis response team due to misclassifying the tone of the message as threatening. This leads to an unnecessary internal incident response, confusion for the customer, wasted staff hours, and reputational damage when the customer shares the interaction online.

Who's Responsible?
Primary accountability may lie with the product/deployment team for failing to implement human review or setting too low a threshold for escalation.

Secondary responsibility could lie with the AI design team, if the model had known deficiencies.

Systemic accountability rests with the organization, for not having oversight, training, and escalation policies in place.

 

Design Safeguards for Transparency and Traceability

Decision Logging - Every AI decision should be logged with context
Explainable Interface - Provide human-readable justifications for escalations

Human-in-the-Loop (HITL) - Require a human reviewer to approve the AI’s recommendation

Feedback Loops - Feed this outcome back into the training dataset, track false positives and use them for retraining.

Clearly Define Roles and responsibilities

 

Imagine an organization using an AI-powered sentiment analysis tool to interpret open-ended customer feedback alongside numerical satisfaction scores like CSAT or NPS. This tool is designed to summarize sentiment trends for executive dashboards and flag "at-risk" customers for retention follow-up. However, the AI starts misclassifying sentiments. For instance, it might mark a comment like, "10/10 because the staff begged me to rate high, but my issue wasn’t resolved at all," as positive due to the high score and phrases like "10/10" or "staff." Consequently, legitimate complaints get overlooked, skewing reports and leading to flawed customer recovery actions.
AI Accountability Concerns:

• The AI is making significant operational misinterpretations.
• Leadership might trust the dashboards at face value, missing deeper dissatisfaction.
• There’s a disconnect between structured data (ratings) and unstructured data (feedback text), and the AI isn’t bridging it well.

Responsibility Assignment: 

• AI Model Designers: They need to ensure the sentiment engine accounts for contextual contradictions, such as sarcasm or coerced ratings.
• VoC Analysts or Human Validators: If the process lacks human review for edge cases or only samples low-scoring feedback, it’s a flaw in the operational design.
• Business Stakeholders: If decisions are made purely on aggregated AI insights without cross-checking anomalies or validating assumptions, there's shared responsibility.

Design Safeguards for Transparency & Traceability in VoC AI:

• Score-Sentiment Conflict Flagging: Implement logic to flag entries where structured scores and sentiment don’t align (e.g., a 9–10 rating with negative sentiment).
• Context-Aware Sentiment Models: Train models using examples with sarcasm, coercion, or common contradictions between numeric scores and written feedback.
• Feedback Review Dashboard: Highlight high-rated yet negatively worded comments for manual review or customer callback queues.
• Sentiment Heatmaps by Segment: Break down sentiment trends by touchpoint, agent, or issue category to surface patterns not visible in averages.
• Explainable Sentiment Outputs: Show which words or phrases influenced sentiment classification to help analysts understand and challenge the AI's reasoning.
• Reinforcement Loop from QA or Callbacks: Feed corrected classifications back into model training to improve accuracy over time.

This approach ensures that the AI system is more reliable and transparent, ultimately leading to better customer satisfaction and retention strategies.

Accountability if an AI going wrong is a joint responsibility right from the designers, reviewers to the leadership and heads. There coukd be a mistake in the design, no proper line of sight and weak controls and flaw in approval mechanism, no governance /review from the leadership, leading to the AI system going wrong. Some of the ways this could be backtracked is audit logs, document control, RACI matrix t check where and what went wrong  

I am using a scenario in healthcare where AI agents are used for reminding outpatients of their next appointment with a doctor. The reminder is based on the severity of their health condition.
The problem, AI agent did not remind some patients with severe conditions about their next appointment.
ASSIGNMENT OF RESPONSIBILITY
-    The main responsibility would be assigned to the designer of the AI agent. This is because there is a possibility that the model the AI agent used to recognize and update severity is not up to date. Newer, undefined health challenges may fall under the category of un-severe.
-    The next line of responsibility will fall on the human that needs to check the category the AI agent places reminder. If for some reason, the human neglects their responsibility, there is a high chance for the AI agent to assign wrong categories.
Ultimately, the AI agent cannot be blamed for the mistakes they made. 
 

If AI goes wrong, the accountability lies in the following questions: who created it, who used it, and how it was used. Considering that a professional user has used the AI tool, the AI tool developer should be held accountable if it goes wrong. This indicates that the tool was badly trained to provide the information. The developer has not tested the tool appropriately before releasing the same for use.  

Example: If a hiring AI wrongly rejects candidates based on gender or race, the developers who didn’t test for bias are responsible.

interesting examples where AI has made a mistake.

 

There are 2 winners for this question - Hardik Joshi and Vinod GC.