Q 819.

In traditional systems, performance audits are conducted by humans at defined intervals.

But as AI systems grow more autonomous, there’s growing interest in whether AI can monitor, assess, and improve its own performance — without losing accountability or transparency.

 

Think of one process in your domain where an AI agent makes regular decisions or recommendations.

How could the AI system perform a self-audit to detect bias, data drift, or declining accuracy?

What boundaries would you put in place to ensure the AI remains responsible, auditable, and aligned with business objectives?

 

⚠️ Note: Any answer that is generic or does not connect with a specific, relevant process will not be approved.

 

🏆 The best answer will be selected on the basis of:

 

• Relevance and clarity of the chosen process
• Practicality and creativity of the self-audit approach
• Thoughtfulness in ensuring transparency and control

 

 

Note for website visitors -

• This platform hosts two weekly questions, one on Monday and the other on Thursday. 
• All previous questions can be found here: https://www.benchmarksixsigma.com/forum/lean-six-sigma-business-excellence-questions/. 
• To participate in the current question, please visit the forum homepage at https://www.benchmarksixsigma.com/forum/.
• The question will be open until Tuesday or Friday at 9:30 AM Indian Standard Time, depending on the launch day. 
• Responses will not be visible until they are reviewed, and only non-plagiarised answers with less than 5-10% plagiarism will be considered for winner selection. 
• If you are unsure about plagiarism, please check your answer using a plagiarism checker tool such as https://smallseotools.com/plagiarism-checker/ before submitting. 
• All correct answers shall be published, and the top-rated answer will be displayed first. The author will receive an honourable mention in our Business Excellence dictionary at https://www.benchmarksixsigma.com/forum/business-excellence-dictionary-glossary/ along with the related term.

• Some people seem to be using AI platforms to find forum answers. This is a risky approach as AI responses are error-prone because our questions are application-oriented (they are never straightforward). Have a look at this funny example - https://www.benchmarksixsigma.com/forum/topic/39458-using-ai-to-respond-to-forum-questions/

• We also use an AI content detector at https://quillbot.com/ai-content-detector. Only answers with less than 45-50% AI-generated content will be considered for winner selection.

Domain: Maps & Navigation apps

Process where AI makes regular decisions

AI manages everything. Finding the shortest route to destination, predicting ETA based on google maps and historic data and showing you navigation / Direction on the map.

---

Scenario (common, not rare)

• A main road is closed for repairs, traffic is diverted through side streets.
• A new flyover opens, but the map still thinks the old junction is the fastest path.
• Result: wrong routes, bad ETA’s and driver frustration.

---

How the AI does a self-audit (detect bias, drift, declining accuracy)

1. Route Reality Check (ETA drift)

·       Continuously compare predicted vs. actual travel times from live GPS traces.

·       If a segment’s error > 15% for 30+ minutes, flag it as suspect.

o   Likely causes: barricade/diversion/new signal/new flyover.

o   Keep monitoring if same delay is observed during peak and non peak hours then report this pattern.

2. Flow Pattern Shift (data drift)

·       Detect sudden, sustained changes in speed profiles and turn behavior:

o   Many cars now avoid the old straight path and turn left / Right at a barricade → diversion from map guidance (not 1 but all vehicles).

o   Cars elevate then rejoin at a different point with higher average speed → flyover inferred.

Report any such patterns.

3. Map Confidence Score

·       For each road segment, keep a freshness score (last verified date + data volume updates of ETA).

·       Low freshness + high ETA error = “re-survey priority” (needs more data / crowd confirmation).

4. Fairness / Coverage Bias

·       Check if errors are concentrated in periphery / new suburbs vs city centers.

·       If yes, schedule extra data collection (prompt drivers) to avoid “downtown bias.”

---

What the AI must not change alone (needs human approval)

·       Permanent geometry edits of the road (closing / opening roads).

·       New flyover addition, lane direction changes, turn restrictions, Speed limits clocks.

·       These go to a map editor queue with evidence:

o   Heatmaps of turns, speed profiles, driver reports, photos (if available).

---

Safeguards & boundaries (keep it responsible)

• No silent map edits: AI proposes; human GIS team approves.
• Audit trail: Every flag, ETA tweak, and approved edit is time-stamped and saved.
• Safety rule: Never route through pedestrian lanes/ Wrong route / private roads even if traffic flows do.

---

Real time example (daily reality)

Morning, 08:00: Barricades divert traffic off Main Rd to Side St.

• ETA error for Main Rd jumps to +25% to 40 minutes full day during both peak and off peak hours.
• 100% of vehicles turn left at the barricade → suspect diversion inferred.
  AI action (auto): Recompute ETAs with live speeds, recommend the diversion, Escalate the new finding.
  Evidence gathered: Turn heatmap + speed drop + user reports.
  Afternoon, 15:00: Human map editor reviews the evidence ands marks Main Rd “under repair,” adds official diversion.
  Two weeks later: New flyover opens. Cars now skip the old junction, elevation profiles & high speeds confirm.
  AI: Improves ETA instantly (auto), submits “flyover candidate” for human approval.
  Map team: Approves the new flyover geometry; routes stabilize, ETAs tighten.

 

AI Self audit for Claims Process Optimization

I see AI audit as evolution of six sigma thinking, where process becomes capable of detecting, measuring and its own variation. Build a system which is powerful enough to learn but transparent enough to be questioned

In Healthcare BPO we frequently encounter AI agents that continuously review claims data and recommend routing logic for claims adjudication process

They recommend if claim should be auto adjudicated or flagged for manual review to SME. Even though these agents learn from historical data patterns, it also makes them prone to bias, drift or increasing become in accurate if underlying data changes. To prevent this, I would design a self-loop for AI in three layers

1.      Automated AI and drift checks- Periodically, AI should compare with a benchmark dataset- - a sample which has been previously audited by trained SME. If AI ‘s decision pattern starts to diverge beyond the defined threshold, it would generate alert (an internal “Bias alert”)

2.      Shadow model validation: A shadow model will run alongside in parallel, generating recommendations that are not acted upon but logged in the database for comparison. This would act like a mirror check – it would highlight performance degradation or data quality issues before they impact real workflows

3.      Explain ability for human oversight: Every decision that AI makes should be explainable in simple language- thus ensuring that SME can trace the logic. These explanations would also go to monthly dashboard for Quality & compliance team to review

To keep the system accountable and aligned with organization boundaries, I would define below controls:

a. Human in loop control- no recommendation should override compliance and policy review without SME review

b. Governance of retraining cycles: Only after validation and sign of by Data Governance team and SME, will an AI be allowed to retrain and that too from approved data sources

c. Audit traceability: Every AI decision, data source, model version, etc. should be logged in as part of audit trail as a part of accountability

 

 

 

Gypsum Board Manufacturing

 

A gypsum board manufacturing plant uses an AI-based production designed to monitor, controls and simultaneously optimize multiple key parameters like board thickness with a target of 12.5 mm, surface finish, and moisture content by analyzing data from sensors. This AI based system covers all the stages of gypsum board manufacturing i.e. slurry preparation, forming, drying, and cutting sections.

 

The key functional parameters which constantly monitors, controls, adjust to optimize are slurry feed rate, additive dosage, line speed, and dryer temperature. Since these parameters directly influence product quality, yield, and significant energy cost, the system must be able to self-audit its own accuracy, bias, and reliability while staying transparent and accountable to operators and management.

 

AI Self-Audit Framework

This self audit system operates within three interconnected layers:

1. Model Integrity  2.Data Integrity 3.Decision Integrity

These three models work in a perfect sync so that AI always remains precise, traceable, auditable, and aligned with plant goals.

 

1. Model Integrity Audit

This ensures that statistical health, accuracy, and fairness of AI prediction and control models are constantly monitored and controlled.

(A) Continuous Performance Tracking:

On real time basis AI compares its predicted control settings like slurry flow, line speed etc against actual outcomes like board thickness and density.

 

There are KPIs such as Mean Absolute Error (MAE) and Process Capability Index (Cpk) are tracked on real time basis and at the end of every shift to observe any drifts.

If Cpk drops below 1.33 or MAE rises by more than 10%, then the system triggers a Model Drift Alert.

 

(B) Adaptive Learning Validation:

Regularly at a predefined interval or post failure alarms AI before any model self-tuning or retraining. The AI runs simulation exercise using historical data to confirm that new parameters would have improved yield and quality. It gives more weightage to most recent data and reduces weightages for older data. 

 

It studies patterns in the data using EWMA and CUSUM charts. It also uses Z-MR charts for shorter runs along with pre-control charts for very simple interpretations. Retraining of AI only proceeds only after supervisor authorization.

 

(C) Bias Detection & Self Correction in the AI model:

It’s possible for an AI model to over-optimize like consistently prioritizing thickness control at the cost of sacrificing the expected surface smoothness or efficient energy consumption. Hence, a fairness score is computed based on internal metrics to ensure balanced performance across all CTQs (Critical to Quality parameters).

 

2. Data Integrity Audit

This ensures that all process data and sensor readings remain reliable, complete, and consistent.

 

(A) Sensor Health Monitoring:

The AI runs SPC (Statistical Process Control) and Z-score analysis on each key sensor like temperature probes, density meters, level transmitters.

Outlier or drifted sensors are automatically isolated, and operators receive a “Sensor Suspect” notification for recalibration.

 

(B) Data Completeness Checks:

Missing or delayed sensor data triggers predictive gap-filling using regression models — but such interpolations are flagged for review.

If missing data exceeds 5% of any critical variable per batch, the AI pauses autonomous control and defers to manual mode.

 

(C) Material Batch Correlation:

The AI validates whether deviations correlate with raw material changes like gypsum purity or starch quality.

This helps ensure process variance due to input parameters isn’t misinterpreted as functional or system drift.

 

3. Decision Integrity Audit

This validates that AI control actions remain explainable, traceable, and aligned with quality objectives.

 

(A) Human-in-the-Loop Control:

Any major automatic adjustment like >3% change in line speed or additive flow must be acknowledged by a production supervisor.

The system logs the decision, timestamp, and operator ID for traceability.

 

(B) Work Performance Reports:

Work performance data goes through expected performance level wherein variance analysis is carried out and any negative variances are highlighted via Work performance report with proper reasoning as to why a particular corrective measure was taken.

 

“Line speed reduced by 2.5% due to rising slurry density i.e. 1.12 g/cc to 1.18 g/cc and increasing moisture variance by 1%.”

 

This builds operator confidence and audit transparency.

 

(C) Self-Validation Sampling:

At a fixed interval of 4 hours i.e. twice in every shift of 8 hours, a random selection of 5 boards are picked up for manual review for the key KPIs like thickness, weight, visual defects, smoothness etc.

Post manual review the AI predictions are compared with manual readings to compute a Validation Accuracy Index.

If this index drops below 95%, the AI triggers a Performance Review Mode.

 

Summary

In nutshell, an AI model self-audits it’s performance by making use of this three layers sytem which is

•            Monitoring model drift and prediction accuracy,

•            Ensuring sensor reliability and data integrity, and

•            Verifying that autonomous decisions remain transparent and traceable.

By ensuring that there is a manual review wherever required, explainable logic and process KPI monitoring and controlling on real time and periodic basis, the system stays responsible, auditable, traceable and aligned with both quality and cost targets.

This framework represents a practical path to Responsible AI in manufacturing — where intelligence enhances control precision without compromising accountability.

 

An AI can reliably flag deviations from predefined metrics like accuracy, data drift, fairness thresholds but true responsible auditing requires ethical, contextual judgement which currently needed human inputs as final authority.

In cold rolling steel strip manufacturing, an AI system often controls rolling mill parameters like tension, speed, and thickness based on sensor data. These decisions affect product quality and energy efficiency.

After every 50 coils processed, the AI runs a self-check,

• Compares predicted vs. actual thickness (in mm).
• Runs statistical tests (e.g. Hypothesis Testing, dynamic SPC) for bias across different suppliers.
• Alerts if rolling force models show drift beyond 2%

In cold rolling, this self-check is highly critical for real time monitoring. However true responsible auditing remains a hybrid process of internal AI control system and human auditor / governance structure.

To keep the AI aligned and auditable,

• AI can flag issues but cannot change control logic without human approval
• Every self-check and alert must be logged for traceability.
• Define clear limits (like max deviation allowed), so that AI cannot override safety or compliance standards
•  Even with self-audit, schedule human audits quarterly / half-yearly to validate AI’s findings.
• AI must provide reasons for alerts in plain language to operator (e.g., “Sensor data drift detected in mill stand # 2”).

The AI continuously checks for bias, data drift, and performance accuracy while adjusting rolling mill parameters. To maintain accountability, boundaries such as human oversight, audit logs and periodic external reviews ensure the AI remains transparent and aligned with business objectives (Human validations and governance as final decision-making authority).

 

🏆 Winner – Shashi Prakash (Gypsum Board Manufacturing) - exemplary self-audit with quantifiable triggers (Cpk/MAE/EWMA/CUSUM), sensor integrity checks, manual sampling, and gated retraining—highly auditable and production-ready.
🥈 Runner-up – Adil Khan (Maps & Navigation) - smart ETA-drift/self-audit logic with freshness scores, fairness checks, and strict “no silent edits” boundaries.
🥉 Special Mention – Sanjib Ghosal (Cold Rolling Steel) - strong hybrid model; periodic self-checks and clear governance—great visual included.
✅ Also approved – Rohan Modak (Healthcare Claims) - benchmark + shadow-model validation with governance controls.