Leaderboard

At our e-commerce product company, we have an AI powered search and recommendation engine feature. It can be configured on each customer project to leverage multiple data sources (ERP, e-commerce, PIM, purchase history) to personalize search and product recommendations. Personalization features include adjusting results based on purchase history, brand preference, and customer profiles. Our learning has been The recommendation engine can personalize shop assortment for different customer segments. While designing customer flows for this feature, we must ensure that the engine does not unintentionally limit catalog visibility or surface exclusive categories disproportionately. If historical purchase data, browsing patterns, or segment profiles reflect societal biases (e.g., preferences along gender, age, ethnicity, or socioeconomic lines), the algorithms can and will replicate and propagate these biases—such as recommending certain products less to some demographic groups or showing limited assortments. Segment-based catalog restriction could reinforce silos and limit choices for certain customer groups, mirroring or reinforcing pre-existing marketplace or data biases. Customizing algorithmic weighting based on customer profiling without scrutiny could favor or disadvantage groups. We had a real example of a sports attire retailer using our product where we experienced that “Inclusive Sizing” (sizes beyond standard American XS–XL, such as plus sizes or petite/tall fit) appeared in only about 10% of products in a given search result. The dynamic facets logic tended to omit these size attribute from the filters entirely. As a result: Customers seeking inclusive sizes were unable to filter effectively. The represented bias favoured mainstream size ranges, thus marginalizing niche segments. The system then further skewed visibility toward products that align with majority sizing, and had potential to worsening representation over time. Some real world complains from users were - "I can never find anything smart with a good price in my size unless they are your top-of-the-line products" - "I see models wearing new designs in the ads but I can't find enough trendy but age-appropriate colours on the website" Additionally, one real risk that was evaluated was that our model/engine might consistently push popular products from high-traffic regions, while under-representing niche or emerging markets. This not only skews visibility but may also limit growth opportunities for less dominant segments. Some steps that we have attempted to apply Design Phase - Curate diverse and representative data inputs - Allow manual overrides for known critical attributes and for attributes deemed socially or commercially significant (e.g., inclusive sizing, accessibility features) were treated as “defined facets,” ensuring consistent visibility regardless of prevalence. - Ethical guardrails in personalization logic: Forbid certain features (like region or size) from driving recommendation weighting unless justified. Testing Phase - Synthetic Test Profiles across demographics - Manual Testing to find if the engine is developing such biases Monitor and Audit Facet Presentation - Track which facets are consistently hidden across queries and evaluate whether they represent systematically underrepresented groups or product lines - Before releasing compliance review is emphasized on Legal, Privacy(GDPR), Security & Accessibility These proactive steps are now taken on early and help ensure our AI serves all buyers fairly, avoiding the “bias in, bias out” trap in new implementation projects.

Is AI solution biased? Well before asking this question, let us dwell more into human nature, is human response or process building biased, it has to be, it forms the basis of selecting criteria, a baseline on which the entire process is set or supposed to operate. Similarly, when we create an AI agent there will be a bias in AI-enabled customer service processes, especially in banking—can have serious consequences, from unfair treatment of customers to regulatory violations. Let’s break this down using your example of a third-party contact center handling banking queries, such as Annual Maintenance Charges (AMC) or unauthorized UPI transactions, and explore how bias can creep in and how to mitigate it. What Bias Can Appear in Banking Customer Service and Where? 1. Case Prioritization Risk of bias: AI may prioritize cases based on customer profile (e.g., high-value customers), potentially delaying resolution for others. E.g: AMC-related queries from senior citizens may be deprioritized if the model learns they are less likely to escalate. 2. Action Recommendations Bias possibility: AI may suggest refunds or escalations based on historical patterns that reflect biased decisions. Example: UPI fraud cases from Tier-2 cities may be less likely to get recommended for escalation due to historical underreporting. 3. Response Generation Bias Risk: Regional models may respond taking into consideration the tone of voice, choice of words, AI agent will respond differently given the tone, politeness and choice of words for customers based in northern part of India versus the same AI agent might find the customer’s similar language or choice of words as rude or condescending and might deny service in southern part of India. Language models may respond differently based on customer name, language, or tone. Example: A polite query may get a more helpful response than an agitated one, even if both are valid. 4. Billing Model Influence Bias Risk: If billing is based on connect minutes, agents may be incentivized to prolong calls. If based on call count, they may rush. Example: AMC queries may be wrapped up quickly without full resolution under a per-call billing model. So, what do we do to minimize bias in Design, Testing, and Monitoring A. Design Phase Diversify Training Data Be it low income customers or high rollers, you might want to include varied customer profiles, geographical regions of customers, languages, net worth of customers, and complaint types. Low amount frauds or frauds based on a certain amount should not matter when a customer is complaining of an unauthorized transaction by a merchant. There is a possibility of bias setting in based on a low or high amount transaction, AI might prioritize only high amount unauthorized transaction cases. We must ensure representation of certain vulnerable groups (e.g.,low income, senior citizens, rural customers). Provide clear objectives that kill bias Design AI models with fairness constraints (e.g., equal resolution rates across demographics). Avoid optimizing solely for efficiency metrics like AHT (Average Handling Time). Human-in-the-Loop Keep humans involved in sensitive decisions (e.g., refund approvals, fraud escalations). B. Testing Phase Inclusion of Bias Audits Test model outputs across different customer segments. Use synthetic data to simulate edge cases (e.g., same query from different regions). Scenario-Based Testing Create test cases for AMC and UPI queries with varying tones, languages, and urgency levels. Check for consistency in response quality and resolution. Metric Diversification Track fairness metrics alongside performance metrics (e.g., resolution equity, escalation parity). C. Monitoring Phase Set up real-time dashboards Monitor call outcomes by customer segment, query type, and agent behavior. Flag anomalies (e.g., unusually short calls for UPI fraud cases). VOC : Feedback Collect customer feedback post-call and correlate with AI decisions. Use feedback to retrain models and adjust flows. Billing Model Alignment Ensure billing models don’t incentivize biased behavior. Consider hybrid models (e.g., quality-adjusted call count) to balance efficiency and fairness. How do we break the “Bias In, Bias Out” Cycle Continuous Learning: Regularly update models with new, unbiased data and feedback. Make it transparent: Make AI decision-making explainable to agents and supervisors. Assign ownership: as a check mechanism, assign accountability for bias monitoring and remediation. Cross-Functional Collaboration: Involve friendly customer base, compliance team, QA team, and customer experience teams in AI governance.

Bias in AI is not a technical bug; it's a systemic threat that can cascade through healthcare outcomes, compliance, and trust. Bias Scenario: Automated Case Prioritization in Medical Coding Suppose an AI tool is employed to rank medical coding cases by urgency, complexity, or reimbursement value. If the training dataset disproportionately represents particular populations ( for example-older adults, inner-city hospitals, affluent zip codes), the AI will tend to rank those cases more often—discriminating against cases from rural, low-income, or minority communities. In this scenario, we should have medical coders from diverse backgrounds involved in the design process to highlight possible blind spots and enable them to alert suspicious prioritizations, inputting those instances back into the model for retraining.