Leaderboard

At our e-commerce product company, we have an AI powered search and recommendation engine feature. It can be configured on each customer project to leverage multiple data sources (ERP, e-commerce, PIM, purchase history) to personalize search and product recommendations. Personalization features include adjusting results based on purchase history, brand preference, and customer profiles. Our learning has been The recommendation engine can personalize shop assortment for different customer segments. While designing customer flows for this feature, we must ensure that the engine does not unintentionally limit catalog visibility or surface exclusive categories disproportionately. If historical purchase data, browsing patterns, or segment profiles reflect societal biases (e.g., preferences along gender, age, ethnicity, or socioeconomic lines), the algorithms can and will replicate and propagate these biases—such as recommending certain products less to some demographic groups or showing limited assortments. Segment-based catalog restriction could reinforce silos and limit choices for certain customer groups, mirroring or reinforcing pre-existing marketplace or data biases. Customizing algorithmic weighting based on customer profiling without scrutiny could favor or disadvantage groups. We had a real example of a sports attire retailer using our product where we experienced that “Inclusive Sizing” (sizes beyond standard American XS–XL, such as plus sizes or petite/tall fit) appeared in only about 10% of products in a given search result. The dynamic facets logic tended to omit these size attribute from the filters entirely. As a result: Customers seeking inclusive sizes were unable to filter effectively. The represented bias favoured mainstream size ranges, thus marginalizing niche segments. The system then further skewed visibility toward products that align with majority sizing, and had potential to worsening representation over time. Some real world complains from users were - "I can never find anything smart with a good price in my size unless they are your top-of-the-line products" - "I see models wearing new designs in the ads but I can't find enough trendy but age-appropriate colours on the website" Additionally, one real risk that was evaluated was that our model/engine might consistently push popular products from high-traffic regions, while under-representing niche or emerging markets. This not only skews visibility but may also limit growth opportunities for less dominant segments. Some steps that we have attempted to apply Design Phase - Curate diverse and representative data inputs - Allow manual overrides for known critical attributes and for attributes deemed socially or commercially significant (e.g., inclusive sizing, accessibility features) were treated as “defined facets,” ensuring consistent visibility regardless of prevalence. - Ethical guardrails in personalization logic: Forbid certain features (like region or size) from driving recommendation weighting unless justified. Testing Phase - Synthetic Test Profiles across demographics - Manual Testing to find if the engine is developing such biases Monitor and Audit Facet Presentation - Track which facets are consistently hidden across queries and evaluate whether they represent systematically underrepresented groups or product lines - Before releasing compliance review is emphasized on Legal, Privacy(GDPR), Security & Accessibility These proactive steps are now taken on early and help ensure our AI serves all buyers fairly, avoiding the “bias in, bias out” trap in new implementation projects.