Bedibrat Kutum

Should AI Be Allowed to Experiment in Live Processes? My answer is Yes. As AI continues to evolve at a rapid pace, one question becoming increasingly relevant in operational circles is: should AI be permitted to test improvements directly within live workflows? My take is yes, but only with the right guardrails in place The case for live experimentation A process improved by an AI has real scope to cut down the need for human work and increase efficiency throughout. The real question is not about experimentation, but rather about implementing that experimentation correctly. Launch with 20% A practical starting point is limiting AI involvement to roughly 20% of the product or workload. This gives us a concrete, observable sample enough to evaluate what's working and identify areas for improvement without putting the entire operation at risk. The goal is a controlled proof of concept, not a full deployment. Choosing the right candidates for experimentation Not all products or tasks are equal. For the pilot group, prioritize items that are: Non-urgent in nature Flexible in delivery timelines Lower in downstream risk if something goes wrong This dramatically reduces exposure while still generating meaningful data. Communication is non-negotiable The most crucial-and commonly overlooked-component of a live experiment is stakeholder management. It is the expectation that you must bring your customer along, directly and honestly and manage their expectations right from day one. Nobody likes a shock and nobody likes being duped: trust is built on honesty. Prepare a team before the experiment begins Warehouse and operational teams should receive thorough documentation and briefings before the experiment kicks off not during or after. A well-prepared team is far better equipped to respond to unexpected developments and adapt quickly when things don't go according to plan. Match expertise to the task When issues arise and in any live experiment, some will resolution speed depends heavily on having the right people assigned to the right problems. Routing urgent or complex cases to subject-matter experts rather than generalists can be the difference between a minor setback and a significant disruption. Document everything, without exception Each action and each result – wins or losses, small or big-must be recorded meticulously. This is not just paper-pushing but is crucial for the next iteration. Even small wins and losses provide a wealth of signals that should inform future decisions. A Real-World Example: AI Copilot in a Customer Retention Team To ground this in something concrete, here's an experiment we ran at my own organization that reflects exactly this approach. The challenge Our retention team operates in one of the most demanding customer-facing environments rebuilding trust with dissatisfied customers requires precision, empathy, and the ability to handle objections that are deeply situational. Every conversation is different. We observed an ongoing deficit with how the agents approached these, in particular regarding objection handling; the correct response to this type of event really depended on the timing and context. The experiment Rather than overhauling the entire team's workflow, we took a measured approach: we deployed an AI copilot to just 15 agents. The brief was straightforward use the tool to reduce manual effort, minimize errors, and get real-time support when handling complex objections during live customer interactions. What we observed The results were notable. Interaction quality improved significantly. Agents moved through conversations with greater confidence, negative interactions decreased, and the overall customer experience saw a measurable lift. But perhaps the most interesting outcome was less expected agents developed a deeper understanding of the product itself. With the copilot surfacing accurate information in real time, they were better positioned to communicate genuine value rather than falling back on scripted responses. The takeaway What this experiment reinforced is that the most effective outcomes come not from replacing human judgment, but from augmenting it. The combination of agent expertise and AI support reduced both workflow friction and human error without sacrificing the human connection that retention work fundamentally depends on. Though very limited, this is a strong argument for the organized 20% discussed above: Begin confined, examine closely, and build outward on what you discover. The bottom line: live AI experimentation is a calculated risk worth taking when managed correctly. A thoughtful 20% pilot, paired with strong communication, team preparation, expert assignment, and rigorous documentation, creates a framework where the potential benefits clearly outweigh the temporary disruption.