Leaderboard

AI-Powered Incident Management System At our e-commerce platform company (PAAS model), we need to support environments of all customers who rely on our system for their online ecommerce sales and we need to have continuous uptime and meet strict SLAs. We use NewRelic as our observability platform which collects massive volumes of logs, traces and metrics. After filtering we are analyzing over 60TB of telemetry data per month. We’re now building a system to query this data for AI model integration, enabling anomaly detection, incident correlation, and predictive analytics. Decision Making Scenario- Analyzing and classifying incidents in real time When an Alert is generated the Operations Manager need to make quick decisions during disruptions like outages, performance degradation, or critical bugs. These incidents needs to be evaluated quickly due to SLA requirements since they directly impact buyer satisfaction, conversion rates, and revenue for our clients. The challenge is to prioritize correctly, weighing severity, scope, resource constraints when choosing action steps. AI Agent Support We have been working on a design of an Incident Response AI Assistant with the final goal that it will - Aggregates and analyzes real-time data, such as error logs, user requests, system metrics, and incident tickets. - Scores incidents by combining: - Business impact (example: conversion drop, cart abandonment) - Customers or geographies affected - Urgency indicators like large number of error log entries, memory or cpu spike - Generates and recommends priority and type based on pre-learned categorization - Suggests response paths to help mitigate the problem - Quick mitigation (rollback the latest patch) - Escalation (pass to development team) - Monitoring (observe performance metrics) - Offers confidence scores on its solution For this we are creating solution with two key components to support the decision-making and the communication 1) Incident Analyzer Bot which will automatically detect system issues using AI/ML Learn from historical incidents to categorize alerts by severity and type and reduce false positives Correlate related events — for example, grouping three different alerts under a single outage — to help managers see the full picture quickly Find and provide reference of related incidents from our historical data to help provide the manager with information about the previous RCA and solution 2) Ops Chatbot is focused on communication (not customer-facing) If a critical issue is detected, it can notify customers automatically and proactively before they notice it themselves. It supports manual overrides, customizable communication methods (like email, message, chat) with pre-defined message templates. Makes message suggestions to the manager, who can review and approve them directly in tools like Microsoft Teams. Manages follow-ups automatically if the issue remains unresolved — for example, sending timed updates like “we are still analyzing your system” every 1, 6, 12 hrs depending on the case. This AI assistant will help to surface and prioritize incidents quickly but the final decision remains with the manager who reviews the categorization, solution recommendation and approves the communication suggested by AI. The manager will be have full discretion to override, approve, or modify the AI’s actions. We have planned to create feedback loop to help the system learn and improve over time and implement accuracy monitoring by comparing AI predictions to actual outcomes, review the AI's confidence scoring especially when AI uncertainty is high. This regular validation against historical incidents should help us ensure that the human/AI-assistant together work towards meeting customer's SLAs.

I recently worked on a project where we built what we called a CO-CEO AI agent for a marketing company. The idea was simple: instead of treating the AI as a background tool, we brought it into the decision-making process almost like another executive. Its main job was to help the CEO design marketing strategies for different clients. Now, here’s the twist. We didn’t just let the AI spit out strategies in isolation. It was invited to client meetings, not literally of course, but through structured prompts where the full context of the client’s business, challenges, and goals was fed in. That way, its recommendations weren’t generic “playbook strategies,” but tailored to the actual discussion the leadership team was having. That made it much more of a trusted advisor than a black-box machine. From this project, I learned that there are two layers to making AI advice truly useful and trustworthy for leaders: 1. Where It Should Assist Pattern spotting across campaigns: Leaders don’t always have the time to compare 50 different client reports. The AI could highlight trends (e.g., “clients in retail are seeing 20% higher engagement when campaigns run mid-week”). Scenario testing: Instead of one “best” strategy, the AI could lay out three options: low-risk, high-growth, or balanced. This gave the CEO choices rather than a single directive. Speed on background research: Before walking into a client strategy session, the AI could summarize competitor campaigns, past results, and market conditions in minutes. These are areas where AI’s scale and speed give leaders an advantage without replacing their judgment. 2. Checks to Keep It Reliable Context gatekeeping: The AI was only as good as the context it had. We made it a rule that client objectives and constraints must be captured first (almost like a briefing note) before the AI gave advice. No context, no strategy. Audit trail of reasoning: Every recommendation had to come with a short rationale, “this works because past campaigns in similar industries showed X, Y, Z.” This gave the CEO confidence in the “why,” not just the “what.” Version control for prompts: As we refined how we asked the AI questions, we tracked changes. For example, when we shifted from “generate campaign ideas” to “act as a CMO and propose three strategies with risks and trade-offs,” we documented it. That way, if a change caused worse outputs, we could roll back quickly. Human override always on: The AI was never treated as final authority. The CEO still made the call, but with stronger input in less time. Honestly, what made this whole setup work wasn’t the AI being “super smart.” It was the way we used it. We never treated it like it was going to run the company or make the final call. Instead, it was more like a second set of eyes, someone in the room who could throw out a few options, show the risks, and spot patterns the rest of us didn’t have time to see. The clever part wasn’t the output, it was the process: making sure it only answered once we’d given it the right context, tracking how we asked questions so we didn’t lose improvements, and always keeping a paper trail of its reasoning so it didn’t feel like magic. At the end of the day, the CEO still made the decisions. The AI just made those decisions faster and more informed. That’s really the trick to building trust. You let the AI contribute, but you don’t hand over the steering wheel. It’s not there to replace judgment, it’s there to make good judgment easier.

Great answers from all respondents. The best answer has been provided by Pavitra Jain. Well done. Answer from Swapnil is also a must read.

Q. Bais in, Bais Out: How to break the cycle? Answer - In a service delivery context given example of prioritizing cases, recommending actions and responding to customers, following steps can be take at various stages of solution development – Design stage – · Assess project objective, scope, metrics and success measures with timelines · Reach out to stakeholders incase of difference of opinion. · Interview and empathize the issues faced · Brainstorm and validate assessment criteria.Design FMEA · Course correct metrics and success measures if required · Involve Developers, testers in the kick off call Testing phase – · Develop use test cases. · Build Agentic AI workflow with what-if scenarios, And OR logic · Link knowledge base repository with correct calibrated clean database Monitoring phase – · Intelligent dashboards with powerapps workflow when any shift in data is observed · Calibrate and retraining AI for precision and accuracy. · Periodic governance This is how one can let the Bias IN and then Bias it Out through careful design, testing and monitoring to break the cycle.