So, here’s the thing. Most AI pilots in ITSM are declared a success and then go nowhere. The conditions that make a pilot succeed are the same conditions a production rollout takes away. A pilot runs on a narrow scope, a motivated team, and close oversight, and none of that survives the move to a live environment at scale. The industry numbers back this up. MIT’s 2025 study The GenAI Divide found that 95% of enterprise AI pilots delivered no measurable financial return, and Gartner is reporting on an AI value crisis.   

This article examines the gap between a successful proof of concept and something the wider organization can rely on and what it takes to close it including:  

• Why pilots succeed, and deployments fail and what the gap between them usually contains 

• What needs to be true about your processes, data, and governance before you scale 

• How to build the internal case for moving from pilot to production 

• What a phased rollout looks like and how to manage the organizational change it requires 

Why pilots succeed, and deployments fail and what the gap between them usually contains 

So many people are asking the question why did our pilot succeed and then why did the subsequent deployment go so badly? The reality is that you can’t compare the two. Pilots are carried out in controlled, safe, sandboxed environments. 

There’s a narrow scope, limited integrations and there are usually lots of people involved in the pilot because, let’s face it, it’s in everyone’s interest for the pilot to go well. Another protecting factor is that the data is limited in most pilots to retain control and ensure the system doesn’t get overloaded.  

Deploying AI to your live production environment is a completely different proposition. Real life isn’t controlled or safe. It’s messy and complicated and there’s a lot more variation once we throw end users into the mix. Given that AI is an amplifier, things can go from controlled to chaotic very, very quickly. 

What needs to be true about your processes, data, and governance before you scale 

Data quality

The success of any AIOps implementation is contingent on data quality. So, if our data is our everything then it needs to be in the best possible state. Have a clear plan for baseline and managing your data so it’s correct and stays that way. If you’re struggling with where to start, go back to value and outcomes. Think about the business outcome you want from your AIOps project. If it’s customer retention, the first area of data to look at could be customer satisfaction metrics. If it’s improved service quality, take a look at your incident, problem, request and change data. By having a focused starting point, you can build a prioritized data quality plan that will support your business outcomes.

Once your data is reviewed, ensure it is optimized for AI usage, for example having standard ranges for numeric data and having naming conventions for category data to ensure AI can recognize it consistently. A final point to remember about data quality is that it’s not a one and done exercise. Like the ITIL practice of continual improvement, data isn’t static. In the case of AIOps, your data is a key input, it will evolve as the business grows and changes over time so now is the time to implement systems to track key quality indicators and set up alerts via your event management process to notify support teams when the data is approaching those quality thresholds. By treating data quality and management as a continuous lifecycle, we can create a sustainable foundation for AIOps delivery.  

Baseline your processes

Ensure they’re fit for purpose and designed with scalability in mind. Make sure your processes are up to date and have been reviewed recently. Everything starts with your people, data and processes. If you don’t have the basics in place, your AIOps project will struggle because there will be gaps.  Before you progress from the pilot stage, review your processes, make sure they still meet the required outcomes and see if there are any gaps.  If you don’t take the time to look at your processes, you will miss things and will end up trying to retrofit process improvements while you’re in the midst of your AIOps implementation and believe me when I say that no one wants that.  

Understand the complexity of your environment

Every IT ecosystem is different, but many organizations have an IT setup that looks like something that contains cloud services, on-prem infrastructure, SaaS platforms, containers, APIs, endpoints, networks, enterprise applications, in-house or other custom applications, printers, AV equipment, and about 47 different places where things can quietly catch fire. That’s a lot and given how AI can amplify what we have already, we need to make sure we have our house in order, take care of system flaws like fragmented data, broken code or clunky workflows to let AI do its thing and improve service delivery rather than adding risk.  

Governance

Have the right guardrails and ownership in place to keep everything on track. When moving from pilot to production, you’ll need to establish a comprehensive governance framework that defines data ownership, quality responsibilities, change enablement processes, and clear escalation procedures. A key tenet of governance is data quality because if AI is going to make decisions that impact on our live environment, then we want those decisions to be made based on the correct information. A key part of your governance strategy should be implementing regular quality checks with defined metrics to ensure your data remains accurate, complete, and up to date. Create trusted data catalogs that map out what data you’ve got, where it comes from, and how it underpins your ITSM tool, so you can see exactly where decisions are being shaped, influenced, or quietly going wrong.

Assign clear ownership for your datasets, with data owners for accountability and dedicated stewards responsible for championing data quality. We’ve already talked about quality KPIs and alerting thresholds, but governance also needs to extend into remediation processes and data retention policies. Once AI moves into production, organizations also need to think much more seriously about auditability and decision ownership. If an AI model suppresses an alert, triggers a change, or decreases the priority of an incident, folks need to understand why that decision was made, who signed off on the level of automation, and who still owns the outcome when something goes wrong. “The AI did it” excuse isn’t going to hold up well in a major incident review or audit. It’s like when we outsource part of our IT offering; we can outsource the day-to-day tasks, but the accountability remains with us. 

You’ll also need to consider regulatory exposure and data handling obligations, particularly where operational data may include customer, financial, or sensitive information. Governance frameworks should define how data is accessed, used, managed, and secured, alongside controls around supplier accountability if third-party platforms or managed service providers are involved in the AI Ops ecosystem. Like data quality and continual improvement, governance can’t be treated as a one-off exercise completed during implementation. We need to plan for change because AI models and our production environments will evolve over time. We will need to build processes for reviewing effectiveness, reassessing risks, and validating ongoing value if we want AIOps to remain trusted once it’s embedded into day-to-day operations.  

How to build the internal case for moving from pilot to production 

When attempting to move from pilot to production it’s important to have a strong case so that you have an agreed baseline and can keep the implementation stage on track. A strong case includes: 

• Focus on value and define what good looks like with solid KPIs and metrics. Quantify the ROI and business benefits. Make it personal to your organization. This isn’t the time for standard measures like “improved operational efficiencies” or “better IT support”, look at what’s important to your organization and its people and work from there. This leads us nicely on to clear, data-driven KPIs and metrics such as reduced alert noise, faster triage, improved incident and request routing accuracy, reduced MTTR, improved major incident response and improved colleague experience. Your pilot was your baseline so use the measurements you’ve built to help level up when you move things into production.  

• Create a technical blueprint. It will include monitoring tools, data sources, service relationships from the CMDB, security controls, automation boundaries and integration points. A solid blueprint will capture the technical landscape, operational dependencies and scalability considerations enabling support teams to get the best possible use from AIOps implementations. 

• Manage risk. Managing risk in AIOps implementations requires organizations to balance innovation and speed with operational control and organizational GRC obligations. Work with your organization to create a dedicated AI Ops risk register so that risks can be captured, discussed, prioritized and acted on.  

• Build a sustainable operating model. It will include setting out clear ownership, proper model management with operational support wrapped around it, accountability for data quality, and continual improvement built into the day job. It also means constantly monitoring, tuning, challenging, and refining how AI recommendations are made, because operational environments change, business priorities shift, and what’s good enough in this moment could quietly become a future risk if we don’t keep on top of things. 

• Make your approach human-centric. It’s all too easy to get caught up with the sparkly new technology but making your approach more human-centric will improve colleague engagement and increase support for the implementation. When moving from pilot to production, make the moves with your people in mind, reducing repetitive tasks, improving visibility, accelerating decision-making because these are the things that will help your techs respond to issues more effectively under pressure. Honest, clear communication regarding AI recommendations, involving operational teams in the rollout, and building trust through practical day-to-day value are all critical to ensuring AI is seen as an enabler for people rather than something being imposed on them. 

What a phased rollout looks like and how to manage the organizational change it requires 

A phased rollout plan gives business time to get used to the concept of AIOps and mitigate risk. A phased rollout could look something like the following: 

Phase 1: Shadow mode

This is the “AI sits quietly in the corner and proves it can behave itself” phase. The platform is connected, watching events, spotting patterns, generating recommendations, and generally trying to impress everyone, but importantly it isn’t making live operational decisions yet. Your support teams still remain fully in control. 

And honestly, this stage is so important because it’s usually where you discover that your monitoring data is noisier than expected, that some of the CI relationships in the CMDB are questionable at best, or that your integration layer has multiple tools alerting on the same thing. You get the idea. Shadow mode gives you the breathing space to test scenarios, tune and refine models, improve data quality, and build confidence without stressing out your techies or risking service disruption. It also helps support teams become familiar with the recommendations in a safe environment. Think of it as a “let it build trust before it touches anything important” stage.  

Phase 2: Limited Scope 

Once the AI has demonstrated some value and stopped terrifying everybody, the next step is to start carefully introducing it into controlled production scenarios. This step is usually where AI Ops starts helping with things like alert duplication, ticket summaries, routing recommendations and root cause suggestions. The key phrase here is “controlled adoption”: you’re still keeping humans firmly involved in decision-making, but you’re starting to allow AIOps to remove some of the repetitive operational pain that support teams deal with every day. This phase is all about demonstrating confidence. Your techs will need to see that AI-driven outputs are useful; IT leaders need evidence that the project can deliver value and your GRC team needs to know that the controls are working. It’s also the stage where we establish what happens when AI gets something wrong in a controlled environment. 

Phase 3: Full Production 

AKA the stage where AIOps has moved beyond being the pilot and become part of normal operational life. Governance models are established. Ownership is clear. Data quality processes are in place. Support teams understand it and are more comfortable using it. The organization has enough confidence to increase and scale automation and predictive capabilities where appropriate.

This stage is where all the work we’ve done around governance is important because autonomous remediation sounds brilliant in a vendor demo, but the reality is very different when the AI decides to restart something important in production at 2 AM. What we need to remember is that full production doesn’t mean finished. AIOps is not a “deploy it and hope for the best” capability. The only thing we can know to be true with absolute certainty is that things will change and evolve over time, so the organizations that succeed are usually the ones that continue tuning, reviewing, governing, and challenging the outputs long after the initial excitement of the rollout has passed. 

That’s my take on moving AI from pilot to production successfully. If you’re working through the same shift, we’d be interested to hear how it’s going.