How to Structure an AI Pilot Project in ITSM: The High-Control Use Case Framework

Almost every large organization has launched at least one artificial intelligence pilot project aimed at impacting IT service management. Many discontinue it after a few months, with results below expectations, and attribute the negative outcome to the technology. In most cases,the main cause is how the project was strategized from the beginning.

There are two recurring causes. The first is excessive ambition, which drives organizations to start with AI agents capable of operating autonomously even when the organization does not have the maturity needed to govern them. The second is the absence of structure and stable, shared governance: no designated owner, no success metrics agreed upon before launch, no criteria for deciding whether to continue.

This article proposes a method for building an AI ITSM Pilot that produces verifiable results and allows useful evidence to be gathered from the early stages of the pilot to build and validate the business case.

Starting from Low-Risk Use Cases

The correct choice of use case largely determines the success of the pilot project. A low-risk use case has three characteristics: results can be verified by a person in a few seconds, errors produce limited and reversible consequences, and the volume of requests handled is sufficient to generate meaningful data within a few weeks.

Three activities meet these conditions.

Ticket triage. The system proposes the category, priority, and assignment team for each incoming request. The operator sees the proposal and confirms or modifies it with a single click. Each interaction generates useful data on the percentage of correct proposals.

Incident summarization. For incidents with many annotations, the system produces a summary of the status and actions taken. This is particularly useful during shift handovers, where manually reconstructing the history of a complex incident takes precious minutes away from resolution.

Knowledge base drafts. Using resolved tickets, the system generates a first draft of the article, which an editor reviews and approves. Creating content for the knowledge base remains one of the most sacrificed activities by operational pressure, and a contribution in this direction frees up qualified time.

These three activities share a valuable property for an AI ITSM Pilot: human review is already a natural part of the workflow, so oversight does not add artificial steps. Fully autonomous workflows, by contrast, can only be activated after the system has demonstrated consistent reliability over time.

Defining Metrics Before Launch

Before the pilot project begins, it is necessary to define three elements for each chosen indicator: the baseline value measured from historical data, the expected target at the end of the trial period, and the translation of that improvement into terms understandable to leadership. Generic formulas such as “greater operational efficiency” or “better IT support” do not survive even the first conversation with finance.

A decision criterion must also be defined in advance: which results justify moving to the next phase and which suggest stopping.Establishing this beforehand prevents the final evaluation from being reduced to a comparison of opinions about how the results are interpreted.

How Many Indicators and Over What Period?

The indicators that are genuinely useful are specific: reduction in non-significant reports, time taken to assign a ticket to the most appropriate team, percentage of correct assignments on the first attempt, average resolution time, quality of response to the most severe incidents, and the perception of the teams involved.

Furthermore, each indicator must be linked to a concrete consequence for the business, recognizable even outside of IT. For example: assignment time corresponds to the wait perceived by the user; the quality of knowledge base drafts affects the ability of service desk staff to close requests without involving specialist teams.

It is better to select two or three indicators, no more. Each metric requires a shared definition, an accurate initial measurement, and an interpretation at the end of the trial. An AI ITSM Pilot that monitors six reduces the care dedicated to each indicator, because it spreads attention across too many fronts.

Since volumes vary significantly throughout the year, measurement must be extended over a period long enough to include both peak periods and periods of lower load.

Responsibility and Governance

An AI ITSM Pilot without a designated owner stalls at the first obstacle. Ownership must be assigned to a specific individual, by name, who is accountable for results and has the authority to decide on any corrective actions.

Alongside ownership, three working tools are needed. The first is a technical map that describes the monitoring tools involved, the data sources used, the service relationships derived from the CMDB, the security controls applied, the boundaries established for the introduction of automation, and the integration points. The second is a dedicated risk register, in which to record identified issues, their priority, and the actions to be taken. The third is an operating model that establishes who maintains the data, who verifies the quality of recommendations, and how frequently the system is reviewed.

The technical map, the risk register, and the operating model only work if the organization has shared rules on the use of AI. In the absence of defined governance, each team adopts its own tools and conventions, resulting in a multiplication of divergences rather than a reduction.

Human-in-the-Loop Controls

Human oversight must be applied where the risk is highest, avoiding its indiscriminate application to every decision. Three elements must be defined and documented before the project begins: which system recommendations require operator approval before becoming operational, who is responsible for approving them, and where that approval is recorded.

It must also be defined what happens when the system makes an error: who detects the error, how the previous situation is restored, and within what timeframe. Defining this procedure during the experimental phase, when the consequences remain contained, avoids having to improvise it in production.

Why the Approval Rate Can Be Misleading

One consideration concerns the effort required to reject a proposal. If approval is obtained with a single action (one click), while correction requires manually re-filling fields, operators under pressure will tend to confirm even inaccurate recommendations.

The data on proposal quality ends up measuring service desk workloads more than system performance: if the percentage of approved proposals is used as an indicator of system accuracy, and if approving requires a quick action while correcting imposes manual work, that percentage will rise during periods of greater pressure on the service desk and fall when the load eases. The solution is to make corrections just as quick as approval — for example, by offering selectable alternatives instead of an empty field.

Introducing Autonomy Gradually

In an AI ITSM Pilot, the expansion of autonomy must proceed through successive versions, each contingent on the results of the previous one.

V1 – Suggestions with Human Review

The system proposes, the person responsible for process oversight decides. Each proposal from the AI agent is compared with the operator’s actual choice, and from this comparison the rate of agreement between the system’s proposal and the operator’s decision is derived. Agreement, however, does not automatically equate to accuracy: a sample of decisions must be verified separately to confirm their correctness.

V2 – Automation of High-Reliability Cases

Once the categories in which the system consistently achieves sufficiently high levels of reliability have been identified, automatic execution is activated only for those. The remaining requests continue to go through human review. The threshold must be set separately for each type of proposal: assigning a ticket requires different guarantees than changing a priority.

V3 – Extended Autonomy

The expansion proceeds as the system’s reliability grows and, consequently, as human operators’ trust increases. Monitoring of results is continuous, and there is always the possibility of rolling back automation when performance degrades.

A path of this kind follows a logic of gradual transition from experimentation to operation, whereby the observation phase precedes introduction within a controlled perimeter and, only thereafter, entry into daily operations.

From Pilot Project to Value Demonstration

The link between experimentation and return on investment (ROI) is built during the project. In other words, the data needed to demonstrate the value of the investment must be collected while the pilot project is underway. The historical data collected before launch constitutes the baseline against which to evaluate the project’s results. The data produced during the V1 phase is used to verify and calibrate the system’s behavior. And the improvements recorded in V2 become the evidence to present when requesting additional budget for the project’s expansion.

The same principles recur in the most widely adopted recommendations for introducing AI into service management: start from high-value use cases, ensure data quality, adopt appropriate tools, train people, experiment at a small scale, and grow gradually.

An AI ITSM Pilot structured in this way remains useful regardless of the outcome. If the project works, the organization will have verifiable data to justify its expansion. If the project does not work, at the cost of limited expenditure, the organization will know which gaps to address before trying again. The reflection that business decision-makers must undertake concerns the degree of maturity of their own organization more than the limitations of the technology.

FAQs

1. How long should an AI pilot project in ITSM last?
   As long as it takes to collect a sufficient number of observations on the chosen use case. For high-volume activities such as ticket triage, a few weeks are enough: shorter periods do not allow a stable result to be distinguished from an occasional variation.
   
2. Why is it advisable to avoid autonomous agents in the initial phase?
   Because control mechanisms, reliability thresholds, and defined responsibilities are needed — and these are built through experience. Starting with suggestions subject to review allows the data needed to calibrate those thresholds to be collected before expanding autonomy.
   
3. Who should lead the project?
   A single individual, supported by designated contacts for data, governance, and the operational teams involved. Shared ownership across multiple functions slows down decision-making processes at the very moment they need to be timely.
   
4. How is return on investment demonstrated at the end of the trial?
   By comparing the values measured before launch with those recorded during the project, against the indicators agreed upon at the outset, and converting the results into hours of work freed up and costs avoided.