Change and Incident Trust Pilots Should Replay the Past Before the Present

An agentic pilot can produce confident recommendations from records that missed a dependency during a previous change. A successful demonstration may leave that gap undiscovered until someone relies on the same records during an incident.

A change and incident trust pilot gives the IT Director another way to evaluate the underlying data. It compares records available at the time of closed events with evidence of what happened. A reviewer can then identify where configuration items, relationships, or ownership details supported the decision, and where they fell short.

The distinction matters beyond agentic IT. The 2025 DORA report on AI-assisted software development describes AI as an amplifier of existing organizational strengths and weaknesses. Its research concerns software delivery, so it does not establish how AI affects IT service management. It does suggest a useful evaluation principle: examine the conditions surrounding a tool before relying on its output.

What a Trust Pilot Tests, and What It Leaves Alone

A trust pilot examines whether records available to an operational decision reflected the relevant configuration at that moment. The reviewer works from closed changes and incidents, where an outcome and supporting evidence may already exist.

For each case, compare the ticket with the historical configuration management database (CMDB) record. A configuration item (CI) is an asset or service component tracked for operational purposes. Review its recorded relationships and ownership alongside the evidence from the closed event.

The reviewer scores the evidence, rather than asking an agent to approve a change or route an incident. Existing change and incident processes continue under their established owners.

Explore Trusted Runtime Truth to see how discovery-sourced records can help teams examine what exists, how components connect, and who owns them.

A historical comparison needs more than a current map. The reviewer needs records for the relevant time, including the last discovery cycle before the event. A later discovery result can help explain what changed, but it should not be presented as evidence of what the team could see earlier.

Why Most Pilots Measure the Wrong Thing

A pilot can demonstrate that an agent produces a plausible answer while leaving the quality of its input unexamined. The evaluator needs separate evidence for each part of the decision.

Four Choices That Weaken the Result

1. Score the agent’s answer without examining its source record. A recommendation may match the eventual outcome for reasons the reviewer cannot establish. Keep a copy of the information available at the decision point, then compare that information with independent evidence.

Result: The team learns that an answer looked useful, but cannot explain whether the underlying record supported it.

2. Begin with active production decisions. A current change has an unknown outcome when the team first evaluates it. Closed cases give reviewers an outcome to investigate before a pilot result influences operational work.

Result: The team has less evidence for judging the recommendation when it first appears.

3. Rely on one average accuracy score. A record can contain many correct attributes while missing a relationship needed for a particular change. Score identity, relevant relationships, and ownership separately, then retain case-level findings.

Result: A high aggregate score can conceal the specific gap that affected a decision.

4. Agree on the pass bar after seeing results. Reviewers may disagree about whether an unknown owner or missing dependency should change the outcome. Set those rules before cases are scored.

Result: The sponsor receives a disputed score rather than a decision they can defend.

For change decisions, relationship coverage in the CMDB deserves its own measure. A correct server name helps identify the target, while its recorded connections help reviewers understand the surrounding service.

The same discipline applies to incident records. A ticket may identify the affected component correctly while leaving the person responsible for it unclear. Those findings need separate entries in the pilot scorecard.

What an Untested Record Costs Each Seat

The cost of an untested record depends on the decision each person must make. A missing relationship may affect a change review differently from an incident handoff.

For the IT Director: A promotion decision can rest on a demonstration that never examined the records behind its answers. The Director then has limited evidence for explaining where the tool should be used.

For the change manager: A review may depend on a colleague remembering a service relationship that the record does not show. The approval discussion can become harder when that colleague is unavailable.

For the incident commander: Responders may spend the opening phase checking whether the ticket identifies the affected CI. That verification competes with diagnosis when a service is already disrupted.

For the change advisory board: Members may spend their meeting resolving disagreements about the asset record before discussing the proposed change. Unresolved questions can delay a decision or return it to the requester.

For an audit reviewer: A team may struggle to show how it evaluated its records before relying on them. A dated scorecard and supporting evidence provide a clearer account of the decision.

These are risks to examine in the selected cases, rather than outcomes to assume for every organization. A pilot should record when evidence supports a concern and when it does not.

Three Rules for a Pilot Designed to Keep Production Decisions Separate

A records pilot can borrow evaluation discipline from release testing. Google Cloud’s canary analysis guidance recommends checking an analysis against known good and known bad versions. Its guidance concerns software releases, so the following application to records is an evaluation design, not a finding from that article.

1. Calibrate Against Known Record Quality

Choose one case whose relevant record is supported by independent evidence. Choose another where a specific record defect is already documented. Ask the reviewer to score both using the proposed criteria.

The known good case should pass the checks it was selected to demonstrate. The known bad case should expose the documented defect. If either result is unexpected, inspect the scoring instructions before evaluating the remaining cases.

A successful change does not automatically prove its record was accurate. An unsuccessful change does not automatically prove its record was defective. The calibration concerns record quality, supported by evidence, rather than whether an event ended well or badly.

For example, a closed change may have caused disruption even though its dependency record was correct. The disruption could have resulted from an execution error. Treating the incident outcome alone as proof of bad data would produce a misleading test.

2. Keep the Review Separate From Operational Decisions

Give a named reviewer access to the approved case materials and a separate scorecard. The reviewer records findings for the pilot sponsor. Existing teams retain responsibility for approvals, incident routing, and production actions.

Agree how case materials will be obtained and protected before the review begins. Ticket exports or other approved records may be suitable if they contain the required history. Confirm the available access method and permissions for the systems involved.

The pilot result should inform a later decision rather than change an active ticket. If the team wants to test an agent afterward, that requires its own scope and review criteria.

3. Set the Pass Criteria Before Scoring

The Director and sponsor should agree on what constitutes a pass before the reviewer sees the evaluation cases. Define separate measures for CI identification, relevant relationships, and ownership.

Each measure needs a denominator and an evidence standard. For example, the relationship measure might compare documented, relevant relationships with those recorded at the time. The sponsor should also decide how to handle cases where the evidence is incomplete.

Do not count an unverified field as correct merely to complete the score. Mark it as unknown and report it separately. A sponsor can then distinguish a record defect from an evidence gap.

When Replay Is the Wrong Choice

Replay depends on enough historical information to reconstruct the record available during a past event. An estate with little history, or tickets without usable historical links, may not support a defensible comparison. In that case, review new changes in shadow mode and compare the saved record with the outcome after closure.

Three Cases a Replay Pilot Could Catch

The following examples are illustrative scenarios. Each shows a different finding that a reviewer could record without assuming that discovery data alone establishes the full outcome.

A dependency identified after implementation: A change record names the application and database. The closed-event evidence identifies an additional connection that affected the service. The reviewer checks whether that connection appeared in the time-matched CMDB record and whether the change record included it.

An ownership field left blank: An incident record identifies the affected CI, but its historical ownership field is empty. The resolution notes identify the team that handled the issue. The reviewer records the missing ownership field and asks the responsible people to confirm the appropriate owner.

Two records for one server: A change references one server record, while supporting evidence points to a duplicate entry. The reviewer uses a durable identifier, such as a serial number, to investigate the match. A hostname alone is not sufficient evidence of a durable match.

Each example needs supporting evidence beyond a ticket summary. If the event record and CMDB disagree, the reviewer should document the disagreement and identify what would resolve it. A finding can remain open when the available evidence does not support a confident judgment.

Virima Gives the Pilot Discovery-Sourced Records to Compare With Tickets

Virima discovery and the CMDB give teams records to examine alongside change and incident evidence. The records reflect the most recent scheduled discovery cycle available for the relevant scope. Reviewers should account for that timing when comparing them with a past event.

Before the pilot: Select one business service and identify the CIs included in the evaluation. Provide the service definition so ViVID™ can build a dependency map from the supplied information. Confirm which historical records and supporting evidence are available for the selected period.

During the pilot: A reviewer compares the historical CI, relationship, and ownership information with the closed case. The reviewer also checks the documented outcome against independent evidence. People confirm ownership; the CMDB records and presents the information available to the team.

At the decision: The Director takes the scorecard to the sponsor with its evidence gaps and unresolved cases. The sponsor can decide whether the records support further evaluation, need correction, or require a different scope.

Virima supplies discovery-sourced CMDB and service-mapping information for the assessment. The pilot team selects cases, judges the evidence, and decides what action follows. A discovery result is one input to that judgment, particularly when an event occurred between discovery cycles.

Five Decisions to Make Before Day One

The Director can put the pilot plan on one page, with supporting evidence held separately.

1. Choose one service and its scope. Identify the business service, its defined components, and the period to examine. Record known gaps in discovery coverage before selecting cases.
2. Select closed cases with usable evidence. Include a known good record and a known defective record for calibration. Select additional cases with documented outcomes and time-matched CMDB information.
3. Name the reviewer and evidence owners. Assign someone to score each case and identify who can resolve questions about its history. Keep ownership confirmation with the people responsible for the service.
4. Write and sign the pass bar. Define the CI, relationship, and ownership measures separately. Record how the team will handle unknown evidence and material defects.
5. Set a stop rule and decision owner. Specify which findings end the pilot or require a pause. Name the person who can make that call and record the reason.

A useful scorecard includes the case identifier, event date, discovery-cycle date, expected evidence, reviewer finding, and supporting source. It also records whether the finding is correct, defective, or unknown under the agreed criteria.

Keep calibration cases identifiable in the final report. The sponsor should be able to see how the scoring method performed before judging the wider set. Retain disagreements and their resolution rather than replacing them with a final average.

Bring the Evidence to Your Next Evaluation

A focused demonstration can help your team examine discovery, CMDB records, and service mapping for a selected service. Bring the questions your reviewers need answered about record history, discovery timing, and the evidence available for closed cases.

Request a Virima demo to discuss the records and service relationships your team wants to evaluate.

Frequently Asked Questions