Back to Blog
Blog

Can historical jobs test whether AI would have recommended the same procedure changes as experts?

Can historical jobs test whether AI would have recommended the same procedure changes as experts?

Historical jobs can test whether AI would have recommended the same procedure changes as experts, and this is one of the most convincing ways to test…

Historical jobs can test whether AI would have recommended the same procedure changes as experts, and this is one of the most convincing ways to test procedure AI, since you already know what the experts decided. You give the AI the same information the experts had at the time, like reports, incidents and lessons learned, and compare its recommendations with the changes that were actually made.

The appeal of this approach is that it replaces abstract claims with a direct comparison on your own work, using your own documents, so the people who will eventually rely on the AI can judge its usefulness against decisions they recognize and respect. It also avoids the problem of evaluating AI on invented examples, which tend to be cleaner and easier than the messy reports and partial information that real procedure reviews are based on.

Setting up the test takes some care, since the AI must only see what was available before the change was made and not the revised procedure or any later documents that reveal the answer. A good test set includes a range of cases, such as changes prompted by incidents, by equipment modifications and by near miss reports, along with some cases where the experts reviewed the information and decided no change was needed, since recommending changes where none were warranted is its own kind of error.

The comparison shows where the AI agrees with experts, where it misses something and where it suggests changes the experts did not make, some of which may turn out to be useful. Those extra suggestions deserve a proper look by someone who knows the procedure, because a few may point to issues the original review overlooked, while others will reveal where the AI lacks site context and needs better information or narrower instructions.

It is a safe way to build confidence before the AI is used on live procedures, since nothing it recommends during the test affects real work. Keeping the results, including the misses, also gives a baseline for checking whether later versions of the AI or its configuration actually improve, and it gives procedure owners concrete examples to discuss when deciding how much weight to give its recommendations.