Practical AI · 3-minute read
The demo has everything: immaculate data, a reassuring presenter and an output that arrives before anyone can ask about the setup. Somewhere in the deck, the word “transformative” is doing unpaid overtime.
I like a good demonstration. It shows what might be possible. The next question is what happens when the tool meets your actual work, including the spreadsheet maintained by three departments with four definitions of “complete.”
Before calling something revolutionary, I want to know what changes on Monday. Who does a task differently? What gets better? What new work or risk comes with it?
Give the pitch an ordinary job
Meet Cinder Mesa Field Services, a fictional oilfield maintenance company invented for this example. Its office team is evaluating a tool that promises to turn incoming maintenance requests into draft job briefs for human review. The demo uses a clear email with every detail included. The company also receives vague messages, contradictory attachments and the occasional “same as last time” from a customer whose last time predates the current filing system.
A useful trial needs examples from that range, using fictional or appropriately approved material. Ask the tool to handle a complete request, a missing detail and a conflict. Notice whether it asks for clarification, flags uncertainty or quietly makes a decision nobody authorized.
NIST's AI Risk Management Framework calls for evaluating performance under conditions resembling the intended setting. Your own task is a better starting point for that conversation than the most impressive slide.
Ask five questions before adding another login
What exact task improves? Name a repeatable activity and the person doing it. “Increase productivity” needs to come down from the balcony and meet a verb.
What information does it need? Check whether your team has suitable inputs and permission to use them. Find out which connections, access rights and setup the demonstration depended on.
What happens when it is wrong? Try missing and conflicting information. Decide which errors a reviewer must catch and which would make the use case unacceptable.
What does the whole job cost? Include preparation, subscription, training, checking, corrections and ongoing upkeep. Measure against the existing process using comparable tasks.
Who owns it after the demo? Name the person responsible for the workflow, the review and the decision to continue, change course or stop.
Turn enthusiasm into a small test
Write down the claim, the task, the current baseline and the evidence you would need before expanding. Define what “good enough” means before seeing the results. Otherwise, a committee can become remarkably flexible about the meaning of success.
Help me design a small evaluation of this AI claim: [claim]. Our task is [task], completed by [role]. Our current process is [process]. Separate vendor claims from evidence we actually have. Suggest realistic test cases, including incomplete and conflicting inputs; measures for quality and total effort; questions about data access and setup; and clear reasons to stop. Flag unknowns. Do not assume the tool has a feature unless we have verified it.
That prompt produces a draft test plan, not a verdict. Check current features, terms and costs with the vendor, and have the people doing the work review the test. A small successful trial supports the next decision; it does not prove the tool will work everywhere.
The result might be a useful purchase. It might be a better process using something you already own. Either outcome beats a revolutionary subscription quietly renewing while everyone goes back to the spreadsheet.
Your turn: What would an AI tool have to improve in your actual work before you would call it worth keeping?
Source: NIST AI RMF Playbook, MEASURE 2.3, for testing in context. The five-question review and fictional oilfield-service-company scenario are original editorial guidance.
