Start with a decision, not a model
Name the work you want to improve and the person accountable for the result. “Add a chatbot” describes an interface. “Help service staff locate an approved procedure with a source reference” describes a task you can evaluate. Compare the task with search, rules or simpler automation before introducing a model.
Find the information boundary
Inventory the documents and systems the workflow needs. Record their owners, access rules, update frequency and retention requirements. A retrieval system must respect the user’s permissions when selecting sources; filtering the final answer alone is insufficient.
Build an evaluation set before the demo
Collect representative questions, difficult edge cases and questions the system should decline. Define what a supported answer looks like. Review retrieval relevance, citation correctness, unsupported statements, latency and cost separately so one average score does not hide a critical failure.
Keep consequential actions reviewable
Separate proposing an action from executing it. A draft response, suggested classification or proposed update may need a human approval gate. Validate structured outputs and apply application permissions before any downstream change.
Make the pilot a decision point
Document the baseline, test population, observation period and acceptance gate. Decide whether to expand, change the use case or stop based on observed performance. A polished demonstration is useful for learning; it is not evidence of production reliability.
Prepare for an AI readiness conversation
- Which task should improve, and how will you compare it with the current process?
- Who owns the source information and authorizes access to it?
- Which errors need review, and which actions must remain under human approval?
A useful starting brief identifies the users, sources, example questions and consequences of an incorrect answer. Use this to define a bounded evaluation before discussing a wider rollout.
Applied AI & intelligent systemsFurther reading
NIST AI Risk Management FrameworkOWASP GenAI Security ProjectAn editorial perspective for project planning. Requirements and implementation choices depend on your organization and systems.