How to scope an AI workflow for production
A production AI workflow needs more than a useful answer from a model. Define the task, the permitted data, the actions it may take and the person accountable for the result. Then test the complete workflow, including missing information and failed integrations, before expanding its use.
Choose a task with a clear finish line
Document what arrives, what a person does today and what counts as completion. Preparing a draft response from approved documents is easier to scope than asking an assistant to handle operations. Identify the volume, exceptions and cost of errors. If fixed rules can reliably solve the task, conventional automation may be the simpler choice. AI is most useful when interpreting variable text or preparing material for review.
Separate finding information from taking action
Retrieval augmented generation, or RAG, supplies relevant material to a model when it prepares an answer. It does not itself establish whether the user is allowed to see that material. Apply access rules before retrieval, retain source references and define what happens when sources disagree. Reading a record and changing it should be separate permissions. In a first release, consequential changes can remain behind human approval.
Test the exceptions before release
Create an evaluation set from representative, authorized examples and keep a separate set for final validation. Agree the expected result before inspecting the model output. Score factual accuracy, useful source references, correct escalation and successful completion of the wider task. Repeat the checks after changing the model, prompt, retrieval settings or tools. A fluent answer does not prove that the surrounding workflow completed correctly.
- Try missing, outdated and conflicting documents.
- Test requests outside the user’s permissions and instructions embedded in retrieved documents.
- Simulate a timeout, a repeated request and a failed write to the destination system.
Agree who operates the system
Before launch, assign an owner for access, incident handling and quality review. Record the cost and latency of a completed task, including human review and retries. Document how to stop the workflow, recover a failed item and return to a manual process. Define retention for inputs, outputs and logs. These choices belong in the scope, alongside the integration and the model.