Quality Control: Testing and Training

Quality Control: Testing and Training

Building an AI agent is often easier than ensuring it works consistently. Unlike traditional software, where 1 + 1 always equals 2, Large Language Models are probabilistic. They can be brilliant one moment and hallucinate the next. In this chapter, we explore how to professionalize your workflow using CrewAI’s built-in quality control tools: the Testing framework and the Training mechanism.

Evaluating Performance with crewai test

How do you know if a change in your prompt improved the result or made it worse? You cannot rely on a single run. The crewai test command allows you to run your crew multiple times systematically to gather metrics on performance.

Testing in CrewAI focuses on benchmarking. When you run a test, the framework executes your crew for a specified number of iterations and aggregates data regarding task duration and execution results. This is comparable to running a stress test on a vehicle prototype.

Running the Test

To execute a test, you use the CLI. You must specify the number of iterations (-n) and the model to be used for evaluation (-m). Note that CrewAI uses an LLM to evaluate the quality of the output produced by your agents.

# Run the crew 3 times and evaluate using GPT-4o
crewai test -n 3 -m gpt-4o

After the command completes, CrewAI will output scores for each task based on semantic consistency and goal completion. This provides a quantitative baseline. If you modify an agent’s backstory and the score drops from 8.5 to 7.0, you know you’ve moved in the wrong direction.

Fine-tuning Behavior with crewai train

While test gives you scores, train allows you to actively improve the agents. The crewai train command facilitates a “Human-in-the-Loop” reinforcement learning process. It runs your crew and then pauses to ask for your feedback on the specific outputs.

This is not technical fine-tuning (updating the LLM weights directly) but rather context optimization. CrewAI saves your feedback and uses it to guide the agents in future executions, effectively building a repository of “good” vs. “bad” examples (few-shot learning context).

The Training Workflow

• Execute: Run crewai train -n <iterations> in your terminal.
• Review: The CLI shows the output of a task.
• Feedback: You are prompted to provide written feedback. For example: “The tone was too informal,” or “You missed the citation in the second paragraph.”
• Persist: CrewAI saves this feedback into a local training file (often a .pkl file).

# Train the crew with 5 iterations of feedback
crewai train -n 5

In subsequent runs, agents will access this training data to understand your preferences better, leading to improved alignment with your expectations over time.