Agent Evaluation

The Problem:

Testing traditional software is straightforward - given input X, expect output Y. But AI agents are non-deterministic (same input can give different outputs), multi-step (errors can cascade), and tool-dependent (real tools have side effects). This makes testing fundamentally different.

Real-World Analogy - Restaurant Quality Testing:

Testing a vending machine is easy - put in Rs 10, check if a Pepsi comes out. Testing a restaurant is harder:

• Was the food good? (subjective quality)
• Was it served in time? (performance)
• Was the waiter polite? (tone and style)
• Was the bill correct? (accuracy)
• Would you come back? (overall satisfaction)

Agent evaluation is like restaurant testing - multiple dimensions, subjective criteria, and the output varies each time.

What Makes Agent Testing Unique:

• Non-deterministic: Same prompt may give different responses each time
• Multi-step: Agent takes 5-10 steps. Error in step 2 can ruin step 5.
• Tool Side Effects: Agent might actually send emails or modify databases during testing
• Subjective Quality: "Is this response good?" depends on context, user, and taste
• Emergent Behavior: Agent might find creative solutions you did not expect (good or bad)

1. Task Completion (Did it achieve the goal?):

The most fundamental metric. Did the agent actually complete what was asked? For a travel booking agent, did it book the correct flight?

• Binary: Success/Failure for clear-cut tasks
• Partial: 0-100% for multi-part tasks (booked flight but wrong meal preference)

2. Correctness (Was the output accurate?):

Even if the agent completes the task, was the information correct? Did it calculate the EMI correctly? Did it cite the right legal section?

• Fact-checking against ground truth
• Hallucination detection
• Numerical accuracy

3. Efficiency (How well did it use resources?):

• Number of LLM calls (fewer is better)
• Number of tool calls (was it efficient?)
• Total tokens consumed (cost)
• Wall-clock time to completion
• Did it avoid unnecessary steps?

4. Safety (Did it follow guardrails?):

• Did it refuse harmful requests?
• Did it ask for human approval for sensitive actions?
• Did it avoid exposing personal data?
• Did it stay within its authorized scope?

5. User Experience (Was it pleasant to use?):

• Response quality and tone
• Did it ask for clarification when needed?
• Was it transparent about its reasoning?
• Latency - did the user wait too long?

1. LLM-as-Judge (Most Popular):

Use a strong LLM (GPT-4, Claude) to evaluate the agent's output. You provide a rubric and the judge LLM scores the response.

Pros: Scalable, handles subjective quality. Cons: Judge LLM has biases, costly for large eval sets.

2. Golden Dataset Testing:

Create a dataset of test cases with known correct answers. Run the agent on each case and compare outputs to expected results. Best for regression testing - ensuring new changes do not break existing behavior.

3. Trajectory Evaluation:

Instead of just checking the final answer, evaluate the entire sequence of actions the agent took. Did it use the right tools in the right order? Did it take unnecessary detours? This catches agents that get the right answer for the wrong reasons.

4. A/B Testing in Production:

Deploy two agent versions (A and B) and route real traffic to both. Measure which version performs better on key metrics (task completion, user satisfaction, cost per task). The gold standard for production optimization.

5. Adversarial Testing:

Deliberately try to break the agent with edge cases, prompt injections, contradictory instructions, and unusual inputs. If the agent handles adversarial inputs gracefully, it will handle normal inputs well too.

Evaluation Pipeline:

[Test Cases Dataset] (50-200 cases per agent)
        |
        v
[Agent Under Test] -- runs each test case -->
        |
        v
[Capture Full Trace]
  - User input
  - Each reasoning step
  - Each tool call + result
  - Final response
  - Token count, latency
        |
        v
[Multi-Evaluator Pipeline]
  |-- Exact Match Evaluator (for factual questions)
  |-- LLM Judge (for subjective quality)
  |-- Trajectory Evaluator (for multi-step correctness)
  |-- Safety Evaluator (for guardrail compliance)
  |-- Cost Evaluator (tokens, API calls)
        |
        v
[Aggregate Scores + Dashboard]
  - Overall score: 87/100
  - Accuracy: 92%, Efficiency: 78%, Safety: 95%
  - Regressions flagged in red
  - Cost per task: Rs 2.50 avg

Evaluation Frameworks:

• Ragas: Popular framework for RAG evaluation. Measures faithfulness, answer relevancy, context precision.
• DeepEval: LLM evaluation framework with 14+ metrics. Supports custom metrics.
• LangSmith: LangChain's evaluation platform. Trace-based evaluation with LLM judges.
• Braintrust: AI evaluation platform with experiment tracking and scoring.
• Promptfoo: Open-source prompt testing tool. Great for comparing different prompts and models.

Best Practices:

• Eval Early and Often: Run evals on every prompt change, model upgrade, or tool modification
• Diverse Test Cases: Include edge cases, adversarial inputs, and multi-language inputs
• Version Everything: Track which prompt version + model + tools produced which eval scores
• Human Spot Checks: LLM judges are good but not perfect. Regularly spot-check with human reviewers
• Regression Alerts: Set up alerts when eval scores drop below thresholds after changes

Multi-Step Agent Testing Strategies:

• Unit Test Each Tool: Test each tool independently with known inputs/outputs. Like unit testing functions before integration testing.
• Mock Tool Responses: Replace real tools with mock responses during testing. This makes tests deterministic and avoids side effects (no real emails sent, no real database changes).
• Trajectory Comparison: Define the expected sequence of tool calls. Compare the agent's actual trajectory against the expected one. Allow some flexibility (order may vary) but flag missing or extra steps.
• End-to-End Testing: Test the complete agent with real (sandboxed) tools. Verify the final outcome is correct. More realistic but slower and harder to maintain.

Dealing with Non-Determinism:

• Temperature 0: Set temperature to 0 for eval runs to reduce randomness (but does not eliminate it completely).
• Multiple Runs: Run each test case 3-5 times and use the median score. This smooths out randomness.
• Semantic Comparison: Instead of exact string matching, use semantic similarity to compare outputs. "Rs 2,847" and "INR 2847" should both be considered correct.
• Assertion-Based: Check for specific assertions rather than exact output. "Response mentions Reliance price AND gives a buy/sell recommendation."

Q: Why is evaluating AI agents harder than testing traditional software?

Q: What is LLM-as-Judge evaluation and what are its limitations?

Q: What five metrics would you track for a production AI agent?