Lab 4: Evaluating a Task-Specific Agent

In this lab, you will evaluate the SLA Agent you built in Lab 2. Instead of repeating multiple HTTP request, you will define a small evaluation dataset, run it automatically with the bat eval tooling, and read deterministic and LLM-judge scores for every task.

This lab assumes the Lab 2 SLA Agent already exists and is well-formed (the sla-agent/ project, with its send_sla_policy tool and SLAClient).

What You Will Do

• Scaffold an evaluation harness inside the agent project with bat eval init
• Write a dataset of tasks that probe the agent's expected behaviour
• Run a deterministic evaluation (status, tool calls, substring checks)
• Add qualitative scoring with an LLM judge
• Visualise results with bat eval plot

Prerequisites

• Completed Lab 2: a runnable sla-agent/ project
• bat-cli installed (the bat command is available)
• An LLM api key or a deployed local model with Ollama.
• Familiarity with the agent's behaviour (see the recap below)

Recap: The Agent Under Test

The SLA Agent maps a high-level user intent to a concrete SLA policy and pushes it to the network's Non-RT RIC. Its single tool is:

send_sla_policy(network_name: str, desired_sla: "low_throughput" | "high_throughput")

Expected behaviour, from its system prompt:

• Intent → policy: heavy use (4K video, professional video calls) ⇒ high_throughput; light use (web browsing, "cheap internet") ⇒ low_throughput.
• Missing info: if the user does not say which network, the agent asks instead of guessing.
• Out of scope: politely declines requests unrelated to SLA management.
• Result: confirms success, or gives a basic explanation on failure.

These four behaviours are exactly what the dataset below checks.

Step 1 — Scaffold the Evaluation Project

From the agent project root (labs/lab2/sla-agent/):

bat eval init

This creates:

sla-agent/
└── eval/
    ├── eval.yaml          # evaluation configuration
    ├── input/
    │   └── tasks.json     # the dataset (one entry per task)
    └── output/            # results land here after a run

You will replace the generated placeholders with the eval.yaml and tasks.json provided alongside this lab. Copy them from here: eval.yaml and tasks.json.

Step 2 — Write the Dataset

A dataset is a JSON array of tasks. Each task describes one conversation and what a correct agent should do. Anatomy of a task:

Field	Meaning
id	Unique task name (also used in the output charts)
turns	List of user messages; more than one ⇒ a multi-turn conversation
expected.status	Final task status to assert (completed, …). Use null to skip the check
expected.expected_outcome	Description of success — scored by the LLM judge
expected.output_must_contain	Substrings that must appear in the final answer (deterministic)
expected.tool_calls	Tools the agent should call

Copy the provided dataset into the agent:

cp ../../lab4/eval_sample/input/tasks.json eval/input/tasks.json
cp ../../lab4/eval_sample/eval.yaml        eval/eval.yaml

The dataset contains six tasks, one per behaviour we care about:

Task id	What it checks
sla_high_throughput_4k	4K intent ⇒ send_sla_policy(..., high_throughput)
sla_low_throughput_browsing	Light-browsing intent ⇒ low_throughput
sla_high_throughput_videocall	Video-call intent ⇒ high_throughput
sla_missing_network_asks	No network given ⇒ agent asks instead of calling the tool
sla_out_of_scope_guardrail	Off-topic request ⇒ polite refusal, no tool call
sla_multi_turn_intent	Asks for the network, then applies high_throughput

The three intent-mapping tasks assert on tool_calls, so they pass without a cluster: we only require that the agent translated the intent into the right desired_sla and network_name. The missing_network and guardrail tasks assert on the absence of a tool call plus the wording of the reply.

Step 3 — Understand eval.yaml

evaluation:
  dataset: eval/input/tasks.json
  output_dir: eval/output
  agent_url: http://127.0.0.1:9900 # must match PORT in the agent's .env
  agent_startup_timeout_s: 45
  k: 1 # replays per task (raise to measure consistency)
  qualitative: false # Phase 1: deterministic only

judge: # used only when qualitative: true
  provider: openai
  model: gpt-4.1-mini
  api_key_env: OPENAI_API_KEY

models: # the agent is evaluated once per model
  - provider: openai
    model: gpt-4.1-mini

Check that everything resolves before running:

bat eval show

Step 4 — Run the Deterministic Evaluation (Phase 1)

With qualitative: false, the engine starts the agent, replays every task, and applies only the deterministic verdict: status, output_must_contain, and tool_calls.

bat eval run

bat eval run starts the agent for you (uv run .), waits for its port, runs the dataset, then stops it.

Results are written under eval/output/<run>/, including a metrics.json with the per-task verdict and the captured trace.

Why deterministic checks matter: a task can return the right status for the wrong reason. Asserting tool_calls (and, where relevant, output_must_contain) pins down why a task passed, not just that it did.

Step 5 — Add Qualitative Scoring (Phase 2)

Deterministic checks confirm the agent did the right action. To also judge how good the natural-language answer is, enable the LLM judge:

evaluation:
  qualitative: true

Make sure the judge's API key is reachable (here via OPENAI_API_KEY), then re-run:

bat eval run

Now each task is additionally scored by the judge against its expected_outcome. This is the right tool for the guardrail and missing_network tasks, whose "correctness" is about tone and clarity rather than an exact string.

The qualitative phase actually runs four independent judges per task — each one specialised on a different aspect of the answer:

Judge name	What it scores
relevance	Does the agent stay on the topic the user raised?
task_completion	Did the agent achieve the expected_outcome?
hallucination	Are the agent's specific claims grounded in user input / tool output?
tool_call	Were tool calls used appropriately for the request?

These show up in metrics.json under qualitative.{response_relevance, task_completion_quality, hallucination_score, tool_call_appropriateness}.

Step 6 — Read the Output

A run lands in eval/output/<run>/, where <run> encodes the timestamp and the evaluated model. The two files you will look at the most are:

eval/output/<run>/
├── metrics.json     # aggregated + per-task scores (start here)
└── episodes/        # per-task traces (full conversation + tool calls)

metrics.json is the main artefact. Each task contributes one entry under episodes, and the file also carries an aggregate summary. Per-task fields:

Field	Meaning
task_id	The id from your dataset
status	Final task status returned by the agent
success	true ⇔ all enabled checks passed (deterministic + qualitative if on)
verdict.passed / verdict.reason	Outcome of the deterministic verdict, with a short reason on failure (e.g. "missing tool call: send_sla_policy" or "output_must_contain not satisfied: ['network']")
time.wall_ms	Wall-clock time the agent took for that episode
tokens.{prompt,completion,total}	Token usage reported by the agent's LLM
qualitative.*	Present only when qualitative: true; one float score per judge plus judge_reasoning

A trimmed example:

{
  "task_id": "sla_high_throughput_4k",
  "status": "completed",
  "success": true,
  "verdict": { "passed": true, "reason": "" },
  "time": { "wall_ms": 1842 },
  "tokens": {
    "prompt_tokens": 612,
    "completion_tokens": 84,
    "total_tokens": 696
  },
  "qualitative": {
    "response_relevance": 1.0,
    "task_completion_quality": 0.9,
    "hallucination_score": 1.0,
    "tool_call_appropriateness": 1.0,
    "judge_reasoning": "Agent applied the correct high_throughput policy on bubbleran1."
  }
}

How to debug a failing task. Look at verdict.reason first — it tells you which deterministic check failed. Then open the matching trace under episodes/ to see the full conversation and the actual tool calls the agent emitted; compare them with the tool_calls block in your task to spot the divergence.

Step 7 — Customize the Judge Prompts (optional)

It is possible that the judges will not behave as expected, for example in our dataset the last query is out of scope and the task relevance judge may be prone to give a low score because the agent will evade the question. The four judges share a fixed scoring rubric (you should not rewrite it — comparisons across runs would stop making sense). What you can do is inject agent-specific context that helps the judge disambiguate domain terms. This goes under judge.prompts in eval.yaml:

judge:
  provider: openai
  model: gpt-4.1-mini
  api_key_env: OPENAI_API_KEY
  prompts:
    relevance: |
      The agent's only legitimate topic is enforcing SLA policies on a 5G network
      via `send_sla_policy`. Treat questions about RIC, slices, throughput or
      "policies" as on-topic; treat anything else (weather, recipes, code, …) as
      a clear off-topic detour.

Only four keys are accepted: relevance, task_completion, hallucination, tool_call. Each value must be a string of at most 1000 characters, and is appended to the judge's system message as an AGENT-SPECIFIC CONTEXT block — the scoring scale itself stays fixed. Anything longer is rejected by bat eval show, so it pays to validate before running.

Step 8 — Visualise the Results

bat eval plot --folder eval/output

This generates summary and per-task charts from every run found under the folder. Use --filter <substring> to focus the per-task charts on a subset of task ids, e.g. --filter intent.

Exercises

1. Add a negative intent task. Write a task where the user clearly wants light usage but phrases it confusingly; verify the agent still chooses low_throughput.
2. Measure consistency. Set k: 5 and re-run. Do the intent-mapping tasks pass every time, or does the agent occasionally pick the wrong SLA?
3. Compare models. Add a second entry under models: and compare the scores of two LLMs on the same dataset.

What You Learned

• Turn an agent's expected behaviour into a structured evaluation dataset
• The difference between deterministic checks (status, output_must_contain, tool_calls) and qualitative LLM-judge scoring
• Assert on tool calls to test intent→action mapping without external infrastructure
• Run, inspect, and visualise evaluations with bat eval init / run / show / plot