Version: v6.0.0 [Falcon]

Lab 5: Evaluate your agent

In Labs 2 to 4 you checked your agent by asking it a question and reading the answer. In this lab you write a small test set of questions, and bat asks them for you, checks every answer, and gives each one a score. You run it again after each change to your agent or its LLM, and you see at once what got better and what broke.

By the end of this lab you will be able to:

write a test set for an agent, with the answer you expect for each question;
run it with bat eval, and read the results;
check exactly what your agent did: which tools and agents it called;
add an LLM judge that scores the quality of each answer;
measure if your agent always answers the same way, and compare two LLMs.

Before you start, you need:

the fabric observability-lab of Lab 1, running;
the blueprint tutorial/ of Lab 4, with the Throughput Checker of Lab 3;
the prerequisites for Labs 2 to 5.

⚠️ An evaluation spends LLM credits. Every question runs your agent, which calls your LLM and the KPI agent, which calls it too. The judge of Part 6 adds up to four LLM calls per question. Keep the test set small while you learn.

Part 1: The idea

   tasks.json: the questions, and what a right answer looks like
        │
   bat eval run: starts your agent, then asks it each question
        │
        ▼
   Throughput Checker ───► KPI agent, in the cluster (through the tunnel)
        │
        │  its answers, and its traces: every LLM call, tool call and agent call
        ▼
   the checks: rules you wrote,           the judge: an LLM that scores
   always the same result                 each answer from 0 to 1
        │                                          │
        └──────────────► metrics.json ◄────────────┘

There are two ways to score an answer:

The checks are exact rules: the answer contains o-ran, the agent called the tool check_target, it asked the KPI agent. A task passes when all its checks hold.
The judge is an LLM that reads the question, what the agent did and its answer, and gives scores from 0 to 1: is the answer on topic, does it do what was asked, did the agent invent anything. It sees what rules cannot, but it costs LLM calls and its scores vary a little.

bat knows what your agent did from its traces, the same record of steps that Phoenix shows. During the run, it writes them to a file next to the results. You don't need Phoenix for this lab.

Part 2: Get ready

Step 1: Open the tunnel to the KPI agent

Your agent asks the KPI agent for each question, as in Lab 3. Start the port-forward of Lab 3, Part 2 (Steps 1 to 4), and keep its terminal open.

Step 2: Stop your own agent

bat eval run starts your agent by itself, on port 9900. If your agent still runs from Lab 3 (make throughput_checker), stop it with Ctrl+C.

Part 3: Write the evaluation

Step 1: Create the evaluation folder

In the tutorial/ folder, run:

bat eval init throughput_checker

You should see:

Created /home/you/tutorial/throughput_checker/eval/eval.yaml
Created /home/you/tutorial/throughput_checker/eval/input/tasks.json
Evaluation scaffold ready in /home/you/tutorial/throughput_checker/eval

throughput_checker tells bat which agent of the blueprint to evaluate. Each agent has its own eval/ folder:

throughput_checker/
└── eval/
    ├── eval.yaml          how to run the evaluation
    ├── input/
    │   └── tasks.json     the test set
    └── output/            the results of each run

Step 2: The settings: `eval.yaml`

Replace the content of throughput_checker/eval/eval.yaml with:

throughput_checker/eval/eval.yaml (click to open)

evaluation:
  dataset: eval/input/tasks.json
  output_dir: eval/output
  k: 1
  qualitative: false

judge:
  provider: openai
  model: gpt-6-luna

models:
  - provider: openai
    model: gpt-5.6-luna

Part	Meaning
`dataset`, `output_dir`	Where the test set and the results are, from the agent's folder.
`k`	How many times each question is asked. `1` for now; Part 7 raises it.
`qualitative`	`false`: only the checks. Part 6 turns the judge on.
`judge`	The LLM that scores the answers, when `qualitative` is `true`. It is not the LLM you test, so it does not score its own answers. It uses the `OPENAI_API_KEY` of your `.env`.
`models`	The LLMs your agent is evaluated with: one run per model. `bat` sets it in your agent for the run, instead of the `model` of `config.yaml`.

Step 3: The test set: `tasks.json`

The test set is a list of tasks. Each task is one question, and what a right answer looks like. Replace the content of throughput_checker/eval/input/tasks.json with:

throughput_checker/eval/input/tasks.json (click to open)

[
  {
    "id": "target_check",
    "turns": ["Does network o-ran meet its throughput target?"],
    "expected": {
      "status": "completed",
      "expected_outcome": "The agent gets the downlink throughput of o-ran from the KPI agent, compares it with the target with check_target, and answers MEETS TARGET or BELOW TARGET, with the measured value and the target.",
      "output_must_contain": ["o-ran", "TARGET"],
      "agent_calls": [{ "agent": "KPI Agent" }],
      "tool_calls": [{ "name": "check_target" }]
    }
  },
  {
    "id": "target_check_reworded",
    "turns": ["Is the downlink throughput of o-ran above the target?"],
    "expected": {
      "status": "completed",
      "expected_outcome": "Same as target_check: the agent answers MEETS TARGET or BELOW TARGET for o-ran, with the measured value and the target.",
      "output_must_contain": ["o-ran", "TARGET"],
      "agent_calls": [{ "agent": "KPI Agent" }],
      "tool_calls": [{ "name": "check_target" }]
    }
  },
  {
    "id": "unknown_network",
    "turns": ["Does network atlantis meet its throughput target?"],
    "expected": {
      "status": "completed",
      "expected_outcome": "There is no network atlantis, so the KPI agent gives no throughput. The agent does not call check_target, answers UNKNOWN and says why, without inventing a number.",
      "output_must_contain": ["UNKNOWN"],
      "agent_calls": [{ "agent": "KPI Agent" }]
    }
  }
]

What each part of a task means:

Part	Meaning
`id`	The name of the task, in the results.
`turns`	The messages sent to the agent, one per turn. Your agent does not remember earlier messages (`checkpoints: false`), so each task has one.
`expected.status`	The status the agent must end with: `completed`, `input-required` or `error`. `null` skips this check.
`expected.output_must_contain`	Words the answer must contain. Upper and lower case count: `TARGET` is in `BELOW TARGET` and `MEETS TARGET`, not in `UNKNOWN`.
`expected.agent_calls`	The agents yours must call, by the name on their agent card (`KPI Agent`). `times` sets how many times at least, `1` if you leave it out.
`expected.tool_calls`	The tools your agent must call, also with `times`. `args_subset` can also check the values the LLM gave the tool, for example `{"measured_mbps": 42}`.
`expected.expected_outcome`	The right answer, in your own words. Only the judge reads it (Part 6).

The three tasks check three behaviours of your agent:

Task	What it checks
`target_check`	The question of Lab 3: the agent asks the KPI agent, calls `check_target`, and gives a verdict.
`target_check_reworded`	The same question in other words: the LLM must still understand it.
`unknown_network`	A network that does not exist: the agent must say `UNKNOWN` and not compare a number it does not have.

The throughput changes all the time, so no task checks the number itself: they check what the agent did with it.

Step 4: Check the settings

bat eval show throughput_checker

You should see:

============================
  EVALUATION CONFIGURATION
============================
Dataset     : /home/you/tutorial/throughput_checker/eval/input/tasks.json
k           : 1
Qualitative : no

Models:
  [1] openai:gpt-5.6-luna

Judge model : openai:gpt-6-luna
============================

Part 4: Run the evaluation

bat eval run throughput_checker

bat starts your agent, asks it the three questions one after the other, checks the answers, and stops the agent. Each question can take up to a minute, because your agent waits for the KPI agent. At the end you should see:

Running evaluation with 1 model(s). task_id=20261016_143005
- openai:gpt-5.6-luna
INFO     ... - Artifacts written to: /home/you/tutorial/throughput_checker/eval/output/20261016_143005/benchmark_openai-gpt-5.6-luna
Evaluation completed. Output: /home/you/tutorial/throughput_checker/eval/output/20261016_143005

ℹ️ During the run, bat changes the telemetry part of throughput_checker/config.yaml, so that your agent writes its traces into the output folder. It puts the file back as it was at the end.

Part 5: Read the results

Each run has its own folder, named after the time it started:

throughput_checker/eval/output/20261016_143005/
├── agent-0.log                    what your agent printed during the run
├── spans-0/agent.jsonl            the traces of your agent
└── benchmark_openai-gpt-5.6-luna/   the results with this LLM
    ├── metrics.json               every task: passed or not, time, tokens
    ├── summary.json               how many attempts of each task passed
    └── episodes/                  one file per task and attempt: what happened

Step 1: `metrics.json`

Open benchmark_openai-gpt-5.6-luna/metrics.json. It has one entry per task under per_episode, and the totals under summary. The first task looks like this:

{
  "task_id": "target_check",
  "expected_outcome": "The agent gets the downlink throughput of o-ran ...",
  "status": "completed",
  "success": true,
  "time": { "wall_ms": 21840.5 },
  "tokens": {
    "prompt_tokens": 1210,
    "completion_tokens": 96,
    "total_tokens": 1306,
    "cached_prompt_pct": 0.0
  },
  "verdict": {
    "passed": true,
    "reason": "status: 'completed'; output[0]: contains 'o-ran'; output[1]: contains 'TARGET'; tool_call:check_target: called 1×; agent_call:KPI Agent: called 1×"
  }
}

Part	Meaning
`success`	`true` when all the checks of the task hold.
`verdict.reason`	The result of each check, in order. A check that fails says what it got and what it expected, for example `tool_call:check_target: called 0×, expected ≥1×`.
`time.wall_ms`	How long the task took, in milliseconds.
`tokens`	The tokens your agent's LLM used: what you pay for. The KPI agent's tokens are not counted: they are spent in the cluster.

At the end of the file, summary.pass_rate is the share of tasks that passed: 1.0 when all of them did.

Step 2: Find out why a task failed

Read verdict.reason first: it names the check that failed. Then open the task's file in episodes/, for example unknown_network__try0.json. Under trace.trajectory.turns, it lists every step of your agent, in order (trimmed here):

"steps": [
  { "kind": "agent", "node": "ask_kpi", "agent": "KPI Agent",
    "asked": "What is the average downlink throughput, in Mbps, ... atlantis meet its throughput target?",
    "answered": "There is no network named atlantis. ..." },
  { "kind": "model", "node": "judge",
    "said": "atlantis: UNKNOWN — the KPI agent found no network named atlantis." }
]

agent: the call to the KPI agent, with the question and its answer;
model: a call to your LLM, with the tools it asked for (called) and what it wrote (said);
tool: a call to a tool, with its values (args) and its result.

Compare these steps with your task. For example, if the KPI agent answered with the throughput of another network, and your agent compared it, the task fails on UNKNOWN: change the instructions of checker_client.py, and run the evaluation again.

Part 6: Add the LLM judge

Step 1: Turn the judge on

The checks tell you what the agent did. To also score how good the answer is, set qualitative to true in eval.yaml:

evaluation:
  ...
  qualitative: true

Run the evaluation again:

bat eval run throughput_checker

After the questions, the judge reads each task and adds a qualitative part to it in metrics.json:

"qualitative": {
  "response_relevance": 1.0,
  "task_completion_quality": 0.9,
  "hallucination_score": 1.0,
  "tool_call_appropriateness": 1.0,
  "judge_reasoning": {
    "relevance": "The agent stayed on the throughput target of o-ran ...",
    "task_completion": "The agent got 42.3 Mbps from the KPI agent, called check_target ...",
    "hallucination": "Every number in the answer comes from the KPI answer or the tool ...",
    "tool_call": "check_target was called once, with the value the KPI agent gave ..."
  }
}

The judge gives four scores, from 0 to 1, higher is better:

Score	The question the judge answers
`response_relevance`	Does the agent stay on the topic the user asked about?
`task_completion_quality`	Does it reach the `expected_outcome` of the task?
`hallucination_score`	Does every value in the answer come from the question, a tool or another agent? `1.0` means nothing was invented.
`tool_call_appropriateness`	Did it call the right tools, with the right values? Only for tasks that have `tool_calls`: for `unknown_network` it is `null`.

judge_reasoning explains each score in one or two sentences. Read it when a score is low. The summary part of metrics.json also gets the average of each score.

Step 2: Tell the judge about your agent (optional)

The judge does not know your agent. For example, it can score UNKNOWN as a task not done. You can give each of its four rubrics a few lines of context, under judge.prompts in eval.yaml:

judge:
  provider: openai
  model: gpt-6-luna
  prompts:
    task_completion: |
      UNKNOWN is the right answer when the KPI agent gives no downlink
      throughput number for the network: it is a success, not a failure.

The keys are the four rubrics: relevance, task_completion, hallucination and tool_call.
Each text has at most 1000 characters. bat eval show throughput_checker tells you if one is too long.
The text is added to the rubric; the scores keep the same meaning, so you can still compare runs.

Part 7: Measure consistency and compare LLMs

Step 1: Ask each question three times

An LLM does not always answer the same way. In eval.yaml, set k: 3 and run the evaluation again. Each task now runs three times, and summary.json tells you how many attempts passed:

{
  "task_id": "unknown_network",
  "attempts": 3,
  "passed": 2,
  "failed": 1,
  "success_percentage": 66.66666666666667
}

A task that passes 3 times out of 3 is reliable. A task that passes 2 times out of 3 shows a weak spot: read the episode that failed (__try0, __try1 or __try2) to see what changed.

Step 2: Add a second LLM

Now compare gpt-5.6-luna with qwen3.8, an open model that runs on the Ollama of your cluster. Only your agent's LLM changes: the KPI agent keeps the LLM of the fabric.

Pull the model in Ollama, once, on the master node. If Ollama is not installed, see the Ollama guide.
```
kubectl -n inference exec deploy/ollama -- ollama pull qwen3.8
```
Open a tunnel to Ollama, as you did for the KPI agent. Open a new terminal on the master node. If you work on your own computer, connect with SSH and carry port 11434:
```
ssh -L 11434:localhost:11434 <user>@<master-node>
```
Then run the command below, and keep the terminal open:
```
kubectl -n inference port-forward svc/ollama 11434
```
Let your agent talk to both providers. In pyproject.toml, add ollama to the extras of bat-adk:
```
dependencies = [
    "bat-adk[openai,ollama,telemetry]==2026.10"
]
```

Add qwen3.8 under models in eval.yaml:

models:
  - provider: openai
    model: gpt-5.6-luna
  - provider: ollama
    model: qwen3.8
    base_url: http://localhost:11434

base_url is the tunnel of item 2. The judge stays on gpt-6-luna, so both models are scored by the same judge.

Run the evaluation again. qwen3.8 runs on your GPU, so expect it to be slower than GPT. The run folder now has one results folder per model, benchmark_openai-gpt-5.6-luna and benchmark_ollama-qwen3.8. Draw the charts that compare them:

bat eval plot --folder throughput_checker/eval/output/<run>

Replace <run> with the name of the run folder, for example 20261016_143005. bat saves the charts as metrics_*.png files in that folder: the time and the tokens of each model, the judge's scores, then one chart per task. Add --filter target to draw the charts per task only for the tasks whose id contains target.

If something goes wrong

What you see	What to do
`Naming an agent ('throughput_checker') works only inside a blueprint`	You are not in the `tutorial/` folder. Go there with `cd`, then run the command again.
`Missing ./eval/eval.yaml. Run 'bat eval init' first.`	Create the evaluation folder first (Part 3, Step 1).
`Model openai:gpt-5.6-luna failed: throughput_checker stopped before it was ready`, with `Network communication error fetching agent card from http://localhost:9310/...`	The tunnel to the KPI agent is not running. Start it again (Part 2, Step 1), then run the evaluation again.
Every task fails with `trace: no spans found for this conversation`, and `agent-0.log` ends with `address already in use`	Your agent of Lab 3 still runs, so `bat` asked it instead of starting its own. Stop it (Part 2, Step 2), then run the evaluation again.
`judge.prompts.task_completion exceeds the 1000-character limit`	Shorten that text to 1000 characters at most.
`Model ollama:qwen3.8 failed`	Check that the tunnel to Ollama is open (Part 7, Step 2, item 2), and that the model is pulled: `kubectl -n inference exec deploy/ollama -- ollama list`.
Still stuck?	Contact BubbleRAN customer support through your usual support channel.

Clean up

The evaluation adds nothing to your cluster. When you are done with the labs, follow the Clean up of Lab 4.

What you learned

A test set (tasks.json) lists questions, and for each one the answer you expect.
bat eval run starts your agent, asks every question, and records the answers and the traces; metrics.json holds the results.
Checks are exact rules on the answer and on what the agent did: which tools and agents it called. Check what the agent did, not only what it said.
The judge is an LLM that scores relevance, task completion, invented facts and tool use from 0 to 1, and explains each score.
k asks each question several times to measure consistency, and models compares LLMs on the same test set; bat eval plot draws the comparison.

Part 1: The idea​

Part 2: Get ready​

Step 1: Open the tunnel to the KPI agent​

Step 2: Stop your own agent​

Part 3: Write the evaluation​

Step 1: Create the evaluation folder​

Step 2: The settings: eval.yaml​

Step 3: The test set: tasks.json​

Step 4: Check the settings​

Part 4: Run the evaluation​

Part 5: Read the results​

Step 1: metrics.json​

Step 2: Find out why a task failed​

Part 6: Add the LLM judge​

Step 1: Turn the judge on​

Step 2: Tell the judge about your agent (optional)​

Part 7: Measure consistency and compare LLMs​

Step 1: Ask each question three times​

Step 2: Add a second LLM​

If something goes wrong​

Clean up​

What you learned​