Agent Labs

Agent Labs

Last updated:

Agent Labs answers one question: how often does my agent get it right?

You write a few questions whose answers you already know. Camber asks your agent each question, checks every answer, and gives you a score. If the score is too low, Camber can tune the agent for you: it studies the wrong answers, improves the agent’s instructions, and tests again until the agent reaches your target.

Flow diagram: 1 Ask, your agent answers each test question in a new chat; 2 Score, the evaluator marks each answer pass or fail; if the target is met or it is the last cycle, done; otherwise, in tuning only, 3 Improve rewrites the instructions as a new agent version and the next cycle starts at Ask

Key Terms

TermWhat it means
EvalA set of test examples for one agent. Create one eval per agent and topic, for example Citi Bike Snowflake - trips and weather.
Test exampleOne question and the answer you expect. The agent sees only the question.
EvaluatorDecides if an answer passes. It reads the expected output; the agent never does.
BenchmarkScores the agent’s current version once. Nothing about the agent changes.
TuningScores the agent, improves its instructions, and repeats until it reaches the target score or the cycle limit.
CycleOne round of tuning: ask every question, score every answer, then improve the instructions.
RepetitionHow many times each question is asked in a cycle. Each repetition is a fresh chat.
ScorePassed answers divided by all answers in a cycle, from 0 to 1. With 4 examples and 1 repetition, 3 passed answers score 0.75.

Before You Start

  • Have an agent to test. It must be active in the current workspace.
  • Make sure the agent can reach the data it needs, for example through a data connector.
  • Know the right answers to your questions. Run the queries yourself first, so you’re sure the expected outputs are correct.

Step 1: Open Agent Labs

Select Agent Labs in the sidebar. The list shows every eval in your workspace, the agent each one tests, and when it was last updated.

Agent Labs page listing evals with their agents and last update times

Step 2: Create an Eval

  1. Select New eval.
  2. Enter an Eval name that says what is tested, for example Citi Bike Snowflake - trips and weather.
  3. Under Agent to test, search for the agent and select it.
  4. Select Create eval.
New eval dialog with an eval name and the Citi Bike Snowflake Analyst agent selected

Step 3: Add Test Examples

Open the eval, select the Test examples tab, then select Add test example.

  1. In Question, write exactly what a user would ask the agent.
  2. Under Evaluator type, choose how the answer is checked:
    • LLM as a Judge: Nova checks the answer with a scoring method you pick. This is the easiest option.
    • Code-based: your own Python script checks the answer.
  3. Fill in the expected output, then select Add test example.
Add test example panel with a question, LLM as a Judge, Exact match, and the expected output 2013-07-18

Choose a Scoring Method

With LLM as a Judge, open Scoring method and pick the one that fits the answer.

Scoring method list showing Exact match, Number with tolerance, Contains all, and Custom rubric
Scoring methodUse it when the answer is…Passes when…Example
Exact matchOne value: a date, a name, an IDThe answer contains the expected value exactly2013-07-18
Number with toleranceA number that may be roundedA number in the answer is within ± the tolerance42.89 ± 0.02
Contains allA list of itemsEvery value appears; with Must appear in this order, in that order tooTop 5 stations, most to fewest trips
Custom rubricAn explanation or a table with several factsNova judges the answer against the rules you write“Pass only if all of these hold…”

Number with tolerance takes the expected number and how far off an answer can be.

Number with tolerance scoring with expected output 42.89 and tolerance 0.02

Contains all takes one value per line. Select Must appear in this order when order matters, such as a ranking.

Contains all scoring with three station names, one per line, and Must appear in this order selected

Custom rubric lets you describe a passing answer in plain words. Be specific: list the numbers that must appear and say what makes the answer fail.

Custom rubric scoring with a rubric that lists the required numbers and when the answer fails
Use your own Python script (Code-based)

Select Code-based, then choose a .py file from your Stash. The file must define a validate(output) function that returns (passed, reason):

def validate(output: str) -> tuple[bool, str]:
    expected = ["Maze Pen", "Google 22 oz Water Bottle", "Google Sunglasses"]
    missing = [name for name in expected if name not in output]
    if missing:
        return False, f"Missing: {', '.join(missing)}"
    return True, "All products found"

output is the agent’s full answer. reason is shown in the results, so make it say what is wrong.

Code-based evaluator with a Python file selected from Stash

When you’re done, the Test examples tab lists each question, its expected output, and its evaluator type. Use the pencil icon to edit an example and the trash icon to delete one.

Test examples tab listing four questions with Exact match, Number, Contains all, and Custom rubric evaluators
ℹ️
The agent under test sees only the question. Only the evaluator reads the expected output, so the agent can’t copy the answer.
Tips for good test examples
  • Ask real questions. Write them the way your users ask them.
  • Make the answer checkable. Ask for a format, for example “Reply with the date as YYYY-MM-DD” or “Round to two decimals”.
  • Fix the time window. If the data keeps changing, add a date range so the right answer stays the same.
  • Cover the hard parts. Include questions that need a tricky join, a unit conversion, or a known data quirk. These are the questions tuning can fix.
  • Mix the scoring methods. One exact value, one number, one list, and one explanation make a good starting set.
  • Don’t reveal the answer in the question. The agent should have to find it.

Step 4: Run a Benchmark

A benchmark tells you how good the agent is today. It doesn’t change the agent.

  1. Select Run benchmark.
  2. Set Repetitions. Use more than 1 to see if the agent answers the same way every time.
  3. Check the total number of runs, then select Run benchmark.
Run benchmark dialog with 1 repetition, showing 4 test examples times 1 repetition equals 4 total

Step 5: Start Tuning

Tuning improves the agent for you. Each cycle, Camber asks every question, scores the answers, and then rewrites the agent’s instructions to fix what went wrong.

  1. Select Start tuning.

  2. Set the options:

    OptionWhat it doesDefault
    RepetitionsHow many times each question is asked per cycle1
    Target scoreTuning stops early once a cycle reaches this score (0 to 1)0.9
    Max cyclesThe most cycles tuning runs, even if it doesn’t reach the target5
  3. Check the total, then select Start tuning.

Start tuning dialog with 1 repetition, target score 0.9, max 5 cycles, and up to 20 runs in total
⚠️
Each improvement is saved as a new active version of the agent. Anyone chatting with the agent gets the new version right away.

An eval runs one benchmark or tuning run at a time. While a run is in progress, the Runs tab shows Running · cycle N of M. To stop it, select Stop run. Answers already scored keep their results, and versions already saved stay active.

Read the Results

Runs

The Runs tab lists every run, newest first, with its result, who started it, and how long it took.

Runs tab listing a completed and a stopped tuning run with who started each and their durations
ResultWhat it means
RunningThe run is in progress. Tuning also shows the current cycle.
Reached targetTuning reached the target score.
Below targetTuning used all its cycles without reaching the target.
CompletedThe benchmark finished.
StoppedSomeone stopped the run.
FailedThe run couldn’t finish. Open it and check the pod.

Run Details

Select a run to see its score, cycles, and repetitions. Under Results, select a cycle number to see how each question did in that cycle.

Tuning run details showing score 1.0000 against target 0.9, 5 of 5 cycles, and all four questions passed in cycle 5

In this example, the agent’s score went from 0.5 in cycle 1 to 1.0 in cycle 5: 0.5 → 0.75 → 0.5 → 0.75 → 1.0.

Why an Answer Failed

Select a red ✕ in the Repetitions column. The details show why the answer failed and the part of the answer with the problem. Select Open chat to read the agent’s full conversation.

Failed repetition details explaining the missing station and wrong order, with the agent's answer and an Open chat link

See What Tuning Changed

Agent Versions

To see the instructions tuning wrote, open Agents, select the agent, then select the settings icon. The Instruction tab shows the current version.

Agent context page of GA BigQuery Analyst at v6, showing the instructions with an Environment section, and the version history icon next to Start Chat

To compare versions, select the version history icon next to Start Chat. Each version made by tuning has a changelog that says what changed and how many failures it addresses. Select a version to see how its instructions differ from the previous one.

Version History showing six versions of GA BigQuery Analyst and the instruction changes from v5 to v6

Tuning keeps what it learns in two sections at the end of the instructions:

  • Environment: facts about your data, such as tables, columns, and formats, so the agent doesn’t have to explore them again.
  • Known pitfalls: mistakes the agent made before and how to avoid them.

Tuning is told to write general rules about your data and never to copy test questions or expected answers into the instructions. Review new versions before you rely on them.

The Pod Behind a Run

Every run is an Agent Pod. Select Open in Pods on the run page to see who did the work:

  • @nova.lead runs the experiment and dispatches the other agents.
  • Your agent answers each question in its own chat.
  • @nova.eval scores each answer.
  • @nova.evolver improves the instructions between cycles (tuning only).
Pod details for a tuning run with nova.lead as supervisor and the agent and nova.eval chats as workers

Select open main chat to read the lead’s summary: the score of each cycle and each version the evolver created.

Main chat of the tuning pod with the lead's summary of cycle scores and the four versions the evolver created

Troubleshooting

ProblemWhat to check
The agent is missing from Agent to testCheck that the agent is active and in the selected workspace.
You can’t start a new runEach eval runs one run at a time. Wait for the current run to finish, or stop it.
A correct answer failedRead the reason in the failure details. For Exact match, the value must appear exactly as written. Use Number with tolerance for rounded numbers, or ask for a format in the question.
Every answer fails the same wayCheck the expected output. Run the query yourself to confirm the right answer.
The score goes up and down between cyclesUse 2 or 3 repetitions so one lucky or unlucky answer counts less.
A run says Failed or the evaluator reports a setup problemSelect Open in Pods and read the lead’s main chat. For a Code-based evaluator, check that the file exists and defines validate(output).
Tuning ended Below targetRead the failures of the last cycle. Start tuning again to continue from the latest version, or make the question clearer if the agent misread it.

Related Documentation