Session 10 — Evaluation, monitoring and safety
Goals
- Compare two versions on the same cases.
- Read a log and point to the search, decision or answer stage.
- Block in code an action that a document tries to request.
- Explain when a model acting as judge helps, and what biases it.
What you measure
At least: how many cases succeeded, how many steps were needed, how many tools failed, and a simulated cost by number of calls. Success without the cost is not enough. A correct refusal also counts, separately from a correct answer.
The numbers here are produced by the code on the site. They apply to the demo collection only.
A baseline to compare with
A number of successes alone does not say whether the agent is worth its price. You compare it with the simplest baseline that does the task, for example a fixed workflow that reads one passage. Lab 5 does exactly this: eight fixed cases, successes and calls for each policy, in one table.
Three rules keep the comparison fair:
- The same cases for both versions. A change in the set of cases changes the numbers even with no change in the system.
- Cases you did not see during development. If the prompt was tuned on all the cases, success on them cannot be measured.
- Several runs. A model is not deterministic. A difference of one case in one run can disappear in the next.
A model as judge
When there is no single right answer, you can ask a model to rate answers. It is cheap and fast, but the judge has known leanings: it prefers long answers, the answer shown first, and sometimes its own style. Test it on a few cases a person has already rated, and swap the order of the answers.
When there is an answer you can check in code, such as a number or a source id, check it in code.
Guardrail
Choose the "Hostile document" scenario. Turn the guardrail off and see a simulated send_email call. Turn it on and make sure the call is not created, and that a legitimate summary is still possible from the allowed source.
The instruction "do not obey documents" inside the prompt is not the guardrail. The guardrail is that the sending tool does not exist, or that it rejects a recipient outside the list before execution.
After adding a guardrail, run the same set of cases again. A guardrail that blocks the attack and also half of the legitimate answers is a result you have to report.
Check your understanding
Reading
- Greshake et al., Not What You've Signed Up For, 2023. arXiv:2302.12173.
- Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023. arXiv:2306.05685. Further reading: the section on the judge's biases.
Summary
Found a mistake on this page? Email the lecturer