Session 9 — Document agent

Session 9 of 13PracticeDraft4 min read

Draft for the lecturer's approval. This is preliminary study material. It does not replace guidance published in Moodle, and it does not set any date or grade.

Goals

• Add retrieval to the agent from session 5.
• Attach a source id to every numeric claim.
• Test missing information and information that was updated.
• Write a question that has no answer in the collection, and make sure the agent says so.

What you build

A collection of four short text files, in the QuickScreen structure or another collection you are allowed to publish. retrieve returns an id, text and a score. The score is not proof of support.

A successful answer to the two-requirements question says: the corrected recall is 90%, the time reduction is 40%, the joint requirement is not met, and the sources are P:1 and C:1. An answer to the cost question says there is no evidence. An answer that computes 95% from A:1, after C:1 was retrieved, fails even if the id is real.

Splitting into passages

A passage is the unit of retrieval, and so also the unit of citation. A passage that is too long brings sentences unrelated to the question. A passage that is too short cuts a number from its context: "36 out of 40" without the sentence that says it is a correction.

Simple rules for a small collection:

• Split by the document's structure, a paragraph or a section, not by a fixed number of characters.
• Every passage gets a stable id that does not change when a new document is added.
• Keep with the passage the document it came from and its version.

An answer with a source

Every number in the answer comes with the id of the passage it appears in. A computed number, such as the time reduction, comes with the ids of every number that went into the calculation. If no passage supports a claim, the claim leaves the answer.

A check in code can enforce part of this: extract the numbers from the answer and make sure each one appears in one of the cited passages, or is the result of a calculation recorded in the log.

A small memory

Keep one note: the source ids and the version. In a new conversation, load the note and compare it with the files. If the correction changed, the old note does not decide the number.

Lab 3 includes these three checks as a script.

Add a fifth file to the collection that corrects the recall figure again. Run the two-requirements question with the note saved from the previous run. Does the agent notice that the version changed? Write one question that has no answer in the collection, and check that the answer says there is no evidence and does not guess.

Check your understanding

The answer says "the time reduction is 40% [P:1]". What is missing?

Pick an answer to see the explanation. There is no grade here.

Summary

What stays. A passage with a stable id is the unit of retrieval and of citation. Every number in the answer comes with its sources.

What breaks. A high score in place of a source. An old note that decides a number after the collection was updated.

Next: session 10, Evaluation, monitoring and safety

The mark is saved only in this browser. It is not an academic record and is not sent to a server.