Rishi Patidar

Trust me, check me

An answer you can't trace back to the page is an answer you can't check.

The last post ended on provenance. It was the part we understood last, and it turned out to be holding up everything else.

Once we had a way to review traces, we sat a person down with them. The pipeline's output was a clean list of fields. The reviewer would read a value, open the PDF, and go looking for where it came from. Every single value. Reviewing a trace meant doing the extraction a second time, by hand.

Then we noticed something worse. Say the pipeline returns a date and an address from the same document. The date was sitting right there on page two. The address wasn't in the document at all; the model had filled one in, because documents like this usually have one. In the output, the two looked exactly the same. Same format, same confidence. Nothing told us which one was real.

That was the moment it clicked. An answer from the model is only worth trusting if you can see where in the document it came from, and something other than the model can confirm it is really there. Without that, every value is a claim, and we had been asking reviewers to go and find the evidence themselves.

Our first fix was the obvious one: ask for a quote next to every value. It helped less than we hoped. A quote the model writes is just more text the model wrote. It can be tidied, paraphrased, or made up, and still look convincing.

So we took the evidence out of the model's hands. Before the model sees anything, code cuts the document into small blocks and gives each one a label, like page 2, block 7. The model is only allowed to answer by naming those labels. Then code checks every answer: is this value actually in the block it named? If not, the answer goes back to the model with the block it got wrong. And because every answer now pointed at a real place, highlighting it in our review screen came for free.

The model can still be wrong. It can no longer be wrong without leaving a trail.

A teammate took it one step further. In their version the model doesn't write the evidence at all. It only says where each value came from, and code attaches the quote and the highlight. The model reads; code cites.

The rule that held it all together was the plainest one: if it isn't in the text, it isn't there. No filling gaps with what is typical. So each value started coming back with a status, not just an answer. Found, found more than once, conflicting, said to be in some other document, or not found. Only the first is a clean answer. Everything else goes to a person as a question.

That made the output look less complete, and we nearly talked ourselves out of it. Then we ran the first five real documents. Every one of them said part of the answer lived in another document, and none of them had that document attached. An address we expected in every file was there in two.

Those aren't edge cases. That is the job. A pipeline that fills the gaps so it looks finished is hiding exactly the thing a reviewer needs to see.

It also made the next step possible. Before you can write evals, someone has to sit with real traces and write down everything that went wrong, and that only works if checking a value takes seconds rather than a hunt through the PDF. With provenance, review stopped being a search: you click the highlight and say yes or no. And an answer that points at the wrong block gets caught by code before a person ever looks at it.

An output tells you what the model thinks. Provenance tells you whether to believe it.

Next: reading the traces. Annotating what went wrong, and turning it into evals.

All writing