When an AI revises a case, what should stay unchanged?
A useful evaluation of an AI assistant can begin after its first answer.
Consider a fictional planning file: one source says a room holds 24 people; another says 18. The event needs 20 places. Keep the conflict visible until the scenario supplies a reason to resolve it. Then introduce an authoritative correction to capacity. The assistant should revise the venue decision, while preserving unrelated conclusions that remain supported.
I would score four behaviours separately: retaining an unresolved conflict; updating the conclusions affected by the correction; avoiding changes to unaffected conclusions; and resuming the case faithfully after an interruption. Human checking time and unresolved items belong in the results too. More confident prose would not, by itself, count as improvement.
For a fair comparison, keep the same LLM, platform, files, memory, search and tools available in both conditions. Declare the configurations and resources. An independent scoring reference should be fixed before inspecting the outputs.…
