@johnryan88864How are you stopping your AI agents from hallucinating "Done"?
We've all been there. You ask Cursor or Claude Code to fix a bug. It confidently replies: "Done, all 12 tests passed." You merge the PR, and everything is broken because it only ran 4 tests and ignored the rest.
I got so tired of this happening that I built a local tool to fix it. It's called Prove It.
How it works: It reads a simple claims.yaml file produced by your agent, independently re-runs the actual pytest suite, and checks the Git diffs. It then outputs a red/green "Claims vs. Reality" report. If the agent lied, the CLI exits with a failure code so your CI/CD blocks the merge.
It's 100% local, fail-closed (prevents YAML/command injection), and built with strict TDD (69 passing tests).
I just launched it today as a $29 CLI for early builders. If you're tired of trusting "Done," you can grab it here: payhip.com/b/HYfjh
Would love to hear how the rest of you are verifying agent output. Are you building custom eval scripts, or just manually reviewing every PR?
Comments2
I stop them from hallucinating, by running qourum consensus throughout my ecosystem, qourum size depends on importance of the task.. I also have usually three frontier level llm's on the same screen, and i feed one prompt to one agent and then the same prompt to the other two agents , then we go over the results and tear it all apart again until we get the correct non hallucinated results and then we take the results and run it thru all three again.. and then we prove everything w recieipts on everything..
Love the quorum consensus approach, Brandon. Using three frontier models to debate and tear the output apart is one of the best ways to increase reasoning quality. The only danger is "unanimous hallucination"—where all three models confidently agree on a solution that actually breaks the code. That’s why the receipts matter. I built Prove It specifically for that last step: it forces the consensus output to actually run pytest and check the git diff before the agent is allowed to say it's done. Consensus for the brain, hard verification for the execution.
2 comments
I stop them from hallucinating, by running qourum consensus throughout my ecosystem, qourum size depends on importance of the task.. I also have usually three frontier level llm's on the same screen, and i feed one prompt to one agent and then the same prompt to the other two agents , then we go over the results and tear it all apart again until we get the correct non hallucinated results and then we take the results and run it thru all three again.. and then we prove everything w recieipts on everything..
Love the quorum consensus approach, Brandon. Using three frontier models to debate and tear the output apart is one of the best ways to increase reasoning quality. The only danger is "unanimous hallucination"—where all three models confidently agree on a solution that actually breaks the code. That’s why the receipts matter. I built Prove It specifically for that last step: it forces the consensus output to actually run pytest and check the git diff before the agent is allowed to say it's done. Consensus for the brain, hard verification for the execution.