slopbench
SlopBench
Code benchmarks measure if a model can solve a task. None measure how much code it wastes.
I got tired of reviewing AI-generated PRs that were technically correct but 5x larger than they needed to be. 🤯
So I built SlopBench: an open-source benchmark that measures code inflation in LLM agents.
The premise is simple. Every model solves the same 5 Python tasks inside a sandboxed Docker environment. The metric is inflation factor = generated LOC ÷ LOC of a hand-written reference solution that passes the exact same tests.
The twist: every model runs twice.
→ Once with a "vanilla" system prompt.
→ Once with a "minimalism" system prompt.
The difference in LOC is how much a simple prompt change shaves off. It's the cheapest lever against AI bloat that any team can pull today, without swapping models.
Here's what nobody else is publishing: inflation is only counted for runs that pass the tests. A 3-line solution that fails 4 out of 5 tries doesn't qualify. See ADR-001 in the repo for the full rationale.…