slopbench
SlopBench
Code benchmarks measure if a model can solve a task. None measure how much code it wastes.
I got tired of reviewing AI-generated PRs that were technically correct but 5x larger than they needed to be. 🤯
So I built SlopBench: an open-source benchmark that measures code inflation in LLM agents.
The premise is simple. Every model solves the same 5 Python tasks inside a sandboxed Docker environment. The metric is inflation factor = generated LOC ÷ LOC of a hand-written reference solution that passes the exact same tests.
The twist: every model runs twice.
→ Once with a "vanilla" system prompt.
→ Once with a "minimalism" system prompt.
The difference in LOC is how much a simple prompt change shaves off. It's the cheapest lever against AI bloat that any team can pull today, without swapping models.
Here's what nobody else is publishing: inflation is only counted for runs that pass the tests. A 3-line solution that fails 4 out of 5 tries doesn't qualify. See ADR-001 in the repo for the full rationale.
Results update weekly via GitHub Actions. Every raw run is committed. Reference solutions are in-repo and contestable via PR — the denominator is public on purpose.
If you use AI agents to write code, this number matters to you.
👉 Repo: lnkd.in/euaYNtQq
If you find the metric useful or want to see more models added, a ⭐ helps more people find it.
What's the worst code inflation you've seen from an LLM? Drop it in the comments. 👇






Comments
0 comments