@darkhater012
Pentester & Backend developer ,, Independent
1h ago · Agents · Cuba

slopbench

SlopBench
Code benchmarks measure if a model can solve a task. None measure how much code it wastes.

I got tired of reviewing AI-generated PRs that were technically correct but 5x larger than they needed to be. 🤯

So I built SlopBench: an open-source benchmark that measures code inflation in LLM agents.

The premise is simple. Every model solves the same 5 Python tasks inside a sandboxed Docker environment. The metric is inflation factor = generated LOC ÷ LOC of a hand-written reference solution that passes the exact same tests.

The twist: every model runs twice.
→ Once with a "vanilla" system prompt.

→ Once with a "minimalism" system prompt.

The difference in LOC is how much a simple prompt change shaves off. It's the cheapest lever against AI bloat that any team can pull today, without swapping models.

Here's what nobody else is publishing: inflation is only counted for runs that pass the tests. A 3-line solution that fails 4 out of 5 tries doesn't qualify. See ADR-001 in the repo for the full rationale.

Results update weekly via GitHub Actions. Every raw run is committed. Reference solutions are in-repo and contestable via PR — the denominator is public on purpose.

If you use AI agents to write code, this number matters to you.

👉 Repo: lnkd.in/euaYNtQq

If you find the metric useful or want to see more models added, a ⭐ helps more people find it.

What's the worst code inflation you've seen from an LLM? Drop it in the comments. 👇

Comments

No comments yet
Start the conversation.
Log in to join the thread.