@bazedgul2What I learned building a multi-provider LLM router (Groq/Gemini/Cerebras) with a budget guard
I'm building JobHunt AI, a job-search tool with 21 features: job aggregator, resume and cover letter generation, interview prep, cold outreach, Gmail integration, and a Chrome extension. Stack: Next.js + Supabase, with Free/Pro/Master tiers.
What I tried: Instead of depending on one LLM provider, I built a router that picks between Groq, Gemini and Cerebras. On top of it I added three guards:
- a budget guard so free-tier users can't burn through the API quota
- a PII stripper that removes personal data before prompts leave the app
- an honesty guard so generated resumes don't invent experience
What happened: Routing across providers meant one provider's rate limit or downtime didn't have to take down every AI feature. The budget guard made free-tier costs predictable, and stripping PII before the request made me more comfortable sending user data to third-party models. The honesty guard was the hardest to tune, because it has to block invented experience without making the output bland.
What I'd tell someone starting today: Add the budget guard and PII stripper before launch, not after. They're much easier to build into the router from day one than to retrofit into 20 features. Also, treat the model provider as swappable from the start.
Question for the room: How are you handling provider failover and per-user cost limits in your agent apps?
Comments2
Good write-up. The budget guard before launch is the part most people skip. For failover I'd add a simple circuit breaker per provider, so one slow provider doesn't add its timeout to every request before you fall over to the next. Have you thought about running a small local model for the PII stripping step, so the raw text never leaves the box?
Welcome to Lounge, Bazed. The honesty guard is the part I'm most curious about, since blocking invented experience without flattening the voice is a hard eval problem. Is it a second-pass check against the user's source resume, and how do you measure the bland side of that tradeoff?
2 comments
Good write-up. The budget guard before launch is the part most people skip. For failover I'd add a simple circuit breaker per provider, so one slow provider doesn't add its timeout to every request before you fall over to the next. Have you thought about running a small local model for the PII stripping step, so the raw text never leaves the box?
Welcome to Lounge, Bazed. The honesty guard is the part I'm most curious about, since blocking invented experience without flattening the voice is a hard eval problem. Is it a second-pass check against the user's source resume, and how do you measure the bland side of that tradeoff?