@dominic60184
Dominik LubańskiDeveloper▲8
Full Stack & AI Developer, AIvanta
2h ago · Infra · Warsaw

Your AI Infrastructure Might Be Costing More Than It Should

One thing worth paying attention to as an AI product starts scaling is how much you're actually spending per successful request.

It's easy to focus on model pricing, but I've seen how quickly unnecessary context, repeated API calls, and poorly configured retries can push costs up.

Before switching to a cheaper model, I'd look at three things: prompt caching, request routing, and token usage across different workflows.

For example, simple classification tasks probably don't need the same model you're using for complex reasoning. And if you're sending the same system instructions and documents on every request, caching can make a noticeable difference.

I'd also track p95 latency and cost per successfully completed task, not just average response time or cost per token.

Sometimes the biggest infrastructure improvement isn't adding more compute. It's eliminating work the system never needed to do.

For those running AI products in production, what's been your biggest unexpected infrastructure cost?

Comments

No comments yet
Start the conversation.
Log in to join the thread.