Your AI agent works in the demo. What happens when an API call fails?

Say you're building an AI agent that qualifies leads, updates a CRM, sends follow-up emails, and books meetings.
Looks great in a demo. Then the CRM API times out after saving a contact. The agent retries, and suddenly you've got duplicate records and two follow-up emails.

That's not an LLM problem. It's a distributed systems problem.
For production, I'd separate the agent's decisions from actual execution. Validate tool calls against strict schemas, use idempotency keys for write operations, and track execution state across retries.

Add exponential backoff, proper webhook deduplication, and human approval for sensitive actions.
And don't just test the happy path. Simulate partial failures, rate limits, expired tokens, and interrupted workflows.
The real test isn't whether an AI agent can complete a task. It's whether it can recover safely when something goes wrong.

Curious how others are handling retries and state management in production AI agents.