Building an AI assistant is easy. Making it safe for real business data is another story.

A lot of companies want an AI assistant connected to their internal docs, CRM, and customer database.
Getting a RAG demo running with embeddings and a vector database isn't particularly difficult. The real engineering starts when different users have different permissions, documents change daily, and the AI needs to give answers people can actually rely on.

One thing I'd never do is retrieve everything first and expect the LLM to figure out what a user is allowed to see.
Access control belongs in the retrieval layer. Tenant isolation, document-level permissions, and metadata filtering need to happen before any context reaches the model.

I'd also build in incremental indexing, source citations, retrieval evaluation, and proper fallback handling when the evidence isn't good enough.

And if the assistant can actually modify CRM records or trigger business actions, those operations need their own authorization and audit trail.

After 10+ years in software engineering, I've learned that the impressive demo is usually the easy part. It's the architecture behind it that determines whether a product is ready for real customers.

Anyone here building production RAG systems? What's been your biggest challenge beyond retrieval accuracy?