Enterprise Chatbot Architecture: Keep RAG Private, Tools Narrow
For knowledge-grounded enterprise chat, we recommend a RAG-enhanced enterprise chatbot architecture, adding agentic capabilities only where the bot must execute tools. The priorities are private network isolation, an API gatekeeper that authorises every retrieval, and full auditability of retrievals and tool calls. Microsoft’s architecture guidance and OWASP’s LLM guidance anchor this.
TL;DR:
Choose a separate store for each tenant when isolation outweighs operating cost; shared stores need row level security or collection level tenant filters.
Apply tenant filters at query time, protect vector collections from manipulated sources, and record the passages behind each answer for review.
Use deterministic checks as defense; reserve model based screening for tool calls and write actions because it adds latency and is vulnerable to injection.
Choose serverless for bursty traffic, container orchestration for stable high volume, and microservices when agentic workflows require complex integrations across multiple systems.
Table of Contents
Which architecture pattern fits your use case?
Common chatbot system architecture patterns, as the Generative AI Atlas describes them, include basic GenAI, RAG-enhanced and agentic. The choice depends on accuracy needs, functional scope, data quality and integration requirements.
Basic GenAI: a direct model inference path. It suits conversational or creative tasks where answers need no strict grounding.
RAG-enhanced: the query is embedded, matching passages are retrieved and the context is assembled into the prompt. It fits knowledge-heavy tasks where answers must be traceable to documents.
Agentic: the model orchestrates tools and reasons iteratively, so it can call APIs, run transactions and complete multi-step processes.
Use these criteria to decide. How costly is a wrong answer? How many systems must the bot touch? How clean is your source data? What can you spend per conversation? Most enterprises land on RAG first, then add agentic steps for specific workflows.
What does each component of a chatbot backend do?
A sound chatbot backend architecture assigns one clear responsibility to each layer.
Application interface: handles authentication, session lifecycle and input sanitisation. It streams responses over REST with server-sent events or over WebSocket.
Application engine (orchestrator): builds prompts, decides whether to call retrieval or a tool, scopes each session and controls concurrency.
LLM gateway: gives unified access to models, manages tokens, selects the model per request and hosts the first guardrails.
Memory system: holds conversation state. Decide what stays ephemeral, what is persisted and how long each is retained.
Knowledge store and vector database: keeps embeddings plus metadata. Collections or namespaces separate tenants.
Tools and connectors: expose explicit interfaces with least-privilege permissions and defined error handling.
For tool design, an example such as the Customerscore shows how a customer success platform exposes its functions to agents through MCP. Keep each tool narrow, so a compromised prompt cannot reach beyond it.
How do you secure a multitenant chatbot?
Start with the network. For enterprise-grade deployments, Microsoft recommends private endpoints and virtual network integration, which keep traffic inside the workload’s private infrastructure and block public access to chatbot portals and agents.
Then control what the model can see. Microsoft’s secure multitenant RAG guidance requires that users only ground on data they are authorised to access, and recommends an API layer for tenant routing, filtering and audit logging. OWASP’s vector and embedding weaknesses guidance adds a warning: relying on the LLM to filter data is weak, so the gatekeeper must authorise and filter before the model sees any grounded content.
Store-per-tenant: strongest isolation, higher operating cost.
Shared multitenant store: cheaper, but needs row-level security or collection discriminators.
Secrets and identity: keep secrets in vaults, use managed identities, retain access logs for audit.
Authorise before you retrieve: the gatekeeper, not the model, decides which grounding data a tenant may use.
How should you design RAG retrieval?
Retrieval quality decides answer quality. Decisions to settle early:
Chunking and embeddings: balance recall, token cost and relevance. Our chunking guide for retrieval-augmented chatbots covers the engineering detail.
Vector database organisation: use namespaces, metadata and fine-grained access controls so tenant filters apply at query time. OWASP’s LLM6 card notes that retrieval sources can be manipulated to change model behaviour, so protect vector collections accordingly.
Provenance: record which documents and passages shaped each response.
Operations: set an index refresh cadence, cache hot retrievals and define fallback behaviour for stale or unavailable indexes.
Which guardrails stop prompt injection?
Layer your defences. Deterministic controls come first: input sanitisation, encoding checks, structured prompts and least-privilege tool scopes. Our engineering taxonomy of LLM guardrails lays out the options.
A model-based guard, where an LLM screens inputs and outputs, adds a further filter. OWASP’s prompt injection cheat sheet cautions that guard models are themselves susceptible to prompt injection and add latency and cost, so treat them as one layer only. The OWASP Top 10 for LLM Applications 2025 flags excessive agency and calls for threat modelling whenever an LLM gets tool access.
Apply heavy checks to high-risk paths such as tool execution.
Set thresholds that trigger alerts and a defined incident response.
Pro Tip: Spend your guard-model budget on tool calls and write actions, not on every harmless read.
For mitigation steps, see our prompt injection prevention guide.
How do you deploy, scale and run it?
Match the platform to the load. Serverless suits bursty traffic, container orchestration suits stable high volume, and microservices suit complex agentic workflows. Microsoft’s Foundry chat reference architecture shows one complete layout, with an application gateway, private endpoints, App Service, Azure Key Vault and Azure Cosmos DB inside a virtual network.
Track the right metrics: time-to-first-token, output tokens per second, total response time, token cost, error rate and concurrent sessions.
Plan graceful degradation: serve cached knowledge or a reduced model set when a dependency fails.
Cache hot retrievals: repeated questions should not pay for repeated searches.
Route by complexity: send simple tasks to smaller models and keep large models for hard ones.
Our post on building an AI operating model covers the surrounding practices.
What governance does an enterprise chatbot need?
Auditors ask who asked what, what the bot retrieved and what it did next. Build for those answers.
Identity: connect your enterprise identity provider end to end, so authorisation flows into retrieval and tool access.
Audit trails: log each request with its grounding sources and tool invocations, so forensic review is possible.
Data residency and retention: make both configurable. Attribute any legal requirement to its own jurisdiction.
Change control: govern prompt edits, model upgrades and runbooks like code releases.
Our note on running LLMs on private data covers compliance controls.
Our pilot-to-production checklist
These steps carry a pilot into production:
Isolate the network with private endpoints.
Put an API gatekeeper with tenant filters in front of every retrieval.
Instrument provenance and monitoring for every retrieval and tool call.
Our team applies this sequence on client engagements, from strategy through managed operations. Further implementation notes are in our insights hub.
Why the model is not your biggest enterprise risk
Most conversations about enterprise chatbots obsess over model choice, yet the failures that matter happen around the model. A leaked tenant document, an over-permissioned tool or an unlogged retrieval will hurt you far more than a slightly weaker model ever will.
Our view: treat the model as replaceable and the gatekeeper as permanent. Spend your first weeks on network boundaries, authorisation before retrieval and audit logs, and only then tune prompts. Agentic features deserve the same suspicion. Grant each tool the smallest permission that works, and add capabilities one at a time, with threat modelling at every step.
Guard models are overrated as a primary defence. They help, but a deterministic check that cannot be talked out of its decision beats a clever judge every time.
— Thomas Samuel
How we help you build secure, production-ready chatbots
We own the work from strategy to operations: data and platform engineering, integration, deployment and managed AI operations, with a senior team accountable throughout. That means the gatekeeper, the isolation and the audit trail you read about here are built and run by the same people. Our services cover each stage, so you can start with a readiness review or go straight to build.
Talk to us about your architecture through our AI & GenAI Solutions team.
FAQ
What is the architecture of a chatbot?
A chatbot architecture combines an application interface, an orchestrator, an LLM gateway, conversation memory, a knowledge store and tools. Enterprise designs add network isolation, an API gatekeeper and audit logging around those layers.
What are the three types of chat architecture patterns?
The three common patterns are basic GenAI, RAG-enhanced and agentic, as described in the Generative AI Atlas. Basic GenAI answers directly, RAG grounds answers in retrieved documents, and agentic designs add tool use and iterative reasoning.
Which AI domain powers chatbots?
Natural language processing powers chatbots, today mostly through large language models. Retrieval and tool orchestration sit on top to ground answers and perform actions.
What is the difference between a chatbot and ChatGPT?
A chatbot is any conversational application, while ChatGPT is one specific product built on a large language model. An enterprise chatbot can use a model like it, but wraps it in your own retrieval, security and governance layers.
Sources
Recommended