Building a genai architecture that survives contact with production
- 1 day ago
- 16 min read
A production genai architecture is an ecosystem, not a single call to a large language model. The pattern to start with is Retrieval-Augmented Generation combined with a dedicated orchestration layer, because it grounds outputs in verifiable data and keeps prompt logic separate from application code. Named entities matter here: RAG, OpenTelemetry, and vector databases such as Milvus or Pinecone form the backbone of almost every serious deployment reviewed across the industry, including the pattern catalogue documented by Martin Fowler’s team.
The minimal set of components you need before calling anything “production ready” is short but non-negotiable:
A client or UI layer that captures intent and renders responses safely.
An orchestration and prompt management layer that decides what to retrieve, what to send the model, and what to do with its output.
A retriever backed by a vector database, paired with an embedding model.
LLM inference, whether API-hosted or self-hosted.
A caching layer to control latency and cost.
Telemetry instrumented with something like OpenTelemetry, tracking token counts and time to first token.
A governance layer covering data provenance, access control, and audit trails.
If you’re briefing a team today, the fastest way to prove out this architecture is a small RAG pilot that exercises retrieval, reranking, and telemetry end to end, before you touch fine-tuning or multi-agent orchestration.
Key Takeaways
Production genai architecture succeeds when retrieval, orchestration, and telemetry are treated as core infrastructure rather than afterthoughts bolted onto a model call.
Point | Details |
Start with RAG and orchestration | Build a small retrieval pipeline with a dedicated orchestration layer before considering fine-tuning or multi-agent systems. |
Instrument telemetry from day one | Track time to first token, token counts, cache hit rates, and retriever precision to catch cost and quality issues early. |
Separate context from knowledge | Store short-term session context and long-term indexed knowledge in different tiers with different access controls. |
Match hosting to governance needs | Choose API-hosted models for speed and self-hosted models when data residency or cost at scale demands full control. |
Sentient Concepts delivers the full lifecycle | Sentient Concepts maps strategy, data engineering, build, and managed operations onto this architecture under one accountable team, with a pilot engagement as the practical starting point. |
Table of Contents
High-level genai architecture components and how they connect
Prompt engineering and orchestration as the application’s control plane
Retrieval-Augmented Generation and vector database architecture
Managing context and knowledge without drowning your prompts
MLOps, governance and observability for non-deterministic systems
Scaling, latency and cost tradeoffs that actually move the needle
How Sentient Concepts turns this reference architecture into a delivered system
Get your genai architecture from reference design to production
High-level genai architecture components and how they connect
A reference genai architecture is best understood as a pipeline with feedback loops, not a straight line. Data enters through the client layer, gets enriched and routed by orchestration, retrieves supporting context from a knowledge store, passes through the model layer, and returns through post-processing before it ever reaches a user. Each stage has a distinct job, and conflating them is where most architectures start to rot.
The GitHub Engineering team’s breakdown of production LLM applications groups components into four practical buckets: input tools, input enrichment and prompt construction, responsible AI tooling (caching, content classifiers), and inference hosting. That framing holds up well because it forces you to separate “getting the prompt right” from “running the model”, which are genuinely different engineering problems with different failure modes.
Component | Role | Typical technology |
Client/UI | Captures user intent, renders streamed output | Web/mobile front end, chat widget |
Orchestration/agent engine | Decides retrieval, tool calls, and prompt assembly | LangChain, custom controller |
Retriever/knowledge store | Supplies grounding context | Milvus, Pinecone |
Embedding service | Converts text to vectors for search | Dedicated embedding model, often smaller than the generation model |
LLM inference | Generates the response | API-hosted or self-hosted model |
Cache | Reduces repeated inference cost and latency | LLM response cache, semantic cache |
Telemetry/observability | Tracks performance and quality | OpenTelemetry, custom dashboards |
MLOps/governance | Versioning, audit, compliance | MLflow, policy engines |
Invariant services such as authentication, quota management, billing, and audit logging should sit outside the GenAI-specific components, treated as shared enterprise infrastructure rather than bolted onto the orchestration layer. Mixing the two creates a maintenance headache the moment you need to swap a model provider or add a second application on the same platform. Data governance controls, meanwhile, belong at every boundary where information crosses from your systems into a third-party model, not just at the point of storage.
Prompt engineering and orchestration as the application’s control plane
Orchestration is where your application’s actual logic lives, not the model itself. A large language model is a stateless function; the orchestration layer decides what context it sees, in what order, and what happens to its output. Get this layer wrong and no amount of model quality fixes the result.
There’s a real distinction between simple chaining, where each step feeds a fixed next step, and agentic approaches like ReAct or Reflexion, where the system decides its own next action based on intermediate results. Research on reflexive agent patterns shows these architectures let systems self-correct and handle multi-step failures that would break a rigid chain outright. Deterministic chains are cheaper to reason about and debug; agentic loops handle ambiguity better but cost more in tokens and latency, and they need tighter guardrails.
Prompt construction itself follows recognisable patterns:
Templates with clearly marked slots for retrieved context, user input, and system instructions.
Dynamic context assembly that ranks and trims retrieved passages before they enter the prompt.
Prompt optimisation loops that test variants against a held-out evaluation set rather than relying on intuition.
Operationally, treat prompts like code. Version them, store them alongside the orchestration logic that calls them, and cache both prompts and their results where the underlying context hasn’t changed. Tools like LangChain exist precisely because hand-rolled orchestration tends to sprawl into unmaintainable spaghetti once you add retries, tool calls, and multiple prompt variants.
Pro Tip: Prompt bloat is the silent killer of latency budgets. Structure prompts so the highest-value context sits closest to the instruction, and hand your reranker a strict token budget, typically 500 to 1,500 tokens for retrieved passages, rather than letting retrieval count dictate prompt length.
Retrieval-Augmented Generation and vector database architecture
RAG is the default production pattern because it solves three problems at once: grounding responses in verifiable data, keeping information current without retraining, and giving you a lever to control what the model can and cannot say. Martin Fowler’s analysis of emerging GenAI patterns treats RAG as the industry standard specifically because it mitigates hallucination more reliably than prompt engineering alone, though it rarely eliminates the problem outright without additional reranking and query rewriting.
The retrieval pipeline has more moving parts than most first drafts assume. Chunking strategy determines what gets embedded and retrieved: too large and you dilute relevance, too small and you lose context that spans paragraphs. Embedding model choice affects both retrieval quality and cost, since embedding calls happen far more often than generation calls. Vector store choice, whether Milvus for self-hosted flexibility or Pinecone for managed simplicity, shapes your operational burden more than most teams expect going in.
Hybrid retrieval, combining dense vector search with traditional keyword search, consistently outperforms either method alone on queries with exact terms like product codes or names. Rerankers, often small cross-encoders, then reorder the hybrid results before they hit the prompt, trimming noise that a pure vector search would have let through.
Approach | Purpose/role | Latency impact | Cost tradeoff | Reliability |
Pure vector search | Semantic similarity retrieval | Low, single query | Vector search cost scales with index size | Misses exact-match terms |
Hybrid (vector + keyword) | Combines semantic and lexical matching | Moderate, two queries merged | Slightly higher compute | Better precision on named entities |
Cross-encoder reranking | Reorders candidates by relevance | Adds a step but shrinks prompt size | Small additional inference cost | Reduces hallucination from noisy context |
Query rewriting | Reformulates ambiguous queries | Adds one LLM call before retrieval | Extra API call per query | Improves recall on vague inputs |
Index freshness is an operational decision as much as a technical one. Some knowledge bases need near-real-time reindexing (pricing, inventory); others tolerate a nightly batch (policy documents, product manuals). Decide this per data source, not once for the whole system, and design your sharding and partition strategy around the update frequency of each source rather than a single global schedule.
Pro Tip: Reserve fine-tuning for cases where RAG genuinely can’t help, typically when you need the model to follow a specific tone, format, or reasoning style rather than recall specific facts. If the problem is “the model doesn’t know X”, that’s a retrieval problem, not a training problem.
Choosing and hosting the LLM layer
The model layer decision splits into API-hosted versus self-hosted, and the trade-offs are sharper than vendor marketing suggests. API-hosted models give you fast iteration, no infrastructure to manage, and a steady upgrade cadence as providers ship improvements. Self-hosted open models give you control over data residency, predictable unit economics at scale, and freedom from provider rate limits, but you inherit the operational burden of serving infrastructure and model updates yourself.
Licensing is not a footnote here. Many self-hosted options ship under the Apache 2.0 licence, which permits commercial use and modification with minimal restriction, but you still need to verify licence terms for every model and toolchain component before committing to a self-hosted path, particularly in regulated industries where redistribution rights matter.
Embedding models deserve a separate decision from generation models. Many teams default to using their generation provider’s embedding endpoint, but a dedicated, smaller embedding model is often cheaper to run at the volume retrieval demands and can be swapped without touching your generation pipeline. Refresh your embeddings whenever you materially change chunking strategy or switch embedding models entirely; mixing embedding spaces from two model versions in one index silently degrades retrieval quality.
Fine-tuning, or lighter approaches like parameter-efficient fine-tuning (PEFT), makes sense when you need consistent formatting, domain-specific terminology, or a narrower behavioural profile than prompting alone reliably delivers. It rarely makes sense as a first move.
Factor | API-hosted | Self-hosted |
Latency | Depends on provider network, variable | Controllable, but requires GPU capacity planning |
Cost profile | Pay per token, scales with usage | High upfront infrastructure cost, lower marginal cost at scale |
Data residency | Depends on provider region options | Full control over where data lives |
Governance | Limited visibility into model internals | Full auditability of weights and behaviour |
Operational complexity | Low, provider manages scaling | High, requires MLOps and GPU orchestration |
Managing context and knowledge without drowning your prompts
Context and knowledge are two different storage problems, and treating them the same is a common architecture mistake. Context is short-term: the current conversation, recently retrieved passages, the last few turns of state. Knowledge is long-term: your indexed document corpus, product catalogues, policy libraries. Context lives in fast, ephemeral storage tied to a session; knowledge lives in a vector database or hybrid index designed for durability and search.
Building a resilient knowledge base means treating documents as versioned assets, not static files. Every chunk needs metadata (source, date, author, access level) and a canonical ID that survives reindexing, so you can trace a generated answer back to the exact document version that produced it.
Enrich each chunk with metadata that supports both filtering and audit, not just retrieval.
Assign canonical, stable IDs to documents and their chunks so reindexing doesn’t break existing citations.
Version embeddings alongside source documents so you can roll back a bad reindex.
Filter personally identifiable information before it ever reaches an embedding index, not after.
Storage tier | Contents | Retention approach | Access control |
Session context | Recent turns, active retrieval results | Cleared at session end or short TTL | Scoped to the active user session |
Knowledge base index | Embedded documents, chunks, metadata | Retained per document lifecycle, reindexed on change | Role-based access, filtered at query time |
Audit log | Prompts, retrieved sources, model outputs | Retained per compliance requirement | Restricted to compliance and engineering teams |
Encryption at rest and in transit is table stakes for any knowledge store feeding a genai architecture, but access controls need to operate at query time too. If a user shouldn’t see a document in a normal search, your retriever shouldn’t be allowed to surface it as context either, and that filter has to sit inside the retrieval layer, not as an afterthought in the UI.
Connecting generative AI to your existing systems
The moment a genai architecture needs to do something, not just say something, the integration layer becomes the riskiest part of the whole design. Connecting an orchestration layer to transactional systems, whether that’s a payment processor, an inventory database, or a claims system, requires more discipline than a typical API integration because the calling logic is driven by a model, not deterministic code.
Synchronous API calls work for low-risk lookups. Event-driven integrations suit higher-volume or asynchronous workflows, where the orchestration layer publishes an intent and a downstream service handles execution independently. Whichever pattern you choose, rate limits, retries, and idempotency need to be explicit design decisions, since a model-driven caller can retry more aggressively or unpredictably than a human-triggered request.
Separate inference from actuation: let the model propose an action, but route anything irreversible through an approval step or a sandboxed execution environment first.
Log every external call with a link back to the source documents or retrieved context that justified it, so a bad outcome can be traced and, where needed, rolled back.
Treat idempotency keys as mandatory for any write operation an agent triggers, since duplicate calls are far more likely with LLM-driven retries than human ones.
Enterprises exploring agentic AI before it enters live workflows consistently find that the traceability question, not the model quality question, is what determines whether a pilot survives contact with compliance review.
MLOps, governance and observability for non-deterministic systems
Running a genai architecture in production means operating a system that behaves differently on the same input from one day to the next. That demands an MLOps lifecycle adapted specifically for generative models, not a repurposed version of classical ML ops. The Snowflake framework for GenAI architecture organises this into five layers: data processing, the model layer, feedback and continuous improvement, deployment and integration, and monitoring and maintenance.
Telemetry needs to be first-class from day one, not retrofitted after a cost overrun. Production teams that treat observability as core infrastructure track time to first token, total token counts, cache hit rates, and retriever precision, correlating all four against user satisfaction and spend. Skipping this instrumentation is one of the most common reasons inference costs spiral without anyone noticing until the invoice arrives.
Key metrics worth dashboarding:
Time to first token (TTFT), the clearest proxy for perceived responsiveness.
Total tokens per request, split between prompt and completion.
Cache hit rate across your LLM cache and semantic cache layers.
Retriever precision, measured against a labelled evaluation set.
User acceptance or thumbs-up rate on generated outputs.
Shadow evaluation results comparing a candidate model or prompt against production before rollout.
Guardrails need to operate at both ends of the pipeline: input sanitisation to catch prompt injection and malicious instructions, and output filtering to catch policy violations or unsafe content before it reaches a user. Human-in-the-loop escalation should trigger automatically when confidence signals drop below a defined threshold, rather than relying on users to report bad answers after the fact.
A governance checklist worth keeping visible to your whole team:
Version every prompt, model, and retrieval index change with a rollback path.
Maintain provenance logs linking outputs to the exact model version and retrieved sources used.
Keep explainability logs sufficient to answer “why did the system say this” for any given output.
Build regulatory auditing hooks into the pipeline from the start, not as a retrofit before an audit deadline.
Tools like MLflow handle much of the experiment tracking and model lifecycle management this layer demands, giving you a single place to trace which model version, prompt template, and retrieval index combination produced a given production output. Teams building out a full AI operating model tend to find this governance layer is what separates a durable deployment from a demo that quietly stops working three months in.
Scaling, latency and cost tradeoffs that actually move the needle
Latency in a genai architecture splits into two very different problems: prefill and decode. Analysis of large-scale inference systems shows prefill is compute-bound, meaning it scales with prompt length and GPU throughput, while decode is memory-bandwidth bound, meaning it scales with how fast you can stream tokens from memory. Continuous batching and KV-cache management address the decode side specifically, keeping GPUs busy across multiple concurrent requests rather than idling between tokens.
Caching remains the single highest-leverage lever available to most teams. An LLM cache that stores full responses for repeated or near-duplicate queries can eliminate a meaningful share of inference calls outright, particularly in customer support or FAQ-style applications where the same questions recur constantly.
Trim prompts aggressively; every unnecessary token in a system prompt costs money on every single call.
Route simpler queries to smaller, cheaper models and reserve your largest model for genuinely complex reasoning tasks.
Use admission control and priority queues so a traffic spike degrades gracefully instead of taking down the whole service.
Apply speculative execution for latency-sensitive paths, where a smaller model drafts a response that a larger model verifies rather than generates from scratch.
Beyond batching and caching, model quantisation and distillation offer real latency gains for teams running self-hosted models. Quantising a model to lower precision reduces memory bandwidth demands during decode, directly attacking the bottleneck described above, while distillation into a smaller student model can cut inference cost substantially for narrower tasks where a full-sized model is overkill.
Pro Tip: Measure cost per useful token, not cost per token. A cached hit or a rejected low-confidence output costs you infrastructure spend without delivering value, so your real unit economics are worse than your raw API bill suggests until you factor in acceptance rate.
Reference architecture patterns and when to use each
Most genai systems fall into one of a handful of recognisable shapes, and picking the wrong one early is expensive to unwind later.
Pattern | Best fit | Key tradeoff |
Monolithic inference service | Early prototypes, single use case | Fast to build, hard to scale independently |
Microservice decomposition | Multiple applications sharing retrieval and model infrastructure | More operational overhead, better isolation and reuse |
GenAI-native cell | Regulated environments needing self-contained reliability | Higher upfront design cost, evolves independently of other systems |
Multi-agent systems | Complex, multi-step tasks with ambiguous paths | Higher token cost and latency, better resilience to partial failure |
Hybrid architectures | Enterprises running both deterministic workflows and agentic tasks | Requires clear boundaries between the two modes |
The GenAI-native cell concept is worth understanding even if you don’t adopt it wholesale: it treats each GenAI service as a self-contained module with its own reliability, assurance, and self-management logic, so it can evolve without dragging the rest of the system along with it. That matters most in regulated environments, where one service’s model upgrade shouldn’t force a re-certification of everything connected to it.
Choose monolithic when you’re validating a single use case and speed to first result matters more than scalability.
Choose microservices when multiple applications will share the same retrieval and model infrastructure.
Choose multi-agent architectures only when the task genuinely requires multiple decision points that can’t be resolved by a single chain, since the added latency and cost aren’t free.
On a reference diagram, annotate three things beyond the obvious data flow arrows: security zones (where PII enters and exits), telemetry capture points (where TTFT and token counts get logged), and approval gates (where an agent’s proposed action needs a human or policy check before execution).
How Sentient Concepts turns this reference architecture into a delivered system
Sentient Concepts maps its service delivery directly onto the lifecycle this guide describes, because a reference architecture only has value once it’s operating in someone’s production environment. The AI strategy and roadmap advisory phase covers scoping and use-case prioritisation, the same groundwork this article’s BLUF recommends before writing a line of orchestration code. From there, readiness and data diligence work assesses whether your existing data estate can actually support the retrieval quality a RAG pipeline needs.
Build phases follow the component map laid out earlier: data and platform engineering stands up the vector store and indexing pipeline, while GenAI solution engineering delivers the orchestration layer, retrieval logic, and integration adapters connecting to transactional systems. In finance, manufacturing, logistics, and insurance deployments, this has translated into automated document processing pipelines and conversational agents that reduce manual review time and cut operational cost, with the specific figures varying by client engagement and workload.
Delivery milestones worth building into any internal roadmap or vendor contract include a working RAG pilot within the first phase, an instrumented telemetry baseline before scaling traffic, a governance checklist signed off before production launch, and a defined handover into managed AI operations so the system keeps improving after go-live rather than degrading quietly once the build team moves on.
What most teams get wrong about genai architecture
The biggest misjudgement I see is treating architecture decisions as model decisions. Teams spend weeks benchmarking which large language model scores best on a leaderboard, then bolt it onto a prompt template with no retrieval strategy, no caching, and no telemetry, and wonder why the pilot stalls in review. The model is rarely the bottleneck. The orchestration layer, the retrieval pipeline, and the observability stack around them are where production systems succeed or fail.
The conventional advice to “start simple” is right, but it gets misapplied. Simple should mean a small RAG pilot with real telemetry from day one, not a bare prompt-and-response loop with no instrumentation at all. I’d rather see a team ship a narrow RAG pipeline with cache hit rates and retriever precision dashboards in week one than a broader multi-agent system with no way to tell why an output went wrong in week eight. The operational discipline is the hard part, not the clever architecture pattern.
If there’s one honest caution worth giving architects evaluating this space: budget real time for the governance and evaluation layer. It’s the part every roadmap underestimates, and it’s the part that determines whether your system survives its first compliance review or its first genuinely bad output in front of a customer.
Get your genai architecture from reference design to production
Reading a reference architecture is one thing; standing up the retrieval pipeline, orchestration layer, and telemetry stack that actually holds together under real traffic is another. Sentient Concepts runs the full lifecycle this guide describes under one accountable team, from initial strategy through build and into ongoing operation, so you’re not handing your RAG pipeline to one vendor, your governance framework to a second, and your monitoring to whoever’s left when things start drifting.
That continuity matters most in the months after launch, when most genai systems either compound in value or quietly degrade because nobody owns the retriever precision dashboard anymore. Our GenAI and AI solutions team builds the architecture; our managed AI operations service keeps it healthy afterwards, tracking the same telemetry this guide recommends instrumenting from day one. If you’re weighing a pilot, the practical next step is a scoping conversation to map your use case against this architecture and identify where a small RAG pilot could prove value within weeks rather than quarters. Get in touch to start that scoping conversation.
Sources
A handful of sources are worth returning to as you move from reference design into build:
FAQ
What is the architecture of GenAI?
A GenAI architecture is a layered system combining a UI, an orchestration layer, retrieval infrastructure (typically a vector database), an LLM inference layer, caching, and telemetry, rather than a single model call. The Snowflake five-layer model is a useful reference for organising these responsibilities across data processing, model, feedback, deployment, and monitoring.
How do I get started designing a genai architecture for my organisation?
Begin with a small RAG pilot that exercises retrieval, reranking, and telemetry end to end, rather than a broad multi-agent build. Firms like Sentient Concepts typically start with a strategy and readiness phase to scope the use case before any engineering work begins.
Is AI replacing architects?
No, in the sense of software architects designing these systems: AI tools accelerate parts of the design and coding process, but the architectural judgement about retrieval strategy, governance, and integration risk still requires human decision-making. In the physical building design sense, AI-driven architecture tools are increasingly used for automated architectural models and generative design exploration, but licensed human architects remain accountable for final designs and compliance.
Can ChatGPT be an architect?
ChatGPT and similar tools can assist with drafting prompts, generating code scaffolding, or exploring design options, but they cannot independently own architectural accountability, compliance sign-off, or the operational judgement a genai architecture requires. Treat it as a component within your orchestration layer, not a replacement for the architect designing the system around it.
What is the difference between RAG and fine-tuning in a genai architecture?
RAG retrieves relevant external context at query time to ground a model’s response in current, verifiable data, while fine-tuning adjusts the model’s underlying weights to change its behaviour or style permanently. Most production systems should default to RAG first and reserve fine-tuning for cases where prompting and retrieval genuinely can’t achieve the required tone or format.
Recommended
