ML System Architecture: Components, Patterns and Enterprise Design
An ML system architecture is the design of the components and data flows that build, train, serve and monitor machine learning models, together with the non-ML software around them. The model is one part. Data pipelines, a model registry, serving infrastructure, monitoring and governance decide whether it still works in month twelve. This guide sets out each layer, the choices that matter most, and what Malaysian enterprises in banking, insurance, manufacturing and logistics should build in from the start.
TL;DR
ML system architecture covers the whole path from data source to monitored prediction, not just the trained model.
A sound enterprise blueprint separates the data estate, administration, model development and deployment, and links them through CI/CD, a model registry, access controls and feedback loops.
Choose batch inference when predictions can be scheduled, and real-time inference only when a decision must return within a request.
Monitoring for data drift, model decay and infrastructure faults is what triggers retraining, so design it on day one.
In Malaysia, voluntary AI ethics guidance sits on top of binding privacy, cybersecurity and sector rules, and the architecture should produce evidence for all three.
Table of Contents
What an ML system architecture is, and what it is not
A model is a trained artefact. An architecture is the full set of parts that make the artefact useful and safe: data ingestion, quality management, training, orchestration, specialised compute and the applications that consume predictions. TechTarget’s guide to planning a machine learning architecture draws exactly this line between an ML workload and a conventional web application.
The difference from traditional software is where behaviour comes from. In ordinary software, engineers specify the logic. In an ML system, functionality is inferred from data. Researchers working on architecture for machine learning-enabled systems argue that this calls for dedicated architectural views for data scientists and data engineers. They also argue that ML artefacts need an identity of their own, separate from source code and raw data, and that versioning, monitoring, retraining, privacy, explainability and fairness belong in the design from the outset.
In practice this means three things for an enterprise team:
First, the data is part of the system. A change in an upstream source can alter predictions without a single line of code changing.
Second, the system keeps changing after launch. Models decay as the world moves, so retraining is a design requirement, not an afterthought.
Third, ownership crosses disciplines. Data engineering, data science, software engineering, security and operations all touch the same pipeline, which is where handoffs tend to cause failures.
The layers of an end-to-end architecture
The major cloud reference designs describe the same flow in slightly different words. Microsoft’s MLOps architecture guidance moves from the data estate through ingestion, exploratory analysis, experimentation, model development and evaluation. From there it goes to model registration, staging tests and approval, then batch or online deployment, monitoring, and potential retraining.
AWS’s machine learning lifecycle architecture adds detail on the supporting components: offline and online feature stores, a model registry, lineage tracking, a scheduler, an alarm manager and feedback loops for performance and drift.
The table below turns these into the layers a design review should cover.
Layer | What it does | Question to settle |
Data estate and ingestion | Brings organisational data into a governed platform | Who owns each source, and how is quality checked on arrival? |
Feature management | Prepares inputs for training and serving | Are training and serving features computed the same way? |
Experimentation and training | Builds and evaluates candidate models | Can any result be reproduced from versioned code and data? |
Model registry | Stores model artefacts and metadata about data, code and models | Can you trace a live prediction back to its training data? |
Release and serving | Promotes approved models to batch or online production | What gates must a model pass before it takes traffic? |
Monitoring and feedback | Watches data, model and infrastructure, and triggers action | Who is alerted, and what happens automatically? |
The feature layer deserves attention. AWS explains that offline stores support training and batch scoring, while online stores support low-latency inference. Mismatches between the two are a common source of quiet errors. Our own write-up on feature store design for sub-10ms serving covers how to keep them consistent.
The registry is the other piece that pays for itself. AWS notes that it stores trained artefacts along with metadata on data, code and models, so that model lineage can be tracked much like version control. For a bank or insurer, that lineage is the beginning of an audit trail.
An ML reference architecture for the enterprise
A reference architecture is a reusable blueprint, so that each new use case does not start from a blank page. The most practical structure separates four areas, as in the Microsoft MLOps v2 blueprint: the data estate, administration and setup, the model-development inner loop, and the deployment outer loop.
The data estate holds organisational data sources and targets. Administration establishes repositories, workspaces, datasets, compute, roles, access controls, CI/CD pipelines and monitoring. The inner loop is where data scientists iterate on experiments. The outer loop takes an approved model into staging and production. Registries, access controls, monitoring and feedback connect them.
The value of this separation is organisational as much as technical. It lets a data science team move quickly in its loop without holding production credentials, and it lets a platform team enforce controls once rather than project by project.
For a first reference architecture, we suggest keeping the scope narrow:
Start with one governed data platform rather than one copy of data per project. Our piece on a lakehouse for AI that avoids a second vector store explains why consolidation also helps audit readiness.
Standardise the path to production before you standardise the tooling. A single, repeatable route from registered model to live endpoint matters more than which orchestrator you pick.
Decide early how identity, secrets and network boundaries work across loops, because retrofitting them is expensive.
Batch or real-time inference
The serving pattern shapes cost, infrastructure and risk more than almost any other decision. According to LaunchDarkly’s guide to model deployment, batch suits work where predictions can be scheduled and consumed later, such as forecasting, reporting, segmentation or portfolio scoring. It has fewer latency constraints and failed jobs can often be rerun. The costs are that predictions can go stale and that batch cannot serve interactive applications.
The same guide describes real-time inference as the right choice when a decision must come back with low latency, for example payment fraud detection, recommendations, dynamic pricing or instant risk scoring. It demands low-latency serving, high availability, scaling, load balancing and request-level observability.
Factor | Batch | Real-time |
Typical Malaysian use | Overnight credit portfolio scoring, demand forecasts, claims triage queues | Card transaction screening, live document classification, conversational agents |
Latency need | Hours acceptable | Milliseconds to seconds |
Failure handling | Rerun the job | Needs redundancy and fallbacks |
Infrastructure | Scheduled compute, cheaper to run | Always-on endpoints, higher operating load |
Main risk | Stale predictions | Outages and latency spikes |
Both patterns are supported by managed platforms. Microsoft lists managed batch endpoints for batch scenarios and managed online endpoints or Kubernetes deployments for near-real-time scenarios.
Many enterprises need both. A document-processing workflow, for instance, may classify incoming trade or insurance documents in real time and re-score the backlog overnight. Our practical guide to model serving architecture walks through the trade-offs in more depth.
Serving, versioning and safe releases
A model should reach production through a gated route, not a manual copy. The Microsoft guidance describes registering candidate models, testing them in a secure staging environment with data-quality, unit, endpoint-performance and responsible-AI checks, and then using gated approval and CI/CD to promote them to batch or online production. Monitoring of the model, the data and the infrastructure follows.
Google’s practitioners guide describes the mature version of this as continuous training: training is operationalised as a pipeline with CI/CD, repeated when new data arrives or performance decays, with candidate models registered and reviewed before deployment through online, batch or streaming patterns.
For real-time models, release control matters. LaunchDarkly recommends separating deployment from release: expose the new model gradually, monitor it in production, compare it with the previous version and keep a rollback ready, rather than sending all traffic to it at once.
This fits regulated environments well. A staged rollout with a recorded approval and a documented rollback is exactly the kind of control an internal auditor expects to see. For a phased approach over six to twelve months, see our MLOps practices for production teams.
Monitoring, drift and retraining
A model that was accurate at launch will not stay that way unattended. The Microsoft blueprint lists what to watch: model and data drift, performance on new data, responsible-AI issues, endpoint response, compute capacity and network problems. Its triggers send model or data issues back to development or retraining, and infrastructure issues to setup and administration.
Google’s guide adds serving logs, explanations, model decay such as data or concept drift, latency, throughput, hardware utilisation and execution errors to the list.
Two habits keep this manageable. First, decide in advance what a breach looks like, so an alert leads to a defined action rather than a debate. Second, route alerts to the team that can fix the cause. We cover the techniques in our guides to data drift detection and to a model retraining strategy that prevents silent failures. If your system includes large language models, LLM observability is the place to begin.
Governance and compliance in Malaysia
Malaysia has no comprehensive AI-specific statute at present. According to Legal 500’s 2026 Malaysia guide, the national AI governance and ethics guidelines cover fairness, reliability, safety, control, privacy, security, inclusiveness, transparency, accountability and human benefit, while existing sectoral laws continue to apply. The Future of Privacy Forum describes the 2024 guidelines as a voluntary framework for responsible AI development and use.
The binding layer is privacy and sector regulation. A January 2026 regional regulatory guide states that AI deployments processing personal data in Malaysia generally remain subject to the Personal Data Protection Act 2010, including its seven data-protection principles, and that organisations should document AI decision processes to demonstrate compliance and accountability.
The practical reading is a layered baseline: voluntary ethics guidance alongside binding privacy, cybersecurity and sector obligations. Banks and insurers should confirm sector-specific requirements with their compliance and legal teams, since those depend on the institution and the use case.
Architecture can make this evidence easy to produce:
The model registry and lineage records show which data and code produced each model.
Staging gates with recorded approvals document how a model reached production.
Access controls and logging show who touched which data and when.
Monitoring records show how the system was supervised after launch.
When the model reads sensitive or private data, the deployment pattern needs the same discipline. Guidance on HIPAA-compliant LLM deployment shows how operational and security controls shape an LLM rollout in a regulated setting, and our own article on running LLMs on private data with three compliance controls applies similar thinking to enterprise data.
Our perspective
Most ML projects we see struggle at the seams, not in the model. Strategy is written by one team, the build is done by another, and operations is inherited by a third. Each handoff loses context: why a feature was chosen, what the data limitations were, what the business was meant to measure. We design the architecture so that one senior team owns strategy, build and operation. That is why we engineer for production from day one, with continuity, governance, auditability and measured business results built in rather than added at the end. A registry, a gated release path and drift monitoring are not extras. They are what let a bank, an insurer, a manufacturer or a logistics operator trust a system enough to rely on it. Our advice to enterprise leaders is to start from the decision the system supports, then choose the batch or real-time pattern, then design the controls your auditors will ask about. Doing it in that order avoids most rework. Thomas Samuel
Where Sentient Concepts fits
If you are designing or inheriting an ML architecture, we cover the whole path: AI strategy and roadmap, readiness and data diligence, solution engineering, and deployment and MLOps, through to managed operations and ongoing optimisation. One team stays accountable from the first design review to the monitored system in production.
Manufacturers can see how this applies to document-heavy operations on our manufacturing page. To discuss your own architecture, start from our services overview and get in touch. For a view of what ongoing operations involve, read what managed MLOps covers and how to buy it.
Where to begin
An architecture review does not need a large programme. A useful first pass answers five questions about one candidate use case.
What decision does the prediction support, and how quickly is it needed? This settles batch versus real-time.
Where does the data live, who owns it, and is it fit for training? Our readiness and data diligence service is built around this question.
How will the model be versioned, approved and rolled back?
What will be monitored, and what happens when a threshold is crossed?
Which privacy, security and sector obligations apply, and what evidence will you need to show?
If several use cases compete for attention, use-case prioritisation and value mapping helps rank them by business value and feasibility, and operating model and enablement clarifies who runs what once the system is live.
FAQ
What is the difference between ML architecture and MLOps?
Architecture is the design: the components, data flows and controls. MLOps is the set of practices that operate that design, such as CI/CD for models, registries, monitoring and retraining. A good architecture makes MLOps practical, and MLOps keeps the architecture working after launch.
What is an ML reference architecture used for?
It is a reusable blueprint that gives every new use case the same route from data to monitored production. Typically it separates the data estate, administration, model development and deployment, and connects them through CI/CD, a registry, access controls and feedback loops. This reduces rework and makes governance consistent.
Should we choose batch or real-time inference?
Choose batch when predictions can be scheduled and consumed later, such as forecasting or portfolio scoring. Choose real-time when a decision must be returned within a request, such as fraud screening. Many enterprises run both, so the architecture should support each without duplicating the data layer.
Does Malaysia regulate ML systems specifically?
According to Legal 500’s 2026 guide, there is no comprehensive AI-specific law, and the national AI governance and ethics guidelines are voluntary. Existing laws still apply, including the Personal Data Protection Act 2010 where personal data is processed. Confirm sector requirements with your legal and compliance teams.
Sources
Machine Learning-Enabled Software and System Architecture (arXiv)
Tips for planning a machine learning architecture (TechTarget)
Machine learning operations, Azure Architecture Center (Microsoft)
Machine Learning Model Deployment: CI/CD and Release Control (LaunchDarkly)
Malaysia Charts Its Digital Course (Future of Privacy Forum)
Legal 500 Country Comparative Guides 2026: Malaysia, Artificial Intelligence
ASEAN guide to how AI is regulated in Southeast Asia, January 2026
Recommended