Data Pipeline for ML: Stages, Controls and Operations
A data pipeline for ML is the automated infrastructure that turns raw application or business data into versioned, validated training and test datasets, and keeps producing fresh ones after the first model ships. Google defines it as the part of the system that processes data into the sets the training and validation pipelines consume. Get it right and models stay current, auditable and reproducible. Get it wrong and the cost compounds: every retraining run inherits the same silent defects.
This guide explains what the pipeline contains, how its stages fit together, where it breaks, and which governance controls Malaysian organisations should build in from the start.
TL;DR
An ML data pipeline prepares and versions training and test data. The wider ML pipeline adds training, evaluation, deployment and monitoring around it.
The core stages are collection, validation, cleaning, transformation, feature preparation and splitting into training, evaluation and test sets.
Reliability comes from reproducing the same transformations in training and production, versioning every dataset, and monitoring data quality and drift.
In Malaysia, the Personal Data Protection Act 2010 and its 2024 amendments shape how personal data may move through the pipeline.
Document-heavy industries such as banking, insurance, logistics and manufacturing gain most when the pipeline is engineered for production from day one.
Table of Contents
What a data pipeline for ML actually is
Three terms are often used as if they were one. They are not, and the difference decides who owns what.
A conventional data pipeline moves and transforms data from sources into a warehouse, lake or lakehouse so that people can run reports and dashboards. It is built for consumption by analysts. An ML data pipeline has a narrower and stricter job: it produces the training and test datasets that a model needs. Teradata’s description of AI data pipelines adds that, unlike pipelines built for reporting, they must support unstructured and multimodal data and continuous retraining cycles.
The ML pipeline is the larger system around it. Google’s Rules of Machine Learning describes the pipeline as everything surrounding the algorithm: gathering data, putting it into training files, training models and exporting them to production. Microsoft’s Azure Machine Learning documentation frames it as an automated workflow for a complete ML task, split into steps that can be developed and automated independently, typically covering data collection, preparation, training, evaluation and deployment.
In practice, the data pipeline is the first segment of that workflow and the one that determines the ceiling for everything after it. A strong algorithm cannot compensate for a training set that does not resemble what the model meets in production.
The stages from source data to model-ready data
The sequence is usually described in four broad steps: collection, cleaning, transformation, and splitting into training and test sets, with a validation set added where needed. TechTarget’s overview of data preparation notes that the sequence is often iterative rather than strictly linear. In an enterprise setting each step needs an owner, a test and a record. The sections below walk through them.
Ingestion and source data
Source data comes from production systems or external providers: core banking platforms, ERP and MES systems in manufacturing, claims systems, shipping documents, call recordings, scanned forms. Microsoft’s Azure Well-Architected guidance separates this from training data, evaluation data and testing data, and the separation matters. Source data is raw material. It is not automatically suitable for AI, and the guidance is explicit that exploration and intermediate processing must remove noise and surface predictive information before the data becomes meaningful training material.
Two design decisions at ingestion are expensive to reverse. The first is whether ingestion is batch, streaming or both, which depends on how fresh the model’s inputs must be. The second is how schema changes are handled. Upstream teams rename columns, add fields and change formats without warning, so ingestion should be built to expect schema evolution rather than fail on it.
Validation and cleaning
Validation asks whether the data is what you think it is. Cleaning decides what to do when it is not. Checks for type mismatches, missing values, out-of-range values and duplicate records belong at the earliest point where they can be run, and a guide on feature engineering pipelines recommends validating both the raw data and the transformed features.
For document-driven use cases the same principle applies to extracted content. A field pulled from an invoice or policy schedule is data like any other, and it should be checked against reference data before it reaches a training set or a downstream system.
Transformation and feature preparation
Transformation and feature engineering are related but distinct. According to TensorFlow’s preprocessing best practices, data engineering converts raw data into prepared data, and feature engineering then tunes that prepared data into the features the model expects. Preprocessing can include cleansing, scaling, imputing missing values and transforming features.
The operational rule that follows is simple and frequently broken: whatever transformation was used during experimentation must be reproduced exactly in production. Google’s guidance says production data pipelines must reproduce the feature and label transformations established during experimentation. When a data scientist scales a field in a notebook and an engineer reimplements that logic separately for serving, the two versions drift apart and accuracy erodes without any error message.
Splitting and versioning datasets
The last stage produces the datasets the rest of the system consumes. Training data teaches the model. Evaluation data supports performance monitoring and hyperparameter tuning. Testing data comes from observations the model never saw during training, so it measures predictive power honestly, as the Azure Well-Architected guidance describes.
Every dataset should be stored and versioned so that a given model can be traced back to the exact data that produced it. MLRun’s workflow documentation describes tracking pipeline executions, data, metadata, code and results for precisely this purpose: reproducing experiments and identifying which data produced each model.
How the stages fit together
The stages above are easier to manage when you read them as a contract between producer and consumer. Each stage promises something to the next one, and a failed promise should stop the line rather than pass through.
Stage | What it produces | What should be checked before handing on |
Ingestion | Raw data landed in a controlled zone | Schema, completeness, source lineage |
Validation and cleaning | Trusted data | Types, ranges, duplicates, missing values |
Transformation | Prepared data | Same logic as used in experimentation |
Feature preparation | Model-ready features | Consistency between training and serving |
Splitting and versioning | Training, evaluation and test sets | No leakage between sets, version recorded |
The most common design failure is leakage: information from the test set, or from the future, reaching the training set. It produces impressive evaluation numbers and disappointing production results. The second is silent schema change. The third is a pipeline that works only on the laptop of the person who built it, because its steps live in notebooks and manual exports rather than in orchestrated, tested code.
Operating the pipeline after launch
A pipeline that runs once is a project. A pipeline that runs reliably every week is a system, and most of the value and most of the risk sit in the second phase.
Google’s perspective is that freshness and consistency are central: the data pipeline continuously creates newer datasets with the same features and labels used originally, so that the training and validation pipelines can stop production models from going stale. The validation pipeline compares a newly trained model against the one in production, using the test set, before anything is promoted.
The wider lifecycle is described in Microsoft’s Azure Databricks documentation: scoping, data exploration, data and feature preparation, training and experiment tracking, evaluation, model registration and staging, production deployment, and monitoring with retraining. Monitoring is not an afterthought in that sequence. It closes the loop back to data preparation.
What should be monitored? MLRun’s documentation lists operational statistics, data quality and model drift, and notes that training can be triggered automatically when drift is identified. Three practical points follow:
Measure data quality at the pipeline, not only model accuracy at the end. By the time accuracy drops, the damage has often been done for weeks.
Define in advance what counts as drift and what the response is: alert, retrain, or roll back to the previous model.
Keep the retraining path as tested as the first training path. A retraining job that has never been exercised will fail on the day it is needed.
We have written separate guides on detecting data drift in production and on building a model retraining strategy that prevents silent failures, and both pick up where this article leaves off.
Document processing: where the pipeline earns its keep
For banks, insurers, logistics operators and manufacturers, the most common entry point to ML is paperwork: invoices, purchase orders, shipping receipts, claims forms, contracts, KYC files. The data pipeline here begins with scanned or digital documents, and the ingestion and validation stages carry more weight than in a tabular project.
One industry view, published by a document AI vendor in 2026, holds that value is shifting from field extraction to reasoning across whole document packets, for example checking that a shipping receipt, an invoice and a purchase order agree with each other. Treat that as one vendor’s perspective rather than settled fact, but the engineering implication is sound. The pipeline should retain the relationships between documents, not just the text pulled out of each one, and it should validate extracted values against source systems before they influence a decision.
That is the approach behind our ICE-Ai intelligent document processing work for enterprises, where extraction, validation and downstream integration are designed as one flow with audit trails rather than as separate tools.
Quality, security and compliance controls in Malaysia
A pipeline that handles customer, employee or counterparty data is subject to data protection law from the first ingestion job. In Malaysia, data protection is primarily governed by the Personal Data Protection Act 2010. According to the Ministry of Digital, amendments passed in July 2024 include mandatory appointment of data protection officers by data controllers and processors, and mandatory breach notification by data controllers. The same source stresses that high data quality and robust data governance are important for reliable AI.
For pipelines that feed automated decisions, the 2026 guideline on automated decision-making and profiling recommends involving the data protection officer early, informing data subjects about the use of AI, mitigating risks, and not relying on AI as the sole factor in a decision. It also allows relevant personnel to be designated as reviewers. In pipeline terms, that means human review points and decision logs are design requirements, not later additions.
Public sector bodies have their own reference point. Malaysia’s February 2025 Public Sector AI Adoption Guidelines rest on seven principles, including privacy and data security, transparency, accountability and fairness, and include AI risk-management guidance and a self-assessment template for ethical compliance. Private enterprises that supply or partner with government will find the principles a useful baseline.
Translating this into pipeline controls:
Lineage and access. Record where each dataset came from, who can read it, and how long it is kept. Google’s guidance links the storing, versioning and management of training and test datasets to reproducibility, compliance, retention, access management and data integrity.
Minimisation. Strip or tokenise personal identifiers at the earliest stage that still lets the model learn what it needs.
Breach readiness. Know which stores hold personal data so a notification obligation can be assessed quickly.
Explainability trail. Keep the dataset version, feature version and model version that produced any decision that affects a person.
For deeper treatment, see our mapping of ISO/IEC 42001 and Malaysia’s PDP requirements to AI data governance, our overview of AI regulation in Malaysia, and the comparison of PDPA and GDPR enforcement. Legal interpretation for a specific processing activity should come from your data protection officer or legal counsel.
A practical build sequence
Teams that succeed rarely start with tooling. They start with a narrow use case and work outward in this order.
Pick a use case with a measurable outcome. Faster underwriting, fewer manual keying hours, shorter invoice cycle time. If the value cannot be measured, the pipeline has no way to prove itself.
Assess readiness and data. Establish what data exists, its quality, who owns it and whether it may lawfully be used for this purpose. This is where many projects change scope.
Design for production first. Decide how data will arrive, how transformations will be shared between training and serving, and how datasets will be versioned, before the first model is trained.
Automate and test. Move steps out of notebooks into orchestrated, tested jobs with validation gates between stages.
Deploy with monitoring. Connect data quality, drift and performance signals to alerts and to a defined retraining route.
Operate and optimise. Assign ownership for the live system, review results against the original business measure, and improve the pipeline on a schedule.
The sequence is deliberately unglamorous. Most failed ML programmes did not fail on algorithm choice. They failed because no one owned the data path once the proof of concept was over.
We build and run AI systems for enterprises, and the pattern we see most often is a gap between the people who design a data pipeline and the people who must live with it. A strategy team defines the use case, an engineering team builds a pipeline, an operations team inherits it, and each handoff loses context: why a field was excluded, which transformation was applied, what the drift threshold was meant to be. That is why we put one senior team in charge of strategy, build and operation. When the people who scoped the use case also own the pipeline in production, validation rules are written with the business consequence in mind, lineage and audit trails are there from day one, and retraining is planned before launch rather than discovered after the first decline in accuracy. For regulated sectors such as banking and insurance, we believe governance is part of the pipeline, not a layer on top. Versioned datasets, documented transformations and human review points are what let a compliance team say yes with confidence. Our approach is designed to deliver measured business results and operating cost savings, and to keep systems maintainable long after launch. Thomas Samuel
Our perspective
How Sentient Concepts can help
If your team is weighing an ML or document-processing project, the quickest way to reduce risk is to settle the data foundations first. Our Readiness & Data Diligence work assesses whether your data, systems and governance can support the use case, and our Data & Platform Engineering team builds the pipelines, platforms and integrations that follow.
After launch, our Deployment & MLOps and Monitoring & Evaluation services keep the pipeline observable and the models current. You can see the full range on our services page, or speak to us about the use case you have in mind.
FAQ
What is the difference between a data pipeline and an ML pipeline?
The data pipeline prepares and versions the datasets used for training and testing. The ML pipeline is the wider workflow that also covers training, evaluation, deployment and monitoring. In most organisations the two are built and owned together, because model quality depends directly on the data path.
How often should an ML data pipeline run?
It depends on how quickly the world the model describes changes. A fraud or pricing model may need frequent refreshes, while a document classifier may change slowly. Set the schedule from measured drift and business tolerance for stale predictions, then confirm it with monitoring rather than assuming it.
Do Malaysian organisations need a data protection officer for ML projects?
Under the July 2024 amendments to the Personal Data Protection Act 2010, data controllers and processors must appoint data protection officers, according to the Ministry of Digital. The 2026 automated decision-making guideline also recommends involving the officer early when AI is used. Confirm how it applies to your processing with your legal counsel.
What breaks ML data pipelines most often?
Upstream schema changes, mismatched transformations between training and serving, leakage between training and test sets, and missing lineage are the usual causes. Validation gates between stages, shared transformation code and dataset versioning address most of them.
Sources
Recommended