Experiment Tracking for ML: A Guide for Malaysian Enterprises
Experiment tracking in ML means recording, for every training run, what went in (code, data, hyperparameters, environment) and what came out (metrics, model files, plots), in a place your team can search and compare. Done from the first run, it is what lets you explain why a model was chosen, reproduce it months later and show an auditor how it was built. Done late, or in spreadsheets, the provenance of your best prototype is usually already lost.
This guide explains what to track, how the tooling works, how to set it up so it survives contact with production, and why it carries extra weight for banks, insurers, manufacturers and logistics firms operating under Malaysian data protection rules.
TL;DR
Experiment tracking is a searchable record of every run: code version, dataset, parameters, metrics and artefacts. It replaces spreadsheets and terminal scrollback.
Start on the first run, not when a model is mature. Early prototypes are the ones whose provenance disappears.
Log the environment as well as the metrics: dependency versions, hardware and data snapshots. Scalars alone do not make a result reproducible.
In Malaysia, tracking doubles as audit evidence. Where models profile individuals or influence credit decisions, you need to show which data was used and why.
Treat tracking as part of the production system, with named owners, retention rules and access control, not as a data scientist’s personal notebook.
Table of Contents
What experiment tracking is, and what it is not
A machine learning experiment is a training run with a specific combination of inputs. Experiment tracking is the practice of saving the metadata of each run and organising those runs so that you can find, compare and revisit them. In effect, it gives the team a queryable history instead of a memory.
It is easy to confuse with three neighbouring practices:
Version control records code changes. Tracking links each run to the exact commit, but also captures data, parameters and outcomes that Git does not hold.
Model registry manages which trained model is approved for which stage of deployment. Tracking feeds it with evidence about how each candidate was produced.
Monitoring watches a model after release. Tracking covers everything that happened before it.
The cost of skipping it compounds. A team that runs fifty experiments in a quarter and cannot say which dataset version produced the best one has to either retrain from scratch or deploy on faith. Neither is acceptable when the model reads loan files or prices insurance risk.
What to record on every run
Most teams begin with accuracy or loss and stop there. A tracking record that holds up in production has five layers.
Inputs: code, data and configuration
Log the commit hash, the configuration file, every hyperparameter and, critically, an identifier for the exact dataset used. A path to a folder is not an identifier, because the folder changes. Use a content hash, a table snapshot or a version tag from your data platform. MLflow’s tracking component describes the same core idea: an API and interface for logging parameters, code versions, metrics and output files.
Outputs: metrics, models and artefacts
Record metrics at each step as well as the final value, so you can see whether training was stable. Store the model weights, evaluation plots, confusion matrices and any sample predictions. For document processing models, keep a fixed evaluation set of anonymised documents so that results across runs are comparable.
Environment: the part most teams skip
Practitioners working on automated performance tooling note that enterprise-grade tracking needs the surrounding context as well as the metrics, such as hardware specifications and dependency versions. Simple metric logging often fails when someone tries to replicate a result on different compute. In practice, record the Python and library versions, the container image digest, the GPU or CPU type and the random seeds.
A 2024 paper on transparent reproducibility in machine learning makes a related point: reproducing an ML experiment means replicating more than the workflow steps, and tracking has to be integrated with the workflow itself.
Lineage: where the data came from
For each dataset, note its source system, extraction date, filters applied and who approved its use. If personal data is involved, record the lawful basis or the internal approval reference. This is the layer that regulators and internal audit ask about first.
Decisions: why a run was chosen
Add a short free-text note and a tag when a run is promoted or rejected. “Rejected: recall on scanned documents fell below threshold” saves the next engineer a week.
How the tooling works
Whatever product you choose, the mechanics are similar. Your training code calls a small client library, which sends key-value pairs for parameters and metrics, plus files, to a tracking server. A web interface then lets you filter and compare runs. Open-source MLflow, for example, offers Python, REST and other APIs to create experiments and runs and to log parameters, metrics and artefacts. It also supports automatic logging for popular libraries, so you do not have to write explicit log statements for every metric.
There are three broad ways to run it.
Approach | Strengths | Watch-outs for Malaysian enterprises |
Self-hosted open-source tracker | Full control of data location and access, no per-seat fees | Your team owns upgrades, backup, authentication and uptime |
Commercial hosted tracker | Fast to start, polished collaboration features | Metadata and artefacts may leave your environment, which needs a cross-border transfer review |
Tracking built into your cloud ML platform | One vendor, integrated permissions and billing | Lock-in, and weaker portability if you later change platform |
For regulated firms, the deciding question is rarely the quality of charts. It is where metadata and artefacts are stored, who can read them and how long they are retained. Artefacts such as sample predictions or feature tables can contain personal data even when the training data itself lives elsewhere.
A setup that survives production
A tracking tool alone does not create discipline. The following sequence works for most enterprise teams, and we would run it in this order.
Step 1: agree the run schema before the first experiment
Decide the mandatory fields: dataset identifier, commit hash, environment image, owner, business use case and data classification. Make the training template refuse to run without them. It is far cheaper to enforce this in a template than to chase it later.
Step 2: capture from the first prototype
Researchers at Vrije Universiteit Amsterdam, writing on continuous experiment-driven MLOps, observe that practitioners focus on performance metrics and often neglect tracking during the research stage. Their argument for treating experimentation as a controllable optimisation process is practical: knowledge carries across model iterations. Waiting until a model looks promising means its origin story is already missing.
Step 3: separate exploration from candidates
Let analysts explore freely in a sandbox project, but require a stricter record, with approved data and a clean environment, for any run that could be promoted. This keeps the audit trail focused without slowing discovery.
Step 4: connect tracking to your registry and deployment pipeline
A model should only reach staging if it points back to a tracked run. This is the link that turns tracking from a convenience into a control. It is also where our work on model retraining strategy begins: retraining decisions rest on being able to compare a new run against the exact baseline that is live.
Step 5: set retention, access and review rules
Decide how long run records are kept, who may view artefacts, and who reviews the tracker quarterly. Tie retention to your records policy and to the lifetime of the model plus any regulatory period that applies to the decisions it supported.
Why Malaysian enterprises should care more
Tracking is good engineering anywhere. In Malaysia, three developments make it a compliance matter for firms handling personal data.
First, the Personal Data Protection (Amendment) Act 2024. According to a summary of the amendment, its headline obligations took effect from 1 June 2025 and include a mandatory Data Protection Officer for qualifying organisations and 72-hour breach notification. If a breach touches a training dataset or a feature store, you need to know quickly which models and runs used that data. A tracker with dataset lineage answers that in minutes. Without it, the answer is a week of interviews. Confirm the exact thresholds that apply to your organisation with your legal counsel and the Personal Data Protection Commissioner’s published guidance.
Second, the Commissioner’s Automated Decision Making and Profiling guideline. It defines profiling as automated processing of personal data to evaluate or predict personal aspects of an individual. A credit scoring model, an insurance underwriting model or a customer propensity model all fall under that description. Where such a model is in scope, you should be able to show why specific data was used and how the model arrived at an output. Experiment tracking is the most natural place to keep that evidence, because it was written at the time the model was built rather than reconstructed afterwards. An earlier consultation version of the guideline explains the Commissioner’s reasoning: more decisions are being made automatically using AI and machine learning.
Third, government messaging on data quality. The Ministry of Digital has stressed that high-quality data underpins AI success, and tracking the exact datasets used in training supports accuracy and helps limit bias in regulated sectors such as finance.
For a deeper view of how these rules fit together, see our pieces on PDPA and AI regulation in Malaysia, mapping ISO/IEC 42001 to Malaysia’s PDP requirements and PDPA compared with GDPR.
What good looks like by industry
The fields to prioritise differ with the use case.
Banking and lending. Document classification and credit models need frozen evaluation sets, per-class metrics and a record of which document templates were in training. When a lender asks why a model misrouted a payslip, you should be able to find the run, the data and the threshold. Our write-up on audit-ready loan document classification shows how this plays out for Malaysian lenders.
Insurance. Underwriting support models should log feature definitions and the version of any rating rules they were compared with. Reviewers need to see how a recommendation relates to the data supplied.
Manufacturing. Predictive models depend on sensor calibration and plant-specific conditions. Log the line, the time window and the sensor firmware version, because a model that works on one line may not on another.
Logistics. Document extraction for bills of lading, invoices and customs forms varies by supplier format. Track performance by document type and language, including mixed Bahasa Malaysia and English, so averages do not hide a weak segment.
In clinical or diagnostic settings, standard MLOps is often not enough. One IEEE paper on medical imaging notes that medical AI workflows are more complex than traditional MLOps, with needs such as calibration, performance monitoring and retraining, and proposes collecting provenance for explanations alongside predictions.
Common mistakes
The same failures appear across organisations.
Tracking only the winner. Failed runs explain why the winner won. Keep them.
Letting each data scientist choose a tool. You end up with five partial histories and none that audit can read. Standardise one tracker per organisation, or at least one schema.
Logging personal data into free-text fields. Notes and sample outputs are searchable and widely visible. Set rules on what may be pasted, and mask identifiers.
No link to deployment. If production models cannot be traced to a run, the tracker is a diary, not a control.
Treating it as a one-off project. Trackers need owners, upgrades and periodic review. That is operations work, and it should be planned and budgeted as such.
Our perspective
We see the same pattern when enterprises move from pilot to production: the model works, but no one can say precisely how it was made. The data scientist who built it has moved on, the notebook is gone, and the question from risk or audit arrives six months later.
That is why we do not treat experiment tracking as a feature of a tool. We treat it as a design decision made on day one of an engagement, because we engineer for production from the start and prioritise continuity, governance and auditability. Because one senior team owns strategy, build and operation, the people who set the run schema are the people who will later answer for the model in production. There is no handoff in which the lineage gets dropped.
Our advice to enterprise leaders is simple. Ask three questions of any AI project before approving go-live. Can the team reproduce the deployed model from the tracked record? Can they say which data trained it? Can they show who approved that data for that purpose? If any answer is no, the cost is deferred, not avoided, and it tends to arrive during an audit or an incident.
For project sponsors, our guide to the AI project lifecycle in Malaysia shows where these checkpoints sit in a delivery plan.
— Thomas Samuel
Putting tracking into your delivery plan
If you are planning an AI programme in finance, insurance, manufacturing or logistics, tracking should be written into scope rather than left to individual teams. We cover it across several services: Readiness & Data Diligence establishes which data can be used and how it is documented, Data & Platform Engineering builds the pipelines and tracking infrastructure, and Deployment & MLOps connects tracked runs to release and monitoring. If you want us to run it after launch, Managed AI Operations keeps the record, the models and the controls maintained. You can see the full range on our services page, or talk to us about an engagement.
A short checklist before your next model review
Use these questions in your next steering meeting. Each should have a one-line answer and a link to evidence.
Does every candidate model point to a tracked run? Is the dataset identified by version rather than by folder name? Is the environment captured well enough to rebuild it? Are failed runs retained? Who may read artefacts that could contain personal data? Where is the tracking server hosted, and does that match your data residency and transfer position? How long are records retained, and who signs off deletion? When did someone last try to reproduce a production model from the record alone?
The last question is the most useful. A reproduction drill, run once a quarter on a model chosen at random, shows quickly whether your tracking is real.
FAQ
How do you track ML experiments?
Add a tracking client to your training code, log parameters, metrics, the dataset version and the code commit, and store model files as artefacts on a tracking server. Then use the interface to filter and compare runs. Begin with the first prototype and enforce a standard schema so records are comparable across teams.
How do you use MLflow to track experiments?
You call its logging functions, or enable automatic logging for a supported library, inside a run. MLflow records the parameters, metrics and files and displays them in its interface, where you can compare runs. Its documentation covers setup, and you should decide early whether to host the server yourself and how to secure it.
Is experiment tracking the same as model monitoring?
No. Tracking records how a model was built, before release. Monitoring observes behaviour after release, such as drift and error rates. They work together: when monitoring flags a problem, the tracked record shows what was trained and on which data, so you can decide how to retrain.
Does experiment tracking matter for PDPA compliance?
It can. Where a model profiles individuals or supports decisions about them, the Commissioner’s automated decision making and profiling guideline is relevant, and tracked lineage helps you show which data was used and why. It does not replace legal advice, so confirm your obligations with your Data Protection Officer or counsel.
Sources
Recommended