Feature Engineering Techniques for Production ML
Feature engineering is the work of turning raw records into inputs a model can learn from, and it usually decides model quality more than the choice of algorithm. The techniques that matter most are encoding categories, scaling numbers, handling missing values, building time and aggregate features, selecting what to keep, and, increasingly, automating the search. This guide covers each one, shows where they fail in production, and adds what Malaysian enterprises must consider under the Personal Data Protection Act 2010.
TL;DR
Start with the basics: encode categories, scale numeric values, treat missing values as information, and validate every transformation on held-out data only.
Time-based features must be point-in-time correct, or offline accuracy will not survive contact with production.
Feature selection (filter, wrapper and embedded methods) reduces cost, noise and audit burden. Automated feature engineering helps, but needs human review.
In Malaysia, features built from personal data fall under the PDPA 2010 and the ADMP Guideline, so privacy design belongs in the feature pipeline, not after it.
A feature is only valuable if it can be recomputed identically in training and in production.
Table of Contents
What feature engineering actually involves
A model never sees your business. It sees columns of numbers. Feature engineering is the translation layer: a transaction timestamp becomes “hour of day” and “days since last payment”, a free-text field becomes a category or an embedding, a customer history becomes a rolling average.
The authors of the well-known book on the subject, Max Kuhn and Kjell Johnson, describe it as closer to an art than a formula. Performance often disappoints because relevant predictors are represented in a way the model cannot use efficiently, and the fix is iterative refinement of transformations, interactions and functional forms. That matches what we see in practice: the first model rarely fails because of the algorithm. It fails because the data was handed to it in the wrong shape.
Practitioners also report that data preparation consumes most of a data scientist’s time, with estimates above 75% depending on the study. That is the commercial reason to treat feature engineering as an engineered, repeatable system rather than a notebook exercise.
Core techniques for tabular data
Most enterprise machine learning still runs on tabular data: loan applications, claims, sensor readings, shipment records. The techniques below cover the large majority of it.
Encoding categorical variables
Models need numbers, so categories must be converted. The right method depends on how many distinct values exist and how the model uses them.
Technique | Best for | Main risk |
One-hot encoding | Low-cardinality fields such as branch type or product line | Column explosion on fields with hundreds of values |
Ordinal encoding | Naturally ordered values such as risk grades | Implies equal spacing between levels |
Target encoding | High-cardinality fields such as merchant or postcode | Leakage if computed on the full dataset |
Frequency encoding | Rare-versus-common signals | Different categories can share a count |
Target encoding deserves particular care. It replaces a category with a statistic of the target, so it must be computed inside cross-validation folds. Computed on the whole dataset, it quietly feeds the answer into the question.
Scaling and transforming numeric values
Distance-based and gradient-based models are sensitive to scale. Standardisation centres each feature and divides by its standard deviation, while min-max scaling maps values to a fixed range; both are documented in the scikit-learn preprocessing guide, which also covers the effect of outliers on each scaler.
Tree-based models are largely indifferent to scaling, but skewed money fields such as transaction amounts or invoice values still benefit from a log transform, because it stops a handful of large values from dominating.
Binning and discretisation
Binning turns a continuous value into ranges: age bands, tenure buckets, order-size tiers. It can capture non-linear behaviour in simple models and makes features easier to explain to a risk committee. The cost is lost information, so choose boundaries from business meaning or quantiles, not by eye.
Missing values as a signal
A blank field is often informative. A missing income on a loan application is not the same as a low income. Impute with a median or a model-based estimate, and add a binary “was missing” indicator alongside. Fit the imputer on training data only, then apply it unchanged to everything else.
Ratios, interactions and aggregates
The features that earn their place are usually domain-driven. Debt-to-income, claim amount over policy limit, defect rate per production run, days between invoice and payment. These ratios encode how the business already thinks about risk. Polynomial and interaction terms can be generated automatically, but a ratio chosen by someone who understands the process will almost always outperform a thousand blind combinations.
Text and document fields
For documents such as invoices, bills of lading or claim forms, the feature is often extracted from text first: amounts, dates, counterparties, clause types. Large language models now do much of this extraction, and we cover how in generative AI for documents. The extracted fields then become ordinary tabular features, which is where the techniques above apply again.
Time-based features and the leakage trap
Time is where feature engineering most often goes quietly wrong. Rolling averages, lag values and “count of events in the last 30 days” are powerful for fraud, demand and maintenance models. They are also the easiest way to let the future leak into training.
The rule is point-in-time correctness. For any prediction timestamp t, every feature must be derived only from data that was available at t. Violate that, and the model learns from information it will never have in production. Offline metrics look excellent, then collapse after deployment. The Tecton documentation on constructing training data explains how feature stores handle this by joining each training row to feature values as they stood at that moment.
Three habits prevent most leakage:
Split training and validation data by time, not at random, whenever the use case is forward-looking.
Record when each source field became available, not just when the event happened. A settlement posted three days late was not known three days earlier.
Compute every aggregate with a window that ends strictly before the prediction time.
In banking and insurance, late-arriving data is the norm, so this is not an edge case. It is the main case.
Feature selection methods
Generating features is only half the job. Keeping every feature makes models slower, harder to audit and more fragile when upstream data changes. Selection methods fall into three families, as laid out in this overview of feature selection in scikit-learn.
Filter methods score each feature independently of any model, using statistics such as correlation with the target, variance thresholds or chi-square tests. They are fast and a sensible first pass for removing constants and obvious duplicates.
Wrapper methods train models on candidate subsets. Recursive feature elimination, which repeatedly drops the weakest feature, is the standard example. They find combinations that filters miss, but at a much higher compute cost.
Embedded methods select during training. L1 regularisation shrinks weak coefficients to zero, and tree ensembles report feature importances as a by-product. They sit in the middle on cost and are often the best default.
A practical sequence is: remove constants and near-duplicates with a filter, let an embedded method rank what is left, and use a wrapper only on a short list when the last point of accuracy justifies the compute. Then test stability. A feature that ranks first on one sample and fortieth on another is not a reliable feature.
Selection is also a governance tool. Every feature you retain is one you must document, monitor and defend. Fewer features means a shorter audit trail.
Automated feature engineering: what it delivers
Automated feature engineering generates and tests candidate transformations at scale. The published evidence is encouraging. The AutoFE system showed an average improvement of 25.24% in predictive performance across varied datasets and classifiers, and its features remain interpretable. A newer approach, CAAFE, uses a large language model to propose features with domain meaning, and lifted mean ROC AUC from 0.798 to 0.822 across 11 of 14 datasets tested.
Treat those as research results, not guarantees. Gains depend on the starting data, and automation also generates plenty of features that look clever and mean nothing. Our rule is that automated tools propose, and a person who understands the process disposes. Each accepted feature gets a plain-language definition, an owner and a test.
Why production changes the rules
A feature that works in a notebook is a hypothesis. A feature that works in production is an asset. The gap between them is mostly engineering.
Training-serving consistency. The code that builds a feature for training must be the same code, or the same definition, that builds it for live scoring. Two implementations will drift apart, and the model will degrade without any error message.
Versioning and data contracts. Version the inputs, the cleaning stages and the parameters. When performance drops, you need to know which of those changed. This applies equally to retrieval systems: in enterprise retrieval-augmented generation research, structured tabular documents were serialised to JSON that preserves row and column relationships and metadata before embedding, because plain text chunking destroys that structure. The same study found hybrid retrieval with metadata filtering lifted Precision@5 from 75% to 90%. If you build document assistants, our notes on chunking for retrieval-augmented chatbots and upstream choices in retrieval-augmented generation go further. Teams that also publish content for AI search can look at how restructuring pages for vector retrieval changes what gets cited.
Monitoring. Features drift when customers, suppliers or systems change. Watch distributions, not just model accuracy. See our production guides on data drift detection and model retraining strategy.
Evaluation. Measure the business metric the feature is meant to move. For generative components, our guide to LLM evaluation metrics sets out how.
A worked example from industry: in quality prediction on a production line, sensor readings sampled at different rates must be aligned, aggregated per batch and joined to inspection outcomes recorded later. Get the alignment wrong and the model learns the inspection schedule instead of the defect. We describe the hybrid approach in quality prediction in manufacturing.
Malaysian considerations: personal data in features
For Malaysian organisations, feature engineering is also a compliance activity. Features about identifiable people, such as repayment behaviour, claims history or employee records, are personal data.
The Personal Data Protection Department’s Automated Decision-Making and Profiling Guideline sets expectations alongside the Personal Data Protection Act 2010 (Act 709). In practice, as the guideline reads:
Engage your Data Protection Officer at the earliest design stage and carry out a Data Protection Impact Assessment before building features from personal data.
Mask, tokenise or pseudonymise personally identifiable information before model training where the feature does not need the raw identity.
Do not rely on AI as the sole factor in decisions about an individual. Designate human reviewers, which in turn means your features must support explanation.
That last point shapes engineering choices. A credit feature such as “months since last missed payment” can be explained to a customer and a reviewer. A 200-dimensional learned embedding of transaction text cannot. Prefer interpretable features where a decision affects a person, and keep the opaque ones for lower-stakes uses. Malaysia’s Ministry of Digital has likewise stressed that high data quality and sound data governance underpin accurate AI outputs. This guidance is current as of the 2026 guideline; check the Personal Data Protection Department for updates and take legal advice on your specific use case.
Our perspective
We see the same pattern in most enterprise engagements. The model is rarely the bottleneck. The bottleneck is that the people who define the features, the people who build the pipeline and the people who run it are three different teams, and each handoff loses meaning. A ratio defined in a workshop becomes a slightly different ratio in code, and a third version in the monitoring dashboard.
Our view is that feature engineering should be owned end to end by one accountable team. The person who chooses “days since last payment” should also care how it is computed at 3 a.m. on a live system, who can explain it to an auditor, and what happens when the source table changes. That is why we design for production from day one, with governance and auditability built into the feature pipeline rather than added at the end.
We are also cautious about automation. Automated feature generation is useful for widening the search, but in banking and insurance a feature nobody can explain is a liability. We would rather ship fifty features someone owns than five hundred nobody does.
— Thomas Samuel
Where to start in your organisation
If you are deciding where to invest first, the order matters more than the sophistication.
First, check readiness: are the source fields complete, consistently defined and legally usable? Our readiness and data diligence work answers that before any modelling begins. Second, build the data and feature pipelines on a platform that guarantees training-serving consistency, which is the focus of data and platform engineering. Third, operate and monitor.
If you want one team to carry a use case from feature design through to running it, see our AI and GenAI solutions and managed AI operations, or talk to us about your project.
FAQ
What is an example of feature engineering?
Turning a raw loan application into “debt-to-income ratio”, “months since last missed payment” and an indicator that income was left blank. Each new column expresses something a credit analyst already reasons about, in a form a model can use. The raw fields alone would hide those relationships.
Is feature engineering still needed with deep learning and LLMs?
For tabular business data, yes. Gradient-boosted trees and linear models remain strong there, and they depend on well-built features. Large language models change the work rather than remove it: they can extract fields from documents and even suggest feature code, but the results still need validation and consistent production computation.
Is feature engineering part of ETL?
They overlap but differ in purpose. ETL moves and cleans data for general use. Feature engineering shapes data for a specific predictive task, and must be reproducible at prediction time as well as in batch. In mature setups, both run on the same governed data platform.
How do I choose between feature selection methods?
Use a filter method first to remove constants and duplicates, then an embedded method such as L1 regularisation or tree importances to rank the rest. Reserve wrapper methods like recursive feature elimination for short lists, as they cost the most compute. Always check that the selected set is stable across different samples.
Sources
Recommended