Model Explainability Techniques: SHAP, LIME and Beyond

8 minutes ago

10 min read

Sketches framing an explainable AI title

Model explainability techniques are the methods used to show why a model produced a given output. They fall into a few families: feature attribution (SHAP), local surrogate models (LIME), counterfactuals, visual saliency maps, concept-based analysis and prototype-based reasoning. No single method suits every model. The right choice depends on the model type, the audience for the explanation and whether you need to explain one decision or the whole system.

For an enterprise, the question is rarely academic. A credit officer, an underwriter or an auditor will ask why a decision was made, and the answer has to be consistent, reproducible and on record. This guide explains the main techniques, compares SHAP and LIME directly, and sets out how to choose and operationalise them in a Malaysian context.

TL;DR

Explainability gives partial, post hoc insight into a model’s behaviour. Interpretability means the model is understandable by design.
SHAP offers theoretically grounded, consistent attributions at a higher computational cost. LIME is faster and local only, but can be unstable between runs.
Use local methods for individual decisions and global methods for model monitoring, bias checks and audit.
Malaysia’s national AI guidelines place transparency and explanations of AI decisions among their principles, so explanations should be designed in, not added later.
Treat explanations as production artefacts: version them, store them and monitor them alongside the model.

Explainability and interpretability: what each one means
The main families of explainability techniques
Local and global explanations
SHAP vs LIME: how they compare
How to choose a technique for your use case
Malaysian governance considerations
Putting explanations into production
Our perspective
Building explainability into your AI programme
FAQ
Sources

Explainability and interpretability: what each one means

The two terms are often used as synonyms, but they describe different things. The Carnegie Mellon textbook on machine learning in production defines a model as intrinsically interpretable when a human can understand its internal workings. Explainability, by contrast, means finding a mechanism that provides partial information about how the model works. Explanations built from external observation of a model, rather than from its internals, are called post hoc explanations.

That distinction matters in practice. A short decision tree or a linear scorecard is interpretable: you can read the logic. A gradient-boosted ensemble or a deep neural network is not, so you apply explainability techniques on top of it.

There is also a relationship with accuracy. Research on the performance-explainability trade-off notes that large and complex models often score highly on performance metrics but lack explainability, while rule-based systems, decision trees and linear classifiers are simpler and more transparent. For some real tasks, raising performance lowers explainability and the reverse. The job is to decide where on that spectrum a given use case should sit.

The main families of explainability techniques

A 2026 survey by Khan groups the field into six families of method. Each answers a slightly different question.

Feature attribution

These methods estimate how much each input variable contributed to a prediction, using gradient-based or game-theoretic principles. Integrated Gradients and SHAP are the best-known examples. For tabular business data such as loan applications, claims or invoices, this is the family most teams reach for first.

Local surrogate models

LIME is the standard example. As Holzinger and colleagues describe it, LIME fits a simple local surrogate model: it generates samples in the neighbourhood of the input of interest and approximates the target model with a simple linear function. The surrogate is easy to explain, even if the original model is not.

Counterfactual explanations

Counterfactuals identify the minimal change to an input that would alter the prediction. They answer the question a customer actually asks: what would have had to be different for this to be approved? That makes them useful wherever a decision may be challenged.

Saliency-based visualisation

Saliency maps generate visual or token-level importance maps. They are widely used for images and text, for example in visual quality inspection, where a reviewer wants to see which region of a part triggered a defect flag. Our guide to visual inspection AI for quality teams covers where this fits in a production line.

Concept-based analysis and prototypes

Concept-based methods map internal representations onto concepts a person understands. Prototype and case-based methods explain a prediction by comparing the input with representative examples the model has learned. Both are more common in deep learning and in specialised domains than in everyday enterprise scoring.

Holzinger and colleagues review 17 methods in total, including Anchors, layer-wise relevance propagation, TCAV and Shapley Flow. Most enterprise programmes will only ever use a small subset, which is why the selection logic below matters more than the catalogue.

Local and global explanations

Before choosing a method, decide what scope of explanation you need. A local explanation relates to a single prediction arising from a single input. A global explanation seeks to explain the model as a whole, showing the range of predictions it can make.

Local explanations serve the people who handle individual cases: the loan officer, the claims handler, the customer who disputes an outcome. Global explanations serve the people who govern the model: risk, compliance, internal audit and the engineers who need to find out whether the model leans on a variable it should not.

Most production systems need both. A model can look sound in aggregate and still behave oddly on a particular segment, and the reverse is also true. Plan for each audience from the start.

SHAP vs LIME: how they compare

These are the two methods enterprise teams meet first, and the comparison is worth setting out carefully.

SHAP is based on game theory. It treats each feature as a player and the model outcome as the payoff. According to a 2024 perspective in Advanced Intelligent Systems, SHAP provides both local and global explanations, while LIME approximates a complex model with a local interpretable model for a specific instance and offers local explanations only. The same paper reports that SHAP takes longer to compute, while LIME is much faster, especially with tree-based models.

On reliability, a 2025 survey of feature attribution techniques states that SHAP gives theoretically grounded attributions with properties such as local accuracy and consistency, and tends to be more stable and better at capturing feature interactions. It also notes that SHAP carries significant computational overhead because Shapley value estimation is combinatorial. LIME can produce different feature importances across repeated runs on the same instance, and its fidelity depends on the model being roughly linear near the point explained.

Criterion	SHAP	LIME
Scope	Local and global	Local only
Basis	Shapley values from game theory	Local linear surrogate fitted on perturbed samples
Consistency between runs	Generally more stable	Can vary with random perturbation
Computing cost	Higher	Lower, especially with tree-based models
Nonlinear behaviour	Depends on the model used	Cannot capture nonlinear associations locally
Typical use	Audit, compliance reporting, model monitoring	Quick checks, prototyping, real-time interfaces

A November 2025 review in MDPI AI reaches the same conclusion from another angle. LIME is efficient and model-agnostic but can be unstable, while SHAP is theoretically consistent at the cost of computational complexity. It adds that gradient-based methods such as Grad-CAM are faster for image data but limited to differentiable models, and that counterfactual approaches are intuitive but can struggle to scale across large datasets.

The practical reading is straightforward. When an explanation may end up in front of an auditor or a regulator, consistency matters more than speed, which favours SHAP. When you need a quick view during development or a lightweight explanation in an interface, LIME is often enough. Neither is a substitute for checking the result: Hall’s KDD 2019 paper recommends assessing how well LIME fits using measures such as RMSE, MAPE or R².

Caveats that belong in the documentation

Both methods are affected by the model they explain and by collinear features. The 2024 perspective recommends presenting SHAP results with plots and plain-language descriptions of the outcomes and the assumptions behind them, including feature independence and model dependence. We would add that this note should sit next to the explanation in the record, not in a separate slide deck.

How to choose a technique for your use case

We suggest working through four questions in order.

Who reads the explanation? A customer needs a short reason and, ideally, a counterfactual. A model risk committee needs global attribution and stability evidence. An engineer needs to debug.

What is the model? A tree ensemble on tabular data is well served by SHAP. A vision model suits saliency maps. A large language model needs different tools again, with attention-based and token-level methods covered in recent surveys, and in many cases retrieval citations and output evaluation matter more than attribution.

How often, and how fast? Real-time explanations at scale push toward cheaper methods or precomputed attributions. Batch audit reports can afford heavier computation.

What is the cost of being wrong? McDermid and colleagues note that where understanding the model is essential, the choice of ML method may be limited largely to interpretable models. Those models give contemporaneous explanations but can be less powerful, so they argue that XAI methods should be weighed explicitly when selecting a model where assurance is a key factor. In other words, explainability belongs in the model selection decision, not after it.

Explanations also have limits of purpose. Stanford HAI reports that Shah separates three kinds of explainability: how a model works, why an input produced an output, and what builds trust. A technique that serves engineers does not automatically serve a frontline user. Decide which kind of explanation each audience needs before you pick tooling.

Malaysian governance considerations

Malaysia has no single statute that prescribes explainability methods, but the direction of travel is clear. On 20 September 2024, the Ministry of Science, Technology and Innovation launched the National Guidelines on AI Governance and Ethics. Commentary on the guidelines describes a transparency principle that calls for clear explanations and justifications for algorithmic decisions, disclosure of AI use in decision-making, and mechanisms for challenging AI decisions.

For financial institutions, Bank Negara Malaysia has published a discussion paper on artificial intelligence in the financial sector. Industry responses to it, such as ACCA’s comments, treat explainability and opacity as a core AI risk area. Lenders should read the current Bank Negara Malaysia publications directly and confirm with their compliance function which requirements apply to their licence.

The practical implication is to build the evidence trail now. That means recording which explanation method was used for which decision type, which version of the model produced it, and who reviewed it. Our guide to credit risk modelling for Malaysian lenders shows how this applies where scorecards and machine learning models meet.

Putting explanations into production

An explanation that works in a notebook often fails in operation. Three habits close the gap.

First, treat explanations as outputs of the system. Store the attribution or counterfactual with each decision so it can be reproduced later, rather than recomputing it against a model that has since changed.

Second, monitor explanations as well as accuracy. If the features driving decisions shift, that is an early signal of drift, and a global attribution view will show it before headline metrics do.

Third, pair explanations with people. An explanation nobody reads is oversight theatre. We set out how to avoid that in four phases for making human-in-the-loop ML work, and the same discipline applies here: define who reviews, on what trigger, and with what authority to override.

These steps sit inside the wider delivery cycle. If you are planning one, our overview of the AI project lifecycle for project managers in Malaysia shows where explainability requirements should be written in, and our guide to model serving architecture covers where explanation services live at runtime.

Our perspective

Most explainability problems we see begin before any method is chosen. A model is built by one team, handed to another to deploy and to a third to run, and the reasoning behind its design is lost on the way. By the time an auditor asks why a decision was made, nobody owns the answer. That is why we keep one senior team accountable for strategy, build and operation. When the people who select the model are the people who run it, explainability becomes a design requirement: the method is chosen with the audience in mind, the explanation is stored with the decision, and governance and auditability are engineered for production from day one. Our view is that explanations should be judged by a simple test. Can the right person, at the right moment, understand enough to act, challenge or sign off? A technically elegant attribution plot that fails that test has not done its job. We would rather agree the audience and the evidence standard first, then pick SHAP, LIME, counterfactuals or an interpretable model to meet it. For regulated sectors such as banking and insurance, and for document-heavy operations in manufacturing and logistics, this is where measured business results and responsible AI meet. The cost of retrofitting explanations compounds, and building them in is cheaper. Thomas Samuel

Building explainability into your AI programme

If you are scoping a model that will touch credit, claims, underwriting or document decisions, we can help you define the explanation requirements before design begins. Our Readiness & Data Diligence work assesses whether your data and governance can support auditable models, and our Deployment & MLOps service keeps explanations, monitoring and model versions together once a system is live.

You can see the full range of our AI strategy, build and run services and talk to us about where explainability fits in your roadmap.

FAQ

Is SHAP better than LIME?

Not in every case. SHAP is generally more consistent and offers both local and global views, which suits audit and monitoring work, but it costs more to compute. LIME is faster and useful for quick checks, though its explanations can vary between runs.

Do I need explainability if my model is accurate?

Accuracy tells you how often the model is right, not why it decides as it does. Explanations let reviewers spot reliance on an inappropriate variable, support customers who query a decision and give auditors a record. Where decisions are consequential, they belong in the design.

Can I use an interpretable model instead of explaining a complex one?

Sometimes, yes. Decision trees, rule-based systems and linear models are transparent by design. They can be less powerful than ensembles or deep networks, so the choice should weigh the performance needed against the level of assurance required.

Which technique suits real-time use?

LIME is lighter and often used when speed and limited computing resources matter. SHAP can also be run at scale if attributions are precomputed or approximated for tree-based models, so test latency against your own service requirements.

Does Malaysia require explainable AI?

The National Guidelines on AI Governance and Ethics set transparency and explanation as principles, and Bank Negara Malaysia has opened a discussion on AI risks in finance. Check the current requirements for your sector and licence with your compliance team.