Feature Attribution in Practice: Selection, Pipelines, and Governance

Introduction

Four articles established what feature attribution can and cannot deliver. Individual attributions from black-box models cannot be verified. Exact computation is possible for specific architectures. No single metric suffices for evaluation. Standard methods conflate association with causation. The practitioner’s question follows: what should you actually do?

The answer is not a single recommendation. It is a decision framework organised around the variables that matter: model access, domain stakes, regulatory requirements, computational budget, and stakeholder needs. The synthesis draws on the XAI-guided feature selection framework of Kost et al. as a worked example, a distillation of domain applications, and the major XAI surveys that establish context.

This article is not legal advice.

Operational and Governance Vocabulary

Feature governance

The organisational practice of documenting, monitoring, and controlling which features are used in ML models, including periodic review of feature relevance, data quality, and drift.

Dominance index

A Herfindahl-Hirschman-based measure of how concentrated feature importance is, where high dominance indicates that a single feature drives most of the predictive power of the model.

Model documentation

The practice of producing structured metadata about the intended use of a model, training data, evaluation results, and limitations, formalised by Mitchell et al. (2019) [2] as Model Cards.

Stakeholder-appropriate explanation

An explanation whose complexity, format, and content match the intended audience's technical expertise and decision-making needs, as opposed to a single explanation format for all audiences.

Cost-aware feature selection

Feature selection that accounts for the acquisition cost of each feature, not just its predictive contribution, enabling trade-offs between model performance and operational expense.

A Worked Example: Kost, Lier and Breitner (2025)

The XAI-FS framework by Kost et al. is the only paper in the selected set that directly addresses how attribution methods should be used within an ML development pipeline [1]. It is therefore the natural starting point for practice-oriented synthesis.

The scenario is a photovoltaic energy forecasting task with 15 features where Global Horizontal Irradiance dominates. The practitioner has two concerns: whether overdependence on GHI creates brittleness, and whether features can be removed to reduce data acquisition costs. The XAI-FS pipeline trains the model and computes attributions (SHAP, ELI5), computes a Herfindahl-Hirschman-based dominance index (GHI scores 0.365, indicating high concentration), tags each feature with acquisition cost, applies a multi-objective decision function balancing accuracy, dominance, and cost, and recommends feature removal. Removing GHI reduces R² from 0.943 to 0.934 while cutting dominance to 0.169 and improving noise robustness.

The framework turns attribution into a design-stage governance tool: the question becomes not “did the model predict correctly?” but “is the feature reliance of the model healthy?”

Scope: Validated on a single photovoltaic forecasting dataset with gradient boosting only. Uses SHAP and ELI5 but not the newer methods reviewed in this series. The dominance index does not capture interaction effects or dynamic importance under distribution shift. It has not been validated on other model types, domains, or with attribution methods beyond SHAP and ELI5. The framework also does not address how attribution-based feature selection interacts with regulatory requirements such as the transparency obligations of the EU AI Act, which may mandate retention of specific features for explainability regardless of predictive importance.

Domain Applications: What the Wider Literature Shows

The tier 3 papers in the selection analysis apply attribution methods to specific domains. While none is a standalone methodological contribution, together they reveal patterns about how attribution works in practice.

Energy and climate (most represented)

The largest domain cluster is energy and climate forecasting, with five papers applying SHAP to wind, solar, heating load, and monsoon prediction. The pattern across all five is consistent: SHAP identifies a small number of dominant physical features that match domain expert intuition. Temporal features (hour of day, day of year) and lagged values of the target variable consistently receive high importance. This alignment with domain knowledge is treated as validation of the attribution method, but it raises a question the papers do not fully address: if attributions merely confirm what domain experts already know, what value do they add?

The answer, from Kost et al., is that the value is in quantification, not discovery. Knowing that GHI is important is not new. Knowing that its dominance index is 0.365 and removing it costs only 0.009 in R² is actionable.

Healthcare and medical imaging (most sensitive)

Four healthcare applications appear, ranging from chronic kidney disease risk prediction (Hina et al. [3], using SHAP) to whole-slide image analysis (Vu et al. [4], using Contrastive Integrated Gradients) and ECG interpretation (van der Valk et al. [5], using VAE disentanglement). The healthcare context imposes requirements that the energy papers do not face: regulatory oversight, the need for causal (not just correlational) explanations, and the requirement that explanations be interpretable by clinicians, not data scientists.

The healthcare papers also reveal a gap. None evaluates whether the provided explanations actually improve clinical decision-making. The assumption that more explainable models lead to better clinician decisions remains untested in the reviewed literature. This is not merely a gap in the papers reviewed here. It reflects a structural limitation of the attribution evaluation field as a whole. User studies are expensive, domain specific, and difficult to publish in ML venues. Until the research community incentivises deployment-stage evaluation, claims that attribution methods improve decision-making will remain unsupported by direct evidence.

Finance and cryptocurrency (most challenging)

Two papers apply SHAP to financial time series (cryptocurrency and stock price forecasting). The short-term, high-noise nature of financial data makes attribution fundamentally harder than in physical domains. Feature importance rankings are less stable. Small changes in input can produce large changes in attributions. The non-stationary nature of financial markets means that feature importance itself is time-dependent. These papers validate the ability of SHAP to produce some attribution, but do not establish its reliability for financial decision-making.

A deeper issue is that financial attribution faces a ground-truth problem that even synthetic benchmarks cannot fully address. In energy forecasting, the physical relationship between irradiance and solar power output provides a natural validation signal. In finance, the data-generating process is itself shaped by human behaviour, which is not decomposable into independent feature contributions. An attribution method that assigns importance to a technical indicator in a price prediction may be correct relative to the model but meaningless relative to the actual market dynamics the model approximates. This does not make financial XAI useless, but it means practitioners must be correspondingly more cautious about interpreting attributions as explanations of real-world phenomena.

Geoscience and remote sensing (scalability test)

Two papers apply SHAP at scale to spatiotemporal Earth observation data: European summer wildfires (24 features, continental scale) and coastal land subsidence (InSAR data). These demonstrate that SHAP can be deployed on realistic spatial scales, but the computational cost of Shapley-value approximation becomes a practical concern at this scale.

Cross-domain patterns

Lessons from the Surveys

Four surveys in the selection provide the broader context within which the methodological papers sit.

Ullah et al. (2025) [6] review 155 XAI papers and establish taxonomy dimensions useful for any practitioner designing an XAI strategy. Their key finding: the field skews toward image domains and gradient-based methods, while tabular and time series (the most common production formats) are underserved. A practitioner building a tabular model cannot rely on the default recommendations of the field.

Laato et al. (2022) [7] systematically review how to explain AI systems to end users and find that most XAI research assumes a technically literate audience. The explanation formats studied rarely match non-expert stakeholder needs. Deploying an attribution method is not the same as providing an explanation: the output must be translated into a stakeholder-appropriate format, and that translation is itself a design problem.

Preet et al. (2025) [8] provide a pedagogical survey of SHAP useful for team onboarding, covering foundations, implementation variants, and the common pitfall Janzing et al. [9] identified: users do not realise they are choosing between interventional and conditional Shapley values.

Mitchell et al. (2019) [2] introduce Model Cards as a documentation standard. Model Cards are complementary to feature attribution: attribution explains individual decisions, while Model Cards explain the model as a whole.

A Decision Framework for Attribution Method Selection

If I were advising a team starting their XAI journey today, I would tell them to start with the decision framework below and budget at least as much for evaluation as for explanation generation. Drawing on all 15 core papers, the domain applications, and the surveys, I propose a structured decision framework organised by three questions.

Question 1: What model access do you have?

Question 2: What does your domain require?

Question 3: What evaluation standard applies?

• Research publication: Use at least two evaluation paradigms. Dehdarirad [10] demonstrates one approach: cross model architectures (classical LR/RF vs. transformer-based DistilBERT/RoBERTa) with dataset characteristics (domain, text length, class balance) and evidence types (positive, negative, all). This triply crossed design reveals that SHAP fails for negative evidence under LR but succeeds under RF. This finding is invisible to single-dimension comparisons. Follow this multi-factorial approach rather than reporting aggregate rankings that mask conditional failures.
• Production deployment: Supplement automated metrics with human evaluation on a representative sample. Use the decision-path inspection of Moreira et al. [11] for any tree-based model. Monitor attribution drift over time using rank correlation between current and baseline feature importance distributions. Design stakeholder-specific explanation formats. The default outputs of the field (heatmaps, bar charts) do not serve non-technical audiences. Log method choice, implementation variant (e.g., interventional vs. conditional SHAP), and known limitations for each production prediction.
• Regulatory compliance: The verification impossibility is a live compliance risk under the EU AI Act. High-risk AI systems (Annex III categories such as biometric identification, critical infrastructure, and access to essential services) must provide transparency under Article 13, including meaningful information about the decision logic. Standard black-box attributions cannot meet this standard for individual predictions. Use Model Cards [2] for model-level documentation alongside per-explanation metadata recording which method was used, what it assumes, and what it cannot guarantee. The developing EN ISO/IEC 42001 AI management system standard may provide a structured audit framework, but it is not yet finalised. Document the rationale for method selection against domain stakes. Accept that individual prediction explanations from black-box models cannot be independently verified and communicate this limitation to auditors and regulators.

Ten Practical Lessons from This Series

1. Do not trust a single attribution you cannot cross-validate. The impossibility result applies to individual explanations. Aggregate statistics across many predictions are more reliable. This recommendation follows from the formal proof in Bhalla et al. Independent replication would strengthen the empirical basis for the aggregate reliability claim.

2. Know your Shapley variant. If you use SHAP, you must know whether your implementation uses interventional or conditional expectations. The difference is not an implementation detail. It changes what the scores mean.

3. Exact attribution is not a luxury. It is a standard. For feedforward networks, FACE [16] shows that exact attribution is computationally cheaper than approximation. If your architecture supports it, use it.

4. Feature interactions will be invisible unless you look for them. First-order attribution methods miss interactions by construction. For high-stakes settings, supplement first-order with higher-order or interaction-aware analysis.

5. Evaluation metrics are not interchangeable. Each metric captures a different aspect of explanation quality. Triangulate across paradigms and acknowledge what each metric misses.

6. Correlation is not causation, even in attribution. Causal SHAP [12] and ExCIR [21] offer different responses to this problem. Choose based on whether your use case requires causal claims or can tolerate correlational awareness.

7. Attribution is a design-stage tool, not just a post-hoc one. The XAI-FS of Kost et al. shows that attribution can guide feature selection, monitor dominance, and manage acquisition costs well before a model reaches production.

8. Domain alignment validates but does not substitute for rigour. Alignment with domain expert intuition is reassuring but not sufficient. Systematic evaluation remains necessary even when attributions look right.

9. Explainability does not equal a good explanation. The attribution output must be translated into a format appropriate for the stakeholder. The default outputs of the field (heatmaps, bar charts) are poorly suited for non-technical audiences.

10. The field is converging on conditional guidance. There is no universal best attribution method. The papers in this series converge on a framework of conditional recommendations: the right method depends on model architecture, domain stakes, evaluation standard, and stakeholder needs.

Questions on Attribution in Practice

Should I use SHAP or LIME as my default?

The evidence in this series shows both have known failure modes. SHAP conflates correlation with causation (Janzing, Ng [12]). LIME underperforms on time series (Troncoso-García [13]). Use the decision framework above to select based on your specific constraints, and always validate with at least one metric from a different paradigm.

How should attribution be integrated into an ML pipeline?

At three points: (1) feature selection stage (Kost’s XAI-FS approach), (2) model validation stage (evaluation against domain expert baselines), (3) production monitoring stage (tracking attribution stability under distribution shift). Production monitoring for attribution drift is analogous to data and concept drift monitoring: track the rank correlation between current and baseline feature importance distributions, and alert when it drops below a threshold. Most current practice only addresses the second stage.

What should I tell a regulator about my explanations?

Be honest about what they cannot do. Standard post-hoc explanations from black-box models cannot be verified for individual cases (Bhalla [14]). They are aggregate statistical summaries, not guarantees of model behaviour. If a regulator requires verifiable explanations, you need either an inherently interpretable model or a VerT-adapted model.

How much should I invest in attribution infrastructure?

Proportional to the cost of an incorrect explanation. For a product recommendation system, lightweight SHAP or LIME may suffice. For a medical diagnostic aid, invest in stronger methods (FACE or Causal SHAP), multi-paradigm evaluation, and stakeholder-specific explanation formats.

What is the single most important thing a practitioner should do?

Triangulate. Do not rely on a single attribution method, a single evaluation metric, or a single validation dataset. The evidence in this series consistently shows that results are conditional on choices that are easy to overlook when using a single method.

Conclusion

The series supports a convergent conclusion: responsible attribution requires accepting that the ambition of the field exceeded its foundations. The response is not abandonment but specialisation. Practitioners who understand the specific constraints of their domain, model, and regulatory context will produce more reliable attributions than those who deploy a general-purpose method and trust its output.

Each new method comes with sharper applicability conditions: exactness for FNNs only, causal discovery under sufficiency assumptions, correlation awareness at the cost of causal claims. This is a sign of a field maturing. The practitioner’s best strategy is to be informed about the limits, rigorous in evaluation, and honest with stakeholders about what attribution can and cannot guarantee.

Concludes a series on feature attribution, explainability, and interpretability. Technical and educational content. Not legal, regulatory, or procurement advice. Claims bounded to the results reported in the cited papers unless explicitly stated otherwise. Organisational deployment decisions should be reviewed under applicable professional and regulatory obligations.