Explainability in Practice: Domains, Evaluation, and Governance

Introduction

The first three articles in this series established what XAI means, how it works, and where it falls short. This article answers the practitioner’s question: what do you actually do? The answer is not a single recommendation but a situated decision framework.

Domain applications demonstrate that XAI requirements vary fundamentally by context. Evaluation frameworks show that metrics must be chosen for the specific use case, not adopted from a menu. Governance structures reveal that organisational commitment matters as much as technical capability. The synthesis draws on domain surveys covering medical imaging, cybersecurity, and IoT; evaluation frameworks linking explainability with uncertainty quantification; and deployment studies that characterise how organisations currently use XAI.

This article is not legal advice.

Vocabulary for XAI Practice

Uncertainty-aware explanation

An explanation that quantifies the confidence in its own attribution scores, following the thesis of Salvi et al. that explainability and uncertainty are "two sides of the same coin."

Model documentation

Structured records of model purpose, development data, performance characteristics, and known limitations. Model Cards and Datasheets are prominent examples. Documentation is a governance mechanism that does not require explanation methods.

Feature governance

The organisational practice of documenting, monitoring, and controlling which features are used in ML models, including periodic review of feature relevance, data quality, and drift.

Accountability

The organisational infrastructure that ensures someone is responsible for model decisions, independent of whether the model is explainable. Accountability structures include escalation paths, audit trails, and designated human-in-the-loop roles.

Domain-Specific XAI Requirements

Medical Imaging and Clinical Decision Support

Medical XAI surveys (Hossain et al. 2025; Tjoa & Guan 2021; Nazir et al. 2023) reveal that healthcare presents uniquely demanding XAI requirements. Explanations must be faithful enough to support clinical decisions, interpretable enough for clinicians with limited ML training, and robust enough to withstand regulatory scrutiny under frameworks like the EU Medical Device Regulation.

The dominant approach in medical imaging combines Grad-CAM or LRP visualisations with clinician evaluation. This approach has documented vulnerabilities: Clever Hans artifacts (hospital markers, imaging equipment watermarks) produce compelling but misleading explanations. Clinicians who trust these visualisations may develop overconfidence in models that rely on spurious correlations.

The critique of Ghassemi et al. (discussed in Critical Perspectives and Limits of Current Explainability Methods) poses a challenge that the medical XAI community has not fully answered: if explanations can be systematically misleading and clinicians cannot evaluate their reliability, do explanations in their current form improve or degrade clinical decision-making?

Cybersecurity

Cybersecurity presents a different set of challenges. The evaluation of XAI metrics in a cybersecurity context by Pawlicki et al. found that explanation speed matters more than in other domains. Threat detection requires real-time or near-real-time explanations. The evaluation metrics that work for offline medical imaging analysis are inappropriate for streaming security data.

Feature attribution methods (SHAP, LIME) dominate security XAI, but the evaluation literature reveals that metric proliferation is particularly acute in this domain. Multiple metrics claiming to measure “faithfulness” produce systematically different rankings of the same methods applied to the same security data.

Internet of Things

IoT XAI (Kök et al. 2023) operates under severe resource constraints. Deep neural network explanations that require gradient computation may be infeasible on edge devices. This has led to interest in intrinsically interpretable models (decision trees, rule lists) that provide explanations as a byproduct of their architecture rather than requiring post-hoc computation.

Geospatial and Environmental XAI

Roussel et al. (2025) propose a quality-aware XAI framework for geospatial analysis that connects explanation quality to downstream decision quality. The framework evaluates whether improving explanation quality actually improves decision outcomes, a direct answer to the challenge of Ghassemi et al.

Evaluation Frameworks

Salvi et al. (2025): Explainability and Uncertainty as Two Sides of the Same Coin

Salvi et al. propose a framework unifying explainability with uncertainty quantification [2]. The thesis is that explanations without uncertainty bounds are fundamentally incomplete. A SHAP value of 0.3 for feature x means something different when the confidence interval is ±0.02 than when it is ±0.25.

The framework maps explanation types to corresponding uncertainty types. Feature attributions require attribution uncertainty. Counterfactual explanations require counterfactual uncertainty. Concept-based explanations require concept uncertainty. This mapping provides a principled basis for selecting evaluation metrics.

Dwivedi et al. (2023): Core Ideas, Techniques, and Solutions

Dwivedi et al. provide a broad survey emphasising practical solutions [3]. The paper organises methods by their deployment readiness and provides selection guidelines matching methods to use cases. The practical emphasis makes it a valuable complement to the conceptual and critical analyses discussed in previous articles.

Hassija et al. (2024): Model-Agnostic Methods for Black-Box Models

Hassija et al. focus specifically on interpreting black-box models [6]. Their survey covers model-agnostic methods including LIME, SHAP, partial dependence plots, accumulated local effects, and permutation feature importance. The emphasis on black-box methods reflects the practical reality that practitioners often cannot access model internals.

Governance Structures

Organisational Accountability

The deployment study of Bhatt et al. (Critical Perspectives and Limits of Current Explainability Methods) established that most XAI serves internal stakeholders. The constructive response is not to abandon XAI but to build organisational structures that redirect explanations toward the stakeholders who need them.

The stakeholder-centred framework of the paper is a governance mechanism: establish clear goals for what the explanation should achieve for a specific stakeholder; design for that stakeholder’s cognitive and domain context; evaluate whether the goals are met. This framework translates technical explainability into organisational accountability.

Access Governance

The three-level access framework of Casper et al. (black-box, white-box, outside-the-box) provides a governance structure for model auditing. Organisations deploying high-risk AI systems should establish policies determining what level of access external auditors require, with the default presumption that black-box access alone is insufficient for rigorous evaluation.

Regulatory Context

The EU AI Act establishes requirements for explainability and transparency that vary by risk category. High-risk AI systems must provide meaningful information about the logic involved in decision-making. The Act does not specify which explanation methods to use, creating both flexibility and uncertainty. Practitioners should design explanation pipelines that can adapt to evolving regulatory requirements.

Synthesis: Practice Requires Integration

The practitioner’s challenge is integrating these dimensions: selecting methods appropriate for the domain, choosing evaluation metrics that measure what matters for the specific use case, and building governance structures that ensure explanations reach the stakeholders who need them.

No single method, metric, or governance structure suffices. Domain determines method requirements. Use case determines metric selection. Organisational context determines governance design. The integration is the practice.

Conclusion

Explainability in practice is a situated, multi-dimensional challenge. Domain applications demonstrate that general-purpose XAI guidance is insufficient. Evaluation frameworks show that metrics must be chosen for the specific context. Governance structures reveal that organisational commitment is as important as technical capability.

The final article in this series examines where XAI is heading: the open challenges identified by the XAI 2.0 manifesto of Longo et al. and the emerging directions that will shape the field.

Frequently Asked Questions

What domain-specific requirements exist for XAI in medical imaging and clinical settings?

Medical imaging requires explanation methods that are robust to Clever Hans artifacts, produce spatially coherent attribution maps, and provide uncertainty quantification alongside explanations. Clinical deployment additionally demands explanations that align with medical reasoning workflows, regulatory compliance under frameworks such as the EU AI Act, and validation against established diagnostic standards rather than purely technical metrics.

How does the EU AI Act affect XAI deployment and governance requirements?

The EU AI Act classifies AI systems by risk level and imposes transparency obligations that include documentation of training data, model architecture, and intended use. For high-risk systems, it requires human oversight mechanisms and technical documentation of explainability properties. The Act creates legal accountability for explanation quality without prescribing specific technical standards, creating both pressure and flexibility.

What evaluation frameworks exist for assessing explanation quality in practice?

Three main frameworks have emerged: functionally-grounded evaluation using formal proxies such as faithfulness and completeness[1], human-grounded evaluation through user studies that measure how explanations affect human decision-making, and application-grounded evaluation in real deployment contexts. Dwivedi et al. (2023)[3] provide a practical synthesis linking method selection to evaluation criteria.

How does Salvi et al. connect explainability and uncertainty in their framework?

Salvi et al. (2025)[2] argue that explanations without uncertainty quantification are fundamentally incomplete because users cannot distinguish between confident model predictions and those near decision boundaries. Their framework integrates uncertainty estimates into explanation outputs by modifying attribution methods to propagate epistemic and aleatoric uncertainty through the explanation computation.

What governance structures are needed for responsible XAI deployment?

Effective governance requires organisational accountability mechanisms including designated explainability officers, access governance that matches explanation detail to stakeholder roles, documented decision frameworks for method selection, and regular audit cycles that verify explanations remain faithful after model updates. These structures bridge the gap between technical capability and responsible practice.