When Explanations Lie: Stress-Testing Saliency Maps Before Clinical Deployment¶

A hands-on MICCAI tutorial on shortcut learning, sanity checks, and trustworthy medical imaging AI

Abstract¶

Saliency maps are widely used to explain medical imaging AI systems, but visually plausible explanations can be misleading. This hands-on notebook teaches students how to implement common saliency methods and, more importantly, how to stress-test them. Using PneumoniaMNIST, participants train a small chest X-ray classifier, implement vanilla gradients, SmoothGrad, Integrated Gradients, occlusion sensitivity, and Grad-CAM, and evaluate explanations using randomization tests, shortcut-learning experiments, deletion curves, stability analysis, and shortcut-region attribution. This is a controlled, clinically motivated teaching experiment rather than a deployable pneumonia detector: participants leave with a reproducible debugging workflow for asking whether explanations are sensitive to model parameters, robust to perturbations, and capable of revealing non-clinical shortcuts.

Reviewer-facing contribution statement. This tutorial contributes:

1. A reproducible saliency stress-test workflow for medical image classifiers.
2. A controlled shortcut-learning experiment with known non-clinical artifacts.
3. Transparent implementations: minimal PyTorch implementations of common saliency methods so learners can see how each explanation is computed.
4. A practical checklist for using explanations as debugging tools before clinical deployment.

Pedagogical implementation note. All explanation methods in this tutorial are implemented from scratch in compact PyTorch code under src/, rather than delegated to a black-box interpretability library. This is intentional: the goal is to make the assumptions, gradients, baselines, smoothing, occlusion masks, and Grad-CAM hooks visible to learners. These implementations are designed for teaching and reproducibility, not as optimized clinical software.

Who is this tutorial for?¶

This notebook is for incoming Master's and PhD students in medical image computing who know basic Python and deep learning. The goal is educational: build enough of the pipeline to understand where saliency maps can fail, not to produce a deployable pneumonia detector.

Learning objectives¶

By the end of this tutorial, you will be able to:

1. Implement common saliency methods for medical image classifiers.
2. Explain why visually plausible heatmaps can be misleading.
3. Run model, layer, and label-randomization sanity checks.
4. Detect shortcut learning using controlled artifacts.
5. Interpret deletion curves, stability tests, and shortcut-region attribution.
6. Describe why explanation quality is not the same as clinical validity.
7. Connect explanation failures to real-world clinical deployment risks.

Clinical motivation: from pixels to patients¶

Imagine a hospital deploys a chest X-ray triage model for pneumonia. The model appears accurate, and its saliency maps seem to highlight the lungs. Later, performance collapses at another hospital. Investigation reveals that the model relied partly on non-clinical image artifacts: scanner-specific borders, preprocessing marks, or labels correlated with the training site.

This tutorial demonstrates why attractive heatmaps are not enough. We will train a simple medical image classifier, generate explanations, and then deliberately stress-test those explanations. The goal is not to build a deployable pneumonia detector, but to learn practical checks that should happen before explainability tools are used to support patient-facing decisions. Similar concerns appear in reports of cross-site pneumonia generalization failures and shortcut use in radiographic COVID-19 models [7, 8].

Setup and reproducibility¶

Run the cells in order. The default configuration uses the full PneumoniaMNIST splits with a moderate epoch budget. Set FAST_DEV_RUN = True for a shorter smoke test while developing or teaching under tight time constraints. Seeds improve reproducibility, but deterministic GPU execution can be slower, so this tutorial leaves deterministic kernels off by default.

Runtime and data box. The default setting uses the full PneumoniaMNIST train, validation, and test splits, 10 baseline epochs, 5 label-noise epochs, and 8 shortcut epochs. Runtime depends on hardware, dependency installation, and the first dataset download. No private data are required.

Materials included. The submission package includes this rendered HTML, the executable notebook, the src/ package used by the notebook, requirements.txt and environment.yml, and a README.md with the local and Colab run paths.

Dataset: PneumoniaMNIST¶

PneumoniaMNIST is part of MedMNIST v2, a lightweight biomedical image classification benchmark [1]. The task is binary pediatric chest X-ray classification: normal versus pneumonia. We normalize images to [-1, 1], which matters later when choosing baselines and occlusion values.

Figure caption: the image grid shows the normalized PneumoniaMNIST samples used in the teaching run, followed by the class distribution of the selected training subset.

Training a baseline pneumonia classifier¶

The classifier is intentionally small so students can inspect the full pipeline. It is a teaching model, not a clinical device. The training curve helps identify obvious underfitting or overfitting before we interpret any explanation.

Clinical metrics beyond accuracy¶

In clinical screening or triage, overall accuracy can hide clinically important errors. Sensitivity reflects how often pneumonia cases are detected, while specificity reflects how often normal cases are correctly ruled out. A model with high accuracy but poor sensitivity may be unsafe in a screening workflow.

Figure caption and clinical note: the confusion matrix summarizes normal-versus-pneumonia errors in the fast run, and the printed metrics report sensitivity and specificity. This model catches most pneumonia cases in the fast run, but it overcalls pneumonia in many normal images; that tradeoff would matter in triage because false positives can burden clinical workflow.

Figure caption: the two example images selected for saliency analysis show one pneumonia case and one normal case with the model's predicted label in each title.

Saliency methods: intuition and implementation¶

For visualization, each saliency map is normalized independently. This improves visual contrast but can make weak and strong explanations look equally intense. Quantitative comparisons should use unnormalized maps or a shared normalization scale.

Taking the absolute value shows sensitivity regardless of direction, but it hides whether a pixel supports or suppresses the predicted class. Signed attributions may be preferable when the direction of evidence matters.

Explanation vocabulary¶

• Plausibility: whether a heatmap looks reasonable to a human observer.
• Faithfulness: whether the map changes when the learned model or target behavior changes.
• Localization: whether attribution falls on a known region of interest, such as our synthetic marker.
• Clinical validity: whether the model and its explanations remain useful, safe, and calibrated in a real clinical workflow.

These concepts are related but not interchangeable. A map can be plausible without being faithful, and a faithful debugging signal is still not proof of clinical validity.

Vanilla gradients¶

Vanilla saliency computes the gradient of a class score with respect to the input pixels:

$$ S(x) = \left| \frac{\partial f_c(x)}{\partial x} \right| $$

Here x is the input image and f_c(x) is the score for class c. Large values indicate pixels where small local changes strongly affect the class score.

SmoothGrad¶

SmoothGrad averages gradients over noisy copies of the same image [3]:

$$ S_{\text{SmoothGrad}}(x) = \frac{1}{N} \sum_{i=1}^{N} \left| \frac{\partial f_c(x + \epsilon_i)}{\partial x} \right|, \quad \epsilon_i \sim \mathcal{N}(0, \sigma^2) $$

Averaging can reduce visual noise. Because our images are normalized to [-1, 1], noisy inputs are clamped back to that range to avoid explaining unrealistic out-of-range images.

Integrated Gradients¶

Integrated Gradients was proposed by Sundararajan et al. [2]. It integrates gradients along a path from a baseline image x' to the input image:

$$ \mathrm{IG}_i(x) = (x_i - x'_i) \int_{\alpha=0}^{1} \frac{\partial f_c(x' + \alpha(x - x'))}{\partial x_i} \, d\alpha $$

Here the default baseline is a mean-gray image in normalized space. A true black image would be represented by -1 after normalization.

Extension: compare Integrated Gradients with Captum¶

Our in-repository Integrated Gradients implementation is intentionally explicit: it builds points along the straight-line path from a baseline image to the input image, backpropagates the target-class logit at each point, averages those gradients, and multiplies by input - baseline. That makes the algorithm inspectable for teaching.

In practice, you may prefer a maintained interpretability library. Captum provides IntegratedGradients for PyTorch models, accepts explicit baselines and target classes, supports several numerical integration rules, and can return a convergence delta. The cell below reproduces the same idea with Captum and compares it with the transparent tutorial implementation.

Captum documentation: https://captum.ai/api/integrated_gradients.html.

Occlusion sensitivity¶

Occlusion sensitivity asks how the target probability changes when a region is hidden:

$$ S_R(x) = f_c(x) - f_c(x_{\setminus R}) $$

R is an image region and x_{\setminus R} is the image with that region occluded. A large drop means the region was important for the prediction. Perturbation methods are related to meaningful perturbation approaches [6].

Grad-CAM¶

Grad-CAM weights convolutional feature maps using gradients flowing into a target layer [4]:

$$ \alpha_k^c = \frac{1}{Z} \sum_i \sum_j \frac{\partial f_c}{\partial A_{ij}^k} $$

$$ L_{\text{Grad-CAM}}^c = \mathrm{ReLU} \left( \sum_k \alpha_k^c A^k \right) $$

A^k is feature map k, and \alpha_k^c measures how important that feature map is for class c. The ReLU keeps features that positively support the class.

Figure caption: the baseline explanation grid compares five saliency methods on the same pneumonia example. Heatmaps are normalized independently for visibility, so color intensity should not be compared as an absolute score across methods.

Why plausible heatmaps can be misleading¶

Visual plausibility is not faithfulness. A map can look anatomical because chest X-rays have strong edges and repeated structures, even if the model parameters or labels are wrong. Sanity checks for saliency maps were popularized by Adebayo et al. [5] and are a practical first line of defense before explanation claims are used in clinical discussions.

Sanity check 1: model randomization¶

We compare the trained model with a randomly initialized model. Using the same target class isolates whether the explanation depends on learned model parameters, rather than changing simply because we explained a different output class.

Figure caption: the model-randomization sanity check compares the trained-model gradient map with a randomly initialized model for the same target class. A faithful explanation should usually change when the learned parameters are replaced.

Sanity check 2: layer randomization¶

Now we randomize parts of the trained network. If explanations barely change after classifier or feature layers are reset, the method may be reflecting image structure more than learned decision logic.

Figure caption: the layer-randomization sanity check shows whether resetting the classifier or feature extractor changes the explanation. High similarity after randomization is a warning that the map may be dominated by image structure rather than learned decision logic.

Sanity check 3: label-noise sanity check¶

This lightweight label-noise demonstration is not a full random-label memorization experiment. The point is narrower: even when the supervised task is corrupted, explanations can still look structured. A stronger memorization experiment would train longer on a small random-label subset.

Figure caption: the label-noise sanity check compares the baseline explanation with a model trained on corrupted labels. Structured-looking heatmaps can still appear even when the supervised task is not clinically meaningful.

Shortcut learning experiment¶

In medical imaging, shortcuts can include scanner artifacts, burned-in text, portable AP markers, scanner borders, institution-specific preprocessing, acquisition protocols, or dataset-source leakage. We simulate a controlled non-clinical marker so we know exactly where the shortcut is and can test whether explanation methods reveal it.

Add a synthetic non-clinical marker¶

Pneumonia images receive a bright top-left marker; normal images receive a bright bottom-right marker. The marker is easy for a CNN to learn but has no clinical meaning.

Figure caption: the shortcut examples show the synthetic label-coded corner marker: pneumonia images receive a bright top-left marker and normal images receive a bright bottom-right marker.

Train a shortcut model¶

A high score on marked data is not evidence of clinically meaningful learning. The clean test set helps reveal whether the classifier depends on the artificial marker.

Compare clean vs marked test performance¶

If the marked and clean test results differ, the marker may be carrying predictive signal. This mirrors the clinical risk of models performing well in one hospital but failing after a shift in scanners, protocols, or preprocessing.

Figure caption: the marked shortcut image is the individual test example used for the shortcut-model explanations below.

Figure caption: shortcut-model explanations are shown for a marked pneumonia image. Grad-CAM can miss tiny corner artifacts because its map is computed from low-resolution convolutional features; this is a limitation of spatial granularity, not automatic evidence that the model ignored the marker.

Remove the marker and measure probability sensitivity¶

If removing the marker changes the prediction or target probability, that is direct evidence that a non-clinical feature influenced the model. A class flip is strong evidence, but it is not required: a smaller probability change still shows marker sensitivity, and no change in one image does not rule out shortcut use across the dataset.

Figure caption and interpretation: the two panels compare the same shortcut image before and after replacing both possible marker corners with the normalized gray fill value. In the saved full-split run, removing the marker changes the selected example's class probabilities substantially. Depending on the run, this may appear as a confidence drop or a class flip; the larger clean-versus-marked test gap above is the stronger dataset-level shortcut signal.

Quantify attribution to the shortcut region¶

The shortcut-region attribution fraction estimates how much attribution lies inside the artificial marker:

$$ F_{\text{shortcut}} = \frac{ \sum_{i \in R_{\text{shortcut}}} S_i }{ \sum_i S_i + \epsilon } $$

This is a simple diagnostic, not a clinical validity score.

Figure caption: the shortcut-region attribution plot reports the fraction of each method's attribution that falls inside the known marker region. Higher values mean the method localized more attribution to the non-clinical artifact.

Quantitative explanation evaluation¶

Visual inspection should be paired with quantitative stress tests. These metrics are diagnostics for model debugging; none of them proves clinical validity.

Map correlation¶

Map correlation compares two flattened heatmaps. We used it above to ask whether explanations change after randomizing model parameters.

Deletion curves¶

A deletion curve removes pixels in order of saliency and tracks the target score:

$$ D(t) = f_c(x_{\text{deleted up to fraction } t}) $$

A sharper confidence drop usually indicates a more class-relevant explanation.

Deletion AUC¶

Deletion curves are useful stress tests, but replacing pixels can create unrealistic images. Interpret deletion AUC as a relative diagnostic, not a direct measure of clinical explanation quality. Lower AUC means the target probability fell faster after deleting high-attribution pixels.

Figure caption: the deletion-curve legends and printed AUC dictionaries are produced by the same cell. Lower AUC means the target probability dropped faster as high-attribution pixels were removed.

What deletion AUC shows in this run¶

In this run, deletion detects the shortcut strongly for several methods, especially occlusion, while Grad-CAM remains relatively high. This illustrates that quantitative explanation tests are method-dependent: some methods expose the shortcut, while coarse spatial methods may still miss small artifacts. The main lesson is not that one metric is definitive, but that visual inspection, deletion behavior, and known-region attribution should be read together.

Stability under perturbations¶

A stable explanation should not change drastically under tiny input perturbations when the prediction remains the same. Instability is a warning that the visual story may be fragile.

Figure caption: the stability plot summarizes how much the selected explanation maps change under small input noise. Low or highly variable correlation suggests that a visual explanation may be fragile even when the prediction is unchanged.

Shortcut-region attribution fraction¶

The marker-fraction plot above is most useful when compared across methods. Grad-CAM is spatially coarse because it is computed from low-resolution convolutional feature maps. Small artifacts such as corner markers may be blurred away or missed, even when the classifier uses them.

Figure caption: the final summary plots compare shortcut-region attribution and deletion AUC across explanation methods. These bars should match the printed summary_rows values above them after a full run-all execution.

Clinical interpretation and limitations¶

This notebook demonstrates failure modes, not deployment readiness. PneumoniaMNIST is small and preprocessed, the CNN is simple, and the marker shortcut is synthetic. Real clinical validation would require external sites, locked preprocessing, subgroup and scanner/protocol analysis, uncertainty and calibration reporting, decision-curve or workflow evaluation, clinician review of successes and failures, and prospective monitoring after deployment. Reporting checklists such as CLAIM help make medical imaging AI studies more transparent [10].

Learner exercises and expected observations¶

1. Change the Integrated Gradients baseline from gray to black. Expected observation: maps can shift because the baseline defines the reference image; this is a reminder that attribution is partly a measurement choice.
2. Change marker_size from 8 to 4 in the shortcut datasets. Expected observation: the shortcut becomes harder to see, and coarse methods such as Grad-CAM may miss it more often.
3. Replace the corner marker with a border stripe or simulated burned-in text. Expected observation: different explanation methods may reveal different artifact shapes, so agreement across methods is more informative than one attractive map.
4. Compare shortcut-model errors on clean data by class. Expected observation: the clean test set often exposes asymmetric false positives or false negatives that are hidden by high marked-test accuracy.
5. Search for the test image with the largest probability drop after marker removal. Expected observation: a single dramatic counterfactual is useful for teaching, but the dataset-level clean-versus-marked gap is better evidence of shortcut reliance.

From pixels to patients: what this tutorial teaches¶

This notebook does not show how to deploy a pneumonia detector. Instead, it shows why deployment requires more than high accuracy and attractive heatmaps.

Before a model influences patient care, teams should ask:

1. Does performance hold across hospitals, scanners, protocols, and patient groups?
2. Are explanations sensitive to the learned model?
3. Could the model be using non-clinical shortcuts?
4. Are uncertainty, calibration, and failure modes reported?
5. Have clinicians reviewed both successes and failures?
6. Would the explanation change a clinical decision, and should it?

Checklist before trusting saliency maps in clinical AI¶

• Evaluate performance on external data.
• Report sensitivity, specificity, balanced accuracy, and uncertainty, not only accuracy.
• Run model-randomization sanity checks.
• Run label-randomization or label-noise sanity checks.
• Test robustness to small perturbations.
• Look for attribution to artifacts, borders, markers, or text.
• Compare multiple explanation methods.
• Involve clinical experts in failure analysis.
• Document dataset provenance and preprocessing.
• Treat explanations as debugging tools, not proof of clinical validity.

Key takeaways¶

1. Saliency maps can be useful debugging tools, but visual plausibility is not evidence of faithful reasoning.
2. Randomization, label-noise, deletion, stability, and shortcut-region tests reveal different failure modes.
3. Accuracy alone is not enough for clinical screening; sensitivity and specificity matter.
4. Synthetic shortcuts are useful teaching tools because the ground-truth artifact is known.
5. Moving from pixels to patients requires external validation, transparent reporting, clinical review, and careful workflow design.

Checklist summary: the final checklist is the practical handoff from the tutorial. It frames saliency maps as one debugging input alongside external validation, clinical metrics, robustness, calibration, provenance, and expert review.

AI assistance disclosure¶

Parts of this tutorial were prepared with assistance from an AI language model for editing, structuring, and clarity. All code, experiments, results, and scientific claims were reviewed and validated by T. Kirscher.

References¶

[1] Yang, J. et al. MedMNIST v2: A large-scale lightweight benchmark for 2D and 3D biomedical image classification. Scientific Data, 2023.

[2] Sundararajan, M., Taly, A., and Yan, Q. Axiomatic Attribution for Deep Networks. ICML, 2017.

[3] Smilkov, D. et al. SmoothGrad: removing noise by adding noise. arXiv, 2017.

[4] Selvaraju, R. R. et al. Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization. ICCV, 2017.

[5] Adebayo, J. et al. Sanity Checks for Saliency Maps. NeurIPS, 2018.

[6] Fong, R. C. and Vedaldi, A. Interpretable Explanations of Black Boxes by Meaningful Perturbation. ICCV, 2017.

[7] Zech, J. R. et al. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study. PLOS Medicine, 2018.

[8] DeGrave, A. J., Janizek, J. D., and Lee, S.-I. AI for radiographic COVID-19 detection selects shortcuts over signal. Nature Machine Intelligence, 2021.

[9] Samek, W. et al., editors. Explainable AI: Interpreting, Explaining and Visualizing Deep Learning. Springer, 2019.

[10] Mongan, J., Moy, L., and Kahn, C. E. Checklist for Artificial Intelligence in Medical Imaging (CLAIM): A Guide for Authors and Reviewers. Radiology: Artificial Intelligence, 2020.