Check out sessions from the WiML Symposium covering diffusion models with KerasCV, on-device ML, and more. Watch on demand
- TensorFlow
- Learn
- TensorFlow Core
- Tutorials
This tutorial demonstrates how to implement Integrated Gradients (IG), an Explainable AI technique introduced in the paper Axiomatic Attribution for Deep Networks. IG aims to explain the relationship between a model's predictions in terms of its features. It has many use cases including understanding feature importances, identifying data skew, and debugging model performance.
IG has become a popular interpretability technique due to its broad applicability to any differentiable model (e.g. images, text, structured data), ease of implementation, theoretical justifications, and computational efficiency relative to alternative approaches that allows it to scale to large networks and feature spaces such as images.
In this tutorial, you will walk through an implementation of IG step-by-step to understand the pixel feature importances of an image classifier. As an example, consider this image of a fireboat spraying jets of water. You would classify this image as a fireboat and might highlight the pixels making up the boat and water cannons as being important to your decision. Your model will also classify this image as a fireboat later on in this tutorial; however, does it highlight the same pixels as important when explaining its decision?
In the images below titled "IG Attribution Mask" and "Original + IG Mask Overlay" you can see that your model instead highlights (in purple) the pixels comprising the boat's water cannons and jets of water as being more important than the boat itself to its decision. How will your model generalize to new fireboats? What about fireboats without water jets? Read on to learn more about how IG works and how to apply IG to your models to better understand the relationship between their predictions and underlying features.
Setup
import matplotlib.pylab as plt import numpy as np import tensorflow as tf import tensorflow_hub as hub 2022-12-14 07:21:01.173720: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory 2022-12-14 07:21:01.173816: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory 2022-12-14 07:21:01.173826: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.Download a pretrained image classifier from TF-Hub
IG can be applied to any differentiable model. In the spirit of the original paper, you will use a pre-trained version of the same model, Inception V1, which you will download from TensorFlow Hub.
model = tf.keras.Sequential([ hub.KerasLayer( name='inception_v1', handle='//tfhub.dev/google/imagenet/inception_v1/classification/4', trainable=False), ]) model.build([None, 224, 224, 3]) model.summary() WARNING:tensorflow:Please fix your imports. Module tensorflow.python.training.tracking.data_structures has been moved to tensorflow.python.trackable.data_structures. The old module will be deleted in version 2.11. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/autograph/pyct/static_analysis/liveness.py:83: Analyzer.lamba_check (from tensorflow.python.autograph.pyct.static_analysis.liveness) is deprecated and will be removed after 2023-09-23. Instructions for updating: Lambda fuctions will be no more assumed to be used in the statement where they are used, or at least in the same block. //github.com/tensorflow/tensorflow/issues/56089 WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/autograph/pyct/static_analysis/liveness.py:83: Analyzer.lamba_check (from tensorflow.python.autograph.pyct.static_analysis.liveness) is deprecated and will be removed after 2023-09-23. Instructions for updating: Lambda fuctions will be no more assumed to be used in the statement where they are used, or at least in the same block. //github.com/tensorflow/tensorflow/issues/56089 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= inception_v1 (KerasLayer) (None, 1001) 6633209 ================================================================= Total params: 6,633,209 Trainable params: 0 Non-trainable params: 6,633,209 _________________________________________________________________From the module page, you need to keep in mind the following about Inception V1:
Inputs: The expected input shape for the model is def load_imagenet_labels(file_path): labels_file = tf.keras.utils.get_file('ImageNetLabels.txt', file_path) with open(labels_file) as reader: f = reader.read() labels = f.splitlines() return np.array(labels) 0. This is a dense 4D tensor of dtype float32 and shape def load_imagenet_labels(file_path): labels_file = tf.keras.utils.get_file('ImageNetLabels.txt', file_path) with open(labels_file) as reader: f = reader.read() labels = f.splitlines() return np.array(labels) 1 whose elements are RGB color values of pixels normalized to the range [0, 1]. The first element is def load_imagenet_labels(file_path): labels_file = tf.keras.utils.get_file('ImageNetLabels.txt', file_path) with open(labels_file) as reader: f = reader.read() labels = f.splitlines() return np.array(labels) 2 to indicate that the model can take any integer batch size.
Outputs: A def load_imagenet_labels(file_path): labels_file = tf.keras.utils.get_file('ImageNetLabels.txt', file_path) with open(labels_file) as reader: f = reader.read() labels = f.splitlines() return np.array(labels) 3 of logits in the shape of def load_imagenet_labels(file_path): labels_file = tf.keras.utils.get_file('ImageNetLabels.txt', file_path) with open(labels_file) as reader: f = reader.read() labels = f.splitlines() return np.array(labels) 4. Each row represents the model's predicted score for each of 1,001 classes from ImageNet. For the model's top predicted class index you can use def load_imagenet_labels(file_path): labels_file = tf.keras.utils.get_file('ImageNetLabels.txt', file_path) with open(labels_file) as reader: f = reader.read() labels = f.splitlines() return np.array(labels) 5. Furthermore, you can also convert the model's logit output to predicted probabilities across all classes using def load_imagenet_labels(file_path): labels_file = tf.keras.utils.get_file('ImageNetLabels.txt', file_path) with open(labels_file) as reader: f = reader.read() labels = f.splitlines() return np.array(labels) 6 to quantify the model's uncertainty as well as explore similar predicted classes for debugging.
def load_imagenet_labels(file_path): labels_file = tf.keras.utils.get_file('ImageNetLabels.txt', file_path) with open(labels_file) as reader: f = reader.read() labels = f.splitlines() return np.array(labels) imagenet_labels = load_imagenet_labels('//storage.googleapis.com/download.tensorflow.org/data/ImageNetLabels.txt')Load and preprocess images with def load_imagenet_labels(file_path): labels_file = tf.keras.utils.get_file('ImageNetLabels.txt', file_path) with open(labels_file) as reader: f = reader.read() labels = f.splitlines() return np.array(labels) 7
You will illustrate IG using two images from Wikimedia Commons: a Fireboat, and a Giant Panda.
def read_image(file_name): image = tf.io.read_file(file_name) image = tf.io.decode_jpeg(image, channels=3) image = tf.image.convert_image_dtype(image, tf.float32) image = tf.image.resize_with_pad(image, target_height=224, target_width=224) return image img_url = { 'Fireboat': '//storage.googleapis.com/download.tensorflow.org/example_images/San_Francisco_fireboat_showing_off.jpg', 'Giant Panda': '//storage.googleapis.com/download.tensorflow.org/example_images/Giant_Panda_2.jpeg', } img_paths = {name: tf.keras.utils.get_file(name, url) for (name, url) in img_url.items()} img_name_tensors = {name: read_image(img_path) for (name, img_path) in img_paths.items()} Downloading data from //storage.googleapis.com/download.tensorflow.org/example_images/San_Francisco_fireboat_showing_off.jpg 3954129/3954129 [==============================] - 0s 0us/step Downloading data from //storage.googleapis.com/download.tensorflow.org/example_images/Giant_Panda_2.jpeg 802859/802859 [==============================] - 0s 0us/step plt.figure(figsize=(8, 8)) for n, (name, img_tensors) in enumerate(img_name_tensors.items()): ax = plt.subplot(1, 2, n+1) ax.imshow(img_tensors) ax.set_title(name) ax.axis('off') plt.tight_layout()Classify images
Let's start by classifying these images and displaying the top 3 most confident predictions. Following is a utility function to retrieve the top k predicted labels and probabilities.
2022-12-14 07:21:01.173720: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory 2022-12-14 07:21:01.173816: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory 2022-12-14 07:21:01.173826: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. 02022-12-14 07:21:01.173720: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory 2022-12-14 07:21:01.173816: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory 2022-12-14 07:21:01.173826: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. 1Calculate Integrated Gradients
Your model, Inception V1, is a learned function that describes a mapping between your input feature space, image pixel values, and an output space defined by ImageNet class probability values between 0 and 1. Early interpretability methods for neural networks assigned feature importance scores using gradients, which tell you which pixels have the steepest local relative to your model's prediction at a given point along your model's prediction function. However, gradients only describe local changes in your model's prediction function with respect to pixel values and do not fully describe your entire model prediction function. As your model fully "learns" the relationship between the range of an individual pixel and the correct ImageNet class, the gradient for this pixel will saturate, meaning become increasingly small and even go to zero. Consider the simple model function below:
2022-12-14 07:21:01.173720: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory 2022-12-14 07:21:01.173816: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory 2022-12-14 07:21:01.173826: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. 4left: Your model's gradients for pixel def load_imagenet_labels(file_path): labels_file = tf.keras.utils.get_file('ImageNetLabels.txt', file_path) with open(labels_file) as reader: f = reader.read() labels = f.splitlines() return np.array(labels) 8 are positive between 0.0 and 0.8 but go to 0.0 between 0.8 and 1.0. Pixel def load_imagenet_labels(file_path): labels_file = tf.keras.utils.get_file('ImageNetLabels.txt', file_path) with open(labels_file) as reader: f = reader.read() labels = f.splitlines() return np.array(labels) 8 clearly has a significant impact on pushing your model toward 80% predicted probability on the true class. Does it make sense that pixel def load_imagenet_labels(file_path): labels_file = tf.keras.utils.get_file('ImageNetLabels.txt', file_path) with open(labels_file) as reader: f = reader.read() labels = f.splitlines() return np.array(labels) 8's importance is small or discontinuous?
right: The intuition behind IG is to accumulate pixel def load_imagenet_labels(file_path): labels_file = tf.keras.utils.get_file('ImageNetLabels.txt', file_path) with open(labels_file) as reader: f = reader.read() labels = f.splitlines() return np.array(labels) 8's local gradients and attribute its importance as a score for how much it adds or subtracts to your model's overall output class probability. You can break down and compute IG in 3 parts:
- interpolate small steps along a straight line in the feature space between 0 (a baseline or starting point) and 1 (input pixel's value)
- compute gradients at each step between your model's predictions with respect to each step
- approximate the integral between your baseline and input by accumulating (cumulative average) these local gradients.
To reinforce this intuition, you will walk through these 3 parts by applying IG to the example "Fireboat" image below.
Establish a baseline
A baseline is an input image used as a starting point for calculating feature importance. Intuitively, you can think of the baseline's explanatory role as representing the impact of the absence of each pixel on the "Fireboat" prediction to contrast with its impact of each pixel on the "Fireboat" prediction when present in the input image. As a result, the choice of the baseline plays a central role in interpreting and visualizing pixel feature importances. For additional discussion of baseline selection, see the resources in the "Next steps" section at the bottom of this tutorial. Here, you will use a black image whose pixel values are all zero.
Other choices you could experiment with include an all white image, or a random image, which you can create with imagenet_labels = load_imagenet_labels('//storage.googleapis.com/download.tensorflow.org/data/ImageNetLabels.txt') 2.
2022-12-14 07:21:01.173720: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory 2022-12-14 07:21:01.173816: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory 2022-12-14 07:21:01.173826: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. 62022-12-14 07:21:01.173720: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory 2022-12-14 07:21:01.173816: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory 2022-12-14 07:21:01.173826: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. 7Unpack formulas into code
The formula for Integrated Gradients is as follows:
\(IntegratedGradients_{i}(x) ::= (x_{i} - x'_{i})\times\int_{\alpha=0}^1\frac{\partial F(x'+\alpha \times (x - x'))}{\partial x_i}{d\alpha}\)
where:
\(_{i}\) = feature
\(x\) = input
\(x'\) = baseline
\(\alpha\) = interpolation constant to perturb features by
In practice, computing a definite integral is not always numerically possible and can be computationally costly, so you compute the following numerical approximation:
\(IntegratedGrads^{approx}_{i}(x)::=(x_{i}-x'_{i})\times\sum_{k=1}^{m}\frac{\partial F(x' + \frac{k}{m}\times(x - x'))}{\partial x_{i} } \times \frac{1}{m}\)
where:
\(_{i}\) = feature (individual pixel)
\(x\) = input (image tensor)
\(x'\) = baseline (image tensor)
\(k\) = scaled feature perturbation constant
\(m\) = number of steps in the Riemann sum approximation of the integral
\((x_{i}-x'_{i})\) = a term for the difference from the baseline. This is necessary to scale the integrated gradients and keep them in terms of the original image. The path from the baseline image to the input is in pixel space. Since with IG you are integrating in a straight line (linear transformation) this ends up being roughly equivalent to the integral term of the derivative of the interpolated image function with respect to \(\alpha\) with enough steps. The integral sums each pixel's gradient times the change in the pixel along the path. It's simpler to implement this integration as uniform steps from one image to the other, substituting \(x := (x' + \alpha(x-x'))\). So the change of variables gives \(dx = (x-x')d\alpha\). The \((x-x')\) term is constant and is factored out of the integral.
Interpolate images
\(IntegratedGrads^{approx}_{i}(x)::=(x_{i}-x'_{i})\times\sum_{k=1}^{m}\frac{\partial F(\overbrace{x' + \frac{k}{m}\times(x - x')}^\text{interpolate m images at k intervals})}{\partial x_{i} } \times \frac{1}{m}\)
First, you will generate a linear interpolation between the baseline and the original image. You can think of interpolated images as small steps in the feature space between your baseline and input, represented by \(\alpha\) in the original equation.
2022-12-14 07:21:01.173720: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory 2022-12-14 07:21:01.173816: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory 2022-12-14 07:21:01.173826: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. 82022-12-14 07:21:01.173720: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory 2022-12-14 07:21:01.173816: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory 2022-12-14 07:21:01.173826: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. 9Let's use the above function to generate interpolated images along a linear path at alpha intervals between a black baseline image and the example "Fireboat" image.
model = tf.keras.Sequential([ hub.KerasLayer( name='inception_v1', handle='//tfhub.dev/google/imagenet/inception_v1/classification/4', trainable=False), ]) model.build([None, 224, 224, 3]) model.summary() 0Let's visualize the interpolated images. Note: another way of thinking about the \(\alpha\) constant is that it is consistently increasing each interpolated image's intensity.
Compute gradients
Now let's take a look at how to calculate gradients in order to measure the relationship between changes to a feature and changes in the model's predictions. In the case of images, the gradient tells us which pixels have the strongest effect on the models predicted class probabilities.
\(IntegratedGrads^{approx}_{i}(x)::=(x_{i}-x'_{i})\times\sum_{k=1}^{m}\frac{\overbrace{\partial F(\text{interpolated images})}^\text{compute gradients} }{\partial x_{i} } \times \frac{1}{m}\)
where:
\(F()\) = your model's prediction function
\(\frac{\partial{F} }{\partial{x_i} }\) = gradient (vector of partial derivatives \(\partial\)) of your model F's prediction function relative to each feature \(x_i\)
TensorFlow makes computing gradients easy for you with a imagenet_labels = load_imagenet_labels('//storage.googleapis.com/download.tensorflow.org/data/ImageNetLabels.txt') 3.
model = tf.keras.Sequential([ hub.KerasLayer( name='inception_v1', handle='//tfhub.dev/google/imagenet/inception_v1/classification/4', trainable=False), ]) model.build([None, 224, 224, 3]) model.summary() 2Let's compute the gradients for each image along the interpolation path with respect to the correct output. Recall that your model returns a imagenet_labels = load_imagenet_labels('//storage.googleapis.com/download.tensorflow.org/data/ImageNetLabels.txt') 4 shaped imagenet_labels = load_imagenet_labels('//storage.googleapis.com/download.tensorflow.org/data/ImageNetLabels.txt') 5 with logits that you convert to predicted probabilities for each class. You need to pass the correct ImageNet target class index to the imagenet_labels = load_imagenet_labels('//storage.googleapis.com/download.tensorflow.org/data/ImageNetLabels.txt') 6 function for your image.
model = tf.keras.Sequential([ hub.KerasLayer( name='inception_v1', handle='//tfhub.dev/google/imagenet/inception_v1/classification/4', trainable=False), ]) model.build([None, 224, 224, 3]) model.summary() 3Note the output shape of imagenet_labels = load_imagenet_labels('//storage.googleapis.com/download.tensorflow.org/data/ImageNetLabels.txt') 7, which gives us the gradient for every pixel of every image along the interpolation path. You can think of these gradients as measuring the change in your model's predictions for each small step in the feature space.
model = tf.keras.Sequential([ hub.KerasLayer( name='inception_v1', handle='//tfhub.dev/google/imagenet/inception_v1/classification/4', trainable=False), ]) model.build([None, 224, 224, 3]) model.summary() 4model = tf.keras.Sequential([ hub.KerasLayer( name='inception_v1', handle='//tfhub.dev/google/imagenet/inception_v1/classification/4', trainable=False), ]) model.build([None, 224, 224, 3]) model.summary() 5Visualizing gradient saturation
Recall that the gradients you just calculated above describe local changes to your model's predicted probability of "Fireboat" and can saturate.
These concepts are visualized using the gradients you calculated above in the 2 plots below.
model = tf.keras.Sequential([ hub.KerasLayer( name='inception_v1', handle='//tfhub.dev/google/imagenet/inception_v1/classification/4', trainable=False), ]) model.build([None, 224, 224, 3]) model.summary() 6left: This plot shows how your model's confidence in the "Fireboat" class varies across alphas. Notice how the gradients, or slope of the line, largely flattens or saturates between 0.6 and 1.0 before settling at the final "Fireboat" predicted probability of about 40%.
right: The right plot shows the average gradients magnitudes over alpha more directly. Note how the values sharply approach and even briefly dip below zero. In fact, your model "learns" the most from gradients at lower values of alpha before saturating. Intuitively, you can think of this as your model has learned the pixels e.g. water cannons to make the correct prediction, sending these pixels gradients to zero, but is still quite uncertain and focused on spurious bridge or water jet pixels as the alpha values approach the original input image.
To make sure these important water cannon pixels are reflected as important to the "Fireboat" prediction, you will continue on below to learn how to accumulate these gradients to accurately approximate how each pixel impacts your "Fireboat" predicted probability.
Accumulate gradients (integral approximation)
There are many different ways you can go about computing the numerical approximation of an integral for IG with different tradeoffs in accuracy and convergence across varying functions. A popular class of methods is called Riemann sums. Here, you will use the Trapezoidal rule (you can find additional code to explore different approximation methods at the end of this tutorial).
\(IntegratedGrads^{approx}_{i}(x)::=(x_{i}-x'_{i})\times \overbrace{\sum_{k=1}^{m} }^\text{Sum m local gradients}\text{gradients(interpolated images)} \times \overbrace{\frac{1}{m} }^\text{Divide by m steps}\)
From the equation, you can see you are summing over imagenet_labels = load_imagenet_labels('//storage.googleapis.com/download.tensorflow.org/data/ImageNetLabels.txt') 8 gradients and dividing by imagenet_labels = load_imagenet_labels('//storage.googleapis.com/download.tensorflow.org/data/ImageNetLabels.txt') 8 steps. You can implement the two operations together for part 3 as an average of the local gradients of imagenet_labels = load_imagenet_labels('//storage.googleapis.com/download.tensorflow.org/data/ImageNetLabels.txt') 8 interpolated predictions and input images.
model = tf.keras.Sequential([ hub.KerasLayer( name='inception_v1', handle='//tfhub.dev/google/imagenet/inception_v1/classification/4', trainable=False), ]) model.build([None, 224, 224, 3]) model.summary() 8The def read_image(file_name): image = tf.io.read_file(file_name) image = tf.io.decode_jpeg(image, channels=3) image = tf.image.convert_image_dtype(image, tf.float32) image = tf.image.resize_with_pad(image, target_height=224, target_width=224) return image 1 function takes the gradients of the predicted probability of the target class with respect to the interpolated images between the baseline and the original image.
model = tf.keras.Sequential([ hub.KerasLayer( name='inception_v1', handle='//tfhub.dev/google/imagenet/inception_v1/classification/4', trainable=False), ]) model.build([None, 224, 224, 3]) model.summary() 9You can confirm averaging across the gradients of imagenet_labels = load_imagenet_labels('//storage.googleapis.com/download.tensorflow.org/data/ImageNetLabels.txt') 8 interpolated images returns an integrated gradients tensor with the same shape as the original "Giant Panda" image.
WARNING:tensorflow:Please fix your imports. Module tensorflow.python.training.tracking.data_structures has been moved to tensorflow.python.trackable.data_structures. The old module will be deleted in version 2.11. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/autograph/pyct/static_analysis/liveness.py:83: Analyzer.lamba_check (from tensorflow.python.autograph.pyct.static_analysis.liveness) is deprecated and will be removed after 2023-09-23. Instructions for updating: Lambda fuctions will be no more assumed to be used in the statement where they are used, or at least in the same block. //github.com/tensorflow/tensorflow/issues/56089 WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/autograph/pyct/static_analysis/liveness.py:83: Analyzer.lamba_check (from tensorflow.python.autograph.pyct.static_analysis.liveness) is deprecated and will be removed after 2023-09-23. Instructions for updating: Lambda fuctions will be no more assumed to be used in the statement where they are used, or at least in the same block. //github.com/tensorflow/tensorflow/issues/56089 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= inception_v1 (KerasLayer) (None, 1001) 6633209 ================================================================= Total params: 6,633,209 Trainable params: 0 Non-trainable params: 6,633,209 _________________________________________________________________ 0WARNING:tensorflow:Please fix your imports. Module tensorflow.python.training.tracking.data_structures has been moved to tensorflow.python.trackable.data_structures. The old module will be deleted in version 2.11. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/autograph/pyct/static_analysis/liveness.py:83: Analyzer.lamba_check (from tensorflow.python.autograph.pyct.static_analysis.liveness) is deprecated and will be removed after 2023-09-23. Instructions for updating: Lambda fuctions will be no more assumed to be used in the statement where they are used, or at least in the same block. //github.com/tensorflow/tensorflow/issues/56089 WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/autograph/pyct/static_analysis/liveness.py:83: Analyzer.lamba_check (from tensorflow.python.autograph.pyct.static_analysis.liveness) is deprecated and will be removed after 2023-09-23. Instructions for updating: Lambda fuctions will be no more assumed to be used in the statement where they are used, or at least in the same block. //github.com/tensorflow/tensorflow/issues/56089 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= inception_v1 (KerasLayer) (None, 1001) 6633209 ================================================================= Total params: 6,633,209 Trainable params: 0 Non-trainable params: 6,633,209 _________________________________________________________________ 1Putting it all together
Now you will combine the 3 previous general parts together into an def read_image(file_name): image = tf.io.read_file(file_name) image = tf.io.decode_jpeg(image, channels=3) image = tf.image.convert_image_dtype(image, tf.float32) image = tf.image.resize_with_pad(image, target_height=224, target_width=224) return image 3 function and utilize a @tf.function decorator to compile it into a high performance callable TensorFlow graph. This is implemented as 5 smaller steps below:
\(IntegratedGrads^{approx}_{i}(x)::=\overbrace{(x_{i}-x'_{i})}^\text{5.}\times \overbrace{\sum_{k=1}^{m} }^\text{4.} \frac{\partial \overbrace{F(\overbrace{x' + \overbrace{\frac{k}{m} }^\text{1.}\times(x - x'))}^\text{2.} }^\text{3.} }{\partial x_{i} } \times \overbrace{\frac{1}{m} }^\text{4.}\)
Generate alphas \(\alpha\)
Generate interpolated images = \((x' + \frac{k}{m}\times(x - x'))\)
Compute gradients between model \(F\) output predictions with respect to input features = \(\frac{\partial F(\text{interpolated path inputs})}{\partial x_{i} }\)
Integral approximation through averaging gradients = \(\sum_{k=1}^m \text{gradients} \times \frac{1}{m}\)
Scale integrated gradients with respect to original image = \((x_{i}-x'_{i}) \times \text{integrated gradients}\). The reason this step is necessary is to make sure that the attribution values accumulated across multiple interpolated images are in the same units and faithfully represent the pixel importances on the original image.
Again, you can check that the IG feature attributions have the same shape as the input "Fireboat" image.
WARNING:tensorflow:Please fix your imports. Module tensorflow.python.training.tracking.data_structures has been moved to tensorflow.python.trackable.data_structures. The old module will be deleted in version 2.11. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/autograph/pyct/static_analysis/liveness.py:83: Analyzer.lamba_check (from tensorflow.python.autograph.pyct.static_analysis.liveness) is deprecated and will be removed after 2023-09-23. Instructions for updating: Lambda fuctions will be no more assumed to be used in the statement where they are used, or at least in the same block. //github.com/tensorflow/tensorflow/issues/56089 WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/autograph/pyct/static_analysis/liveness.py:83: Analyzer.lamba_check (from tensorflow.python.autograph.pyct.static_analysis.liveness) is deprecated and will be removed after 2023-09-23. Instructions for updating: Lambda fuctions will be no more assumed to be used in the statement where they are used, or at least in the same block. //github.com/tensorflow/tensorflow/issues/56089 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= inception_v1 (KerasLayer) (None, 1001) 6633209 ================================================================= Total params: 6,633,209 Trainable params: 0 Non-trainable params: 6,633,209 _________________________________________________________________ 5WARNING:tensorflow:Please fix your imports. Module tensorflow.python.training.tracking.data_structures has been moved to tensorflow.python.trackable.data_structures. The old module will be deleted in version 2.11. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/autograph/pyct/static_analysis/liveness.py:83: Analyzer.lamba_check (from tensorflow.python.autograph.pyct.static_analysis.liveness) is deprecated and will be removed after 2023-09-23. Instructions for updating: Lambda fuctions will be no more assumed to be used in the statement where they are used, or at least in the same block. //github.com/tensorflow/tensorflow/issues/56089 WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/autograph/pyct/static_analysis/liveness.py:83: Analyzer.lamba_check (from tensorflow.python.autograph.pyct.static_analysis.liveness) is deprecated and will be removed after 2023-09-23. Instructions for updating: Lambda fuctions will be no more assumed to be used in the statement where they are used, or at least in the same block. //github.com/tensorflow/tensorflow/issues/56089 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= inception_v1 (KerasLayer) (None, 1001) 6633209 ================================================================= Total params: 6,633,209 Trainable params: 0 Non-trainable params: 6,633,209 _________________________________________________________________ 1The paper suggests the number of steps to range between 20 to 300 depending upon the example (although in practice this can be higher in the 1,000s to accurately approximate the integral). You can find additional code to check for the appropriate number of steps in the "Next steps" resources at the end of this tutorial.
Visualize attributions
You are ready to visualize attributions, and overlay them on the original image. The code below sums the absolute values of the integrated gradients across the color channels to produce an attribution mask. This plotting method captures the relative impact of pixels on the model's predictions.
Looking at the attributions on the "Fireboat" image, you can see the model identifies the water cannons and spouts as contributing to its correct prediction.
WARNING:tensorflow:Please fix your imports. Module tensorflow.python.training.tracking.data_structures has been moved to tensorflow.python.trackable.data_structures. The old module will be deleted in version 2.11. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/autograph/pyct/static_analysis/liveness.py:83: Analyzer.lamba_check (from tensorflow.python.autograph.pyct.static_analysis.liveness) is deprecated and will be removed after 2023-09-23. Instructions for updating: Lambda fuctions will be no more assumed to be used in the statement where they are used, or at least in the same block. //github.com/tensorflow/tensorflow/issues/56089 WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/autograph/pyct/static_analysis/liveness.py:83: Analyzer.lamba_check (from tensorflow.python.autograph.pyct.static_analysis.liveness) is deprecated and will be removed after 2023-09-23. Instructions for updating: Lambda fuctions will be no more assumed to be used in the statement where they are used, or at least in the same block. //github.com/tensorflow/tensorflow/issues/56089 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= inception_v1 (KerasLayer) (None, 1001) 6633209 ================================================================= Total params: 6,633,209 Trainable params: 0 Non-trainable params: 6,633,209 _________________________________________________________________ 8On the "Giant Panda" image, the attributions highlight the texture, nose, and the fur of the Panda's face.
WARNING:tensorflow:Please fix your imports. Module tensorflow.python.training.tracking.data_structures has been moved to tensorflow.python.trackable.data_structures. The old module will be deleted in version 2.11. WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/autograph/pyct/static_analysis/liveness.py:83: Analyzer.lamba_check (from tensorflow.python.autograph.pyct.static_analysis.liveness) is deprecated and will be removed after 2023-09-23. Instructions for updating: Lambda fuctions will be no more assumed to be used in the statement where they are used, or at least in the same block. //github.com/tensorflow/tensorflow/issues/56089 WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow/python/autograph/pyct/static_analysis/liveness.py:83: Analyzer.lamba_check (from tensorflow.python.autograph.pyct.static_analysis.liveness) is deprecated and will be removed after 2023-09-23. Instructions for updating: Lambda fuctions will be no more assumed to be used in the statement where they are used, or at least in the same block. //github.com/tensorflow/tensorflow/issues/56089 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= inception_v1 (KerasLayer) (None, 1001) 6633209 ================================================================= Total params: 6,633,209 Trainable params: 0 Non-trainable params: 6,633,209 _________________________________________________________________ 9Uses and limitations
Use cases
- Employing techniques like Integrated Gradients before deploying your model can help you develop intuition for how and why it works. Do the features highlighted by this technique match your intuition? If not, that may be indicative of a bug in your model or dataset, or overfitting.
Limitations
Integrated Gradients provides feature importances on individual examples, however, it does not provide global feature importances across an entire dataset.
Integrated Gradients provides individual feature importances, but it does not explain feature interactions and combinations.
Next steps
This tutorial presented a basic implementation of Integrated Gradients. As a next step, you can use this notebook to try this technique with different models and images yourself.
For interested readers, there is a lengthier version of this tutorial (which includes code for different baselines, to compute integral approximations, and to determine a sufficient number of steps) which you can find here.
To deepen your understanding, check out the paper Axiomatic Attribution for Deep Networks and Github repository, which contains an implementation in a previous version of TensorFlow. You can also explore feature attribution, and the impact of different baselines, on distill.pub.
Interested in incorporating IG into your production machine learning workflows for feature importances, model error analysis, and data skew monitoring? Check out Google Cloud's Explainable AI product that supports IG attributions. The Google AI PAIR research group also open-sourced the which can be used for model debugging, including visualizing IG feature attributions.