{ "cells": [ { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# For tips on running notebooks in Google Colab, see\n# https://codelin.vip/beginner/colab\n%matplotlib inline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[Introduction](introyt1_tutorial.html) \\|\\|\n[Tensors](tensors_deeper_tutorial.html) \\|\\|\n[Autograd](autogradyt_tutorial.html) \\|\\| [Building\nModels](modelsyt_tutorial.html) \\|\\| [TensorBoard\nSupport](tensorboardyt_tutorial.html) \\|\\| [Training\nModels](trainingyt.html) \\|\\| **Model Understanding**\n\nModel Understanding with Captum\n===============================\n\nFollow along with the video below or on\n[youtube](https://www.youtube.com/watch?v=Am2EF9CLu-g). Download the\nnotebook and corresponding files\n[here](https://pytorch-tutorial-assets.s3.amazonaws.com/youtube-series/video7.zip).\n\n``` {.python .jupyter-code-cell}\nfrom IPython.display import display, HTML\nhtml_code = \"\"\"\n
\n \n
\n\"\"\"\ndisplay(HTML(html_code))\n```\n\n[Captum](https://captum.ai/) (\"comprehension\" in Latin) is an open\nsource, extensible library for model interpretability built on PyTorch.\n\nWith the increase in model complexity and the resulting lack of\ntransparency, model interpretability methods have become increasingly\nimportant. Model understanding is both an active area of research as\nwell as an area of focus for practical applications across industries\nusing machine learning. Captum provides state-of-the-art algorithms,\nincluding Integrated Gradients, to provide researchers and developers\nwith an easy way to understand which features are contributing to a\nmodel's output.\n\nFull documentation, an API reference, and a suite of tutorials on\nspecific topics are available at the [captum.ai](https://captum.ai/)\nwebsite.\n\nIntroduction\n------------\n\nCaptum's approach to model interpretability is in terms of\n*attributions.* There are three kinds of attributions available in\nCaptum:\n\n- **Feature Attribution** seeks to explain a particular output in\n terms of features of the input that generated it. Explaining whether\n a movie review was positive or negative in terms of certain words in\n the review is an example of feature attribution.\n- **Layer Attribution** examines the activity of a model's hidden\n layer subsequent to a particular input. Examining the\n spatially-mapped output of a convolutional layer in response to an\n input image in an example of layer attribution.\n- **Neuron Attribution** is analagous to layer attribution, but\n focuses on the activity of a single neuron.\n\nIn this interactive notebook, we'll look at Feature Attribution and\nLayer Attribution.\n\nEach of the three attribution types has multiple **attribution\nalgorithms** associated with it. Many attribution algorithms fall into\ntwo broad categories:\n\n- **Gradient-based algorithms** calculate the backward gradients of a\n model output, layer output, or neuron activation with respect to the\n input. **Integrated Gradients** (for features), **Layer Gradient \\*\n Activation**, and **Neuron Conductance** are all gradient-based\n algorithms.\n- **Perturbation-based algorithms** examine the changes in the output\n of a model, layer, or neuron in response to changes in the input.\n The input perturbations may be directed or random. **Occlusion,**\n **Feature Ablation,** and **Feature Permutation** are all\n perturbation-based algorithms.\n\nWe'll be examining algorithms of both types below.\n\nEspecially where large models are involved, it can be valuable to\nvisualize attribution data in ways that relate it easily to the input\nfeatures being examined. While it is certainly possible to create your\nown visualizations with Matplotlib, Plotly, or similar tools, Captum\noffers enhanced tools specific to its attributions:\n\n- The `captum.attr.visualization` module (imported below as `viz`)\n provides helpful functions for visualizing attributions related to\n images.\n- **Captum Insights** is an easy-to-use API on top of Captum that\n provides a visualization widget with ready-made visualizations for\n image, text, and arbitrary model types.\n\nBoth of these visualization toolsets will be demonstrated in this\nnotebook. The first few examples will focus on computer vision use\ncases, but the Captum Insights section at the end will demonstrate\nvisualization of attributions in a multi-model, visual\nquestion-and-answer model.\n\nInstallation\n------------\n\nBefore you get started, you need to have a Python environment with:\n\n- Python version 3.6 or higher\n- For the Captum Insights example, Flask 1.1 or higher and\n Flask-Compress (the latest version is recommended)\n- PyTorch version 1.2 or higher (the latest version is recommended)\n- TorchVision version 0.6 or higher (the latest version is\n recommended)\n- Captum (the latest version is recommended)\n- Matplotlib version 3.3.4, since Captum currently uses a Matplotlib\n function whose arguments have been renamed in later versions\n\nTo install Captum in an Anaconda or pip virtual environment, use the\nappropriate command for your environment below:\n\nWith `conda`:\n\n``` {.sh}\nconda install pytorch torchvision captum flask-compress matplotlib=3.3.4 -c pytorch\n```\n\nWith `pip`:\n\n``` {.sh}\npip install torch torchvision captum matplotlib==3.3.4 Flask-Compress\n```\n\nRestart this notebook in the environment you set up, and you're ready to\ngo!\n\nA First Example\n---------------\n\nTo start, let's take a simple, visual example. We'll start with a ResNet\nmodel pretrained on the ImageNet dataset. We'll get a test input, and\nuse different **Feature Attribution** algorithms to examine how the\ninput images affect the output, and see a helpful visualization of this\ninput attribution map for some test images.\n\nFirst, some imports:\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "import torch\nimport torch.nn.functional as F\nimport torchvision.transforms as transforms\nimport torchvision.models as models\n\nimport captum\nfrom captum.attr import IntegratedGradients, Occlusion, LayerGradCam, LayerAttribution\nfrom captum.attr import visualization as viz\n\nimport os, sys\nimport json\n\nimport numpy as np\nfrom PIL import Image\nimport matplotlib.pyplot as plt\nfrom matplotlib.colors import LinearSegmentedColormap" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we'll use the TorchVision model library to download a pretrained\nResNet. Since we're not training, we'll place it in evaluation mode for\nnow.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "model = models.resnet18(weights='IMAGENET1K_V1')\nmodel = model.eval()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The place where you got this interactive notebook should also have an\n`img` folder with a file `cat.jpg` in it.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "test_img = Image.open('img/cat.jpg')\ntest_img_data = np.asarray(test_img)\nplt.imshow(test_img_data)\nplt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Our ResNet model was trained on the ImageNet dataset, and expects images\nto be of a certain size, with the channel data normalized to a specific\nrange of values. We'll also pull in the list of human-readable labels\nfor the categories our model recognizes - that should be in the `img`\nfolder as well.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# model expects 224x224 3-color image\ntransform = transforms.Compose([\n transforms.Resize(224),\n transforms.CenterCrop(224),\n transforms.ToTensor()\n])\n\n# standard ImageNet normalization\ntransform_normalize = transforms.Normalize(\n mean=[0.485, 0.456, 0.406],\n std=[0.229, 0.224, 0.225]\n )\n\ntransformed_img = transform(test_img)\ninput_img = transform_normalize(transformed_img)\ninput_img = input_img.unsqueeze(0) # the model requires a dummy batch dimension\n\nlabels_path = 'img/imagenet_class_index.json'\nwith open(labels_path) as json_data:\n idx_to_labels = json.load(json_data)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, we can ask the question: What does our model think this image\nrepresents?\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "output = model(input_img)\noutput = F.softmax(output, dim=1)\nprediction_score, pred_label_idx = torch.topk(output, 1)\npred_label_idx.squeeze_()\npredicted_label = idx_to_labels[str(pred_label_idx.item())][1]\nprint('Predicted:', predicted_label, '(', prediction_score.squeeze().item(), ')')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We've confirmed that ResNet thinks our image of a cat is, in fact, a\ncat. But *why* does the model think this is an image of a cat?\n\nFor the answer to that, we turn to Captum.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Feature Attribution with Integrated Gradients\n=============================================\n\n**Feature attribution** attributes a particular output to features of\nthe input. It uses a specific input - here, our test image - to generate\na map of the relative importance of each input feature to a particular\noutput feature.\n\n[Integrated Gradients](https://captum.ai/api/integrated_gradients.html)\nis one of the feature attribution algorithms available in Captum.\nIntegrated Gradients assigns an importance score to each input feature\nby approximating the integral of the gradients of the model's output\nwith respect to the inputs.\n\nIn our case, we're going to be taking a specific element of the output\nvector - that is, the one indicating the model's confidence in its\nchosen category - and use Integrated Gradients to understand what parts\nof the input image contributed to this output.\n\nOnce we have the importance map from Integrated Gradients, we'll use the\nvisualization tools in Captum to give a helpful representation of the\nimportance map. Captum's `visualize_image_attr()` function provides a\nvariety of options for customizing display of your attribution data.\nHere, we pass in a custom Matplotlib color map.\n\nRunning the cell with the `integrated_gradients.attribute()` call will\nusually take a minute or two.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# Initialize the attribution algorithm with the model\nintegrated_gradients = IntegratedGradients(model)\n\n# Ask the algorithm to attribute our output target to \nattributions_ig = integrated_gradients.attribute(input_img, target=pred_label_idx, n_steps=200)\n\n# Show the original image for comparison\n_ = viz.visualize_image_attr(None, np.transpose(transformed_img.squeeze().cpu().detach().numpy(), (1,2,0)), \n method=\"original_image\", title=\"Original Image\")\n\ndefault_cmap = LinearSegmentedColormap.from_list('custom blue', \n [(0, '#ffffff'),\n (0.25, '#0000ff'),\n (1, '#0000ff')], N=256)\n\n_ = viz.visualize_image_attr(np.transpose(attributions_ig.squeeze().cpu().detach().numpy(), (1,2,0)),\n np.transpose(transformed_img.squeeze().cpu().detach().numpy(), (1,2,0)),\n method='heat_map',\n cmap=default_cmap,\n show_colorbar=True,\n sign='positive',\n title='Integrated Gradients')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In the image above, you should see that Integrated Gradients gives us\nthe strongest signal around the cat's location in the image.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Feature Attribution with Occlusion\n==================================\n\nGradient-based attribution methods help to understand the model in terms\nof directly computing out the output changes with respect to the input.\n*Perturbation-based attribution* methods approach this more directly, by\nintroducing changes to the input to measure the effect on the output.\n[Occlusion](https://captum.ai/api/occlusion.html) is one such method. It\ninvolves replacing sections of the input image, and examining the effect\non the output signal.\n\nBelow, we set up Occlusion attribution. Similarly to configuring a\nconvolutional neural network, you can specify the size of the target\nregion, and a stride length to determine the spacing of individual\nmeasurements. We'll visualize the output of our Occlusion attribution\nwith `visualize_image_attr_multiple()`, showing heat maps of both\npositive and negative attribution by region, and by masking the original\nimage with the positive attribution regions. The masking gives a very\ninstructive view of what regions of our cat photo the model found to be\nmost \"cat-like\".\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "occlusion = Occlusion(model)\n\nattributions_occ = occlusion.attribute(input_img,\n target=pred_label_idx,\n strides=(3, 8, 8),\n sliding_window_shapes=(3,15, 15),\n baselines=0)\n\n\n_ = viz.visualize_image_attr_multiple(np.transpose(attributions_occ.squeeze().cpu().detach().numpy(), (1,2,0)),\n np.transpose(transformed_img.squeeze().cpu().detach().numpy(), (1,2,0)),\n [\"original_image\", \"heat_map\", \"heat_map\", \"masked_image\"],\n [\"all\", \"positive\", \"negative\", \"positive\"],\n show_colorbar=True,\n titles=[\"Original\", \"Positive Attribution\", \"Negative Attribution\", \"Masked\"],\n fig_size=(18, 6)\n )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Again, we see greater significance placed on the region of the image\nthat contains the cat.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Layer Attribution with Layer GradCAM\n====================================\n\n**Layer Attribution** allows you to attribute the activity of hidden\nlayers within your model to features of your input. Below, we'll use a\nlayer attribution algorithm to examine the activity of one of the\nconvolutional layers within our model.\n\nGradCAM computes the gradients of the target output with respect to the\ngiven layer, averages for each output channel (dimension 2 of output),\nand multiplies the average gradient for each channel by the layer\nactivations. The results are summed over all channels. GradCAM is\ndesigned for convnets; since the activity of convolutional layers often\nmaps spatially to the input, GradCAM attributions are often upsampled\nand used to mask the input.\n\nLayer attribution is set up similarly to input attribution, except that\nin addition to the model, you must specify a hidden layer within the\nmodel that you wish to examine. As above, when we call `attribute()`, we\nspecify the target class of interest.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "layer_gradcam = LayerGradCam(model, model.layer3[1].conv2)\nattributions_lgc = layer_gradcam.attribute(input_img, target=pred_label_idx)\n\n_ = viz.visualize_image_attr(attributions_lgc[0].cpu().permute(1,2,0).detach().numpy(),\n sign=\"all\",\n title=\"Layer 3 Block 1 Conv 2\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We'll use the convenience method `interpolate()` in the\n[LayerAttribution](https://captum.ai/api/base_classes.html?highlight=layerattribution#captum.attr.LayerAttribution)\nbase class to upsample this attribution data for comparison to the input\nimage.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "upsamp_attr_lgc = LayerAttribution.interpolate(attributions_lgc, input_img.shape[2:])\n\nprint(attributions_lgc.shape)\nprint(upsamp_attr_lgc.shape)\nprint(input_img.shape)\n\n_ = viz.visualize_image_attr_multiple(upsamp_attr_lgc[0].cpu().permute(1,2,0).detach().numpy(),\n transformed_img.permute(1,2,0).numpy(),\n [\"original_image\",\"blended_heat_map\",\"masked_image\"],\n [\"all\",\"positive\",\"positive\"],\n show_colorbar=True,\n titles=[\"Original\", \"Positive Attribution\", \"Masked\"],\n fig_size=(18, 6))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Visualizations such as this can give you novel insights into how your\nhidden layers respond to your input.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Visualization with Captum Insights\n==================================\n\nCaptum Insights is an interpretability visualization widget built on top\nof Captum to facilitate model understanding. Captum Insights works\nacross images, text, and other features to help users understand feature\nattribution. It allows you to visualize attribution for multiple\ninput/output pairs, and provides visualization tools for image, text,\nand arbitrary data.\n\nIn this section of the notebook, we'll visualize multiple image\nclassification inferences with Captum Insights.\n\nFirst, let's gather some image and see what the model thinks of them.\nFor variety, we'll take our cat, a teapot, and a trilobite fossil:\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "imgs = ['img/cat.jpg', 'img/teapot.jpg', 'img/trilobite.jpg']\n\nfor img in imgs:\n img = Image.open(img)\n transformed_img = transform(img)\n input_img = transform_normalize(transformed_img)\n input_img = input_img.unsqueeze(0) # the model requires a dummy batch dimension\n\n output = model(input_img)\n output = F.softmax(output, dim=1)\n prediction_score, pred_label_idx = torch.topk(output, 1)\n pred_label_idx.squeeze_()\n predicted_label = idx_to_labels[str(pred_label_idx.item())][1]\n print('Predicted:', predicted_label, '/', pred_label_idx.item(), ' (', prediction_score.squeeze().item(), ')')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "...and it looks like our model is identifying them all correctly - but\nof course, we want to dig deeper. For that we'll use the Captum Insights\nwidget, which we configure with an `AttributionVisualizer` object,\nimported below. The `AttributionVisualizer` expects batches of data, so\nwe'll bring in Captum's `Batch` helper class. And we'll be looking at\nimages specifically, so well also import `ImageFeature`.\n\nWe configure the `AttributionVisualizer` with the following arguments:\n\n- An array of models to be examined (in our case, just the one)\n- A scoring function, which allows Captum Insights to pull out the\n top-k predictions from a model\n- An ordered, human-readable list of classes our model is trained on\n- A list of features to look for - in our case, an `ImageFeature`\n- A dataset, which is an iterable object returning batches of inputs\n and labels - just like you'd use for training\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from captum.insights import AttributionVisualizer, Batch\nfrom captum.insights.attr_vis.features import ImageFeature\n\n# Baseline is all-zeros input - this may differ depending on your data\ndef baseline_func(input):\n return input * 0\n\n# merging our image transforms from above\ndef full_img_transform(input):\n i = Image.open(input)\n i = transform(i)\n i = transform_normalize(i)\n i = i.unsqueeze(0)\n return i\n\n\ninput_imgs = torch.cat(list(map(lambda i: full_img_transform(i), imgs)), 0)\n\nvisualizer = AttributionVisualizer(\n models=[model],\n score_func=lambda o: torch.nn.functional.softmax(o, 1),\n classes=list(map(lambda k: idx_to_labels[k][1], idx_to_labels.keys())),\n features=[\n ImageFeature(\n \"Photo\",\n baseline_transforms=[baseline_func],\n input_transforms=[],\n )\n ],\n dataset=[Batch(input_imgs, labels=[282,849,69])]\n)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note that running the cell above didn't take much time at all, unlike\nour attributions above. That's because Captum Insights lets you\nconfigure different attribution algorithms in a visual widget, after\nwhich it will compute and display the attributions. *That* process will\ntake a few minutes.\n\nRunning the cell below will render the Captum Insights widget. You can\nthen choose attributions methods and their arguments, filter model\nresponses based on predicted class or prediction correctness, see the\nmodel's predictions with associated probabilities, and view heatmaps of\nthe attribution compared with the original image.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "visualizer.render()" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.12" } }, "nbformat": 4, "nbformat_minor": 0 }