{ "cells": [ { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# For tips on running notebooks in Google Colab, see\n# https://pytorch.org/tutorials/beginner/colab\n%matplotlib inline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[Introduction](introyt1_tutorial.html) \\|\\|\n[Tensors](tensors_deeper_tutorial.html) \\|\\|\n[Autograd](autogradyt_tutorial.html) \\|\\| [Building\nModels](modelsyt_tutorial.html) \\|\\| **TensorBoard Support** \\|\\|\n[Training Models](trainingyt.html) \\|\\| [Model\nUnderstanding](captumyt.html)\n\nPyTorch TensorBoard Support\n===========================\n\nFollow along with the video below or on\n[youtube](https://www.youtube.com/watch?v=6CEld3hZgqc).\n\n``` {.python .jupyter-code-cell}\nfrom IPython.display import display, HTML\nhtml_code = \"\"\"\n
\n \n
\n\"\"\"\ndisplay(HTML(html_code))\n```\n\nBefore You Start\n----------------\n\nTo run this tutorial, you'll need to install PyTorch, TorchVision,\nMatplotlib, and TensorBoard.\n\nWith `conda`:\n\n``` {.sh}\nconda install pytorch torchvision -c pytorch\nconda install matplotlib tensorboard\n```\n\nWith `pip`:\n\n``` {.sh}\npip install torch torchvision matplotlib tensorboard\n```\n\nOnce the dependencies are installed, restart this notebook in the Python\nenvironment where you installed them.\n\nIntroduction\n------------\n\nIn this notebook, we'll be training a variant of LeNet-5 against the\nFashion-MNIST dataset. Fashion-MNIST is a set of image tiles depicting\nvarious garments, with ten class labels indicating the type of garment\ndepicted.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# PyTorch model and training necessities\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\n\n# Image datasets and image manipulation\nimport torchvision\nimport torchvision.transforms as transforms\n\n# Image display\nimport matplotlib.pyplot as plt\nimport numpy as np\n\n# PyTorch TensorBoard support\nfrom torch.utils.tensorboard import SummaryWriter\n\n# In case you are using an environment that has TensorFlow installed,\n# such as Google Colab, uncomment the following code to avoid\n# a bug with saving embeddings to your TensorBoard directory\n\n# import tensorflow as tf\n# import tensorboard as tb\n# tf.io.gfile = tb.compat.tensorflow_stub.io.gfile" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Showing Images in TensorBoard\n=============================\n\nLet's start by adding sample images from our dataset to TensorBoard:\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# Gather datasets and prepare them for consumption\ntransform = transforms.Compose(\n [transforms.ToTensor(),\n transforms.Normalize((0.5,), (0.5,))])\n\n# Store separate training and validations splits in ./data\ntraining_set = torchvision.datasets.FashionMNIST('./data',\n download=True,\n train=True,\n transform=transform)\nvalidation_set = torchvision.datasets.FashionMNIST('./data',\n download=True,\n train=False,\n transform=transform)\n\ntraining_loader = torch.utils.data.DataLoader(training_set,\n batch_size=4,\n shuffle=True,\n num_workers=2)\n\n\nvalidation_loader = torch.utils.data.DataLoader(validation_set,\n batch_size=4,\n shuffle=False,\n num_workers=2)\n\n# Class labels\nclasses = ('T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',\n 'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle Boot')\n\n# Helper function for inline image display\ndef matplotlib_imshow(img, one_channel=False):\n if one_channel:\n img = img.mean(dim=0)\n img = img / 2 + 0.5 # unnormalize\n npimg = img.numpy()\n if one_channel:\n plt.imshow(npimg, cmap=\"Greys\")\n else:\n plt.imshow(np.transpose(npimg, (1, 2, 0)))\n\n# Extract a batch of 4 images\ndataiter = iter(training_loader)\nimages, labels = next(dataiter)\n\n# Create a grid from the images and show them\nimg_grid = torchvision.utils.make_grid(images)\nmatplotlib_imshow(img_grid, one_channel=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Above, we used TorchVision and Matplotlib to create a visual grid of a\nminibatch of our input data. Below, we use the `add_image()` call on\n`SummaryWriter` to log the image for consumption by TensorBoard, and we\nalso call `flush()` to make sure it's written to disk right away.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# Default log_dir argument is \"runs\" - but it's good to be specific\n# torch.utils.tensorboard.SummaryWriter is imported above\nwriter = SummaryWriter('runs/fashion_mnist_experiment_1')\n\n# Write image data to TensorBoard log dir\nwriter.add_image('Four Fashion-MNIST Images', img_grid)\nwriter.flush()\n\n# To view, start TensorBoard on the command line with:\n# tensorboard --logdir=runs\n# ...and open a browser tab to http://localhost:6006/" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you start TensorBoard at the command line and open it in a new\nbrowser tab (usually at [localhost:6006](localhost:6006)), you should\nsee the image grid under the IMAGES tab.\n\nGraphing Scalars to Visualize Training\n======================================\n\nTensorBoard is useful for tracking the progress and efficacy of your\ntraining. Below, we'll run a training loop, track some metrics, and save\nthe data for TensorBoard's consumption.\n\nLet's define a model to categorize our image tiles, and an optimizer and\nloss function for training:\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "class Net(nn.Module):\n def __init__(self):\n super(Net, self).__init__()\n self.conv1 = nn.Conv2d(1, 6, 5)\n self.pool = nn.MaxPool2d(2, 2)\n self.conv2 = nn.Conv2d(6, 16, 5)\n self.fc1 = nn.Linear(16 * 4 * 4, 120)\n self.fc2 = nn.Linear(120, 84)\n self.fc3 = nn.Linear(84, 10)\n\n def forward(self, x):\n x = self.pool(F.relu(self.conv1(x)))\n x = self.pool(F.relu(self.conv2(x)))\n x = x.view(-1, 16 * 4 * 4)\n x = F.relu(self.fc1(x))\n x = F.relu(self.fc2(x))\n x = self.fc3(x)\n return x\n \n\nnet = Net()\ncriterion = nn.CrossEntropyLoss()\noptimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let's train a single epoch, and evaluate the training vs. validation\nset losses every 1000 batches:\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "print(len(validation_loader))\nfor epoch in range(1): # loop over the dataset multiple times\n running_loss = 0.0\n\n for i, data in enumerate(training_loader, 0):\n # basic training loop\n inputs, labels = data\n optimizer.zero_grad()\n outputs = net(inputs)\n loss = criterion(outputs, labels)\n loss.backward()\n optimizer.step()\n\n running_loss += loss.item()\n if i % 1000 == 999: # Every 1000 mini-batches...\n print('Batch {}'.format(i + 1))\n # Check against the validation set\n running_vloss = 0.0\n \n # In evaluation mode some model specific operations can be omitted eg. dropout layer\n net.train(False) # Switching to evaluation mode, eg. turning off regularisation\n for j, vdata in enumerate(validation_loader, 0):\n vinputs, vlabels = vdata\n voutputs = net(vinputs)\n vloss = criterion(voutputs, vlabels)\n running_vloss += vloss.item()\n net.train(True) # Switching back to training mode, eg. turning on regularisation\n \n avg_loss = running_loss / 1000\n avg_vloss = running_vloss / len(validation_loader)\n \n # Log the running loss averaged per batch\n writer.add_scalars('Training vs. Validation Loss',\n { 'Training' : avg_loss, 'Validation' : avg_vloss },\n epoch * len(training_loader) + i)\n\n running_loss = 0.0\nprint('Finished Training')\n\nwriter.flush()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Switch to your open TensorBoard and have a look at the SCALARS tab.\n\nVisualizing Your Model\n======================\n\nTensorBoard can also be used to examine the data flow within your model.\nTo do this, call the `add_graph()` method with a model and sample input:\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# Again, grab a single mini-batch of images\ndataiter = iter(training_loader)\nimages, labels = next(dataiter)\n\n# add_graph() will trace the sample input through your model,\n# and render it as a graph.\nwriter.add_graph(net, images)\nwriter.flush()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When you switch over to TensorBoard, you should see a GRAPHS tab.\nDouble-click the \"NET\" node to see the layers and data flow within your\nmodel.\n\nVisualizing Your Dataset with Embeddings\n========================================\n\nThe 28-by-28 image tiles we're using can be modeled as 784-dimensional\nvectors (28 \\* 28 = 784). It can be instructive to project this to a\nlower-dimensional representation. The `add_embedding()` method will\nproject a set of data onto the three dimensions with highest variance,\nand display them as an interactive 3D chart. The `add_embedding()`\nmethod does this automatically by projecting to the three dimensions\nwith highest variance.\n\nBelow, we'll take a sample of our data, and generate such an embedding:\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# Select a random subset of data and corresponding labels\ndef select_n_random(data, labels, n=100):\n assert len(data) == len(labels)\n\n perm = torch.randperm(len(data))\n return data[perm][:n], labels[perm][:n]\n\n# Extract a random subset of data\nimages, labels = select_n_random(training_set.data, training_set.targets)\n\n# get the class labels for each image\nclass_labels = [classes[label] for label in labels]\n\n# log embeddings\nfeatures = images.view(-1, 28 * 28)\nwriter.add_embedding(features,\n metadata=class_labels,\n label_img=images.unsqueeze(1))\nwriter.flush()\nwriter.close()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now if you switch to TensorBoard and select the PROJECTOR tab, you\nshould see a 3D representation of the projection. You can rotate and\nzoom the model. Examine it at large and small scales, and see whether\nyou can spot patterns in the projected data and the clustering of\nlabels.\n\nFor better visibility, it's recommended to:\n\n- Select \"label\" from the \"Color by\" drop-down on the left.\n- Toggle the Night Mode icon along the top to place the light-colored\n images on a dark background.\n\nOther Resources\n===============\n\nFor more information, have a look at:\n\n- PyTorch documentation on\n [torch.utils.tensorboard.SummaryWriter](https://pytorch.org/docs/stable/tensorboard.html?highlight=summarywriter)\n- Tensorboard tutorial content in the [PyTorch.org\n Tutorials](https://pytorch.org/tutorials/)\n- For more information about TensorBoard, see the [TensorBoard\n documentation](https://www.tensorflow.org/tensorboard)\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.12" } }, "nbformat": 4, "nbformat_minor": 0 }