diff --git a/index.rst b/index.rst index afde4e103fb..b7570e30224 100644 --- a/index.rst +++ b/index.rst @@ -1,288 +1,401 @@ Welcome to PyTorch Tutorials ============================ -To learn how to use PyTorch, begin with our Getting Started Tutorials. -The :doc:`60-minute blitz ` is the most common -starting point, and provides a broad view into how to use PyTorch from the basics all the way into constructing deep neural networks. - -Some considerations: - -* We’ve added a new feature to tutorials that allows users to open the notebook associated with a tutorial in Google Colab. - Visit `this page `_ for more information. -* If you would like to do the tutorials interactively via IPython / Jupyter, - each tutorial has a download link for a Jupyter Notebook and Python source code. -* Additional high-quality examples are available, including image classification, - unsupervised learning, reinforcement learning, machine translation, and - many other applications, in `PyTorch Examples - `_. -* You can find reference documentation for the PyTorch API and layers in `PyTorch Docs - `_ or via inline help. -* If you would like the tutorials section improved, please open a github issue - `here `_ with your feedback. -* Check out our - `PyTorch Cheat Sheet `_ - for additional useful information. -* Finally, here's a link to the - `PyTorch Release Notes `_ - -Learning PyTorch ------------------- - -.. customgalleryitem:: - :figure: /_static/img/thumbnails/pytorch-logo-flat.png - :tooltip: Understand PyTorch’s Tensor library and neural networks at a high level - :description: :doc:`/beginner/deep_learning_60min_blitz` - -.. customgalleryitem:: - :tooltip: This tutorial introduces the fundamental concepts of PyTorch through self-contained examples - :figure: /_static/img/thumbnails/examples.png - :description: :doc:`/beginner/pytorch_with_examples` - -.. customgalleryitem:: - :figure: /_static/img/torch.nn.png - :tooltip: Use torch.nn to create and train a neural network - :description: :doc:`beginner/nn_tutorial` - -.. customgalleryitem:: - :figure: /_static/img/thumbnails/pytorch_tensorboard.png - :tooltip: Learn to use TensorBoard to visualize data and model training - :description: :doc:`intermediate/tensorboard_tutorial` - -.. raw:: html - -
- - -Image/Video ----------------------- - -.. customgalleryitem:: - :figure: /_static/img/thumbnails/tv-img.png - :tooltip: Finetuning a pre-trained Mask R-CNN model - :description: :doc:`intermediate/torchvision_tutorial` - -.. customgalleryitem:: - :figure: /_static/img/thumbnails/sphx_glr_transfer_learning_tutorial_001.png - :tooltip: In transfer learning, a model created from one task is used in another - :description: :doc:`beginner/transfer_learning_tutorial` - -.. customgalleryitem:: - :figure: /_static/img/panda.png - :tooltip: Raise your awareness to the security vulnerabilities of ML models, and get insight into the hot topic of adversarial machine learning - :description: :doc:`beginner/fgsm_tutorial` - -.. customgalleryitem:: - :tooltip: Train a generative adversarial network (GAN) to generate new celebrities - :figure: /_static/img/dcgan_generator.png - :description: :doc:`beginner/dcgan_faces_tutorial` - - -.. raw:: html - -
- -Audio ----------------------- - -.. customgalleryitem:: - :figure: /_static/img/audio_preprocessing_tutorial_waveform.png - :tooltip: Preprocessing with torchaudio Tutorial - :description: :doc:`beginner/audio_preprocessing_tutorial` - .. raw:: html -
- +
+
-Text ----------------------- +.. Add callout items below this line -.. customgalleryitem:: - :tooltip: Transformer Tutorial - :figure: /_static/img/transformer_architecture.jpg - :description: :doc:`/beginner/transformer_tutorial` +.. customcalloutitem:: + :description: The 60 min blitz is the most common starting point and provides a broad view on how to use PyTorch. It covers the basics all to the way constructing deep neural networks. + :header: New to PyTorch? + :button_link: beginner/deep_learning_60min_blitz.html + :button_text: Start 60-min blitz -.. customgalleryitem:: - :figure: /_static/img/rnnclass.png - :tooltip: Build and train a basic character-level RNN to classify words - :description: :doc:`intermediate/char_rnn_classification_tutorial` +.. customcalloutitem:: + :description: Bite-size, ready-to-deploy PyTorch code examples. + :header: PyTorch Recipes + :button_link: recipes/recipes_index.html + :button_text: Explore Recipes -.. customgalleryitem:: - :figure: /_static/img/char_rnn_generation.png - :tooltip: Generate names from languages - :description: :doc:`intermediate/char_rnn_generation_tutorial` - -.. galleryitem:: intermediate/seq2seq_translation_tutorial.py - :figure: _static/img/seq2seq_flat.png - -.. customgalleryitem:: - :tooltip: Sentiment Ngrams with Torchtext - :figure: /_static/img/text_sentiment_ngrams_model.png - :description: :doc:`/beginner/text_sentiment_ngrams_tutorial` - -.. customgalleryitem:: - :tooltip: Language Translation with Torchtext - :figure: /_static/img/thumbnails/german_to_english_translation.png - :description: :doc:`/beginner/torchtext_translation_tutorial` +.. End of callout item section .. raw:: html -
- -Reinforcement Learning ----------------------- - -.. customgalleryitem:: - :tooltip: Use PyTorch to train a Deep Q Learning (DQN) agent - :figure: /_static/img/cartpole.gif - :description: :doc:`intermediate/reinforcement_q_learning` +
+
+ +
+ + + +
+ +
+ +
+
+ +.. Add tutorial cards below this line + +.. Learning PyTorch + +.. customcarditem:: + :header: Deep Learning with PyTorch: A 60 Minute Blitz + :card_description: Understand PyTorch’s Tensor library and neural networks at a high level. + :image: _static/img/thumbnails/cropped/60-min-blitz.png + :link: beginner/deep_learning_60min_blitz.html + :tags: Getting-Started + +.. customcarditem:: + :header: Learning PyTorch with Examples + :card_description: This tutorial introduces the fundamental concepts of PyTorch through self-contained examples. + :image: _static/img/thumbnails/cropped/learning-pytorch-with-examples.png + :link: beginner/pytorch_with_examples.html + :tags: Getting-Started + +.. customcarditem:: + :header: What is torch.nn really? + :card_description: Use torch.nn to create and train a neural network. + :image: _static/img/thumbnails/cropped/torch-nn.png + :link: beginner/nn_tutorial.html + :tags: Getting-Started + +.. customcarditem:: + :header: Visualizing Models, Data, and Training with Tensorboard + :card_description: Learn to use TensorBoard to visualize data and model training. + :image: _static/img/thumbnails/cropped/visualizing-with-tensorboard.png + :link: intermediate/tensorboard_tutorial.html + :tags: Interpretability,Getting-Started,Tensorboard + +.. Image/Video + +.. customcarditem:: + :header: TorchVision Object Detection Finetuning Tutorial + :card_description: Finetune a pre-trained Mask R-CNN model. + :image: _static/img/thumbnails/cropped/TorchVision-Object-Detection-Finetuning-Tutorial.png + :link: intermediate/torchvision_tutorial.html + :tags: Image/Video + +.. customcarditem:: + :header: Transfer Learning for Computer Vision Tutorial + :card_description: Train a convolutional neural network for image classification using transfer learning. + :image: _static/img/thumbnails/cropped/Transfer-Learning-for-Computer-Vision-Tutorial.png + :link: beginner/transfer_learning_tutorial.html + :tags: Image/Video + +.. customcarditem:: + :header: Adversarial Example Generation + :card_description: Train a convolutional neural network for image classification using transfer learning. + :image: _static/img/thumbnails/cropped/Adversarial-Example-Generation.png + :link: beginner/fgsm_tutorial.html + :tags: Image/Video + +.. customcarditem:: + :header: DCGAN Tutorial + :card_description: Train a generative adversarial network (GAN) to generate new celebrities. + :image: _static/img/thumbnails/cropped/DCGAN-Tutorial.png + :link: beginner/dcgan_faces_tutorial.html + :tags: Image/Video + +.. Audio + +.. customcarditem:: + :header: torchaudio Tutorial + :card_description: Learn to load and preprocess data from a simple dataset with PyTorch's torchaudio library. + :image: _static/img/thumbnails/cropped/torchaudio-Tutorial.png + :link: beginner/audio_preprocessing_tutorial.html + :tags: Audio + +.. Text + +.. customcarditem:: + :header: Sequence-to-Sequence Modeling with nn.Transformer and torchtext + :card_description: Learn how to train a sequence-to-sequence model that uses the nn.Transformer module. + :image: _static/img/thumbnails/cropped/Sequence-to-Sequence-Modeling-with-nnTransformer-andTorchText.png + :link: beginner/transformer_tutorial.html + :tags: Text + +.. customcarditem:: + :header: NLP from Scratch: Classifying Names with a Character-level RNN + :card_description: Build and train a basic character-level RNN to classify word from scratch without the use of torchtext. First in a series of three tutorials. + :image: _static/img/thumbnails/cropped/NLP-From-Scratch-Classifying-Names-with-a-Character-Level-RNN.png + :link: intermediate/char_rnn_classification_tutorial + :tags: Text + +.. customcarditem:: + :header: NLP from Scratch: Generating Names with a Character-level RNN + :card_description: After using character-level RNN to classify names, leanr how to generate names from languages. Second in a series of three tutorials. + :image: _static/img/thumbnails/cropped/NLP-From-Scratch-Generating-Names-with-a-Character-Level-RNN.png + :link: intermediate/char_rnn_generation_tutorial.html + :tags: Text + +.. customcarditem:: + :header: NLP from Scratch: Translation with a Sequence-to-sequence Network and Attention + :card_description: This is the third and final tutorial on doing “NLP From Scratch”, where we write our own classes and functions to preprocess the data to do our NLP modeling tasks. + :image: _static/img/thumbnails/cropped/NLP-From-Scratch-Translation-with-a-Sequence-to-Sequence-Network-and-Attention.png + :link: intermediate/seq2seq_translation_tutorial.html + :tags: Text + +.. customcarditem:: + :header: Text Classification with Torchtext + :card_description: This is the third and final tutorial on doing “NLP From Scratch”, where we write our own classes and functions to preprocess the data to do our NLP modeling tasks. + :image: _static/img/thumbnails/cropped/Text-Classification-with-TorchText.png + :link: beginner/text_sentiment_ngrams_tutorial.html + :tags: Text + +.. customcarditem:: + :header: Language Translation with Torchtext + :card_description: Use torchtext to reprocess data from a well-known datasets containing both English and German. Then use it to train a sequence-to-sequence model. + :image: _static/img/thumbnails/cropped/Language-Translation-with-TorchText.png + :link: beginner/torchtext_translation_tutorial.html + :tags: Text + +.. Reinforcement Learning + +.. customcarditem:: + :header: Reinforcement Learning (DQN) + :card_description: Learn how to use PyTorch to train a Deep Q Learning (DQN) agent on the CartPole-v0 task from the OpenAI Gym. + :image: _static/img/cartpole.gif + :link: intermediate/reinforcement_q_learning.html + :tags: Reinforcement-Learning + +.. Deploying PyTorch Models in Production + +.. customcarditem:: + :header: Deploying PyTorch in Python via a REST API with Flask + :card_description: Deploy a PyTorch model using Flask and expose a REST API for model inference using the example of a pretrained DenseNet 121 model which detects the image. + :image: _static/img/thumbnails/cropped/Deploying-PyTorch-in-Python-via-a-REST-API-with-Flask.png + :link: intermediate/flask_rest_api_tutorial.html + :tags: Production + +.. customcarditem:: + :header: Introduction to TorchScript + :card_description: Introduction to TorchScript, an intermediate representation of a PyTorch model (subclass of nn.Module) that can then be run in a high-performance environment such as C++. + :image: _static/img/thumbnails/cropped/Introduction-to-TorchScript.png + :link: beginner/Intro_to_TorchScript_tutorial.html + :tags: Production + +.. customcarditem:: + :header: Loading a TorchScript Model in C++ + :card_description: Learn how PyTorch provides to go from an existing Python model to a serialized representation that can be loaded and executed purely from C++, with no dependency on Python. + :image: _static/img/thumbnails/cropped/Loading-a-TorchScript-Model-in-Cpp.png + :link: advanced/cpp_export.html + :tags: Production + +.. customcarditem:: + :header: (optional) Exporting a Model from PyTorch to ONNX and Running it using ONNX Runtime + :card_description: Convert a model defined in PyTorch into the ONNX format and then run it with ONNX Runtime. + :image: _static/img/thumbnails/cropped/optional-Exporting-a-Model-from-PyTorch-to-ONNX-and-Running-it-using-ONNX-Runtime.png + :link: advanced/super_resolution_with_onnxruntime.html + :tags: Production + +.. Frontend APIs + +.. customcarditem:: + :header: (experimental) Introduction to Named Tensors in PyTorch + :card_description: Learn how to use PyTorch to train a Deep Q Learning (DQN) agent on the CartPole-v0 task from the OpenAI Gym. + :image: _static/img/thumbnails/cropped/experimental-Introduction-to-Named-Tensors-in-PyTorch.png + :link: intermediate/memory_format_tutorial.html + :tags: Frontend-APIs,Named-Tensor,Best-Practice + +.. customcarditem:: + :header: (experimental) Channels Last Memory Format in PyTorch + :card_description: Get an overview of Channels Last memory format and understand how it is used to order NCHW tensors in memory preserving dimensions. + :image: _static/img/thumbnails/cropped/experimental-Channels-Last-Memory-Format-in-PyTorch.png + :link: intermediate/memory_format_tutorial.html + :tags: Memory-Format,Best-Practice + +.. customcarditem:: + :header: Using the PyTorch C++ Frontend + :card_description: Walk through an end-to-end example of training a model with the C++ frontend by training a DCGAN – a kind of generative model – to generate images of MNIST digits. + :image: _static/img/thumbnails/cropped/Using-the-PyTorch-Cpp-Frontend.png + :link: advanced/cpp_frontend.html + :tags: Frontend-APIs,C++ + +.. customcarditem:: + :header: Custom C++ and CUDA Extensions + :card_description: Create a neural network layer with no parameters using numpy. Then use scipy to create a neural network layer that has learnable weights. + :image: _static/img/thumbnails/cropped/Custom-Cpp-and-CUDA-Extensions.png + :link: advanced/cpp_extension.html + :tags: Frontend-APIs,C++,CUDA + +.. customcarditem:: + :header: Extending TorchScript with Custom C++ Operators + :card_description: Implement a custom TorchScript operator in C++, how to build it into a shared library, how to use it in Python to define TorchScript models and lastly how to load it into a C++ application for inference workloads. + :image: _static/img/thumbnails/cropped/Extending-TorchScript-with-Custom-Cpp-Operators.png + :link: advanced/torch_script_custom_ops.html + :tags: Frontend-APIs,TorchScript,C++ + +.. customcarditem:: + :header: Extending TorchScript with Custom C++ Classes + :card_description: This is a continuation of the custom operator tutorial, and introduces the API we’ve built for binding C++ classes into TorchScript and Python simultaneously. + :image: _static/img/thumbnails/cropped/Extending-TorchScript-with-Custom-Cpp-Classes.png + :link: advanced/torch_script_custom_classes.html + :tags: Frontend-APIs,TorchScript,C++ + +.. customcarditem:: + :header: Autograd in C++ Frontend + :card_description: The autograd package helps build flexible and dynamic nerural netorks. In this tutorial, exploreseveral examples of doing autograd in PyTorch C++ frontend + :image: _static/img/thumbnails/cropped/Autograd-in-Cpp-Frontend.png + :link: advanced/cpp_autograd.html + :tags: Frontend-APIs,C++ + +.. Model Optimization + +.. customcarditem:: + :header: Pruning Tutorial + :card_description: Learn how to use torch.nn.utils.prune to sparsify your neural networks, and how to extend it to implement your own custom pruning technique. + :image: _static/img/thumbnails/cropped/Pruning-Tutorial.png + :link: intermediate/pruning_tutorial.html + :tags: Model-Optimization,Best-Practice + +.. customcarditem:: + :header: (experimental) Dynamic Quantization on an LSTM Word Language Model + :card_description: Apply dynamic quantization, the easiest form of quantization, to a LSTM-based next word prediction model. + :image: _static/img/thumbnails/cropped/experimental-Dynamic-Quantization-on-an-LSTM-Word-Language-Model.png + :link: advanced/dynamic_quantization_tutorial.html + :tags: Text,Quantization,Model-Optimization + +.. customcarditem:: + :header: (experimental) Dynamic Quantization on BERT + :card_description: Apply the dynamic quantization on a BERT (Bidirectional Embedding Representations from Transformers) model. + :image: _static/img/thumbnails/cropped/experimental-Dynamic-Quantization-on-BERT.png + :link: intermediate/dynamic_quantization_bert_tutorial.html + :tags: Text,Quantization,Model-Optimization + +.. customcarditem:: + :header: (experimental) Static Quantization with Eager Mode in PyTorch + :card_description: Learn techniques to impove a model's accuracy = post-training static quantization, per-channel quantization, and quantization-aware training. + :image: _static/img/thumbnails/cropped/experimental-Static-Quantization-with-Eager-Mode-in-PyTorch.png + :link: advanced/static_quantization_tutorial.html + :tags: Image/Video,Quantization,Model-Optimization + +.. customcarditem:: + :header: (experimental) Quantized Transfer Learning for Computer Vision Tutorial + :card_description: Learn techniques to impove a model's accuracy - post-training static quantization, per-channel quantization, and quantization-aware training. + :image: _static/img/thumbnails/cropped/experimental-Quantized-Transfer-Learning-for-Computer-Vision-Tutorial.png + :link: advanced/static_quantization_tutorial.html + :tags: Image/Video,Quantization,Model-Optimization + +.. Parallel-and-Distributed-Training + +.. customcarditem:: + :header: Single-Machine Model Parallel Best Practices + :card_description: Learn how to implement model parallel, a distributed training technique which splits a single model onto different GPUs, rather than replicating the entire model on each GPU + :image: _static/img/thumbnails/cropped/Model-Parallel-Best-Practices.png + :link: intermediate/model_parallel_tutorial.html + :tags: Parallel-and-Distributed-Training + +.. customcarditem:: + :header: Getting Started with Distributed Data Parallel + :card_description: Learn the basics of when to use distributed data paralle versus data parallel and work through an example to set it up. + :image: _static/img/thumbnails/cropped/Getting-Started-with-Distributed-Data-Parallel.png + :link: intermediate/ddp_tutorial.html + :tags: Parallel-and-Distributed-Training + +.. customcarditem:: + :header: Writing Distributed Applications with PyTorch + :card_description: Set up the distributed package of PyTorch, use the different communication strategies, and go over some the internals of the package. + :image: _static/img/thumbnails/cropped/Writing-Distributed-Applications-with-PyTorch.png + :link: intermediate/dist_tuto.html + :tags: Parallel-and-Distributed-Training + +.. customcarditem:: + :header: Getting Started with Distributed RPC Framework + :card_description: Learn how to build distributed training using the torch.distributed.rpc package. + :image: _static/img/thumbnails/cropped/Getting Started with Distributed-RPC-Framework.png + :link: intermediate/rpc_tutorial.html + :tags: Parallel-and-Distributed-Training + +.. customcarditem:: + :header: (advanced) PyTorch 1.0 Distributed Trainer with Amazon AWS + :card_description: Set up the distributed package of PyTorch, use the different communication strategies, and go over some the internals of the package. + :image: _static/img/thumbnails/cropped/advanced-PyTorch-1point0-Distributed-Trainer-with-Amazon-AWS.png + :link: beginner/aws_distributed_training_tutorial.html + :tags: Parallel-and-Distributed-Training + +.. customcarditem:: + :header: Implementing a Parameter Server Using Distributed RPC Framework + :card_description: Walk through a through a simple example of implementing a parameter server using PyTorch’s Distributed RPC framework. + :image: _static/img/thumbnails/cropped/Implementing-a-Parameter-Server-Using-Distributed-RPC-Framework.png + :link: intermediate/rpc_param_server_tutorial.html + :tags: Parallel-and-Distributed-Training + +.. End of tutorial card section .. raw:: html -
+
-Deploying PyTorch Models in Production --------------------------------------- + -.. customgalleryitem:: - :tooltip: Deploying PyTorch and Building a REST API using Flask - :description: :doc:`/intermediate/flask_rest_api_tutorial` - :figure: _static/img/flask.png +
-.. customgalleryitem:: - :tooltip: Introduction to TorchScript - :description: :doc:`beginner/Intro_to_TorchScript_tutorial` - :figure: _static/img/torchscript.png +
+
+
-.. customgalleryitem:: - :tooltip: Loading a PyTorch model in C++ - :description: :doc:`advanced/cpp_export` - :figure: _static/img/torchscript_to_cpp.png -.. customgalleryitem:: - :figure: /_static/img/cat.jpg - :tooltip: Exporting a Model from PyTorch to ONNX and Running it using ONNXRuntime - :description: :doc:`advanced/super_resolution_with_onnxruntime` +Additional Resources +============================ .. raw:: html -
- -Frontend APIs ----------------------- -.. customgalleryitem:: - :figure: /_static/img/named_tensor.png - :tooltip: Named Tensor - :description: :doc:`intermediate/named_tensor_tutorial` - -.. customgalleryitem:: - :figure: /_static/img/memory_format_logo.png - :tooltip: Memory Format - :description: :doc:`intermediate/memory_format_tutorial` - -.. customgalleryitem:: - :tooltip: Using the PyTorch C++ Frontend - :figure: /_static/img/cpp-pytorch.png - :description: :doc:`advanced/cpp_frontend` - -.. customgalleryitem:: - :tooltip: Implement custom extensions in C++ or CUDA for eager PyTorch - :description: :doc:`/advanced/cpp_extension` - :figure: _static/img/cpp_logo.png - -.. customgalleryitem:: - :tooltip: Implement custom operators in C++ or CUDA for TorchScript - :description: :doc:`/advanced/torch_script_custom_ops` - :figure: _static/img/cpp_logo.png - -.. customgalleryitem:: - :tooltip: Implement custom classes in C++ for TorchScript - :description: :doc:`/advanced/torch_script_custom_classes` - :figure: _static/img/cpp_logo.png - -.. customgalleryitem:: - :tooltip: Autograd in C++ Frontend - :figure: /_static/img/cpp-pytorch.png - :description: :doc:`advanced/cpp_autograd` +
+
-.. raw:: html - -
+.. Add callout items below this line +.. customcalloutitem:: + :header: Examples of PyTorch + :description: A set of examples around pytorch in Vision, Text, Reinforcement Learning, etc. + :button_link: https://github.com/pytorch/examples + :button_text: Checkout Examples -Model Optimization ---------------------------- +.. customcalloutitem:: + :header: PyTorch Cheat Sheet + :description: Quick overview to essential PyTorch elements. + :button_link: beginner/ptcheat.html + :button_text: Download -.. customgalleryitem:: - :tooltip: Use pruning to sparsify your neural networks - :description: :doc:`/intermediate/pruning_tutorial` - :figure: _static/img/pruning.png +.. customcalloutitem:: + :header: Tutorials on GitHub + :description: Access PyTorch Tutorials from GitHub. + :button_link: https://github.com/pytorch/tutorials + :button_text: Go To GitHub -.. customgalleryitem:: - :tooltip: Perform dynamic quantization on a pre-trained PyTorch model - :description: :doc:`/advanced/dynamic_quantization_tutorial` - :figure: _static/img/quant_asym.png -.. customgalleryitem:: - :tooltip: Convert a well-known state-of-the-art model like BERT into dynamic quantized model - :description: :doc:`/intermediate/dynamic_quantization_bert_tutorial` - :figure: /_static/img/bert.png - -.. customgalleryitem:: - :tooltip: (experimental) Static Quantization with Eager Mode in PyTorch - :figure: /_static/img/qat.png - :description: :doc:`advanced/static_quantization_tutorial` - -.. customgalleryitem:: - :tooltip: Perform quantized transfer learning with feature extractor - :description: :doc:`/intermediate/quantized_transfer_learning_tutorial` - :figure: /_static/img/quantized_transfer_learning.png +.. End of callout section .. raw:: html -
- -Parallel and Distributed Training ---------------------------------- - -.. customgalleryitem:: - :tooltip: Model parallel training on multiple GPUs - :description: :doc:`/intermediate/model_parallel_tutorial` - :figure: _static/img/distributed/DistPyTorch.jpg - -.. customgalleryitem:: - :tooltip: Getting started with DistributedDataParallel - :description: :doc:`/intermediate/ddp_tutorial` - :figure: _static/img/distributed/DistPyTorch.jpg - -.. customgalleryitem:: - :tooltip: Parallelize computations across processes and clusters of machines - :description: :doc:`/intermediate/dist_tuto` - :figure: _static/img/distributed/DistPyTorch.jpg - -.. customgalleryitem:: - :tooltip: Getting Started with Distributed RPC Framework - :description: :doc:`/intermediate/rpc_tutorial` - :figure: _static/img/distributed/DistPyTorch.jpg - -.. customgalleryitem:: - :tooltip: PyTorch distributed trainer with Amazon AWS - :description: :doc:`/beginner/aws_distributed_training_tutorial` - :figure: _static/img/distributed/DistPyTorch.jpg - -.. customgalleryitem:: - :tooltip: Implementing a Parameter Server Using Distributed RPC Framework - :description: :doc:`/intermediate/rpc_param_server_tutorial` - :figure: _static/img/distributed/DistPyTorch.jpg - -.. raw:: html +
+
- .. ----------------------------------------- .. Page TOC .. ----------------------------------------- +.. toctree:: + :maxdepth: 2 + :hidden: + :includehidden: + :caption: PyTorch Recipes + + See All Recipes + .. toctree:: :maxdepth: 2 :hidden: diff --git a/recipes/deployment_with_flask.rst b/recipes/deployment_with_flask.rst deleted file mode 100644 index 0d1291c8d89..00000000000 --- a/recipes/deployment_with_flask.rst +++ /dev/null @@ -1,284 +0,0 @@ -Deploying with Flask -==================== - -In this recipe, you will learn: - -- How to wrap your trained PyTorch model in a Flask container to expose - it via a web API -- How to translate incoming web requests into PyTorch tensors for your - model -- How to package your model’s output for an HTTP response - -Requirements ------------- - -You will need a Python 3 environment with the following packages (and -their dependencies) installed: - -- PyTorch 1.5 -- TorchVision 0.6.0 -- Flask 1.1 - -Optionally, to get some of the supporting files, you'll need git. - -The instructions for installing PyTorch and TorchVision are available at -`pytorch.org`_. Instructions for installing Flask are available on `the -Flask site`_. - -What is Flask? --------------- - -Flask is a lightweight web server written in Python. It provides a -convenient way for you to quickly set up a web API for predictions from -your trained PyTorch model, either for direct use, or as a web service -within a larger system. - -Setup and Supporting Files --------------------------- - -We're going to create a web service that takes in images, and maps them -to one of the 1000 classes of the ImageNet dataset. To do this, you'll -need an image file for testing. Optionally, you can also get a file that -will map the class index output by the model to a human-readable class -name. - -Option 1: To Get Both Files Quickly -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -You can pull both of the supporting files quickly by checking out the -TorchServe repository and copying them to your working folder. *(NB: -There is no dependency on TorchServe for this tutorial - it's just a -quick way to get the files.)* Issue the following commands from your -shell prompt: - -:: - - git clone https://github.com/pytorch/serve - cp serve/examples/image_classifier/kitten.jpg . - cp serve/examples/image_classifier/index_to_name.json . - -And you've got them! - -Option 2: Bring Your Own Image -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The ``index_to_name.json`` file is optional in the Flask service below. -You can test your service with your own image - just make sure it's a -3-color JPEG. - -Building Your Flask Service ---------------------------- - -The full Python script for the Flask service is shown at the end of this -recipe; you can copy and paste that into your own ``app.py`` file. Below -we'll look at individual sections to make their functions clear. - -Imports -~~~~~~~ - -:: - - import torchvision.models as models - import torchvision.transforms as transforms - from PIL import Image - from flask import Flask, jsonify, request - -In order: - -- We'll be using a pre-trained DenseNet model from - ``torchvision.models`` -- ``torchvision.transforms`` contains tools for manipulating your image - data -- Pillow (``PIL``) is what we'll use to load the image file initially -- And of course we'll need classes from ``flask`` - -Pre-Processing -~~~~~~~~~~~~~~ - -:: - - def transform_image(infile): - input_transforms = [transforms.Resize(255), - transforms.CenterCrop(224), - transforms.ToTensor(), - transforms.Normalize([0.485, 0.456, 0.406], - [0.229, 0.224, 0.225])] - my_transforms = transforms.Compose(input_transforms) - image = Image.open(infile) - timg = my_transforms(image) - timg.unsqueeze_(0) - return timg - -The web request gave us an image file, but our model expects a PyTorch -tensor of shape (N, 3, 224, 224) where *N* is the number of items in the -input batch. (We will just have a batch size of 1.) The first thing we -do is compose a set of TorchVision transforms that resize and crop the -image, convert it to a tensor, then normalize the values in the tensor. -(For more information on this normalization, see the documentation for -``torchvision.models_``.) - -After that, we open the file and apply the transforms. The transforms -return a tensor of shape (3, 224, 224) - the 3 color channels of a -224x224 image. Because we need to make this single image a batch, we use -the ``unsqueeze_(0)`` call to modify the tensor in place by adding a new -first dimension. The tensor contains the same data, but now has shape -(1, 3, 224, 224). - -In general, even if you're not working with image data, you will need to -transform the input from your HTTP request into a tensor that PyTorch -can consume. - -Inference -~~~~~~~~~ - -:: - - def get_prediction(input_tensor): - outputs = model.forward(input_tensor) - _, y_hat = outputs.max(1) - prediction = y_hat.item() - return prediction - -The inference itself is the simplest part: When we pass the input tensor -to them model, we get back a tensor of values that represent the model's -estimated likelihood that the image belongs to a particular class. The -``max()`` call finds the class with the maximum likelihood value, and -returns that value with the ImageNet class index. Finally, we extract -that class index from the tensor containing it with the ``item()`` call, and -return it. - -Post-Processing -~~~~~~~~~~~~~~~ - -:: - - def render_prediction(prediction_idx): - stridx = str(prediction_idx) - class_name = 'Unknown' - if img_class_map is not None: - if stridx in img_class_map is not None: - class_name = img_class_map[stridx][1] - - return prediction_idx, class_name - -The ``render_prediction()`` method maps the predicted class index to a -human-readable class label. It's typical, after getting the prediction -from your model, to perform post-processing to make the prediction ready -for either human consumption, or for another piece of software. - -Running The Full Flask App --------------------------- - -Paste the following into a file called ``app.py``: - -:: - - import io - import json - import os - - import torchvision.models as models - import torchvision.transforms as transforms - from PIL import Image - from flask import Flask, jsonify, request - - - app = Flask(__name__) - model = models.densenet121(pretrained=True) # Trained on 1000 classes from ImageNet - model.eval() # Turns off autograd and - - - - img_class_map = None - mapping_file_path = 'index_to_name.json' # Human-readable names for Imagenet classes - if os.path.isfile(mapping_file_path): - with open (mapping_file_path) as f: - img_class_map = json.load(f) - - - - # Transform input into the form our model expects - def transform_image(infile): - input_transforms = [transforms.Resize(255), # We use multiple TorchVision transforms to ready the image - transforms.CenterCrop(224), - transforms.ToTensor(), - transforms.Normalize([0.485, 0.456, 0.406], # Standard normalization for ImageNet model input - [0.229, 0.224, 0.225])] - my_transforms = transforms.Compose(input_transforms) - image = Image.open(infile) # Open the image file - timg = my_transforms(image) # Transform PIL image to appropriately-shaped PyTorch tensor - timg.unsqueeze_(0) # PyTorch models expect batched input; create a batch of 1 - return timg - - - # Get a prediction - def get_prediction(input_tensor): - outputs = model.forward(input_tensor) # Get likelihoods for all ImageNet classes - _, y_hat = outputs.max(1) # Extract the most likely class - prediction = y_hat.item() # Extract the int value from the PyTorch tensor - return prediction - - # Make the prediction human-readable - def render_prediction(prediction_idx): - stridx = str(prediction_idx) - class_name = 'Unknown' - if img_class_map is not None: - if stridx in img_class_map is not None: - class_name = img_class_map[stridx][1] - - return prediction_idx, class_name - - - @app.route('/', methods=['GET']) - def root(): - return jsonify({'msg' : 'Try POSTing to the /predict endpoint with an RGB image attachment'}) - - - @app.route('/predict', methods=['POST']) - def predict(): - if request.method == 'POST': - file = request.files['file'] - if file is not None: - input_tensor = transform_image(file) - prediction_idx = get_prediction(input_tensor) - class_id, class_name = render_prediction(prediction_idx) - return jsonify({'class_id': class_id, 'class_name': class_name}) - - - if __name__ == '__main__': - app.run() - -To start the server from your shell prompt, issue the following command: - -:: - - FLASK_APP=app.py flask run - -By default, your Flask server is listening on port 5000. Once the server -is running, open another terminal window, and test your new inference -server: - -:: - - curl -X POST -H "Content-Type: multipart/form-data" http://localhost:5000/predict -F "file=@kitten.jpg" - -If everything is set up correctly, you should recevie a response similar -to the following: - -:: - - {"class_id":285,"class_name":"Egyptian_cat"} - -Important Resources -------------------- - -- `pytorch.org`_ for installation instructions, and more documentation - and tutorials -- The `Flask site`_ has a `Quick Start guide`_ that goes into more - detail on setting up a simple Flask service - -.. _pytorch.org: https://pytorch.org -.. _Flask site: https://flask.palletsprojects.com/en/1.1.x/ -.. _Quick Start guide: https://flask.palletsprojects.com/en/1.1.x/quickstart/ -.. _torchvision.models: https://pytorch.org/docs/stable/torchvision/models.html -.. _the Flask site: https://flask.palletsprojects.com/en/1.1.x/installation/ diff --git a/recipes/index.rst b/recipes/index.rst deleted file mode 100644 index 9c0450f4d99..00000000000 --- a/recipes/index.rst +++ /dev/null @@ -1,367 +0,0 @@ -:orphan: - - - -.. _sphx_glr_recipes: - -Recipes ------------------- -1. recipes/* and recipes_index.rst - PyTorch Recipes - https://pytorch.org/tutorials/recipes/recipes_index.html - - - - -.. raw:: html - -
- - - -.. _sphx_glr_recipes_recipes: - -PyTorch Recipes ---------------------------------------------- -1. loading_data_recipe.py - Loading Data in PyTorch - https://pytorch.org/tutorials/recipes/recipes/loading_data_recipe.html - -2. defining_a_neural_network.py - Defining a Neural Network in PyTorch - https://pytorch.org/tutorials/recipes/recipes/defining_a_neural_network.html - -3. what_is_state_dict.py - What is a state_dict in PyTorch - https://pytorch.org/tutorials/recipes/recipes/what_is_state_dict.html - -4. saving_and_loading_models_for_inference.py - Saving and loading models for inference in PyTorch - https://pytorch.org/tutorials/recipes/recipes/saving_and_loading_models_for_inference.html - -5. custom_dataset_transforms_loader.py - Developing Custom PyTorch Dataloaders - https://pytorch.org/tutorials/recipes/recipes/custom_dataset_transforms_loader.html - - -6. Captum_Recipe.py - Model Interpretability using Captum - https://pytorch.org/tutorials/recipes/recipes/Captum_Recipe.html - -7. dynamic_quantization.py - Dynamic Quantization - https://pytorch.org/tutorials/recipes/recipes/dynamic_quantization.html - -8. save_load_across_devices.py - Saving and loading models across devices in PyTorch - https://pytorch.org/tutorials/recipes/recipes/save_load_across_devices.html - -9. saving_and_loading_a_general_checkpoint.py - Saving and loading a general checkpoint in PyTorch - https://pytorch.org/tutorials/recipes/recipes/saving_and_loading_a_general_checkpoint.html - -10. saving_and_loading_models_for_inference.py - Saving and loading models for inference in PyTorch - https://pytorch.org/tutorials/recipes/recipes/saving_and_loading_models_for_inference.html - -11. saving_multiple_models_in_one_file.py - Saving and loading multiple models in one file using PyTorch - https://pytorch.org/tutorials/recipes/recipes/saving_multiple_models_in_one_file.html - -12. warmstarting_model_using_parameters_from_a_different_model.py - Warmstarting models using parameters from different model - https://pytorch.org/tutorials/recipes/recipes/warmstarting_model_using_parameters_from_a_different_model.html - -13. zeroing_out_gradients.py - Zeroing out gradients - https://pytorch.org/tutorials/recipes/recipes/zeroing_out_gradients.html - - - -.. raw:: html - -
- -.. only:: html - - .. figure:: /recipes/recipes/images/thumb/sphx_glr_tensorboard_with_pytorch_thumb.png - - :ref:`sphx_glr_recipes_recipes_tensorboard_with_pytorch.py` - -.. raw:: html - -
- - -.. toctree:: - :hidden: - - /recipes/recipes/tensorboard_with_pytorch - -.. raw:: html - -
- -.. only:: html - - .. figure:: /recipes/recipes/images/thumb/sphx_glr_saving_and_loading_models_for_inference_thumb.png - - :ref:`sphx_glr_recipes_recipes_saving_and_loading_models_for_inference.py` - -.. raw:: html - -
- - -.. toctree:: - :hidden: - - /recipes/recipes/saving_and_loading_models_for_inference - -.. raw:: html - -
- -.. only:: html - - .. figure:: /recipes/recipes/images/thumb/sphx_glr_what_is_state_dict_thumb.png - - :ref:`sphx_glr_recipes_recipes_what_is_state_dict.py` - -.. raw:: html - -
- - -.. toctree:: - :hidden: - - /recipes/recipes/what_is_state_dict - -.. raw:: html - -
- -.. only:: html - - .. figure:: /recipes/recipes/images/thumb/sphx_glr_saving_and_loading_a_general_checkpoint_thumb.png - - :ref:`sphx_glr_recipes_recipes_saving_and_loading_a_general_checkpoint.py` - -.. raw:: html - -
- - -.. toctree:: - :hidden: - - /recipes/recipes/saving_and_loading_a_general_checkpoint - -.. raw:: html - -
- -.. only:: html - - .. figure:: /recipes/recipes/images/thumb/sphx_glr_warmstarting_model_using_parameters_from_a_different_model_thumb.png - - :ref:`sphx_glr_recipes_recipes_warmstarting_model_using_parameters_from_a_different_model.py` - -.. raw:: html - -
- - -.. toctree:: - :hidden: - - /recipes/recipes/warmstarting_model_using_parameters_from_a_different_model - -.. raw:: html - -
- -.. only:: html - - .. figure:: /recipes/recipes/images/thumb/sphx_glr_loading_data_recipe_thumb.png - - :ref:`sphx_glr_recipes_recipes_loading_data_recipe.py` - -.. raw:: html - -
- - -.. toctree:: - :hidden: - - /recipes/recipes/loading_data_recipe - -.. raw:: html - -
- -.. only:: html - - .. figure:: /recipes/recipes/images/thumb/sphx_glr_save_load_across_devices_thumb.png - - :ref:`sphx_glr_recipes_recipes_save_load_across_devices.py` - -.. raw:: html - -
- - -.. toctree:: - :hidden: - - /recipes/recipes/save_load_across_devices - -.. raw:: html - -
- -.. only:: html - - .. figure:: /recipes/recipes/images/thumb/sphx_glr_saving_multiple_models_in_one_file_thumb.png - - :ref:`sphx_glr_recipes_recipes_saving_multiple_models_in_one_file.py` - -.. raw:: html - -
- - -.. toctree:: - :hidden: - - /recipes/recipes/saving_multiple_models_in_one_file - -.. raw:: html - -
- -.. only:: html - - .. figure:: /recipes/recipes/images/thumb/sphx_glr_defining_a_neural_network_thumb.png - - :ref:`sphx_glr_recipes_recipes_defining_a_neural_network.py` - -.. raw:: html - -
- - -.. toctree:: - :hidden: - - /recipes/recipes/defining_a_neural_network - -.. raw:: html - -
- -.. only:: html - - .. figure:: /recipes/recipes/images/thumb/sphx_glr_zeroing_out_gradients_thumb.png - - :ref:`sphx_glr_recipes_recipes_zeroing_out_gradients.py` - -.. raw:: html - -
- - -.. toctree:: - :hidden: - - /recipes/recipes/zeroing_out_gradients - -.. raw:: html - -
- -.. only:: html - - .. figure:: /recipes/recipes/images/thumb/sphx_glr_dynamic_quantization_thumb.png - - :ref:`sphx_glr_recipes_recipes_dynamic_quantization.py` - -.. raw:: html - -
- - -.. toctree:: - :hidden: - - /recipes/recipes/dynamic_quantization - -.. raw:: html - -
- -.. only:: html - - .. figure:: /recipes/recipes/images/thumb/sphx_glr_Captum_Recipe_thumb.png - - :ref:`sphx_glr_recipes_recipes_Captum_Recipe.py` - -.. raw:: html - -
- - -.. toctree:: - :hidden: - - /recipes/recipes/Captum_Recipe - -.. raw:: html - -
- -.. only:: html - - .. figure:: /recipes/recipes/images/thumb/sphx_glr_custom_dataset_transforms_loader_thumb.png - - :ref:`sphx_glr_recipes_recipes_custom_dataset_transforms_loader.py` - -.. raw:: html - -
- - -.. toctree:: - :hidden: - - /recipes/recipes/custom_dataset_transforms_loader -.. raw:: html - -
- - - -.. only :: html - - .. container:: sphx-glr-footer - :class: sphx-glr-footer-gallery - - - .. container:: sphx-glr-download - - :download:`Download all examples in Python source code: recipes_python.zip ` - - - - .. container:: sphx-glr-download - - :download:`Download all examples in Jupyter notebooks: recipes_jupyter.zip ` - - -.. only:: html - - .. rst-class:: sphx-glr-signature - - `Gallery generated by Sphinx-Gallery `_ diff --git a/recipes/recipes/Captum_Recipe.ipynb b/recipes/recipes/Captum_Recipe.ipynb deleted file mode 100644 index 9feb2f0bc4a..00000000000 --- a/recipes/recipes/Captum_Recipe.ipynb +++ /dev/null @@ -1,160 +0,0 @@ -{ - "cells": [ - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "%matplotlib inline" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\nModel Interpretability using Captum\n===================================\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Captum helps you understand how the data features impact your model\npredictions or neuron activations, shedding light on how your model\noperates.\n\nUsing Captum, you can apply a wide range of state-of-the-art feature\nattribution algorithms such as \\ ``Guided GradCam``\\ and\n\\ ``Integrated Gradients``\\ in a unified way.\n\nIn this recipe you will learn how to use Captum to: \\* attribute the\npredictions of an image classifier to their corresponding image\nfeatures. \\* visualize the attribution results.\n\n\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Before you begin\n----------------\n\n\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Make sure Captum is installed in your active Python environment. Captum\nis available both on GitHub, as a ``pip`` package, or as a ``conda``\npackage. For detailed instructions, consult the installation guide at\nhttps://captum.ai/\n\n\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "For a model, we use a built-in image classifier in PyTorch. Captum can\nreveal which parts of a sample image support certain predictions made by\nthe model.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "import torchvision\nfrom torchvision import transforms\nfrom PIL import Image\nimport requests\nfrom io import BytesIO\n\nmodel = torchvision.models.resnet18(pretrained=True).eval()\n\nresponse = requests.get(\"https://image.freepik.com/free-photo/two-beautiful-puppies-cat-dog_58409-6024.jpg\")\nimg = Image.open(BytesIO(response.content))\n\ncenter_crop = transforms.Compose([\n transforms.Resize(256),\n transforms.CenterCrop(224),\n])\n\nnormalize = transforms.Compose([\n transforms.ToTensor(), # converts the image to a tensor with values between 0 and 1\n transforms.Normalize( # normalize to follow 0-centered imagenet pixel rgb distribution\n mean=[0.485, 0.456, 0.406],\n std=[0.229, 0.224, 0.225]\n )\n])\ninput_img = normalize(center_crop(img)).unsqueeze(0)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Computing Attribution\n---------------------\n\n\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Among the top-3 predictions of the models are classes 208 and 283 which\ncorrespond to dog and cat.\n\nLet us attribute each of these predictions to the corresponding part of\nthe input, using Captum\u2019s \\ ``Occlusion``\\ algorithm.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "from captum.attr import Occlusion \n\nocclusion = Occlusion(model)\n\nstrides = (3, 9, 9) # smaller = more fine-grained attribution but slower\ntarget=208, # Labrador index in ImageNet \nsliding_window_shapes=(3,45, 45) # choose size enough to change object appearance\nbaselines = 0 # values to occlude the image with. 0 corresponds to gray\n\nattribution_dog = occlusion.attribute(input_img,\n strides = strides,\n target=target,\n sliding_window_shapes=sliding_window_shapes,\n baselines=baselines)\n\n\ntarget=283, # Persian cat index in ImageNet \nattribution_cat = occlusion.attribute(input_img,\n strides = strides,\n target=target,\n sliding_window_shapes=sliding_window_shapes,\n baselines=0)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Besides ``Occlusion``, Captum features many algorithms such as\n\\ ``Integrated Gradients``\\ , \\ ``Deconvolution``\\ ,\n\\ ``GuidedBackprop``\\ , \\ ``Guided GradCam``\\ , \\ ``DeepLift``\\ , and\n\\ ``GradientShap``\\ . All of these algorithms are subclasses of\n``Attribution`` which expects your model as a callable ``forward_func``\nupon initialization and has an ``attribute(...)`` method which returns\nthe attribution result in a unified format.\n\nLet us visualize the computed attribution results in case of images.\n\n\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Visualizing the Results\n-----------------------\n\n\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Captum\u2019s \\ ``visualization``\\ utility provides out-of-the-box methods\nto visualize attribution results both for pictorial and for textual\ninputs.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "import numpy as np\nfrom captum.attr import visualization as viz\n\n# Convert the compute attribution tensor into an image-like numpy array\nattribution_dog = np.transpose(attribution_dog.squeeze().cpu().detach().numpy(), (1,2,0))\n\nvis_types = [\"heat_map\", \"original_image\"]\nvis_signs = [\"all\", \"all\"], # \"positive\", \"negative\", or \"all\" to show both\n# positive attribution indicates that the presence of the area increases the prediction score\n# negative attribution indicates distractor areas whose absence increases the score\n\n_ = viz.visualize_image_attr_multiple(attribution_dog,\n center_crop(img),\n vis_types,\n vis_signs,\n [\"attribution for dog\", \"image\"],\n show_colorbar = True\n )\n\n\nattribution_cat = np.transpose(attribution_cat.squeeze().cpu().detach().numpy(), (1,2,0))\n\n_ = viz.visualize_image_attr_multiple(attribution_cat,\n center_crop(img),\n [\"heat_map\", \"original_image\"], \n [\"all\", \"all\"], # positive/negative attribution or all\n [\"attribution for cat\", \"image\"],\n show_colorbar = True\n )" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "If your data is textual, ``visualization.visualize_text()`` offers a\ndedicated view to explore attribution on top of the input text. Find out\nmore at http://captum.ai/tutorials/IMDB_TorchText_Interpret\n\n\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Final Notes\n-----------\n\n\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Captum can handle most model types in PyTorch across modalities\nincluding vision, text, and more. With Captum you can: \\* Attribute a\nspecific output to the model input as illustrated above. \\* Attribute a\nspecific output to a hidden-layer neuron (see Captum API reference). \\*\nAttribute a hidden-layer neuron response to the model input (see Captum\nAPI reference).\n\nFor complete API of the supported methods and a list of tutorials,\nconsult our website http://captum.ai\n\nAnother useful post by Gilbert Tanner:\nhttps://gilberttanner.com/blog/interpreting-pytorch-models-with-captum\n\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.4" - } - }, - "nbformat": 4, - "nbformat_minor": 0 -} \ No newline at end of file diff --git a/recipes/recipes/Captum_Recipe.py b/recipes/recipes/Captum_Recipe.py deleted file mode 100644 index 105c8cd5f67..00000000000 --- a/recipes/recipes/Captum_Recipe.py +++ /dev/null @@ -1,189 +0,0 @@ -""" -Model Interpretability using Captum -=================================== - -""" - - -###################################################################### -# Captum helps you understand how the data features impact your model -# predictions or neuron activations, shedding light on how your model -# operates. -# -# Using Captum, you can apply a wide range of state-of-the-art feature -# attribution algorithms such as \ ``Guided GradCam``\ and -# \ ``Integrated Gradients``\ in a unified way. -# -# In this recipe you will learn how to use Captum to: \* attribute the -# predictions of an image classifier to their corresponding image -# features. \* visualize the attribution results. -# - - -###################################################################### -# Before you begin -# ---------------- -# - - -###################################################################### -# Make sure Captum is installed in your active Python environment. Captum -# is available both on GitHub, as a ``pip`` package, or as a ``conda`` -# package. For detailed instructions, consult the installation guide at -# https://captum.ai/ -# - - -###################################################################### -# For a model, we use a built-in image classifier in PyTorch. Captum can -# reveal which parts of a sample image support certain predictions made by -# the model. -# - -import torchvision -from torchvision import transforms -from PIL import Image -import requests -from io import BytesIO - -model = torchvision.models.resnet18(pretrained=True).eval() - -response = requests.get("https://image.freepik.com/free-photo/two-beautiful-puppies-cat-dog_58409-6024.jpg") -img = Image.open(BytesIO(response.content)) - -center_crop = transforms.Compose([ - transforms.Resize(256), - transforms.CenterCrop(224), -]) - -normalize = transforms.Compose([ - transforms.ToTensor(), # converts the image to a tensor with values between 0 and 1 - transforms.Normalize( # normalize to follow 0-centered imagenet pixel rgb distribution - mean=[0.485, 0.456, 0.406], - std=[0.229, 0.224, 0.225] - ) -]) -input_img = normalize(center_crop(img)).unsqueeze(0) - - -###################################################################### -# Computing Attribution -# --------------------- -# - - -###################################################################### -# Among the top-3 predictions of the models are classes 208 and 283 which -# correspond to dog and cat. -# -# Let us attribute each of these predictions to the corresponding part of -# the input, using Captum’s \ ``Occlusion``\ algorithm. -# - -from captum.attr import Occlusion - -occlusion = Occlusion(model) - -strides = (3, 9, 9) # smaller = more fine-grained attribution but slower -target=208, # Labrador index in ImageNet -sliding_window_shapes=(3,45, 45) # choose size enough to change object appearance -baselines = 0 # values to occlude the image with. 0 corresponds to gray - -attribution_dog = occlusion.attribute(input_img, - strides = strides, - target=target, - sliding_window_shapes=sliding_window_shapes, - baselines=baselines) - - -target=283, # Persian cat index in ImageNet -attribution_cat = occlusion.attribute(input_img, - strides = strides, - target=target, - sliding_window_shapes=sliding_window_shapes, - baselines=0) - - -###################################################################### -# Besides ``Occlusion``, Captum features many algorithms such as -# \ ``Integrated Gradients``\ , \ ``Deconvolution``\ , -# \ ``GuidedBackprop``\ , \ ``Guided GradCam``\ , \ ``DeepLift``\ , and -# \ ``GradientShap``\ . All of these algorithms are subclasses of -# ``Attribution`` which expects your model as a callable ``forward_func`` -# upon initialization and has an ``attribute(...)`` method which returns -# the attribution result in a unified format. -# -# Let us visualize the computed attribution results in case of images. -# - - -###################################################################### -# Visualizing the Results -# ----------------------- -# - - -###################################################################### -# Captum’s \ ``visualization``\ utility provides out-of-the-box methods -# to visualize attribution results both for pictorial and for textual -# inputs. -# - -import numpy as np -from captum.attr import visualization as viz - -# Convert the compute attribution tensor into an image-like numpy array -attribution_dog = np.transpose(attribution_dog.squeeze().cpu().detach().numpy(), (1,2,0)) - -vis_types = ["heat_map", "original_image"] -vis_signs = ["all", "all"], # "positive", "negative", or "all" to show both -# positive attribution indicates that the presence of the area increases the prediction score -# negative attribution indicates distractor areas whose absence increases the score - -_ = viz.visualize_image_attr_multiple(attribution_dog, - center_crop(img), - vis_types, - vis_signs, - ["attribution for dog", "image"], - show_colorbar = True - ) - - -attribution_cat = np.transpose(attribution_cat.squeeze().cpu().detach().numpy(), (1,2,0)) - -_ = viz.visualize_image_attr_multiple(attribution_cat, - center_crop(img), - ["heat_map", "original_image"], - ["all", "all"], # positive/negative attribution or all - ["attribution for cat", "image"], - show_colorbar = True - ) - - -###################################################################### -# If your data is textual, ``visualization.visualize_text()`` offers a -# dedicated view to explore attribution on top of the input text. Find out -# more at http://captum.ai/tutorials/IMDB_TorchText_Interpret -# - - -###################################################################### -# Final Notes -# ----------- -# - - -###################################################################### -# Captum can handle most model types in PyTorch across modalities -# including vision, text, and more. With Captum you can: \* Attribute a -# specific output to the model input as illustrated above. \* Attribute a -# specific output to a hidden-layer neuron (see Captum API reference). \* -# Attribute a hidden-layer neuron response to the model input (see Captum -# API reference). -# -# For complete API of the supported methods and a list of tutorials, -# consult our website http://captum.ai -# -# Another useful post by Gilbert Tanner: -# https://gilberttanner.com/blog/interpreting-pytorch-models-with-captum -# \ No newline at end of file diff --git a/recipes/recipes/Captum_Recipe.rst b/recipes/recipes/Captum_Recipe.rst deleted file mode 100644 index aff05ebd305..00000000000 --- a/recipes/recipes/Captum_Recipe.rst +++ /dev/null @@ -1,220 +0,0 @@ -.. note:: - :class: sphx-glr-download-link-note - - Click :ref:`here ` to download the full example code -.. rst-class:: sphx-glr-example-title - -.. _sphx_glr_recipes_recipes_Captum_Recipe.py: - - -Model Interpretability using Captum -=================================== -Captum helps you understand how the data features impact your model -predictions or neuron activations, shedding light on how your model -operates. - -Using Captum, you can apply a wide range of state-of-the-art feature -attribution algorithms such as \ ``Guided GradCam``\ and -\ ``Integrated Gradients``\ in a unified way. - -In this recipe you will learn how to use Captum to: \* attribute the -predictions of an image classifier to their corresponding image -features. \* visualize the attribution results. - - -Before you begin ----------------- - - -Make sure Captum is installed in your active Python environment. Captum -is available both on GitHub, as a ``pip`` package, or as a ``conda`` -package. For detailed instructions, consult the installation guide at -https://captum.ai/ - - -For a model, we use a built-in image classifier in PyTorch. Captum can -reveal which parts of a sample image support certain predictions made by -the model. - - - -.. code-block:: default - - - import torchvision - from torchvision import transforms - from PIL import Image - import requests - from io import BytesIO - - model = torchvision.models.resnet18(pretrained=True).eval() - - response = requests.get("https://image.freepik.com/free-photo/two-beautiful-puppies-cat-dog_58409-6024.jpg") - img = Image.open(BytesIO(response.content)) - - center_crop = transforms.Compose([ - transforms.Resize(256), - transforms.CenterCrop(224), - ]) - - normalize = transforms.Compose([ - transforms.ToTensor(), # converts the image to a tensor with values between 0 and 1 - transforms.Normalize( # normalize to follow 0-centered imagenet pixel rgb distribution - mean=[0.485, 0.456, 0.406], - std=[0.229, 0.224, 0.225] - ) - ]) - input_img = normalize(center_crop(img)).unsqueeze(0) - - - -Computing Attribution ---------------------- - - -Among the top-3 predictions of the models are classes 208 and 283 which -correspond to dog and cat. - -Let us attribute each of these predictions to the corresponding part of -the input, using Captum’s \ ``Occlusion``\ algorithm. - - - -.. code-block:: default - - - from captum.attr import Occlusion - - occlusion = Occlusion(model) - - strides = (3, 9, 9) # smaller = more fine-grained attribution but slower - target=208, # Labrador index in ImageNet - sliding_window_shapes=(3,45, 45) # choose size enough to change object appearance - baselines = 0 # values to occlude the image with. 0 corresponds to gray - - attribution_dog = occlusion.attribute(input_img, - strides = strides, - target=target, - sliding_window_shapes=sliding_window_shapes, - baselines=baselines) - - - target=283, # Persian cat index in ImageNet - attribution_cat = occlusion.attribute(input_img, - strides = strides, - target=target, - sliding_window_shapes=sliding_window_shapes, - baselines=0) - - - -Besides ``Occlusion``, Captum features many algorithms such as -\ ``Integrated Gradients``\ , \ ``Deconvolution``\ , -\ ``GuidedBackprop``\ , \ ``Guided GradCam``\ , \ ``DeepLift``\ , and -\ ``GradientShap``\ . All of these algorithms are subclasses of -``Attribution`` which expects your model as a callable ``forward_func`` -upon initialization and has an ``attribute(...)`` method which returns -the attribution result in a unified format. - -Let us visualize the computed attribution results in case of images. - - -Visualizing the Results ------------------------ - - -Captum’s \ ``visualization``\ utility provides out-of-the-box methods -to visualize attribution results both for pictorial and for textual -inputs. - - - -.. code-block:: default - - - import numpy as np - from captum.attr import visualization as viz - - # Convert the compute attribution tensor into an image-like numpy array - attribution_dog = np.transpose(attribution_dog.squeeze().cpu().detach().numpy(), (1,2,0)) - - vis_types = ["heat_map", "original_image"] - vis_signs = ["all", "all"], # "positive", "negative", or "all" to show both - # positive attribution indicates that the presence of the area increases the prediction score - # negative attribution indicates distractor areas whose absence increases the score - - _ = viz.visualize_image_attr_multiple(attribution_dog, - center_crop(img), - vis_types, - vis_signs, - ["attribution for dog", "image"], - show_colorbar = True - ) - - - attribution_cat = np.transpose(attribution_cat.squeeze().cpu().detach().numpy(), (1,2,0)) - - _ = viz.visualize_image_attr_multiple(attribution_cat, - center_crop(img), - ["heat_map", "original_image"], - ["all", "all"], # positive/negative attribution or all - ["attribution for cat", "image"], - show_colorbar = True - ) - - - -If your data is textual, ``visualization.visualize_text()`` offers a -dedicated view to explore attribution on top of the input text. Find out -more at http://captum.ai/tutorials/IMDB_TorchText_Interpret - - -Final Notes ------------ - - -Captum can handle most model types in PyTorch across modalities -including vision, text, and more. With Captum you can: \* Attribute a -specific output to the model input as illustrated above. \* Attribute a -specific output to a hidden-layer neuron (see Captum API reference). \* -Attribute a hidden-layer neuron response to the model input (see Captum -API reference). - -For complete API of the supported methods and a list of tutorials, -consult our website http://captum.ai - -Another useful post by Gilbert Tanner: -https://gilberttanner.com/blog/interpreting-pytorch-models-with-captum - - -.. rst-class:: sphx-glr-timing - - **Total running time of the script:** ( 0 minutes 0.000 seconds) - - -.. _sphx_glr_download_recipes_recipes_Captum_Recipe.py: - - -.. only :: html - - .. container:: sphx-glr-footer - :class: sphx-glr-footer-example - - - - .. container:: sphx-glr-download - - :download:`Download Python source code: Captum_Recipe.py ` - - - - .. container:: sphx-glr-download - - :download:`Download Jupyter notebook: Captum_Recipe.ipynb ` - - -.. only:: html - - .. rst-class:: sphx-glr-signature - - `Gallery generated by Sphinx-Gallery `_ diff --git a/recipes/recipes/custom_dataset_transforms_loader.ipynb b/recipes/recipes/custom_dataset_transforms_loader.ipynb deleted file mode 100644 index 207d56dc868..00000000000 --- a/recipes/recipes/custom_dataset_transforms_loader.ipynb +++ /dev/null @@ -1,261 +0,0 @@ -{ - "cells": [ - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "%matplotlib inline" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\nDeveloping Custom PyTorch Dataloaders\n=====================================\n\nA significant amount of the effort applied to developing machine\nlearning algorithms is related to data preparation. PyTorch provides\nmany tools to make data loading easy and hopefully, makes your code more\nreadable. In this recipe, you will learn how to:\n\n 1. Create a custom dataset leveraging the PyTorch dataset APIs;\n 2. Create callable custom transforms that can be composable; and\n 3. Put these components together to create a custom dataloader.\n\nPlease note, to run this tutorial, ensure the following packages are\ninstalled:\n - ``scikit-image``: For image io and transforms\n - ``pandas``: For easier csv parsing\n\nAs a point of attribution, this recipe is based on the original tutorial\nfrom `Sasank Chilamkurthy `__ and was later\nedited by `Joe Spisak `__.\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Setup\n----------------------\nFirst let\u2019s import all of the needed libraries for this recipe.\n\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "from __future__ import print_function, division\nimport os\nimport torch\nimport pandas as pd\nfrom skimage import io, transform\nimport numpy as np\nimport matplotlib.pyplot as plt\nfrom torch.utils.data import Dataset, DataLoader\nfrom torchvision import transforms, utils\n\n# Ignore warnings\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\nplt.ion() # interactive mode" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Part 1: The Dataset\n-------------------\n\n\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The dataset we are going to deal with is that of facial pose. Overall,\n68 different landmark points are annotated for each face.\n\nAs a next step, please download the dataset from\n`here `_ so that the\nimages are in a directory named \u2018data/faces/\u2019.\n\n**Note:** This dataset was actually generated by applying\n`dlib's pose estimation `_\non images from the imagenet dataset containing the \u2018face\u2019 tag.\n\n::\n\n !wget https://download.pytorch.org/tutorial/faces.zip\n !mkdir data/faces/\n import zipfile\n with zipfile.ZipFile(\"faces.zip\",\"r\") as zip_ref:\n zip_ref.extractall(\"/data/faces/\")\n %cd /data/faces/\n\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The dataset comes with a csv file with annotations which looks like\nthis:\n\n::\n\n image_name,part_0_x,part_0_y,part_1_x,part_1_y,part_2_x, ... ,part_67_x,part_67_y\n 0805personali01.jpg,27,83,27,98, ... 84,134\n 1084239450_e76e00b7e7.jpg,70,236,71,257, ... ,128,312\n\nLet\u2019s quickly read the CSV and get the annotations in an (N, 2) array\nwhere N is the number of landmarks.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "landmarks_frame = pd.read_csv('faces/face_landmarks.csv')\n\nn = 65\nimg_name = landmarks_frame.iloc[n, 0]\nlandmarks = landmarks_frame.iloc[n, 1:]\nlandmarks = np.asarray(landmarks)\nlandmarks = landmarks.astype('float').reshape(-1, 2)\n\nprint('Image name: {}'.format(img_name))\nprint('Landmarks shape: {}'.format(landmarks.shape))\nprint('First 4 Landmarks: {}'.format(landmarks[:4]))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "1.1 Write a simple helper function to show an image\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nNext let\u2019s write a simple helper function to show an image, its landmarks and use it to show a sample.\n\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "def show_landmarks(image, landmarks):\n \"\"\"Show image with landmarks\"\"\"\n plt.imshow(image)\n plt.scatter(landmarks[:, 0], landmarks[:, 1], s=10, marker='.', c='r')\n plt.pause(0.001) # pause a bit so that plots are updated\n\nplt.figure()\nshow_landmarks(io.imread(os.path.join('faces/', img_name)),\n landmarks)\nplt.show()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "1.2 Create a dataset class\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nNow lets talk about the PyTorch dataset class\n\n\n\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "``torch.utils.data.Dataset`` is an abstract class representing a\ndataset. Your custom dataset should inherit ``Dataset`` and override the\nfollowing methods:\n\n- ``__len__`` so that ``len(dataset)`` returns the size of the dataset.\n- ``__getitem__`` to support indexing such that ``dataset[i]`` can be\n used to get $`i$`\u00a0th sample\n\nLet\u2019s create a dataset class for our face landmarks dataset. We will\nread the csv in ``__init__`` but leave the reading of images to\n``__getitem__``. This is memory efficient because all the images are not\nstored in the memory at once but read as required.\n\nHere we show a sample of our dataset in the forma of a dict\n``{'image': image, 'landmarks': landmarks}``. Our dataset will take an\noptional argument ``transform`` so that any required processing can be\napplied on the sample. We will see the usefulness of ``transform`` in\nanother recipe.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "class FaceLandmarksDataset(Dataset):\n \"\"\"Face Landmarks dataset.\"\"\"\n\n def __init__(self, csv_file, root_dir, transform=None):\n \"\"\"\n Args:\n csv_file (string): Path to the csv file with annotations.\n root_dir (string): Directory with all the images.\n transform (callable, optional): Optional transform to be applied\n on a sample.\n \"\"\"\n self.landmarks_frame = pd.read_csv(csv_file)\n self.root_dir = root_dir\n self.transform = transform\n\n def __len__(self):\n return len(self.landmarks_frame)\n\n def __getitem__(self, idx):\n if torch.is_tensor(idx):\n idx = idx.tolist()\n\n img_name = os.path.join(self.root_dir,\n self.landmarks_frame.iloc[idx, 0])\n image = io.imread(img_name)\n landmarks = self.landmarks_frame.iloc[idx, 1:]\n landmarks = np.array([landmarks])\n landmarks = landmarks.astype('float').reshape(-1, 2)\n sample = {'image': image, 'landmarks': landmarks}\n\n if self.transform:\n sample = self.transform(sample)\n\n return sample" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "1.3 Iterate through data samples\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Next let\u2019s instantiate this class and iterate through the data samples.\nWe will print the sizes of first 4 samples and show their landmarks.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "face_dataset = FaceLandmarksDataset(csv_file='faces/face_landmarks.csv',\n root_dir='faces/')\n\nfig = plt.figure()\n\nfor i in range(len(face_dataset)):\n sample = face_dataset[i]\n\n print(i, sample['image'].shape, sample['landmarks'].shape)\n\n ax = plt.subplot(1, 4, i + 1)\n plt.tight_layout()\n ax.set_title('Sample #{}'.format(i))\n ax.axis('off')\n show_landmarks(**sample)\n\n if i == 3:\n plt.show()\n break" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Part 2: Data Tranformations\n---------------------------\n\n\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Now that we have a dataset to work with and have done some level of\ncustomization, we can move to creating custom transformations. In\ncomputer vision, these come in handy to help generalize algorithms and\nimprove accuracy. A suite of transformations used at training time is\ntypically referred to as data augmentation and is a common practice for\nmodern model development.\n\nOne issue common in handling datasets is that the samples may not all be\nthe same size. Most neural networks expect the images of a fixed size.\nTherefore, we will need to write some prepocessing code. Let\u2019s create\nthree transforms:\n\n- ``Rescale``: to scale the image\n- ``RandomCrop``: to crop from image randomly. This is data\n augmentation.\n- ``ToTensor``: to convert the numpy images to torch images (we need to\n swap axes).\n\nWe will write them as callable classes instead of simple functions so\nthat parameters of the transform need not be passed everytime it\u2019s\ncalled. For this, we just need to implement ``__call__`` method and if\nrequired, ``__init__`` method. We can then use a transform like this:\n\n::\n\n tsfm = Transform(params)\n transformed_sample = tsfm(sample)\n\nObserve below how these transforms had to be applied both on the image\nand landmarks.\n\n\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "2.1 Create callable classes\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nLet\u2019s start with creating callable classes for each transform\n\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "class Rescale(object):\n \"\"\"Rescale the image in a sample to a given size.\n\n Args:\n output_size (tuple or int): Desired output size. If tuple, output is\n matched to output_size. If int, smaller of image edges is matched\n to output_size keeping aspect ratio the same.\n \"\"\"\n\n def __init__(self, output_size):\n assert isinstance(output_size, (int, tuple))\n self.output_size = output_size\n\n def __call__(self, sample):\n image, landmarks = sample['image'], sample['landmarks']\n\n h, w = image.shape[:2]\n if isinstance(self.output_size, int):\n if h > w:\n new_h, new_w = self.output_size * h / w, self.output_size\n else:\n new_h, new_w = self.output_size, self.output_size * w / h\n else:\n new_h, new_w = self.output_size\n\n new_h, new_w = int(new_h), int(new_w)\n\n img = transform.resize(image, (new_h, new_w))\n\n # h and w are swapped for landmarks because for images,\n # x and y axes are axis 1 and 0 respectively\n landmarks = landmarks * [new_w / w, new_h / h]\n\n return {'image': img, 'landmarks': landmarks}\n\n\nclass RandomCrop(object):\n \"\"\"Crop randomly the image in a sample.\n\n Args:\n output_size (tuple or int): Desired output size. If int, square crop\n is made.\n \"\"\"\n\n def __init__(self, output_size):\n assert isinstance(output_size, (int, tuple))\n if isinstance(output_size, int):\n self.output_size = (output_size, output_size)\n else:\n assert len(output_size) == 2\n self.output_size = output_size\n\n def __call__(self, sample):\n image, landmarks = sample['image'], sample['landmarks']\n\n h, w = image.shape[:2]\n new_h, new_w = self.output_size\n\n top = np.random.randint(0, h - new_h)\n left = np.random.randint(0, w - new_w)\n\n image = image[top: top + new_h,\n left: left + new_w]\n\n landmarks = landmarks - [left, top]\n\n return {'image': image, 'landmarks': landmarks}\n\n\nclass ToTensor(object):\n \"\"\"Convert ndarrays in sample to Tensors.\"\"\"\n\n def __call__(self, sample):\n image, landmarks = sample['image'], sample['landmarks']\n\n # swap color axis because\n # numpy image: H x W x C\n # torch image: C X H X W\n image = image.transpose((2, 0, 1))\n return {'image': torch.from_numpy(image),\n 'landmarks': torch.from_numpy(landmarks)}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "2.2 Compose transforms and apply to a sample\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nNext let\u2019s compose these transforms and apply to a sample\n\n\nLet\u2019s say we want to rescale the shorter side of the image to 256 and\nthen randomly crop a square of size 224 from it. i.e, we want to compose\n``Rescale`` and ``RandomCrop`` transforms.\n``torchvision.transforms.Compose`` is a simple callable class which\nallows us to do this.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "scale = Rescale(256)\ncrop = RandomCrop(128)\ncomposed = transforms.Compose([Rescale(256),\n RandomCrop(224)])\n\n# Apply each of the above transforms on sample.\nfig = plt.figure()\nsample = face_dataset[65]\nfor i, tsfrm in enumerate([scale, crop, composed]):\n transformed_sample = tsfrm(sample)\n\n ax = plt.subplot(1, 3, i + 1)\n plt.tight_layout()\n ax.set_title(type(tsfrm).__name__)\n show_landmarks(**transformed_sample)\n\nplt.show()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "2.3 Iterate through the dataset\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nNext we will iterate through the dataset\n\n\nLet\u2019s put this all together to create a dataset with composed\ntransforms. To summarize, every time this dataset is sampled:\n\n- An image is read from the file on the fly\n- Transforms are applied on the read image\n- Since one of the transforms is random, data is augmentated on\n sampling\n\nWe can iterate over the created dataset with a ``for i in range`` loop\nas before.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "transformed_dataset = FaceLandmarksDataset(csv_file='faces/face_landmarks.csv',\n root_dir='faces/',\n transform=transforms.Compose([\n Rescale(256),\n RandomCrop(224),\n ToTensor()\n ]))\n\nfor i in range(len(transformed_dataset)):\n sample = transformed_dataset[i]\n\n print(i, sample['image'].size(), sample['landmarks'].size())\n\n if i == 3:\n break" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Part 3: The Dataloader\n----------------------\n\n\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "By operating on the dataset directly, we are losing out on a lot of\nfeatures by using a simple ``for`` loop to iterate over the data. In\nparticular, we are missing out on:\n\n- Batching the data\n- Shuffling the data\n- Load the data in parallel using ``multiprocessing`` workers.\n\n``torch.utils.data.DataLoader`` is an iterator which provides all these\nfeatures. Parameters used below should be clear. One parameter of\ninterest is ``collate_fn``. You can specify how exactly the samples need\nto be batched using ``collate_fn``. However, default collate should work\nfine for most use cases.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "dataloader = DataLoader(transformed_dataset, batch_size=4,\n shuffle=True, num_workers=4)\n\n\n# Helper function to show a batch\ndef show_landmarks_batch(sample_batched):\n \"\"\"Show image with landmarks for a batch of samples.\"\"\"\n images_batch, landmarks_batch = \\\n sample_batched['image'], sample_batched['landmarks']\n batch_size = len(images_batch)\n im_size = images_batch.size(2)\n\n grid = utils.make_grid(images_batch)\n plt.imshow(grid.numpy().transpose((1, 2, 0)))\n\n for i in range(batch_size):\n plt.scatter(landmarks_batch[i, :, 0].numpy() + i * im_size,\n landmarks_batch[i, :, 1].numpy(),\n s=10, marker='.', c='r')\n\n plt.title('Batch from dataloader')\n\nfor i_batch, sample_batched in enumerate(dataloader):\n print(i_batch, sample_batched['image'].size(),\n sample_batched['landmarks'].size())\n\n # observe 4th batch and stop.\n if i_batch == 3:\n plt.figure()\n show_landmarks_batch(sample_batched)\n plt.axis('off')\n plt.ioff()\n plt.show()\n break" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Now that you\u2019ve learned how to create a custom dataloader with PyTorch,\nwe recommend diving deeper into the docs and customizing your workflow\neven further. You can learn more in the ``torch.utils.data`` docs\n`here `__.\n\n\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.4" - } - }, - "nbformat": 4, - "nbformat_minor": 0 -} \ No newline at end of file diff --git a/recipes/recipes/custom_dataset_transforms_loader.py b/recipes/recipes/custom_dataset_transforms_loader.py deleted file mode 100644 index 077dcee8001..00000000000 --- a/recipes/recipes/custom_dataset_transforms_loader.py +++ /dev/null @@ -1,480 +0,0 @@ -""" -Developing Custom PyTorch Dataloaders -===================================== - -A significant amount of the effort applied to developing machine -learning algorithms is related to data preparation. PyTorch provides -many tools to make data loading easy and hopefully, makes your code more -readable. In this recipe, you will learn how to: - - 1. Create a custom dataset leveraging the PyTorch dataset APIs; - 2. Create callable custom transforms that can be composable; and - 3. Put these components together to create a custom dataloader. - -Please note, to run this tutorial, ensure the following packages are -installed: - - ``scikit-image``: For image io and transforms - - ``pandas``: For easier csv parsing - -As a point of attribution, this recipe is based on the original tutorial -from `Sasank Chilamkurthy `__ and was later -edited by `Joe Spisak `__. -""" - - -###################################################################### -# Setup -# ---------------------- -# First let’s import all of the needed libraries for this recipe. -# -# - -from __future__ import print_function, division -import os -import torch -import pandas as pd -from skimage import io, transform -import numpy as np -import matplotlib.pyplot as plt -from torch.utils.data import Dataset, DataLoader -from torchvision import transforms, utils - -# Ignore warnings -import warnings -warnings.filterwarnings("ignore") - -plt.ion() # interactive mode - - -###################################################################### -# Part 1: The Dataset -# ------------------- -# - -###################################################################### -# The dataset we are going to deal with is that of facial pose. Overall, -# 68 different landmark points are annotated for each face. -# -# As a next step, please download the dataset from -# `here `_ so that the -# images are in a directory named ‘data/faces/’. -# -# **Note:** This dataset was actually generated by applying -# `dlib's pose estimation `_ -# on images from the imagenet dataset containing the ‘face’ tag. -# -# :: -# -# !wget https://download.pytorch.org/tutorial/faces.zip -# !mkdir data/faces/ -# import zipfile -# with zipfile.ZipFile("faces.zip","r") as zip_ref: -# zip_ref.extractall("/data/faces/") -# %cd /data/faces/ - - - -###################################################################### -# The dataset comes with a csv file with annotations which looks like -# this: -# -# :: -# -# image_name,part_0_x,part_0_y,part_1_x,part_1_y,part_2_x, ... ,part_67_x,part_67_y -# 0805personali01.jpg,27,83,27,98, ... 84,134 -# 1084239450_e76e00b7e7.jpg,70,236,71,257, ... ,128,312 -# -# Let’s quickly read the CSV and get the annotations in an (N, 2) array -# where N is the number of landmarks. -# - - -landmarks_frame = pd.read_csv('faces/face_landmarks.csv') - -n = 65 -img_name = landmarks_frame.iloc[n, 0] -landmarks = landmarks_frame.iloc[n, 1:] -landmarks = np.asarray(landmarks) -landmarks = landmarks.astype('float').reshape(-1, 2) - -print('Image name: {}'.format(img_name)) -print('Landmarks shape: {}'.format(landmarks.shape)) -print('First 4 Landmarks: {}'.format(landmarks[:4])) - -###################################################################### -# 1.1 Write a simple helper function to show an image -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# Next let’s write a simple helper function to show an image, its landmarks and use it to show a sample. -# -# - -def show_landmarks(image, landmarks): - """Show image with landmarks""" - plt.imshow(image) - plt.scatter(landmarks[:, 0], landmarks[:, 1], s=10, marker='.', c='r') - plt.pause(0.001) # pause a bit so that plots are updated - -plt.figure() -show_landmarks(io.imread(os.path.join('faces/', img_name)), - landmarks) -plt.show() - - -###################################################################### -# 1.2 Create a dataset class -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# Now lets talk about the PyTorch dataset class -# -# - - -###################################################################### -# ``torch.utils.data.Dataset`` is an abstract class representing a -# dataset. Your custom dataset should inherit ``Dataset`` and override the -# following methods: -# -# - ``__len__`` so that ``len(dataset)`` returns the size of the dataset. -# - ``__getitem__`` to support indexing such that ``dataset[i]`` can be -# used to get :math:``i`` th sample -# -# Let’s create a dataset class for our face landmarks dataset. We will -# read the csv in ``__init__`` but leave the reading of images to -# ``__getitem__``. This is memory efficient because all the images are not -# stored in the memory at once but read as required. -# -# Here we show a sample of our dataset in the forma of a dict -# ``{'image': image, 'landmarks': landmarks}``. Our dataset will take an -# optional argument ``transform`` so that any required processing can be -# applied on the sample. We will see the usefulness of ``transform`` in -# another recipe. -# - -class FaceLandmarksDataset(Dataset): - """Face Landmarks dataset.""" - - def __init__(self, csv_file, root_dir, transform=None): - """ - Args: - csv_file (string): Path to the csv file with annotations. - root_dir (string): Directory with all the images. - transform (callable, optional): Optional transform to be applied - on a sample. - """ - self.landmarks_frame = pd.read_csv(csv_file) - self.root_dir = root_dir - self.transform = transform - - def __len__(self): - return len(self.landmarks_frame) - - def __getitem__(self, idx): - if torch.is_tensor(idx): - idx = idx.tolist() - - img_name = os.path.join(self.root_dir, - self.landmarks_frame.iloc[idx, 0]) - image = io.imread(img_name) - landmarks = self.landmarks_frame.iloc[idx, 1:] - landmarks = np.array([landmarks]) - landmarks = landmarks.astype('float').reshape(-1, 2) - sample = {'image': image, 'landmarks': landmarks} - - if self.transform: - sample = self.transform(sample) - - return sample - - -###################################################################### -# 1.3 Iterate through data samples -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# - - -###################################################################### -# Next let’s instantiate this class and iterate through the data samples. -# We will print the sizes of first 4 samples and show their landmarks. -# - -face_dataset = FaceLandmarksDataset(csv_file='faces/face_landmarks.csv', - root_dir='faces/') - -fig = plt.figure() - -for i in range(len(face_dataset)): - sample = face_dataset[i] - - print(i, sample['image'].shape, sample['landmarks'].shape) - - ax = plt.subplot(1, 4, i + 1) - plt.tight_layout() - ax.set_title('Sample #{}'.format(i)) - ax.axis('off') - show_landmarks(**sample) - - if i == 3: - plt.show() - break - - -###################################################################### -# Part 2: Data Tranformations -# --------------------------- -# - - -###################################################################### -# Now that we have a dataset to work with and have done some level of -# customization, we can move to creating custom transformations. In -# computer vision, these come in handy to help generalize algorithms and -# improve accuracy. A suite of transformations used at training time is -# typically referred to as data augmentation and is a common practice for -# modern model development. -# -# One issue common in handling datasets is that the samples may not all be -# the same size. Most neural networks expect the images of a fixed size. -# Therefore, we will need to write some prepocessing code. Let’s create -# three transforms: -# -# - ``Rescale``: to scale the image -# - ``RandomCrop``: to crop from image randomly. This is data -# augmentation. -# - ``ToTensor``: to convert the numpy images to torch images (we need to -# swap axes). -# -# We will write them as callable classes instead of simple functions so -# that parameters of the transform need not be passed everytime it’s -# called. For this, we just need to implement ``__call__`` method and if -# required, ``__init__`` method. We can then use a transform like this: -# -# :: -# -# tsfm = Transform(params) -# transformed_sample = tsfm(sample) -# -# Observe below how these transforms had to be applied both on the image -# and landmarks. -# - - -###################################################################### -# 2.1 Create callable classes -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# Let’s start with creating callable classes for each transform -# -# - -class Rescale(object): - """Rescale the image in a sample to a given size. - - Args: - output_size (tuple or int): Desired output size. If tuple, output is - matched to output_size. If int, smaller of image edges is matched - to output_size keeping aspect ratio the same. - """ - - def __init__(self, output_size): - assert isinstance(output_size, (int, tuple)) - self.output_size = output_size - - def __call__(self, sample): - image, landmarks = sample['image'], sample['landmarks'] - - h, w = image.shape[:2] - if isinstance(self.output_size, int): - if h > w: - new_h, new_w = self.output_size * h / w, self.output_size - else: - new_h, new_w = self.output_size, self.output_size * w / h - else: - new_h, new_w = self.output_size - - new_h, new_w = int(new_h), int(new_w) - - img = transform.resize(image, (new_h, new_w)) - - # h and w are swapped for landmarks because for images, - # x and y axes are axis 1 and 0 respectively - landmarks = landmarks * [new_w / w, new_h / h] - - return {'image': img, 'landmarks': landmarks} - - -class RandomCrop(object): - """Crop randomly the image in a sample. - - Args: - output_size (tuple or int): Desired output size. If int, square crop - is made. - """ - - def __init__(self, output_size): - assert isinstance(output_size, (int, tuple)) - if isinstance(output_size, int): - self.output_size = (output_size, output_size) - else: - assert len(output_size) == 2 - self.output_size = output_size - - def __call__(self, sample): - image, landmarks = sample['image'], sample['landmarks'] - - h, w = image.shape[:2] - new_h, new_w = self.output_size - - top = np.random.randint(0, h - new_h) - left = np.random.randint(0, w - new_w) - - image = image[top: top + new_h, - left: left + new_w] - - landmarks = landmarks - [left, top] - - return {'image': image, 'landmarks': landmarks} - - -class ToTensor(object): - """Convert ndarrays in sample to Tensors.""" - - def __call__(self, sample): - image, landmarks = sample['image'], sample['landmarks'] - - # swap color axis because - # numpy image: H x W x C - # torch image: C X H X W - image = image.transpose((2, 0, 1)) - return {'image': torch.from_numpy(image), - 'landmarks': torch.from_numpy(landmarks)} - - -###################################################################### -# 2.2 Compose transforms and apply to a sample -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# Next let’s compose these transforms and apply to a sample -# -# -# Let’s say we want to rescale the shorter side of the image to 256 and -# then randomly crop a square of size 224 from it. i.e, we want to compose -# ``Rescale`` and ``RandomCrop`` transforms. -# ``torchvision.transforms.Compose`` is a simple callable class which -# allows us to do this. -# - -scale = Rescale(256) -crop = RandomCrop(128) -composed = transforms.Compose([Rescale(256), - RandomCrop(224)]) - -# Apply each of the above transforms on sample. -fig = plt.figure() -sample = face_dataset[65] -for i, tsfrm in enumerate([scale, crop, composed]): - transformed_sample = tsfrm(sample) - - ax = plt.subplot(1, 3, i + 1) - plt.tight_layout() - ax.set_title(type(tsfrm).__name__) - show_landmarks(**transformed_sample) - -plt.show() - - -###################################################################### -# 2.3 Iterate through the dataset -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# Next we will iterate through the dataset -# -# -# Let’s put this all together to create a dataset with composed -# transforms. To summarize, every time this dataset is sampled: -# -# - An image is read from the file on the fly -# - Transforms are applied on the read image -# - Since one of the transforms is random, data is augmentated on -# sampling -# -# We can iterate over the created dataset with a ``for i in range`` loop -# as before. -# - -transformed_dataset = FaceLandmarksDataset(csv_file='faces/face_landmarks.csv', - root_dir='faces/', - transform=transforms.Compose([ - Rescale(256), - RandomCrop(224), - ToTensor() - ])) - -for i in range(len(transformed_dataset)): - sample = transformed_dataset[i] - - print(i, sample['image'].size(), sample['landmarks'].size()) - - if i == 3: - break - - -###################################################################### -# Part 3: The Dataloader -# ---------------------- -# - - -###################################################################### -# By operating on the dataset directly, we are losing out on a lot of -# features by using a simple ``for`` loop to iterate over the data. In -# particular, we are missing out on: -# -# - Batching the data -# - Shuffling the data -# - Load the data in parallel using ``multiprocessing`` workers. -# -# ``torch.utils.data.DataLoader`` is an iterator which provides all these -# features. Parameters used below should be clear. One parameter of -# interest is ``collate_fn``. You can specify how exactly the samples need -# to be batched using ``collate_fn``. However, default collate should work -# fine for most use cases. -# - -dataloader = DataLoader(transformed_dataset, batch_size=4, - shuffle=True, num_workers=4) - - -# Helper function to show a batch -def show_landmarks_batch(sample_batched): - """Show image with landmarks for a batch of samples.""" - images_batch, landmarks_batch = \ - sample_batched['image'], sample_batched['landmarks'] - batch_size = len(images_batch) - im_size = images_batch.size(2) - - grid = utils.make_grid(images_batch) - plt.imshow(grid.numpy().transpose((1, 2, 0))) - - for i in range(batch_size): - plt.scatter(landmarks_batch[i, :, 0].numpy() + i * im_size, - landmarks_batch[i, :, 1].numpy(), - s=10, marker='.', c='r') - - plt.title('Batch from dataloader') - -for i_batch, sample_batched in enumerate(dataloader): - print(i_batch, sample_batched['image'].size(), - sample_batched['landmarks'].size()) - - # observe 4th batch and stop. - if i_batch == 3: - plt.figure() - show_landmarks_batch(sample_batched) - plt.axis('off') - plt.ioff() - plt.show() - break - - -###################################################################### -# Now that you’ve learned how to create a custom dataloader with PyTorch, -# we recommend diving deeper into the docs and customizing your workflow -# even further. You can learn more in the ``torch.utils.data`` docs -# `here `__. -# diff --git a/recipes/recipes/custom_dataset_transforms_loader.rst b/recipes/recipes/custom_dataset_transforms_loader.rst deleted file mode 100644 index 5122e4ad931..00000000000 --- a/recipes/recipes/custom_dataset_transforms_loader.rst +++ /dev/null @@ -1,539 +0,0 @@ -.. note:: - :class: sphx-glr-download-link-note - - Click :ref:`here ` to download the full example code -.. rst-class:: sphx-glr-example-title - -.. _sphx_glr_recipes_recipes_custom_dataset_transforms_loader.py: - - -Developing Custom PyTorch Dataloaders -===================================== - -A significant amount of the effort applied to developing machine -learning algorithms is related to data preparation. PyTorch provides -many tools to make data loading easy and hopefully, makes your code more -readable. In this recipe, you will learn how to: - - 1. Create a custom dataset leveraging the PyTorch dataset APIs; - 2. Create callable custom transforms that can be composable; and - 3. Put these components together to create a custom dataloader. - -Please note, to run this tutorial, ensure the following packages are -installed: - - ``scikit-image``: For image io and transforms - - ``pandas``: For easier csv parsing - -As a point of attribution, this recipe is based on the original tutorial -from `Sasank Chilamkurthy `__ and was later -edited by `Joe Spisak `__. -Setup ----------------------- -First let’s import all of the needed libraries for this recipe. - - - - -.. code-block:: default - - - from __future__ import print_function, division - import os - import torch - import pandas as pd - from skimage import io, transform - import numpy as np - import matplotlib.pyplot as plt - from torch.utils.data import Dataset, DataLoader - from torchvision import transforms, utils - - # Ignore warnings - import warnings - warnings.filterwarnings("ignore") - - plt.ion() # interactive mode - - - -Part 1: The Dataset -------------------- - - -The dataset we are going to deal with is that of facial pose. Overall, -68 different landmark points are annotated for each face. - -As a next step, please download the dataset from -`here `_ so that the -images are in a directory named ‘data/faces/’. - -**Note:** This dataset was actually generated by applying -`dlib's pose estimation `_ -on images from the imagenet dataset containing the ‘face’ tag. - -:: - - !wget https://download.pytorch.org/tutorial/faces.zip - !mkdir data/faces/ - import zipfile - with zipfile.ZipFile("faces.zip","r") as zip_ref: - zip_ref.extractall("/data/faces/") - %cd /data/faces/ - -The dataset comes with a csv file with annotations which looks like -this: - -:: - - image_name,part_0_x,part_0_y,part_1_x,part_1_y,part_2_x, ... ,part_67_x,part_67_y - 0805personali01.jpg,27,83,27,98, ... 84,134 - 1084239450_e76e00b7e7.jpg,70,236,71,257, ... ,128,312 - -Let’s quickly read the CSV and get the annotations in an (N, 2) array -where N is the number of landmarks. - - - -.. code-block:: default - - - - landmarks_frame = pd.read_csv('faces/face_landmarks.csv') - - n = 65 - img_name = landmarks_frame.iloc[n, 0] - landmarks = landmarks_frame.iloc[n, 1:] - landmarks = np.asarray(landmarks) - landmarks = landmarks.astype('float').reshape(-1, 2) - - print('Image name: {}'.format(img_name)) - print('Landmarks shape: {}'.format(landmarks.shape)) - print('First 4 Landmarks: {}'.format(landmarks[:4])) - - -1.1 Write a simple helper function to show an image -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Next let’s write a simple helper function to show an image, its landmarks and use it to show a sample. - - - - -.. code-block:: default - - - def show_landmarks(image, landmarks): - """Show image with landmarks""" - plt.imshow(image) - plt.scatter(landmarks[:, 0], landmarks[:, 1], s=10, marker='.', c='r') - plt.pause(0.001) # pause a bit so that plots are updated - - plt.figure() - show_landmarks(io.imread(os.path.join('faces/', img_name)), - landmarks) - plt.show() - - - -1.2 Create a dataset class -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Now lets talk about the PyTorch dataset class - - - -``torch.utils.data.Dataset`` is an abstract class representing a -dataset. Your custom dataset should inherit ``Dataset`` and override the -following methods: - -- ``__len__`` so that ``len(dataset)`` returns the size of the dataset. -- ``__getitem__`` to support indexing such that ``dataset[i]`` can be - used to get :math:``i`` th sample - -Let’s create a dataset class for our face landmarks dataset. We will -read the csv in ``__init__`` but leave the reading of images to -``__getitem__``. This is memory efficient because all the images are not -stored in the memory at once but read as required. - -Here we show a sample of our dataset in the forma of a dict -``{'image': image, 'landmarks': landmarks}``. Our dataset will take an -optional argument ``transform`` so that any required processing can be -applied on the sample. We will see the usefulness of ``transform`` in -another recipe. - - - -.. code-block:: default - - - class FaceLandmarksDataset(Dataset): - """Face Landmarks dataset.""" - - def __init__(self, csv_file, root_dir, transform=None): - """ - Args: - csv_file (string): Path to the csv file with annotations. - root_dir (string): Directory with all the images. - transform (callable, optional): Optional transform to be applied - on a sample. - """ - self.landmarks_frame = pd.read_csv(csv_file) - self.root_dir = root_dir - self.transform = transform - - def __len__(self): - return len(self.landmarks_frame) - - def __getitem__(self, idx): - if torch.is_tensor(idx): - idx = idx.tolist() - - img_name = os.path.join(self.root_dir, - self.landmarks_frame.iloc[idx, 0]) - image = io.imread(img_name) - landmarks = self.landmarks_frame.iloc[idx, 1:] - landmarks = np.array([landmarks]) - landmarks = landmarks.astype('float').reshape(-1, 2) - sample = {'image': image, 'landmarks': landmarks} - - if self.transform: - sample = self.transform(sample) - - return sample - - - -1.3 Iterate through data samples -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - -Next let’s instantiate this class and iterate through the data samples. -We will print the sizes of first 4 samples and show their landmarks. - - - -.. code-block:: default - - - face_dataset = FaceLandmarksDataset(csv_file='faces/face_landmarks.csv', - root_dir='faces/') - - fig = plt.figure() - - for i in range(len(face_dataset)): - sample = face_dataset[i] - - print(i, sample['image'].shape, sample['landmarks'].shape) - - ax = plt.subplot(1, 4, i + 1) - plt.tight_layout() - ax.set_title('Sample #{}'.format(i)) - ax.axis('off') - show_landmarks(**sample) - - if i == 3: - plt.show() - break - - - -Part 2: Data Tranformations ---------------------------- - - -Now that we have a dataset to work with and have done some level of -customization, we can move to creating custom transformations. In -computer vision, these come in handy to help generalize algorithms and -improve accuracy. A suite of transformations used at training time is -typically referred to as data augmentation and is a common practice for -modern model development. - -One issue common in handling datasets is that the samples may not all be -the same size. Most neural networks expect the images of a fixed size. -Therefore, we will need to write some prepocessing code. Let’s create -three transforms: - -- ``Rescale``: to scale the image -- ``RandomCrop``: to crop from image randomly. This is data - augmentation. -- ``ToTensor``: to convert the numpy images to torch images (we need to - swap axes). - -We will write them as callable classes instead of simple functions so -that parameters of the transform need not be passed everytime it’s -called. For this, we just need to implement ``__call__`` method and if -required, ``__init__`` method. We can then use a transform like this: - -:: - - tsfm = Transform(params) - transformed_sample = tsfm(sample) - -Observe below how these transforms had to be applied both on the image -and landmarks. - - -2.1 Create callable classes -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Let’s start with creating callable classes for each transform - - - - -.. code-block:: default - - - class Rescale(object): - """Rescale the image in a sample to a given size. - - Args: - output_size (tuple or int): Desired output size. If tuple, output is - matched to output_size. If int, smaller of image edges is matched - to output_size keeping aspect ratio the same. - """ - - def __init__(self, output_size): - assert isinstance(output_size, (int, tuple)) - self.output_size = output_size - - def __call__(self, sample): - image, landmarks = sample['image'], sample['landmarks'] - - h, w = image.shape[:2] - if isinstance(self.output_size, int): - if h > w: - new_h, new_w = self.output_size * h / w, self.output_size - else: - new_h, new_w = self.output_size, self.output_size * w / h - else: - new_h, new_w = self.output_size - - new_h, new_w = int(new_h), int(new_w) - - img = transform.resize(image, (new_h, new_w)) - - # h and w are swapped for landmarks because for images, - # x and y axes are axis 1 and 0 respectively - landmarks = landmarks * [new_w / w, new_h / h] - - return {'image': img, 'landmarks': landmarks} - - - class RandomCrop(object): - """Crop randomly the image in a sample. - - Args: - output_size (tuple or int): Desired output size. If int, square crop - is made. - """ - - def __init__(self, output_size): - assert isinstance(output_size, (int, tuple)) - if isinstance(output_size, int): - self.output_size = (output_size, output_size) - else: - assert len(output_size) == 2 - self.output_size = output_size - - def __call__(self, sample): - image, landmarks = sample['image'], sample['landmarks'] - - h, w = image.shape[:2] - new_h, new_w = self.output_size - - top = np.random.randint(0, h - new_h) - left = np.random.randint(0, w - new_w) - - image = image[top: top + new_h, - left: left + new_w] - - landmarks = landmarks - [left, top] - - return {'image': image, 'landmarks': landmarks} - - - class ToTensor(object): - """Convert ndarrays in sample to Tensors.""" - - def __call__(self, sample): - image, landmarks = sample['image'], sample['landmarks'] - - # swap color axis because - # numpy image: H x W x C - # torch image: C X H X W - image = image.transpose((2, 0, 1)) - return {'image': torch.from_numpy(image), - 'landmarks': torch.from_numpy(landmarks)} - - - -2.2 Compose transforms and apply to a sample -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Next let’s compose these transforms and apply to a sample - - -Let’s say we want to rescale the shorter side of the image to 256 and -then randomly crop a square of size 224 from it. i.e, we want to compose -``Rescale`` and ``RandomCrop`` transforms. -``torchvision.transforms.Compose`` is a simple callable class which -allows us to do this. - - - -.. code-block:: default - - - scale = Rescale(256) - crop = RandomCrop(128) - composed = transforms.Compose([Rescale(256), - RandomCrop(224)]) - - # Apply each of the above transforms on sample. - fig = plt.figure() - sample = face_dataset[65] - for i, tsfrm in enumerate([scale, crop, composed]): - transformed_sample = tsfrm(sample) - - ax = plt.subplot(1, 3, i + 1) - plt.tight_layout() - ax.set_title(type(tsfrm).__name__) - show_landmarks(**transformed_sample) - - plt.show() - - - -2.3 Iterate through the dataset -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Next we will iterate through the dataset - - -Let’s put this all together to create a dataset with composed -transforms. To summarize, every time this dataset is sampled: - -- An image is read from the file on the fly -- Transforms are applied on the read image -- Since one of the transforms is random, data is augmentated on - sampling - -We can iterate over the created dataset with a ``for i in range`` loop -as before. - - - -.. code-block:: default - - - transformed_dataset = FaceLandmarksDataset(csv_file='faces/face_landmarks.csv', - root_dir='faces/', - transform=transforms.Compose([ - Rescale(256), - RandomCrop(224), - ToTensor() - ])) - - for i in range(len(transformed_dataset)): - sample = transformed_dataset[i] - - print(i, sample['image'].size(), sample['landmarks'].size()) - - if i == 3: - break - - - -Part 3: The Dataloader ----------------------- - - -By operating on the dataset directly, we are losing out on a lot of -features by using a simple ``for`` loop to iterate over the data. In -particular, we are missing out on: - -- Batching the data -- Shuffling the data -- Load the data in parallel using ``multiprocessing`` workers. - -``torch.utils.data.DataLoader`` is an iterator which provides all these -features. Parameters used below should be clear. One parameter of -interest is ``collate_fn``. You can specify how exactly the samples need -to be batched using ``collate_fn``. However, default collate should work -fine for most use cases. - - - -.. code-block:: default - - - dataloader = DataLoader(transformed_dataset, batch_size=4, - shuffle=True, num_workers=4) - - - # Helper function to show a batch - def show_landmarks_batch(sample_batched): - """Show image with landmarks for a batch of samples.""" - images_batch, landmarks_batch = \ - sample_batched['image'], sample_batched['landmarks'] - batch_size = len(images_batch) - im_size = images_batch.size(2) - - grid = utils.make_grid(images_batch) - plt.imshow(grid.numpy().transpose((1, 2, 0))) - - for i in range(batch_size): - plt.scatter(landmarks_batch[i, :, 0].numpy() + i * im_size, - landmarks_batch[i, :, 1].numpy(), - s=10, marker='.', c='r') - - plt.title('Batch from dataloader') - - for i_batch, sample_batched in enumerate(dataloader): - print(i_batch, sample_batched['image'].size(), - sample_batched['landmarks'].size()) - - # observe 4th batch and stop. - if i_batch == 3: - plt.figure() - show_landmarks_batch(sample_batched) - plt.axis('off') - plt.ioff() - plt.show() - break - - - -Now that you’ve learned how to create a custom dataloader with PyTorch, -we recommend diving deeper into the docs and customizing your workflow -even further. You can learn more in the ``torch.utils.data`` docs -`here `__. - - - -.. rst-class:: sphx-glr-timing - - **Total running time of the script:** ( 0 minutes 0.000 seconds) - - -.. _sphx_glr_download_recipes_recipes_custom_dataset_transforms_loader.py: - - -.. only :: html - - .. container:: sphx-glr-footer - :class: sphx-glr-footer-example - - - - .. container:: sphx-glr-download - - :download:`Download Python source code: custom_dataset_transforms_loader.py ` - - - - .. container:: sphx-glr-download - - :download:`Download Jupyter notebook: custom_dataset_transforms_loader.ipynb ` - - -.. only:: html - - .. rst-class:: sphx-glr-signature - - `Gallery generated by Sphinx-Gallery `_ diff --git a/recipes/recipes/defining_a_neural_network.ipynb b/recipes/recipes/defining_a_neural_network.ipynb deleted file mode 100644 index 7709cfaf1c5..00000000000 --- a/recipes/recipes/defining_a_neural_network.ipynb +++ /dev/null @@ -1,122 +0,0 @@ -{ - "cells": [ - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "%matplotlib inline" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\nDefining a Neural Network in PyTorch\n====================================\nDeep learning uses artificial neural networks (models), which are\ncomputing systems that are composed of many layers of interconnected\nunits. By passing data through these interconnected units, a neural\nnetwork is able to learn how to approximate the computations required to\ntransform inputs into outputs. In PyTorch, neural networks can be\nconstructed using the ``torch.nn`` package.\n\nIntroduction\n------------\nPyTorch provides the elegantly designed modules and classes, including\n``torch.nn``, to help you create and train neural networks. An\n``nn.Module`` contains layers, and a method ``forward(input)`` that\nreturns the ``output``.\n\nIn this recipe, we will use ``torch.nn`` to define a neural network\nintended for the `MNIST\ndataset `__.\n\nSetup\n-----\nBefore we begin, we need to install ``torch`` if it isn\u2019t already\navailable.\n\n::\n\n pip install torchaudio\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Steps\n-----\n\n1. Import all necessary libraries for loading our data\n2. Define and intialize the neural network\n3. Specify how data will pass through your model\n4. [Optional] Pass data through your model to test\n\n1. Import necessary libraries for loading our data\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nFor this recipe, we will use ``torch`` and its subsidiaries ``torch.nn``\nand ``torch.nn.functional``.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "import torch\nimport torch.nn as nn\nimport torch.nn.functional as F" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "2. Define and intialize the neural network\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nOur network will recognize images. We will use a process built into\nPyTorch called convolution. Convolution adds each element of an image to\nits local neighbors, weighted by a kernel, or a small martrix, that\nhelps us extract certain features (like edge detection, sharpness,\nblurriness, etc.) from the input image.\n\nThere are two requirements for defining the ``Net`` class of your model.\nThe first is writing an ``__init__`` function that references\n``nn.Module``. This function is where you define the fully connected\nlayers in your neural network.\n\nUsing convolution, we will define our model to take 1 input image\nchannel, and output match our target of 10 labels representing numbers 0\nthrough 9. This algorithm is yours to create, we will follow a standard\nMNIST algorithm.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "class Net(nn.Module):\n def __init__(self):\n super(Net, self).__init__()\n\n # First 2D convolutional layer, taking in 1 input channel (image),\n # outputting 32 convolutional features, with a square kernel size of 3\n self.conv1 = nn.Conv2d(1, 32, 3, 1)\n # Second 2D convolutional layer, taking in the 32 input layers,\n # outputting 64 convolutional features, with a square kernel size of 3\n self.conv2 = nn.Conv2d(32, 64, 3, 1)\n\n # Designed to ensure that adjacent pixels are either all 0s or all active\n # with an input probability\n self.dropout1 = nn.Dropout2d(0.25)\n self.dropout2 = nn.Dropout2d(0.5)\n\n # First fully connected layer\n self.fc1 = nn.Linear(9216, 128)\n # Second fully connected layer that outputs our 10 labels\n self.fc2 = nn.Linear(128, 10)\n\nmy_nn = Net()\nprint(my_nn)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We have finished defining our neural network, now we have to define how\nour data will pass through it.\n\n3. Specify how data will pass through your model\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nWhen you use PyTorch to build a model, you just have to define the\n``forward`` function, that will pass the data into the computation graph\n(i.e. our neural network). This will represent our feed-forward\nalgorithm.\n\nYou can use any of the Tensor operations in the ``forward`` function.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "class Net(nn.Module):\n def __init__(self):\n super(Net, self).__init__()\n self.conv1 = nn.Conv2d(1, 32, 3, 1)\n self.conv2 = nn.Conv2d(32, 64, 3, 1)\n self.dropout1 = nn.Dropout2d(0.25)\n self.dropout2 = nn.Dropout2d(0.5)\n self.fc1 = nn.Linear(9216, 128)\n self.fc2 = nn.Linear(128, 10)\n\n # x represents our data\n def forward(self, x):\n # Pass data through conv1\n x = self.conv1(x)\n # Use the rectified-linear activation function over x\n x = F.relu(x)\n\n x = self.conv2(x)\n x = F.relu(x)\n\n # Run max pooling over x\n x = F.max_pool2d(x, 2)\n # Pass data through dropout1\n x = self.dropout1(x)\n # Flatten x with start_dim=1\n x = torch.flatten(x, 1)\n # Pass data through fc1\n x = self.fc1(x)\n x = F.relu(x)\n x = self.dropout2(x)\n x = self.fc2(x)\n\n # Apply softmax to x \n output = F.log_softmax(x, dim=1)\n return output" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "4. [Optional] Pass data through your model to test\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nTo ensure we receive our desired output, let\u2019s test our model by passing\nsome random data through it.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "# Equates to one random 28x28 image\nrandom_data = torch.rand((1, 1, 28, 28))\n\nmy_nn = Net()\nresult = my_nn(random_data)\nprint (result)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Each number in this resulting tensor equates to the prediction of the\nlabel the random tensor is associated to.\n\nCongratulations! You have successfully defined a neural network in\nPyTorch.\n\nLearn More\n----------\n\nTake a look at these other recipes to continue your learning:\n\n- `What is a state_dict in PyTorch `__\n- `Saving and loading models for inference in PyTorch `__\n\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.4" - } - }, - "nbformat": 4, - "nbformat_minor": 0 -} \ No newline at end of file diff --git a/recipes/recipes/defining_a_neural_network.py b/recipes/recipes/defining_a_neural_network.py deleted file mode 100644 index 42e2d3370ca..00000000000 --- a/recipes/recipes/defining_a_neural_network.py +++ /dev/null @@ -1,183 +0,0 @@ -""" -Defining a Neural Network in PyTorch -==================================== -Deep learning uses artificial neural networks (models), which are -computing systems that are composed of many layers of interconnected -units. By passing data through these interconnected units, a neural -network is able to learn how to approximate the computations required to -transform inputs into outputs. In PyTorch, neural networks can be -constructed using the ``torch.nn`` package. - -Introduction ------------- -PyTorch provides the elegantly designed modules and classes, including -``torch.nn``, to help you create and train neural networks. An -``nn.Module`` contains layers, and a method ``forward(input)`` that -returns the ``output``. - -In this recipe, we will use ``torch.nn`` to define a neural network -intended for the `MNIST -dataset `__. - -Setup ------ -Before we begin, we need to install ``torch`` if it isn’t already -available. - -:: - - pip install torchaudio - - -""" - - -###################################################################### -# Steps -# ----- -# -# 1. Import all necessary libraries for loading our data -# 2. Define and intialize the neural network -# 3. Specify how data will pass through your model -# 4. [Optional] Pass data through your model to test -# -# 1. Import necessary libraries for loading our data -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# For this recipe, we will use ``torch`` and its subsidiaries ``torch.nn`` -# and ``torch.nn.functional``. -# - -import torch -import torch.nn as nn -import torch.nn.functional as F - - -###################################################################### -# 2. Define and intialize the neural network -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# Our network will recognize images. We will use a process built into -# PyTorch called convolution. Convolution adds each element of an image to -# its local neighbors, weighted by a kernel, or a small martrix, that -# helps us extract certain features (like edge detection, sharpness, -# blurriness, etc.) from the input image. -# -# There are two requirements for defining the ``Net`` class of your model. -# The first is writing an ``__init__`` function that references -# ``nn.Module``. This function is where you define the fully connected -# layers in your neural network. -# -# Using convolution, we will define our model to take 1 input image -# channel, and output match our target of 10 labels representing numbers 0 -# through 9. This algorithm is yours to create, we will follow a standard -# MNIST algorithm. -# - -class Net(nn.Module): - def __init__(self): - super(Net, self).__init__() - - # First 2D convolutional layer, taking in 1 input channel (image), - # outputting 32 convolutional features, with a square kernel size of 3 - self.conv1 = nn.Conv2d(1, 32, 3, 1) - # Second 2D convolutional layer, taking in the 32 input layers, - # outputting 64 convolutional features, with a square kernel size of 3 - self.conv2 = nn.Conv2d(32, 64, 3, 1) - - # Designed to ensure that adjacent pixels are either all 0s or all active - # with an input probability - self.dropout1 = nn.Dropout2d(0.25) - self.dropout2 = nn.Dropout2d(0.5) - - # First fully connected layer - self.fc1 = nn.Linear(9216, 128) - # Second fully connected layer that outputs our 10 labels - self.fc2 = nn.Linear(128, 10) - -my_nn = Net() -print(my_nn) - - -###################################################################### -# We have finished defining our neural network, now we have to define how -# our data will pass through it. -# -# 3. Specify how data will pass through your model -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# When you use PyTorch to build a model, you just have to define the -# ``forward`` function, that will pass the data into the computation graph -# (i.e. our neural network). This will represent our feed-forward -# algorithm. -# -# You can use any of the Tensor operations in the ``forward`` function. -# - -class Net(nn.Module): - def __init__(self): - super(Net, self).__init__() - self.conv1 = nn.Conv2d(1, 32, 3, 1) - self.conv2 = nn.Conv2d(32, 64, 3, 1) - self.dropout1 = nn.Dropout2d(0.25) - self.dropout2 = nn.Dropout2d(0.5) - self.fc1 = nn.Linear(9216, 128) - self.fc2 = nn.Linear(128, 10) - - # x represents our data - def forward(self, x): - # Pass data through conv1 - x = self.conv1(x) - # Use the rectified-linear activation function over x - x = F.relu(x) - - x = self.conv2(x) - x = F.relu(x) - - # Run max pooling over x - x = F.max_pool2d(x, 2) - # Pass data through dropout1 - x = self.dropout1(x) - # Flatten x with start_dim=1 - x = torch.flatten(x, 1) - # Pass data through fc1 - x = self.fc1(x) - x = F.relu(x) - x = self.dropout2(x) - x = self.fc2(x) - - # Apply softmax to x - output = F.log_softmax(x, dim=1) - return output - - -###################################################################### -# 4. [Optional] Pass data through your model to test -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# To ensure we receive our desired output, let’s test our model by passing -# some random data through it. -# - -# Equates to one random 28x28 image -random_data = torch.rand((1, 1, 28, 28)) - -my_nn = Net() -result = my_nn(random_data) -print (result) - - -###################################################################### -# Each number in this resulting tensor equates to the prediction of the -# label the random tensor is associated to. -# -# Congratulations! You have successfully defined a neural network in -# PyTorch. -# -# Learn More -# ---------- -# -# Take a look at these other recipes to continue your learning: -# -# - `What is a state_dict in PyTorch `__ -# - `Saving and loading models for inference in PyTorch `__ diff --git a/recipes/recipes/defining_a_neural_network.rst b/recipes/recipes/defining_a_neural_network.rst deleted file mode 100644 index 6097b49d295..00000000000 --- a/recipes/recipes/defining_a_neural_network.rst +++ /dev/null @@ -1,234 +0,0 @@ -.. note:: - :class: sphx-glr-download-link-note - - Click :ref:`here ` to download the full example code -.. rst-class:: sphx-glr-example-title - -.. _sphx_glr_recipes_recipes_defining_a_neural_network.py: - - -Defining a Neural Network in PyTorch -==================================== -Deep learning uses artificial neural networks (models), which are -computing systems that are composed of many layers of interconnected -units. By passing data through these interconnected units, a neural -network is able to learn how to approximate the computations required to -transform inputs into outputs. In PyTorch, neural networks can be -constructed using the ``torch.nn`` package. - -Introduction ------------- -PyTorch provides the elegantly designed modules and classes, including -``torch.nn``, to help you create and train neural networks. An -``nn.Module`` contains layers, and a method ``forward(input)`` that -returns the ``output``. - -In this recipe, we will use ``torch.nn`` to define a neural network -intended for the `MNIST -dataset `__. - -Setup ------ -Before we begin, we need to install ``torch`` if it isn’t already -available. - -:: - - pip install torchaudio -Steps ------ - -1. Import all necessary libraries for loading our data -2. Define and intialize the neural network -3. Specify how data will pass through your model -4. [Optional] Pass data through your model to test - -1. Import necessary libraries for loading our data -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -For this recipe, we will use ``torch`` and its subsidiaries ``torch.nn`` -and ``torch.nn.functional``. - - - -.. code-block:: default - - - import torch - import torch.nn as nn - import torch.nn.functional as F - - - -2. Define and intialize the neural network -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Our network will recognize images. We will use a process built into -PyTorch called convolution. Convolution adds each element of an image to -its local neighbors, weighted by a kernel, or a small martrix, that -helps us extract certain features (like edge detection, sharpness, -blurriness, etc.) from the input image. - -There are two requirements for defining the ``Net`` class of your model. -The first is writing an ``__init__`` function that references -``nn.Module``. This function is where you define the fully connected -layers in your neural network. - -Using convolution, we will define our model to take 1 input image -channel, and output match our target of 10 labels representing numbers 0 -through 9. This algorithm is yours to create, we will follow a standard -MNIST algorithm. - - - -.. code-block:: default - - - class Net(nn.Module): - def __init__(self): - super(Net, self).__init__() - - # First 2D convolutional layer, taking in 1 input channel (image), - # outputting 32 convolutional features, with a square kernel size of 3 - self.conv1 = nn.Conv2d(1, 32, 3, 1) - # Second 2D convolutional layer, taking in the 32 input layers, - # outputting 64 convolutional features, with a square kernel size of 3 - self.conv2 = nn.Conv2d(32, 64, 3, 1) - - # Designed to ensure that adjacent pixels are either all 0s or all active - # with an input probability - self.dropout1 = nn.Dropout2d(0.25) - self.dropout2 = nn.Dropout2d(0.5) - - # First fully connected layer - self.fc1 = nn.Linear(9216, 128) - # Second fully connected layer that outputs our 10 labels - self.fc2 = nn.Linear(128, 10) - - my_nn = Net() - print(my_nn) - - - -We have finished defining our neural network, now we have to define how -our data will pass through it. - -3. Specify how data will pass through your model -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -When you use PyTorch to build a model, you just have to define the -``forward`` function, that will pass the data into the computation graph -(i.e. our neural network). This will represent our feed-forward -algorithm. - -You can use any of the Tensor operations in the ``forward`` function. - - - -.. code-block:: default - - - class Net(nn.Module): - def __init__(self): - super(Net, self).__init__() - self.conv1 = nn.Conv2d(1, 32, 3, 1) - self.conv2 = nn.Conv2d(32, 64, 3, 1) - self.dropout1 = nn.Dropout2d(0.25) - self.dropout2 = nn.Dropout2d(0.5) - self.fc1 = nn.Linear(9216, 128) - self.fc2 = nn.Linear(128, 10) - - # x represents our data - def forward(self, x): - # Pass data through conv1 - x = self.conv1(x) - # Use the rectified-linear activation function over x - x = F.relu(x) - - x = self.conv2(x) - x = F.relu(x) - - # Run max pooling over x - x = F.max_pool2d(x, 2) - # Pass data through dropout1 - x = self.dropout1(x) - # Flatten x with start_dim=1 - x = torch.flatten(x, 1) - # Pass data through fc1 - x = self.fc1(x) - x = F.relu(x) - x = self.dropout2(x) - x = self.fc2(x) - - # Apply softmax to x - output = F.log_softmax(x, dim=1) - return output - - - -4. [Optional] Pass data through your model to test -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -To ensure we receive our desired output, let’s test our model by passing -some random data through it. - - - -.. code-block:: default - - - # Equates to one random 28x28 image - random_data = torch.rand((1, 1, 28, 28)) - - my_nn = Net() - result = my_nn(random_data) - print (result) - - - -Each number in this resulting tensor equates to the prediction of the -label the random tensor is associated to. - -Congratulations! You have successfully defined a neural network in -PyTorch. - -Learn More ----------- - -Take a look at these other recipes to continue your learning: - -- `What is a state_dict in PyTorch `__ -- `Saving and loading models for inference in PyTorch `__ - - -.. rst-class:: sphx-glr-timing - - **Total running time of the script:** ( 0 minutes 0.000 seconds) - - -.. _sphx_glr_download_recipes_recipes_defining_a_neural_network.py: - - -.. only :: html - - .. container:: sphx-glr-footer - :class: sphx-glr-footer-example - - - - .. container:: sphx-glr-download - - :download:`Download Python source code: defining_a_neural_network.py ` - - - - .. container:: sphx-glr-download - - :download:`Download Jupyter notebook: defining_a_neural_network.ipynb ` - - -.. only:: html - - .. rst-class:: sphx-glr-signature - - `Gallery generated by Sphinx-Gallery `_ diff --git a/recipes/recipes/dynamic_quantization.ipynb b/recipes/recipes/dynamic_quantization.ipynb deleted file mode 100644 index 1f49e4d1dae..00000000000 --- a/recipes/recipes/dynamic_quantization.ipynb +++ /dev/null @@ -1,133 +0,0 @@ -{ - "cells": [ - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "%matplotlib inline" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\nDynamic Quantization\n====================\n\n--------------\n\nIn this recipe you will see how to take advantage of Dynamic\nQuantization to accelerate inference on an LSTM-style recurrent neural\nnetwork. This reduces the size of the model weights and speeds up model\nexecution.\n\n**Introduction**\n\nThere are a number of trade-offs that can be made when designing neural\nnetworks. During model developmenet and training you can alter the\nnumber of layers and number of parameters in a recurrent neural network\nand trade-off accuracy against model size and/or model latency or\nthroughput. Such changes can take lot of time and compute resources\nbecause you are iterating over the model training. Quantization gives\nyou a way to make a similar trade off between performance and model\naccuracy with a known model after training is completed.\n\nYou can give it a try in a single session and you will certainly reduce\nyour model size significantly and may get a significant latency\nreduction without losing a lot of accuracy.\n\n**What is dynamic quantization?**\n\nQuantizing a network means converting it to use a reduced precision\ninteger representation for the weights and/or activations. This saves on\nmodel size and allows the use of higher throughput math operations on\nyour CPU or GPU.\n\nWhen converting from floating point to integer values you are\nessentially multiplying the floating point value by some scale factor\nand rounding the result to a whole number. The various quantization\napproaches differ in the way they approach determining that scale\nfactor.\n\nThe key idea with dynamic quantization as described here is that we are\ngoing to determine the scale factor for activations dynamically based on\nthe data range observed at runtime. This ensures that the scale factor\nis \"tuned\" so that as much signal as possible about each observed\ndataset is preserved.\n\nThe model parameters on the other hand are known during model conversion\nand they are converted ahead of time and stored in INT8 form.\n\nArithmetic in the quantized model is done using vectorized INT8\ninstructions. Accumulation is typically done with INT16 or INT32 to\navoid overflow. This higher precision value is scaled back to INT8 if\nthe next layer is quantized or converted to FP32 for output.\n\nDynamic quantization is relatively free of tuning parameters which makes\nit well suited to be added into production pipelines as a standard part\nof converting LSTM models to deployment.\n\n--------------\n\n**Note: Limitations on the approach taken here**\n\nThis recipe provides a quick introduction to the dynamic quantization\nfeatures in PyTorch and the workflow for using it. Our focus is on\nexplaining the specific functions used to convert the model. We will\nmake a number of significant simplifications in the interest of brevity\nand clarity\n\n1. You will start with a minimal LSTM network\n2. You are simply going to initialize the network with a random hidden\n state\n3. You are going to test the network with random inputs\n4. You are not going to train the network in this tutorial\n5. You will see that the quantized form of this network is smaller and\n runs faster than the floating point network we started with\n6. You will see that the output values are generally in the same\n ballpark as the output of the FP32 network, but we are not\n demonstrating here the expected accuracy loss on a real trained\n network\n\nYou will see how dynamic quantization is done and be able to see\nsuggestive reductions in memory use and latency times. Providing a\ndemonstration that the technique can preserve high levels of model\naccuracy on a trained LSTM is left to a more advanced tutorial. If you\nwant to move right away to that more rigorous treatment please proceed\nto the `advanced dynamic quantization\ntutorial `__.\n\n--------------\n\n**Recipe Structure**\n\nThis recipe has 5 steps.\n\n1. Set up\n\n Here you define a very simple LSTM, import modules, and establish\n some random input tensors.\n\n2. Do the quantization\n\n Here you instantiate a floating point model and then create quantized\n version of it.\n\n3. Look at model size\n\n Here you show that the model size gets smaller.\n\n4. Look at latency\n\n Here you run the two models and compare model runtime (latency).\n\n5. Look at accuracy\n\n Here you run the two models and compare outputs.\n\n**Step 1: set up**\n\nThis is a straightfoward bit of code to set up for the rest of the\nrecipe.\n\nThe unique module we are importing here is torch.quantization which\nincludes PyTorch's quantized operators and conversion functions. We also\ndefine a very simple LSTM model and set up some inputs.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "# import the modules used here in this recipe\nimport torch\nimport torch.quantization\nimport torch.nn as nn\nimport copy\nimport os\nimport time\n\n# define a very, very simple LSTM for demonstration purposes\n# in this case, we are wrapping nn.LSTM, one layer, no pre or post processing\n# inspired by \n# https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html, by Robert Guthrie\n# and https://pytorch.org/tutorials/advanced/dynamic_quantization_tutorial.html\nclass lstm_for_demonstration(nn.Module):\n \"\"\"Elementary Long Short Term Memory style model which simply wraps nn.LSTM\n Not to be used for anything other than demonstration. \n \"\"\" \n def __init__(self,in_dim,out_dim,depth):\n super(lstm_for_demonstration,self).__init__()\n self.lstm = nn.LSTM(in_dim,out_dim,depth)\n\n def forward(self,inputs,hidden):\n out,hidden = self.lstm(inputs,hidden)\n return out, hidden\n\n \ntorch.manual_seed(29592) # set the seed for reproducibility\n\n#shape parameters\nmodel_dimension=8\nsequence_length=20\nbatch_size=1\nlstm_depth=1\n\n# random data for input\ninputs = torch.randn(sequence_length,batch_size,model_dimension)\n# hidden is actually is a tuple of the initial hidden state and the initial cell state\nhidden = (torch.randn(lstm_depth,batch_size,model_dimension), torch.randn(lstm_depth,batch_size,model_dimension))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "**Step 2: Do the quantization**\n\nNow we get to the fun part. First we create an instance of the model\ncalled float\\_lstm then we are going to quantize it. We're going to use\nthe\n\n::\n\n torch.quantization.quantize_dynamic()\n\nfunction here (`see\ndocumentation `__)\nwhich takes the model, then a list of the submodules which we want to\nhave quantized if they appear, then the datatype we are targeting. This\nfunction returns a quantized version of the original model as a new\nmodule.\n\nThat's all it takes.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "# here is our floating point instance \nfloat_lstm = lstm_for_demonstration(model_dimension, model_dimension,lstm_depth)\n\n# this is the call that does the work\nquantized_lstm = torch.quantization.quantize_dynamic(\n float_lstm, {nn.LSTM, nn.Linear}, dtype=torch.qint8\n)\n\n# show the changes that were made\nprint('Here is the floating point version of this module:')\nprint(float_lstm)\nprint('')\nprint('and now the quantized version:')\nprint(quantized_lstm)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "**Step 4. Look at the model size**\n\nOk, so we've quantized the model. What does that get us? Well the first\nbenefit is that we've replaced the FP32 model parameters with INT8\nvalues (and some recorded scale factors). This means about 75% less data\nto store and move around. With the default values the reduction shown\nbelow will be less than 75% but if you increase the model size above\n(for example you can set model dimension to something like 80) this will\nconverge towards 4x smaller as the stored model size dominated more and\nmore by the parameter values.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "def print_size_of_model(model, label=\"\"):\n torch.save(model.state_dict(), \"temp.p\")\n size=os.path.getsize(\"temp.p\")\n print(\"model: \",label,' \\t','Size (KB):', size/1e3)\n os.remove('temp.p')\n return size\n\n# compare the sizes\nf=print_size_of_model(float_lstm,\"fp32\")\nq=print_size_of_model(quantized_lstm,\"int8\")\nprint(\"{0:.2f} times smaller\".format(f/q))\n\n# note that this value is wrong in PyTorch 1.4 due to https://github.com/pytorch/pytorch/issues/31468\n# this will be fixed in 1.5 with https://github.com/pytorch/pytorch/pull/31540" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "**Step 4: Look at latency**\n\nThe second benefit is that the quantized model will typically run\nfaster. This is due to a combinations of effects including at least:\n\n1. Less time spent moving parameter data in\n2. Faster INT8 operations\n\nAs you will see the quantized version of this super-simple network runs\nfaster. This will generally be true of more complex networks but as they\nsay \"your milage may vary\" depending on a number of factors including\nthe structure of the model and the hardware you are running on.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "# compare the performance\nprint(\"Floating point FP32\")\n# %timeit float_lstm.forward(inputs, hidden)\n\nprint(\"Quantized INT8\")\n# %timeit quantized_lstm.forward(inputs,hidden)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "**Step 5: Look at accuracy**\n\nWe are not going to do a careful look at accuracy here because we are\nworking with a randomly initialized network rather than a properly\ntrained one. However, I think it is worth quickly showing that the\nquantized network does produce output tensors that are \"in the same\nballpark\" as the original one.\n\nFor a more detailed analysis please see the more advanced tutorials\nreferenced at the end of this recipe.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "# run the float model\nout1, hidden1 = float_lstm(inputs, hidden)\nmag1 = torch.mean(abs(out1)).item()\nprint('mean absolute value of output tensor values in the FP32 model is {0:.5f} '.format(mag1))\n\n# run the quantized model\nout2, hidden2 = quantized_lstm(inputs, hidden)\nmag2 = torch.mean(abs(out2)).item()\nprint('mean absolute value of output tensor values in the INT8 model is {0:.5f}'.format(mag2))\n\n# compare them \nmag3 = torch.mean(abs(out1-out2)).item()\nprint('mean absolute value of the difference between the output tensors is {0:.5f} or {1:.2f} percent'.format(mag3,mag3/mag1*100))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "**Summary**\n\nWe've explained what dynamic quantization is, what beenefits it brings,\nand you have used the ``torch.quantization.quantize_dynamic()`` function\nto quickly quantize a simple LSTM model.\n\n**To continue learning about dynamic quantization**\n\nThis was a fast and high level treatment of this material; for more\ndetail please continue learning with:\n\nhttps://pytorch.org/tutorials/advanced/dynamic\\_quantization\\_tutorial.html\n\n**Other resources:**\n\nDocs\n\nhttps://pytorch.org/docs/stable/quantization.html\n\nTutorials\n\nhttps://pytorch.org/tutorials/intermediate/dynamic\\_quantization\\_bert\\_tutorial.html\n\nhttps://pytorch.org/tutorials/advanced/dynamic\\_quantization\\_tutorial.html\n\nBlogs\n\nhttps://pytorch.org/blog/introduction-to-quantization-on-pytorch/\n\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.4" - } - }, - "nbformat": 4, - "nbformat_minor": 0 -} \ No newline at end of file diff --git a/recipes/recipes/dynamic_quantization.py b/recipes/recipes/dynamic_quantization.py deleted file mode 100644 index 73811daa94c..00000000000 --- a/recipes/recipes/dynamic_quantization.py +++ /dev/null @@ -1,313 +0,0 @@ -""" -Dynamic Quantization -==================== - --------------- - -In this recipe you will see how to take advantage of Dynamic -Quantization to accelerate inference on an LSTM-style recurrent neural -network. This reduces the size of the model weights and speeds up model -execution. - -**Introduction** - -There are a number of trade-offs that can be made when designing neural -networks. During model developmenet and training you can alter the -number of layers and number of parameters in a recurrent neural network -and trade-off accuracy against model size and/or model latency or -throughput. Such changes can take lot of time and compute resources -because you are iterating over the model training. Quantization gives -you a way to make a similar trade off between performance and model -accuracy with a known model after training is completed. - -You can give it a try in a single session and you will certainly reduce -your model size significantly and may get a significant latency -reduction without losing a lot of accuracy. - -**What is dynamic quantization?** - -Quantizing a network means converting it to use a reduced precision -integer representation for the weights and/or activations. This saves on -model size and allows the use of higher throughput math operations on -your CPU or GPU. - -When converting from floating point to integer values you are -essentially multiplying the floating point value by some scale factor -and rounding the result to a whole number. The various quantization -approaches differ in the way they approach determining that scale -factor. - -The key idea with dynamic quantization as described here is that we are -going to determine the scale factor for activations dynamically based on -the data range observed at runtime. This ensures that the scale factor -is "tuned" so that as much signal as possible about each observed -dataset is preserved. - -The model parameters on the other hand are known during model conversion -and they are converted ahead of time and stored in INT8 form. - -Arithmetic in the quantized model is done using vectorized INT8 -instructions. Accumulation is typically done with INT16 or INT32 to -avoid overflow. This higher precision value is scaled back to INT8 if -the next layer is quantized or converted to FP32 for output. - -Dynamic quantization is relatively free of tuning parameters which makes -it well suited to be added into production pipelines as a standard part -of converting LSTM models to deployment. - --------------- - -**Note: Limitations on the approach taken here** - -This recipe provides a quick introduction to the dynamic quantization -features in PyTorch and the workflow for using it. Our focus is on -explaining the specific functions used to convert the model. We will -make a number of significant simplifications in the interest of brevity -and clarity - -1. You will start with a minimal LSTM network -2. You are simply going to initialize the network with a random hidden - state -3. You are going to test the network with random inputs -4. You are not going to train the network in this tutorial -5. You will see that the quantized form of this network is smaller and - runs faster than the floating point network we started with -6. You will see that the output values are generally in the same - ballpark as the output of the FP32 network, but we are not - demonstrating here the expected accuracy loss on a real trained - network - -You will see how dynamic quantization is done and be able to see -suggestive reductions in memory use and latency times. Providing a -demonstration that the technique can preserve high levels of model -accuracy on a trained LSTM is left to a more advanced tutorial. If you -want to move right away to that more rigorous treatment please proceed -to the `advanced dynamic quantization -tutorial `__. - --------------- - -**Recipe Structure** - -This recipe has 5 steps. - -1. Set up - - Here you define a very simple LSTM, import modules, and establish - some random input tensors. - -2. Do the quantization - - Here you instantiate a floating point model and then create quantized - version of it. - -3. Look at model size - - Here you show that the model size gets smaller. - -4. Look at latency - - Here you run the two models and compare model runtime (latency). - -5. Look at accuracy - - Here you run the two models and compare outputs. - -**Step 1: set up** - -This is a straightfoward bit of code to set up for the rest of the -recipe. - -The unique module we are importing here is torch.quantization which -includes PyTorch's quantized operators and conversion functions. We also -define a very simple LSTM model and set up some inputs. - -""" - -# import the modules used here in this recipe -import torch -import torch.quantization -import torch.nn as nn -import copy -import os -import time - -# define a very, very simple LSTM for demonstration purposes -# in this case, we are wrapping nn.LSTM, one layer, no pre or post processing -# inspired by -# https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html, by Robert Guthrie -# and https://pytorch.org/tutorials/advanced/dynamic_quantization_tutorial.html -class lstm_for_demonstration(nn.Module): - """Elementary Long Short Term Memory style model which simply wraps nn.LSTM - Not to be used for anything other than demonstration. - """ - def __init__(self,in_dim,out_dim,depth): - super(lstm_for_demonstration,self).__init__() - self.lstm = nn.LSTM(in_dim,out_dim,depth) - - def forward(self,inputs,hidden): - out,hidden = self.lstm(inputs,hidden) - return out, hidden - - -torch.manual_seed(29592) # set the seed for reproducibility - -#shape parameters -model_dimension=8 -sequence_length=20 -batch_size=1 -lstm_depth=1 - -# random data for input -inputs = torch.randn(sequence_length,batch_size,model_dimension) -# hidden is actually is a tuple of the initial hidden state and the initial cell state -hidden = (torch.randn(lstm_depth,batch_size,model_dimension), torch.randn(lstm_depth,batch_size,model_dimension)) - - -###################################################################### -# **Step 2: Do the quantization** -# -# Now we get to the fun part. First we create an instance of the model -# called float\_lstm then we are going to quantize it. We're going to use -# the -# -# :: -# -# torch.quantization.quantize_dynamic() -# -# function here (`see -# documentation `__) -# which takes the model, then a list of the submodules which we want to -# have quantized if they appear, then the datatype we are targeting. This -# function returns a quantized version of the original model as a new -# module. -# -# That's all it takes. -# - - # here is our floating point instance -float_lstm = lstm_for_demonstration(model_dimension, model_dimension,lstm_depth) - -# this is the call that does the work -quantized_lstm = torch.quantization.quantize_dynamic( - float_lstm, {nn.LSTM, nn.Linear}, dtype=torch.qint8 -) - -# show the changes that were made -print('Here is the floating point version of this module:') -print(float_lstm) -print('') -print('and now the quantized version:') -print(quantized_lstm) - - -###################################################################### -# **Step 4. Look at the model size** -# -# Ok, so we've quantized the model. What does that get us? Well the first -# benefit is that we've replaced the FP32 model parameters with INT8 -# values (and some recorded scale factors). This means about 75% less data -# to store and move around. With the default values the reduction shown -# below will be less than 75% but if you increase the model size above -# (for example you can set model dimension to something like 80) this will -# converge towards 4x smaller as the stored model size dominated more and -# more by the parameter values. -# - -def print_size_of_model(model, label=""): - torch.save(model.state_dict(), "temp.p") - size=os.path.getsize("temp.p") - print("model: ",label,' \t','Size (KB):', size/1e3) - os.remove('temp.p') - return size - -# compare the sizes -f=print_size_of_model(float_lstm,"fp32") -q=print_size_of_model(quantized_lstm,"int8") -print("{0:.2f} times smaller".format(f/q)) - -# note that this value is wrong in PyTorch 1.4 due to https://github.com/pytorch/pytorch/issues/31468 -# this will be fixed in 1.5 with https://github.com/pytorch/pytorch/pull/31540 - - -###################################################################### -# **Step 4: Look at latency** -# -# The second benefit is that the quantized model will typically run -# faster. This is due to a combinations of effects including at least: -# -# 1. Less time spent moving parameter data in -# 2. Faster INT8 operations -# -# As you will see the quantized version of this super-simple network runs -# faster. This will generally be true of more complex networks but as they -# say "your milage may vary" depending on a number of factors including -# the structure of the model and the hardware you are running on. -# - -# compare the performance -print("Floating point FP32") -# %timeit float_lstm.forward(inputs, hidden) - -print("Quantized INT8") -# %timeit quantized_lstm.forward(inputs,hidden) - - -###################################################################### -# **Step 5: Look at accuracy** -# -# We are not going to do a careful look at accuracy here because we are -# working with a randomly initialized network rather than a properly -# trained one. However, I think it is worth quickly showing that the -# quantized network does produce output tensors that are "in the same -# ballpark" as the original one. -# -# For a more detailed analysis please see the more advanced tutorials -# referenced at the end of this recipe. -# - -# run the float model -out1, hidden1 = float_lstm(inputs, hidden) -mag1 = torch.mean(abs(out1)).item() -print('mean absolute value of output tensor values in the FP32 model is {0:.5f} '.format(mag1)) - -# run the quantized model -out2, hidden2 = quantized_lstm(inputs, hidden) -mag2 = torch.mean(abs(out2)).item() -print('mean absolute value of output tensor values in the INT8 model is {0:.5f}'.format(mag2)) - -# compare them -mag3 = torch.mean(abs(out1-out2)).item() -print('mean absolute value of the difference between the output tensors is {0:.5f} or {1:.2f} percent'.format(mag3,mag3/mag1*100)) - - -###################################################################### -# **Summary** -# -# We've explained what dynamic quantization is, what beenefits it brings, -# and you have used the ``torch.quantization.quantize_dynamic()`` function -# to quickly quantize a simple LSTM model. -# -# **To continue learning about dynamic quantization** -# -# This was a fast and high level treatment of this material; for more -# detail please continue learning with: -# -# https://pytorch.org/tutorials/advanced/dynamic\_quantization\_tutorial.html -# -# **Other resources:** -# -# Docs -# -# https://pytorch.org/docs/stable/quantization.html -# -# Tutorials -# -# https://pytorch.org/tutorials/intermediate/dynamic\_quantization\_bert\_tutorial.html -# -# https://pytorch.org/tutorials/advanced/dynamic\_quantization\_tutorial.html -# -# Blogs -# -# https://pytorch.org/blog/introduction-to-quantization-on-pytorch/ -# \ No newline at end of file diff --git a/recipes/recipes/dynamic_quantization.rst b/recipes/recipes/dynamic_quantization.rst deleted file mode 100644 index 4aaafb64023..00000000000 --- a/recipes/recipes/dynamic_quantization.rst +++ /dev/null @@ -1,370 +0,0 @@ -.. note:: - :class: sphx-glr-download-link-note - - Click :ref:`here ` to download the full example code -.. rst-class:: sphx-glr-example-title - -.. _sphx_glr_recipes_recipes_dynamic_quantization.py: - - -Dynamic Quantization -==================== - --------------- - -In this recipe you will see how to take advantage of Dynamic -Quantization to accelerate inference on an LSTM-style recurrent neural -network. This reduces the size of the model weights and speeds up model -execution. - -**Introduction** - -There are a number of trade-offs that can be made when designing neural -networks. During model developmenet and training you can alter the -number of layers and number of parameters in a recurrent neural network -and trade-off accuracy against model size and/or model latency or -throughput. Such changes can take lot of time and compute resources -because you are iterating over the model training. Quantization gives -you a way to make a similar trade off between performance and model -accuracy with a known model after training is completed. - -You can give it a try in a single session and you will certainly reduce -your model size significantly and may get a significant latency -reduction without losing a lot of accuracy. - -**What is dynamic quantization?** - -Quantizing a network means converting it to use a reduced precision -integer representation for the weights and/or activations. This saves on -model size and allows the use of higher throughput math operations on -your CPU or GPU. - -When converting from floating point to integer values you are -essentially multiplying the floating point value by some scale factor -and rounding the result to a whole number. The various quantization -approaches differ in the way they approach determining that scale -factor. - -The key idea with dynamic quantization as described here is that we are -going to determine the scale factor for activations dynamically based on -the data range observed at runtime. This ensures that the scale factor -is "tuned" so that as much signal as possible about each observed -dataset is preserved. - -The model parameters on the other hand are known during model conversion -and they are converted ahead of time and stored in INT8 form. - -Arithmetic in the quantized model is done using vectorized INT8 -instructions. Accumulation is typically done with INT16 or INT32 to -avoid overflow. This higher precision value is scaled back to INT8 if -the next layer is quantized or converted to FP32 for output. - -Dynamic quantization is relatively free of tuning parameters which makes -it well suited to be added into production pipelines as a standard part -of converting LSTM models to deployment. - --------------- - -**Note: Limitations on the approach taken here** - -This recipe provides a quick introduction to the dynamic quantization -features in PyTorch and the workflow for using it. Our focus is on -explaining the specific functions used to convert the model. We will -make a number of significant simplifications in the interest of brevity -and clarity - -1. You will start with a minimal LSTM network -2. You are simply going to initialize the network with a random hidden - state -3. You are going to test the network with random inputs -4. You are not going to train the network in this tutorial -5. You will see that the quantized form of this network is smaller and - runs faster than the floating point network we started with -6. You will see that the output values are generally in the same - ballpark as the output of the FP32 network, but we are not - demonstrating here the expected accuracy loss on a real trained - network - -You will see how dynamic quantization is done and be able to see -suggestive reductions in memory use and latency times. Providing a -demonstration that the technique can preserve high levels of model -accuracy on a trained LSTM is left to a more advanced tutorial. If you -want to move right away to that more rigorous treatment please proceed -to the `advanced dynamic quantization -tutorial `__. - --------------- - -**Recipe Structure** - -This recipe has 5 steps. - -1. Set up - - Here you define a very simple LSTM, import modules, and establish - some random input tensors. - -2. Do the quantization - - Here you instantiate a floating point model and then create quantized - version of it. - -3. Look at model size - - Here you show that the model size gets smaller. - -4. Look at latency - - Here you run the two models and compare model runtime (latency). - -5. Look at accuracy - - Here you run the two models and compare outputs. - -**Step 1: set up** - -This is a straightfoward bit of code to set up for the rest of the -recipe. - -The unique module we are importing here is torch.quantization which -includes PyTorch's quantized operators and conversion functions. We also -define a very simple LSTM model and set up some inputs. - -.. code-block:: default - - - # import the modules used here in this recipe - import torch - import torch.quantization - import torch.nn as nn - import copy - import os - import time - - # define a very, very simple LSTM for demonstration purposes - # in this case, we are wrapping nn.LSTM, one layer, no pre or post processing - # inspired by - # https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html, by Robert Guthrie - # and https://pytorch.org/tutorials/advanced/dynamic_quantization_tutorial.html - class lstm_for_demonstration(nn.Module): - """Elementary Long Short Term Memory style model which simply wraps nn.LSTM - Not to be used for anything other than demonstration. - """ - def __init__(self,in_dim,out_dim,depth): - super(lstm_for_demonstration,self).__init__() - self.lstm = nn.LSTM(in_dim,out_dim,depth) - - def forward(self,inputs,hidden): - out,hidden = self.lstm(inputs,hidden) - return out, hidden - - - torch.manual_seed(29592) # set the seed for reproducibility - - #shape parameters - model_dimension=8 - sequence_length=20 - batch_size=1 - lstm_depth=1 - - # random data for input - inputs = torch.randn(sequence_length,batch_size,model_dimension) - # hidden is actually is a tuple of the initial hidden state and the initial cell state - hidden = (torch.randn(lstm_depth,batch_size,model_dimension), torch.randn(lstm_depth,batch_size,model_dimension)) - - - -**Step 2: Do the quantization** - -Now we get to the fun part. First we create an instance of the model -called float\_lstm then we are going to quantize it. We're going to use -the - -:: - - torch.quantization.quantize_dynamic() - -function here (`see -documentation `__) -which takes the model, then a list of the submodules which we want to -have quantized if they appear, then the datatype we are targeting. This -function returns a quantized version of the original model as a new -module. - -That's all it takes. - - - -.. code-block:: default - - - # here is our floating point instance - float_lstm = lstm_for_demonstration(model_dimension, model_dimension,lstm_depth) - - # this is the call that does the work - quantized_lstm = torch.quantization.quantize_dynamic( - float_lstm, {nn.LSTM, nn.Linear}, dtype=torch.qint8 - ) - - # show the changes that were made - print('Here is the floating point version of this module:') - print(float_lstm) - print('') - print('and now the quantized version:') - print(quantized_lstm) - - - -**Step 4. Look at the model size** - -Ok, so we've quantized the model. What does that get us? Well the first -benefit is that we've replaced the FP32 model parameters with INT8 -values (and some recorded scale factors). This means about 75% less data -to store and move around. With the default values the reduction shown -below will be less than 75% but if you increase the model size above -(for example you can set model dimension to something like 80) this will -converge towards 4x smaller as the stored model size dominated more and -more by the parameter values. - - - -.. code-block:: default - - - def print_size_of_model(model, label=""): - torch.save(model.state_dict(), "temp.p") - size=os.path.getsize("temp.p") - print("model: ",label,' \t','Size (KB):', size/1e3) - os.remove('temp.p') - return size - - # compare the sizes - f=print_size_of_model(float_lstm,"fp32") - q=print_size_of_model(quantized_lstm,"int8") - print("{0:.2f} times smaller".format(f/q)) - - # note that this value is wrong in PyTorch 1.4 due to https://github.com/pytorch/pytorch/issues/31468 - # this will be fixed in 1.5 with https://github.com/pytorch/pytorch/pull/31540 - - - -**Step 4: Look at latency** - -The second benefit is that the quantized model will typically run -faster. This is due to a combinations of effects including at least: - -1. Less time spent moving parameter data in -2. Faster INT8 operations - -As you will see the quantized version of this super-simple network runs -faster. This will generally be true of more complex networks but as they -say "your milage may vary" depending on a number of factors including -the structure of the model and the hardware you are running on. - - - -.. code-block:: default - - - # compare the performance - print("Floating point FP32") - # %timeit float_lstm.forward(inputs, hidden) - - print("Quantized INT8") - # %timeit quantized_lstm.forward(inputs,hidden) - - - -**Step 5: Look at accuracy** - -We are not going to do a careful look at accuracy here because we are -working with a randomly initialized network rather than a properly -trained one. However, I think it is worth quickly showing that the -quantized network does produce output tensors that are "in the same -ballpark" as the original one. - -For a more detailed analysis please see the more advanced tutorials -referenced at the end of this recipe. - - - -.. code-block:: default - - - # run the float model - out1, hidden1 = float_lstm(inputs, hidden) - mag1 = torch.mean(abs(out1)).item() - print('mean absolute value of output tensor values in the FP32 model is {0:.5f} '.format(mag1)) - - # run the quantized model - out2, hidden2 = quantized_lstm(inputs, hidden) - mag2 = torch.mean(abs(out2)).item() - print('mean absolute value of output tensor values in the INT8 model is {0:.5f}'.format(mag2)) - - # compare them - mag3 = torch.mean(abs(out1-out2)).item() - print('mean absolute value of the difference between the output tensors is {0:.5f} or {1:.2f} percent'.format(mag3,mag3/mag1*100)) - - - -**Summary** - -We've explained what dynamic quantization is, what beenefits it brings, -and you have used the ``torch.quantization.quantize_dynamic()`` function -to quickly quantize a simple LSTM model. - -**To continue learning about dynamic quantization** - -This was a fast and high level treatment of this material; for more -detail please continue learning with: - -https://pytorch.org/tutorials/advanced/dynamic\_quantization\_tutorial.html - -**Other resources:** - -Docs - -https://pytorch.org/docs/stable/quantization.html - -Tutorials - -https://pytorch.org/tutorials/intermediate/dynamic\_quantization\_bert\_tutorial.html - -https://pytorch.org/tutorials/advanced/dynamic\_quantization\_tutorial.html - -Blogs - -https://pytorch.org/blog/introduction-to-quantization-on-pytorch/ - - -.. rst-class:: sphx-glr-timing - - **Total running time of the script:** ( 0 minutes 0.000 seconds) - - -.. _sphx_glr_download_recipes_recipes_dynamic_quantization.py: - - -.. only :: html - - .. container:: sphx-glr-footer - :class: sphx-glr-footer-example - - - - .. container:: sphx-glr-download - - :download:`Download Python source code: dynamic_quantization.py ` - - - - .. container:: sphx-glr-download - - :download:`Download Jupyter notebook: dynamic_quantization.ipynb ` - - -.. only:: html - - .. rst-class:: sphx-glr-signature - - `Gallery generated by Sphinx-Gallery `_ diff --git a/recipes/recipes/images/thumb/sphx_glr_Captum_Recipe_thumb.png b/recipes/recipes/images/thumb/sphx_glr_Captum_Recipe_thumb.png deleted file mode 100644 index 233f8e605ef..00000000000 Binary files a/recipes/recipes/images/thumb/sphx_glr_Captum_Recipe_thumb.png and /dev/null differ diff --git a/recipes/recipes/images/thumb/sphx_glr_custom_dataset_transforms_loader_thumb.png b/recipes/recipes/images/thumb/sphx_glr_custom_dataset_transforms_loader_thumb.png deleted file mode 100644 index 233f8e605ef..00000000000 Binary files a/recipes/recipes/images/thumb/sphx_glr_custom_dataset_transforms_loader_thumb.png and /dev/null differ diff --git a/recipes/recipes/images/thumb/sphx_glr_defining_a_neural_network_thumb.png b/recipes/recipes/images/thumb/sphx_glr_defining_a_neural_network_thumb.png deleted file mode 100644 index 233f8e605ef..00000000000 Binary files a/recipes/recipes/images/thumb/sphx_glr_defining_a_neural_network_thumb.png and /dev/null differ diff --git a/recipes/recipes/images/thumb/sphx_glr_dynamic_quantization_thumb.png b/recipes/recipes/images/thumb/sphx_glr_dynamic_quantization_thumb.png deleted file mode 100644 index 233f8e605ef..00000000000 Binary files a/recipes/recipes/images/thumb/sphx_glr_dynamic_quantization_thumb.png and /dev/null differ diff --git a/recipes/recipes/images/thumb/sphx_glr_loading_data_recipe_thumb.png b/recipes/recipes/images/thumb/sphx_glr_loading_data_recipe_thumb.png deleted file mode 100644 index 233f8e605ef..00000000000 Binary files a/recipes/recipes/images/thumb/sphx_glr_loading_data_recipe_thumb.png and /dev/null differ diff --git a/recipes/recipes/images/thumb/sphx_glr_save_load_across_devices_thumb.png b/recipes/recipes/images/thumb/sphx_glr_save_load_across_devices_thumb.png deleted file mode 100644 index 233f8e605ef..00000000000 Binary files a/recipes/recipes/images/thumb/sphx_glr_save_load_across_devices_thumb.png and /dev/null differ diff --git a/recipes/recipes/images/thumb/sphx_glr_saving_and_loading_a_general_checkpoint_thumb.png b/recipes/recipes/images/thumb/sphx_glr_saving_and_loading_a_general_checkpoint_thumb.png deleted file mode 100644 index 233f8e605ef..00000000000 Binary files a/recipes/recipes/images/thumb/sphx_glr_saving_and_loading_a_general_checkpoint_thumb.png and /dev/null differ diff --git a/recipes/recipes/images/thumb/sphx_glr_saving_and_loading_models_for_inference_thumb.png b/recipes/recipes/images/thumb/sphx_glr_saving_and_loading_models_for_inference_thumb.png deleted file mode 100644 index 233f8e605ef..00000000000 Binary files a/recipes/recipes/images/thumb/sphx_glr_saving_and_loading_models_for_inference_thumb.png and /dev/null differ diff --git a/recipes/recipes/images/thumb/sphx_glr_saving_multiple_models_in_one_file_thumb.png b/recipes/recipes/images/thumb/sphx_glr_saving_multiple_models_in_one_file_thumb.png deleted file mode 100644 index 233f8e605ef..00000000000 Binary files a/recipes/recipes/images/thumb/sphx_glr_saving_multiple_models_in_one_file_thumb.png and /dev/null differ diff --git a/recipes/recipes/images/thumb/sphx_glr_tensorboard_with_pytorch_thumb.png b/recipes/recipes/images/thumb/sphx_glr_tensorboard_with_pytorch_thumb.png deleted file mode 100644 index 233f8e605ef..00000000000 Binary files a/recipes/recipes/images/thumb/sphx_glr_tensorboard_with_pytorch_thumb.png and /dev/null differ diff --git a/recipes/recipes/images/thumb/sphx_glr_warmstarting_model_using_parameters_from_a_different_model_thumb.png b/recipes/recipes/images/thumb/sphx_glr_warmstarting_model_using_parameters_from_a_different_model_thumb.png deleted file mode 100644 index 233f8e605ef..00000000000 Binary files a/recipes/recipes/images/thumb/sphx_glr_warmstarting_model_using_parameters_from_a_different_model_thumb.png and /dev/null differ diff --git a/recipes/recipes/images/thumb/sphx_glr_what_is_state_dict_thumb.png b/recipes/recipes/images/thumb/sphx_glr_what_is_state_dict_thumb.png deleted file mode 100644 index 233f8e605ef..00000000000 Binary files a/recipes/recipes/images/thumb/sphx_glr_what_is_state_dict_thumb.png and /dev/null differ diff --git a/recipes/recipes/images/thumb/sphx_glr_zeroing_out_gradients_thumb.png b/recipes/recipes/images/thumb/sphx_glr_zeroing_out_gradients_thumb.png deleted file mode 100644 index 233f8e605ef..00000000000 Binary files a/recipes/recipes/images/thumb/sphx_glr_zeroing_out_gradients_thumb.png and /dev/null differ diff --git a/recipes/recipes/loading_data_recipe.ipynb b/recipes/recipes/loading_data_recipe.ipynb deleted file mode 100644 index 8332ae5a58f..00000000000 --- a/recipes/recipes/loading_data_recipe.ipynb +++ /dev/null @@ -1,140 +0,0 @@ -{ - "cells": [ - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "%matplotlib inline" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\nLoading data in PyTorch\n=======================\nPyTorch features extensive neural network building blocks with a simple,\nintuitive, and stable API. PyTorch includes packages to prepare and load\ncommon datasets for your model.\n\nIntroduction\n------------\nAt the heart of PyTorch data loading utility is the\n`torch.utils.data.DataLoader `__\nclass. It represents a Python iterable over a dataset. Libraries in\nPyTorch offer built-in high-quality datasets for you to use in\n`torch.utils.data.Dataset `__.\nThese datasets are currently available in:\n\n* `torchvision `__\n* `torchaudio `__\n* `torchtext `__\n\nwith more to come.\nUsing the Yesno dataset from ``torchaudio.datasets.YESNO``, we will\ndemonstrate how to effectively and efficiently load data from a PyTorch\n``Dataset`` into a PyTorch ``DataLoader``.\n\nSetup\n-----\nBefore we begin, we need to install ``torchaudio`` to have access to the\ndataset.\n\n::\n\n pip install torchaudio\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Steps\n-----\n\n1. Import all necessary libraries for loading our data\n2. Access the data in the dataset\n3. Loading the data\n4. Iterate over the data\n5. [Optional] Visualize the data\n\n\n1. Import necessary libraries for loading our data\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nFor this recipe, we will use ``torch`` and ``torchaudio``. Depending on\nwhat built-in datasets you use, you can also install and import\n``torchvision`` or ``torchtext``.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "import torch\nimport torchaudio" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "2. Access the data in the dataset\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nThe Yesno dataset in ``torchaudio`` features sixty recordings of one\nindividual saying yes or no in Hebrew; with each recording being eight\nwords long (`read more here `__).\n\n``torchaudio.datasets.YESNO`` creates a dataset for YesNo.\n\n::\n\n torchaudio.datasets.YESNO(\n root,\n url='http://www.openslr.org/resources/1/waves_yesno.tar.gz',\n folder_in_archive='waves_yesno',\n download=False,\n transform=None,\n target_transform=None)\n\nEach item in the dataset is a tuple of the form: (waveform, sample_rate,\nlabels).\n\nYou must set a ``root`` for the Yesno dataset, which is where the\ntraining and testing dataset will exist. The other parameters are\noptional, with their default values shown. Here is some additional\nuseful info on the other parameters:\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "# * ``download``: If true, downloads the dataset from the internet and puts it in root directory. If dataset is already downloaded, it is not downloaded again.\n# * ``transform``: Using transforms on your data allows you to take it from its source state and transform it into data that\u2019s joined together, de-normalized, and ready for training. Each library in PyTorch supports a growing list of transformations.\n# * ``target_transform``: A function/transform that takes in the target and transforms it.\n# \n# Let\u2019s access our Yesno data:\n# \n\n# A data point in Yesno is a tuple (waveform, sample_rate, labels) where labels \n# is a list of integers with 1 for yes and 0 for no.\nyesno_data_trainset = torchaudio.datasets.YESNO('./', download=True)\n\n# Pick data point number 3 to see an example of the the yesno_data:\nn = 3\nwaveform, sample_rate, labels = yesno_data[n]\nprint(\"Waveform: {}\\nSample rate: {}\\nLabels: {}\".format(waveform, sample_rate, labels))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "When using this data in practice, it is best practice to provision the\ndata into a \u201ctraining\u201d dataset and a \u201ctesting\u201d dataset. This ensures\nthat you have out-of-sample data to test the performance of your model.\n\n3. Loading the data\n~~~~~~~~~~~~~~~~~~~~~~~\n\nNow that we have access to the dataset, we must pass it through\n``torch.utils.data.DataLoader``. The ``DataLoader`` combines the dataset\nand a sampler, returning an iterable over the dataset.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "data_loader = torch.utils.data.DataLoader(yesno_data,\n batch_size=1,\n shuffle=True)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "4. Iterate over the data\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nOur data is now iterable using the ``data_loader``. This will be\nnecessary when we begin training our model! You will notice that now\neach data entry in the ``data_loader`` object is converted to a tensor\ncontaining tensors representing our waveform, sample rate, and labels.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "for data in data_loader:\n print(\"Data: \", data)\n print(\"Waveform: {}\\nSample rate: {}\\nLabels: {}\".format(data[0], data[1], data[2]))\n break" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "5. [Optional] Visualize the data\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nYou can optionally visualize your data to further understand the output\nfrom your ``DataLoader``.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "import matplotlib.pyplot as plt\n\nprint(data[0][0].numpy())\n\nplt.figure()\nplt.plot(waveform.t().numpy())" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Congratulations! You have successfully loaded data in PyTorch.\n\nLearn More\n----------\n\nTake a look at these other recipes to continue your learning:\n\n- `Defining a Neural Network `__\n- `What is a state_dict in PyTorch `__\n\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.4" - } - }, - "nbformat": 4, - "nbformat_minor": 0 -} \ No newline at end of file diff --git a/recipes/recipes/loading_data_recipe.py b/recipes/recipes/loading_data_recipe.py deleted file mode 100644 index 99e685897c3..00000000000 --- a/recipes/recipes/loading_data_recipe.py +++ /dev/null @@ -1,171 +0,0 @@ -""" -Loading data in PyTorch -======================= -PyTorch features extensive neural network building blocks with a simple, -intuitive, and stable API. PyTorch includes packages to prepare and load -common datasets for your model. - -Introduction ------------- -At the heart of PyTorch data loading utility is the -`torch.utils.data.DataLoader `__ -class. It represents a Python iterable over a dataset. Libraries in -PyTorch offer built-in high-quality datasets for you to use in -`torch.utils.data.Dataset `__. -These datasets are currently available in: - -* `torchvision `__ -* `torchaudio `__ -* `torchtext `__ - -with more to come. -Using the Yesno dataset from ``torchaudio.datasets.YESNO``, we will -demonstrate how to effectively and efficiently load data from a PyTorch -``Dataset`` into a PyTorch ``DataLoader``. - -Setup ------ -Before we begin, we need to install ``torchaudio`` to have access to the -dataset. - -:: - - pip install torchaudio - - -""" - - - - - - - -###################################################################### -# Steps -# ----- -# -# 1. Import all necessary libraries for loading our data -# 2. Access the data in the dataset -# 3. Loading the data -# 4. Iterate over the data -# 5. [Optional] Visualize the data -# -# -# 1. Import necessary libraries for loading our data -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# For this recipe, we will use ``torch`` and ``torchaudio``. Depending on -# what built-in datasets you use, you can also install and import -# ``torchvision`` or ``torchtext``. -# - -import torch -import torchaudio - - -###################################################################### -# 2. Access the data in the dataset -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# The Yesno dataset in ``torchaudio`` features sixty recordings of one -# individual saying yes or no in Hebrew; with each recording being eight -# words long (`read more here `__). -# -# ``torchaudio.datasets.YESNO`` creates a dataset for YesNo. -# -# :: -# -# torchaudio.datasets.YESNO( -# root, -# url='http://www.openslr.org/resources/1/waves_yesno.tar.gz', -# folder_in_archive='waves_yesno', -# download=False, -# transform=None, -# target_transform=None) -# -# Each item in the dataset is a tuple of the form: (waveform, sample_rate, -# labels). -# -# You must set a ``root`` for the Yesno dataset, which is where the -# training and testing dataset will exist. The other parameters are -# optional, with their default values shown. Here is some additional -# useful info on the other parameters: - -# * ``download``: If true, downloads the dataset from the internet and puts it in root directory. If dataset is already downloaded, it is not downloaded again. -# * ``transform``: Using transforms on your data allows you to take it from its source state and transform it into data that’s joined together, de-normalized, and ready for training. Each library in PyTorch supports a growing list of transformations. -# * ``target_transform``: A function/transform that takes in the target and transforms it. -# -# Let’s access our Yesno data: -# - -# A data point in Yesno is a tuple (waveform, sample_rate, labels) where labels -# is a list of integers with 1 for yes and 0 for no. -yesno_data_trainset = torchaudio.datasets.YESNO('./', download=True) - -# Pick data point number 3 to see an example of the the yesno_data: -n = 3 -waveform, sample_rate, labels = yesno_data[n] -print("Waveform: {}\nSample rate: {}\nLabels: {}".format(waveform, sample_rate, labels)) - - -###################################################################### -# When using this data in practice, it is best practice to provision the -# data into a “training” dataset and a “testing” dataset. This ensures -# that you have out-of-sample data to test the performance of your model. -# -# 3. Loading the data -# ~~~~~~~~~~~~~~~~~~~~~~~ -# -# Now that we have access to the dataset, we must pass it through -# ``torch.utils.data.DataLoader``. The ``DataLoader`` combines the dataset -# and a sampler, returning an iterable over the dataset. -# - -data_loader = torch.utils.data.DataLoader(yesno_data, - batch_size=1, - shuffle=True) - - -###################################################################### -# 4. Iterate over the data -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# Our data is now iterable using the ``data_loader``. This will be -# necessary when we begin training our model! You will notice that now -# each data entry in the ``data_loader`` object is converted to a tensor -# containing tensors representing our waveform, sample rate, and labels. -# - -for data in data_loader: - print("Data: ", data) - print("Waveform: {}\nSample rate: {}\nLabels: {}".format(data[0], data[1], data[2])) - break - - -###################################################################### -# 5. [Optional] Visualize the data -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# You can optionally visualize your data to further understand the output -# from your ``DataLoader``. -# - -import matplotlib.pyplot as plt - -print(data[0][0].numpy()) - -plt.figure() -plt.plot(waveform.t().numpy()) - - -###################################################################### -# Congratulations! You have successfully loaded data in PyTorch. -# -# Learn More -# ---------- -# -# Take a look at these other recipes to continue your learning: -# -# - `Defining a Neural Network `__ -# - `What is a state_dict in PyTorch `__ diff --git a/recipes/recipes/loading_data_recipe.rst b/recipes/recipes/loading_data_recipe.rst deleted file mode 100644 index e2d70679b68..00000000000 --- a/recipes/recipes/loading_data_recipe.rst +++ /dev/null @@ -1,221 +0,0 @@ -.. note:: - :class: sphx-glr-download-link-note - - Click :ref:`here ` to download the full example code -.. rst-class:: sphx-glr-example-title - -.. _sphx_glr_recipes_recipes_loading_data_recipe.py: - - -Loading data in PyTorch -======================= -PyTorch features extensive neural network building blocks with a simple, -intuitive, and stable API. PyTorch includes packages to prepare and load -common datasets for your model. - -Introduction ------------- -At the heart of PyTorch data loading utility is the -`torch.utils.data.DataLoader `__ -class. It represents a Python iterable over a dataset. Libraries in -PyTorch offer built-in high-quality datasets for you to use in -`torch.utils.data.Dataset `__. -These datasets are currently available in: - -* `torchvision `__ -* `torchaudio `__ -* `torchtext `__ - -with more to come. -Using the Yesno dataset from ``torchaudio.datasets.YESNO``, we will -demonstrate how to effectively and efficiently load data from a PyTorch -``Dataset`` into a PyTorch ``DataLoader``. - -Setup ------ -Before we begin, we need to install ``torchaudio`` to have access to the -dataset. - -:: - - pip install torchaudio -Steps ------ - -1. Import all necessary libraries for loading our data -2. Access the data in the dataset -3. Loading the data -4. Iterate over the data -5. [Optional] Visualize the data - - -1. Import necessary libraries for loading our data -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -For this recipe, we will use ``torch`` and ``torchaudio``. Depending on -what built-in datasets you use, you can also install and import -``torchvision`` or ``torchtext``. - - - -.. code-block:: default - - - import torch - import torchaudio - - - -2. Access the data in the dataset -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The Yesno dataset in ``torchaudio`` features sixty recordings of one -individual saying yes or no in Hebrew; with each recording being eight -words long (`read more here `__). - -``torchaudio.datasets.YESNO`` creates a dataset for YesNo. - -:: - - torchaudio.datasets.YESNO( - root, - url='http://www.openslr.org/resources/1/waves_yesno.tar.gz', - folder_in_archive='waves_yesno', - download=False, - transform=None, - target_transform=None) - -Each item in the dataset is a tuple of the form: (waveform, sample_rate, -labels). - -You must set a ``root`` for the Yesno dataset, which is where the -training and testing dataset will exist. The other parameters are -optional, with their default values shown. Here is some additional -useful info on the other parameters: - - -.. code-block:: default - - - # * ``download``: If true, downloads the dataset from the internet and puts it in root directory. If dataset is already downloaded, it is not downloaded again. - # * ``transform``: Using transforms on your data allows you to take it from its source state and transform it into data that’s joined together, de-normalized, and ready for training. Each library in PyTorch supports a growing list of transformations. - # * ``target_transform``: A function/transform that takes in the target and transforms it. - # - # Let’s access our Yesno data: - # - - # A data point in Yesno is a tuple (waveform, sample_rate, labels) where labels - # is a list of integers with 1 for yes and 0 for no. - yesno_data_trainset = torchaudio.datasets.YESNO('./', download=True) - - # Pick data point number 3 to see an example of the the yesno_data: - n = 3 - waveform, sample_rate, labels = yesno_data[n] - print("Waveform: {}\nSample rate: {}\nLabels: {}".format(waveform, sample_rate, labels)) - - - -When using this data in practice, it is best practice to provision the -data into a “training” dataset and a “testing” dataset. This ensures -that you have out-of-sample data to test the performance of your model. - -3. Loading the data -~~~~~~~~~~~~~~~~~~~~~~~ - -Now that we have access to the dataset, we must pass it through -``torch.utils.data.DataLoader``. The ``DataLoader`` combines the dataset -and a sampler, returning an iterable over the dataset. - - - -.. code-block:: default - - - data_loader = torch.utils.data.DataLoader(yesno_data, - batch_size=1, - shuffle=True) - - - -4. Iterate over the data -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Our data is now iterable using the ``data_loader``. This will be -necessary when we begin training our model! You will notice that now -each data entry in the ``data_loader`` object is converted to a tensor -containing tensors representing our waveform, sample rate, and labels. - - - -.. code-block:: default - - - for data in data_loader: - print("Data: ", data) - print("Waveform: {}\nSample rate: {}\nLabels: {}".format(data[0], data[1], data[2])) - break - - - -5. [Optional] Visualize the data -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -You can optionally visualize your data to further understand the output -from your ``DataLoader``. - - - -.. code-block:: default - - - import matplotlib.pyplot as plt - - print(data[0][0].numpy()) - - plt.figure() - plt.plot(waveform.t().numpy()) - - - -Congratulations! You have successfully loaded data in PyTorch. - -Learn More ----------- - -Take a look at these other recipes to continue your learning: - -- `Defining a Neural Network `__ -- `What is a state_dict in PyTorch `__ - - -.. rst-class:: sphx-glr-timing - - **Total running time of the script:** ( 0 minutes 0.000 seconds) - - -.. _sphx_glr_download_recipes_recipes_loading_data_recipe.py: - - -.. only :: html - - .. container:: sphx-glr-footer - :class: sphx-glr-footer-example - - - - .. container:: sphx-glr-download - - :download:`Download Python source code: loading_data_recipe.py ` - - - - .. container:: sphx-glr-download - - :download:`Download Jupyter notebook: loading_data_recipe.ipynb ` - - -.. only:: html - - .. rst-class:: sphx-glr-signature - - `Gallery generated by Sphinx-Gallery `_ diff --git a/recipes/recipes/save_load_across_devices.ipynb b/recipes/recipes/save_load_across_devices.ipynb deleted file mode 100644 index 9ad519d95f5..00000000000 --- a/recipes/recipes/save_load_across_devices.ipynb +++ /dev/null @@ -1,158 +0,0 @@ -{ - "cells": [ - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "%matplotlib inline" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\nSaving and loading models across devices in PyTorch\n===================================================\n\nThere may be instances where you want to save and load your neural\nnetworks across different devices.\n\nIntroduction\n------------\n\nSaving and loading models across devices is relatively straightforward\nusing PyTorch. In this recipe, we will experiment with saving and\nloading models across CPUs and GPUs.\n\nSetup\n-----\n\nIn order for every code block to run properly in this recipe, you must\nfirst change the runtime to \u201cGPU\u201d or higher. Once you do, we need to\ninstall ``torch`` if it isn\u2019t already available.\n\n::\n\n pip install torch\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Steps\n-----\n\n1. Import all necessary libraries for loading our data\n2. Define and intialize the neural network\n3. Save on a GPU, load on a CPU\n4. Save on a GPU, load on a GPU\n5. Save on a CPU, load on a GPU\n6. Saving and loading ``DataParallel`` models\n\n1. Import necessary libraries for loading our data\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nFor this recipe, we will use ``torch`` and its subsidiaries ``torch.nn``\nand ``torch.optim``.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "import torch\nimport torch.nn as nn\nimport torch.optim as optim" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "2. Define and intialize the neural network\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nFor sake of example, we will create a neural network for training\nimages. To learn more see the Defining a Neural Network recipe.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "class Net(nn.Module):\n def __init__(self):\n super(Net, self).__init__()\n self.conv1 = nn.Conv2d(3, 6, 5)\n self.pool = nn.MaxPool2d(2, 2)\n self.conv2 = nn.Conv2d(6, 16, 5)\n self.fc1 = nn.Linear(16 * 5 * 5, 120)\n self.fc2 = nn.Linear(120, 84)\n self.fc3 = nn.Linear(84, 10)\n\n def forward(self, x):\n x = self.pool(F.relu(self.conv1(x)))\n x = self.pool(F.relu(self.conv2(x)))\n x = x.view(-1, 16 * 5 * 5)\n x = F.relu(self.fc1(x))\n x = F.relu(self.fc2(x))\n x = self.fc3(x)\n return x\n\nnet = Net()\nprint(net)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "3. Save on GPU, Load on CPU\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nWhen loading a model on a CPU that was trained with a GPU, pass\n``torch.device('cpu')`` to the ``map_location`` argument in the\n``torch.load()`` function.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "# Specify a path to save to\nPATH = \"model.pt\"\n\n# Save\ntorch.save(net.state_dict(), PATH)\n\n# Load\ndevice = torch.device('cpu')\nmodel = Net()\nmodel.load_state_dict(torch.load(PATH, map_location=device))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In this case, the storages underlying the tensors are dynamically\nremapped to the CPU device using the ``map_location`` argument.\n\n4. Save on GPU, Load on GPU\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nWhen loading a model on a GPU that was trained and saved on GPU, simply\nconvert the initialized model to a CUDA optimized model using\n``model.to(torch.device('cuda'))``.\n\nBe sure to use the ``.to(torch.device('cuda'))`` function on all model\ninputs to prepare the data for the model.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "# Save\ntorch.save(net.state_dict(), PATH)\n\n# Load\ndevice = torch.device(\"cuda\")\nmodel = Net()\nmodel.load_state_dict(torch.load(PATH))\nmodel.to(device)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Note that calling ``my_tensor.to(device)`` returns a new copy of\n``my_tensor`` on GPU. It does NOT overwrite ``my_tensor``. Therefore,\nremember to manually overwrite tensors:\n``my_tensor = my_tensor.to(torch.device('cuda'))``.\n\n5. Save on CPU, Load on GPU\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nWhen loading a model on a GPU that was trained and saved on CPU, set the\n``map_location`` argument in the ``torch.load()`` function to\n``cuda:device_id``. This loads the model to a given GPU device.\n\nBe sure to call ``model.to(torch.device('cuda'))`` to convert the\nmodel\u2019s parameter tensors to CUDA tensors.\n\nFinally, also be sure to use the ``.to(torch.device('cuda'))`` function\non all model inputs to prepare the data for the CUDA optimized model.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "# Save\ntorch.save(net.state_dict(), PATH)\n\n# Load\ndevice = torch.device(\"cuda\")\nmodel = Net()\n# Choose whatever GPU device number you want\nmodel.load_state_dict(torch.load(PATH, map_location=\"cuda:0\"))\n# Make sure to call input = input.to(device) on any input tensors that you feed to the model\nmodel.to(device)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "6. Saving ``torch.nn.DataParallel`` Models\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n``torch.nn.DataParallel`` is a model wrapper that enables parallel GPU\nutilization.\n\nTo save a ``DataParallel`` model generically, save the\n``model.module.state_dict()``. This way, you have the flexibility to\nload the model any way you want to any device you want.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "# Save\ntorch.save(net.module.state_dict(), PATH)\n\n# Load to whatever device you want" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Congratulations! You have successfully saved and loaded models across\ndevices in PyTorch.\n\nLearn More\n----------\n\nTake a look at these other recipes to continue your learning:\n\n- TBD\n- TBD\n\n\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.4" - } - }, - "nbformat": 4, - "nbformat_minor": 0 -} \ No newline at end of file diff --git a/recipes/recipes/save_load_across_devices.py b/recipes/recipes/save_load_across_devices.py deleted file mode 100644 index c2d86fbab50..00000000000 --- a/recipes/recipes/save_load_across_devices.py +++ /dev/null @@ -1,190 +0,0 @@ -""" -Saving and loading models across devices in PyTorch -=================================================== - -There may be instances where you want to save and load your neural -networks across different devices. - -Introduction ------------- - -Saving and loading models across devices is relatively straightforward -using PyTorch. In this recipe, we will experiment with saving and -loading models across CPUs and GPUs. - -Setup ------ - -In order for every code block to run properly in this recipe, you must -first change the runtime to “GPU” or higher. Once you do, we need to -install ``torch`` if it isn’t already available. - -:: - - pip install torch - -""" - - -###################################################################### -# Steps -# ----- -# -# 1. Import all necessary libraries for loading our data -# 2. Define and intialize the neural network -# 3. Save on a GPU, load on a CPU -# 4. Save on a GPU, load on a GPU -# 5. Save on a CPU, load on a GPU -# 6. Saving and loading ``DataParallel`` models -# -# 1. Import necessary libraries for loading our data -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# For this recipe, we will use ``torch`` and its subsidiaries ``torch.nn`` -# and ``torch.optim``. -# - -import torch -import torch.nn as nn -import torch.optim as optim - - -###################################################################### -# 2. Define and intialize the neural network -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# For sake of example, we will create a neural network for training -# images. To learn more see the Defining a Neural Network recipe. -# - -class Net(nn.Module): - def __init__(self): - super(Net, self).__init__() - self.conv1 = nn.Conv2d(3, 6, 5) - self.pool = nn.MaxPool2d(2, 2) - self.conv2 = nn.Conv2d(6, 16, 5) - self.fc1 = nn.Linear(16 * 5 * 5, 120) - self.fc2 = nn.Linear(120, 84) - self.fc3 = nn.Linear(84, 10) - - def forward(self, x): - x = self.pool(F.relu(self.conv1(x))) - x = self.pool(F.relu(self.conv2(x))) - x = x.view(-1, 16 * 5 * 5) - x = F.relu(self.fc1(x)) - x = F.relu(self.fc2(x)) - x = self.fc3(x) - return x - -net = Net() -print(net) - - -###################################################################### -# 3. Save on GPU, Load on CPU -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# When loading a model on a CPU that was trained with a GPU, pass -# ``torch.device('cpu')`` to the ``map_location`` argument in the -# ``torch.load()`` function. -# - -# Specify a path to save to -PATH = "model.pt" - -# Save -torch.save(net.state_dict(), PATH) - -# Load -device = torch.device('cpu') -model = Net() -model.load_state_dict(torch.load(PATH, map_location=device)) - - -###################################################################### -# In this case, the storages underlying the tensors are dynamically -# remapped to the CPU device using the ``map_location`` argument. -# -# 4. Save on GPU, Load on GPU -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# When loading a model on a GPU that was trained and saved on GPU, simply -# convert the initialized model to a CUDA optimized model using -# ``model.to(torch.device('cuda'))``. -# -# Be sure to use the ``.to(torch.device('cuda'))`` function on all model -# inputs to prepare the data for the model. -# - -# Save -torch.save(net.state_dict(), PATH) - -# Load -device = torch.device("cuda") -model = Net() -model.load_state_dict(torch.load(PATH)) -model.to(device) - - -###################################################################### -# Note that calling ``my_tensor.to(device)`` returns a new copy of -# ``my_tensor`` on GPU. It does NOT overwrite ``my_tensor``. Therefore, -# remember to manually overwrite tensors: -# ``my_tensor = my_tensor.to(torch.device('cuda'))``. -# -# 5. Save on CPU, Load on GPU -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# When loading a model on a GPU that was trained and saved on CPU, set the -# ``map_location`` argument in the ``torch.load()`` function to -# ``cuda:device_id``. This loads the model to a given GPU device. -# -# Be sure to call ``model.to(torch.device('cuda'))`` to convert the -# model’s parameter tensors to CUDA tensors. -# -# Finally, also be sure to use the ``.to(torch.device('cuda'))`` function -# on all model inputs to prepare the data for the CUDA optimized model. -# - -# Save -torch.save(net.state_dict(), PATH) - -# Load -device = torch.device("cuda") -model = Net() -# Choose whatever GPU device number you want -model.load_state_dict(torch.load(PATH, map_location="cuda:0")) -# Make sure to call input = input.to(device) on any input tensors that you feed to the model -model.to(device) - - -###################################################################### -# 6. Saving ``torch.nn.DataParallel`` Models -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# ``torch.nn.DataParallel`` is a model wrapper that enables parallel GPU -# utilization. -# -# To save a ``DataParallel`` model generically, save the -# ``model.module.state_dict()``. This way, you have the flexibility to -# load the model any way you want to any device you want. -# - -# Save -torch.save(net.module.state_dict(), PATH) - -# Load to whatever device you want - - -###################################################################### -# Congratulations! You have successfully saved and loaded models across -# devices in PyTorch. -# -# Learn More -# ---------- -# -# Take a look at these other recipes to continue your learning: -# -# - TBD -# - TBD -# diff --git a/recipes/recipes/save_load_across_devices.rst b/recipes/recipes/save_load_across_devices.rst deleted file mode 100644 index 30f23e353e7..00000000000 --- a/recipes/recipes/save_load_across_devices.rst +++ /dev/null @@ -1,250 +0,0 @@ -.. note:: - :class: sphx-glr-download-link-note - - Click :ref:`here ` to download the full example code -.. rst-class:: sphx-glr-example-title - -.. _sphx_glr_recipes_recipes_save_load_across_devices.py: - - -Saving and loading models across devices in PyTorch -=================================================== - -There may be instances where you want to save and load your neural -networks across different devices. - -Introduction ------------- - -Saving and loading models across devices is relatively straightforward -using PyTorch. In this recipe, we will experiment with saving and -loading models across CPUs and GPUs. - -Setup ------ - -In order for every code block to run properly in this recipe, you must -first change the runtime to “GPU” or higher. Once you do, we need to -install ``torch`` if it isn’t already available. - -:: - - pip install torch -Steps ------ - -1. Import all necessary libraries for loading our data -2. Define and intialize the neural network -3. Save on a GPU, load on a CPU -4. Save on a GPU, load on a GPU -5. Save on a CPU, load on a GPU -6. Saving and loading ``DataParallel`` models - -1. Import necessary libraries for loading our data -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -For this recipe, we will use ``torch`` and its subsidiaries ``torch.nn`` -and ``torch.optim``. - - - -.. code-block:: default - - - import torch - import torch.nn as nn - import torch.optim as optim - - - -2. Define and intialize the neural network -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -For sake of example, we will create a neural network for training -images. To learn more see the Defining a Neural Network recipe. - - - -.. code-block:: default - - - class Net(nn.Module): - def __init__(self): - super(Net, self).__init__() - self.conv1 = nn.Conv2d(3, 6, 5) - self.pool = nn.MaxPool2d(2, 2) - self.conv2 = nn.Conv2d(6, 16, 5) - self.fc1 = nn.Linear(16 * 5 * 5, 120) - self.fc2 = nn.Linear(120, 84) - self.fc3 = nn.Linear(84, 10) - - def forward(self, x): - x = self.pool(F.relu(self.conv1(x))) - x = self.pool(F.relu(self.conv2(x))) - x = x.view(-1, 16 * 5 * 5) - x = F.relu(self.fc1(x)) - x = F.relu(self.fc2(x)) - x = self.fc3(x) - return x - - net = Net() - print(net) - - - -3. Save on GPU, Load on CPU -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -When loading a model on a CPU that was trained with a GPU, pass -``torch.device('cpu')`` to the ``map_location`` argument in the -``torch.load()`` function. - - - -.. code-block:: default - - - # Specify a path to save to - PATH = "model.pt" - - # Save - torch.save(net.state_dict(), PATH) - - # Load - device = torch.device('cpu') - model = Net() - model.load_state_dict(torch.load(PATH, map_location=device)) - - - -In this case, the storages underlying the tensors are dynamically -remapped to the CPU device using the ``map_location`` argument. - -4. Save on GPU, Load on GPU -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -When loading a model on a GPU that was trained and saved on GPU, simply -convert the initialized model to a CUDA optimized model using -``model.to(torch.device('cuda'))``. - -Be sure to use the ``.to(torch.device('cuda'))`` function on all model -inputs to prepare the data for the model. - - - -.. code-block:: default - - - # Save - torch.save(net.state_dict(), PATH) - - # Load - device = torch.device("cuda") - model = Net() - model.load_state_dict(torch.load(PATH)) - model.to(device) - - - -Note that calling ``my_tensor.to(device)`` returns a new copy of -``my_tensor`` on GPU. It does NOT overwrite ``my_tensor``. Therefore, -remember to manually overwrite tensors: -``my_tensor = my_tensor.to(torch.device('cuda'))``. - -5. Save on CPU, Load on GPU -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -When loading a model on a GPU that was trained and saved on CPU, set the -``map_location`` argument in the ``torch.load()`` function to -``cuda:device_id``. This loads the model to a given GPU device. - -Be sure to call ``model.to(torch.device('cuda'))`` to convert the -model’s parameter tensors to CUDA tensors. - -Finally, also be sure to use the ``.to(torch.device('cuda'))`` function -on all model inputs to prepare the data for the CUDA optimized model. - - - -.. code-block:: default - - - # Save - torch.save(net.state_dict(), PATH) - - # Load - device = torch.device("cuda") - model = Net() - # Choose whatever GPU device number you want - model.load_state_dict(torch.load(PATH, map_location="cuda:0")) - # Make sure to call input = input.to(device) on any input tensors that you feed to the model - model.to(device) - - - -6. Saving ``torch.nn.DataParallel`` Models -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -``torch.nn.DataParallel`` is a model wrapper that enables parallel GPU -utilization. - -To save a ``DataParallel`` model generically, save the -``model.module.state_dict()``. This way, you have the flexibility to -load the model any way you want to any device you want. - - - -.. code-block:: default - - - # Save - torch.save(net.module.state_dict(), PATH) - - # Load to whatever device you want - - - -Congratulations! You have successfully saved and loaded models across -devices in PyTorch. - -Learn More ----------- - -Take a look at these other recipes to continue your learning: - -- TBD -- TBD - - - -.. rst-class:: sphx-glr-timing - - **Total running time of the script:** ( 0 minutes 0.000 seconds) - - -.. _sphx_glr_download_recipes_recipes_save_load_across_devices.py: - - -.. only :: html - - .. container:: sphx-glr-footer - :class: sphx-glr-footer-example - - - - .. container:: sphx-glr-download - - :download:`Download Python source code: save_load_across_devices.py ` - - - - .. container:: sphx-glr-download - - :download:`Download Jupyter notebook: save_load_across_devices.ipynb ` - - -.. only:: html - - .. rst-class:: sphx-glr-signature - - `Gallery generated by Sphinx-Gallery `_ diff --git a/recipes/recipes/saving_and_loading_a_general_checkpoint.ipynb b/recipes/recipes/saving_and_loading_a_general_checkpoint.ipynb deleted file mode 100644 index 0952465a851..00000000000 --- a/recipes/recipes/saving_and_loading_a_general_checkpoint.ipynb +++ /dev/null @@ -1,140 +0,0 @@ -{ - "cells": [ - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "%matplotlib inline" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\nSaving and loading a general checkpoint in PyTorch\n==================================================\nSaving and loading a general checkpoint model for inference or \nresuming training can be helpful for picking up where you last left off.\nWhen saving a general checkpoint, you must save more than just the\nmodel\u2019s state_dict. It is important to also save the optimizer\u2019s\nstate_dict, as this contains buffers and parameters that are updated as\nthe model trains. Other items that you may want to save are the epoch\nyou left off on, the latest recorded training loss, external\n``torch.nn.Embedding`` layers, and more, based on your own algorithm.\n\nIntroduction\n------------\nTo save multiple checkpoints, you must organize them in a dictionary and\nuse ``torch.save()`` to serialize the dictionary. A common PyTorch\nconvention is to save these checkpoints using the ``.tar`` file\nextension. To load the items, first initialize the model and optimizer,\nthen load the dictionary locally using torch.load(). From here, you can\neasily access the saved items by simply querying the dictionary as you\nwould expect.\n\nIn this recipe, we will explore how to save and load multiple\ncheckpoints.\n\nSetup\n-----\nBefore we begin, we need to install ``torch`` if it isn\u2019t already\navailable.\n\n::\n\n pip install torch\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Steps\n-----\n\n1. Import all necessary libraries for loading our data\n2. Define and intialize the neural network\n3. Initialize the optimizer\n4. Save the general checkpoint\n5. Load the general checkpoint\n\n1. Import necessary libraries for loading our data\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nFor this recipe, we will use ``torch`` and its subsidiaries ``torch.nn``\nand ``torch.optim``.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "import torch\nimport torch.nn as nn\nimport torch.optim as optim" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "2. Define and intialize the neural network\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nFor sake of example, we will create a neural network for training\nimages. To learn more see the Defining a Neural Network recipe.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "class Net(nn.Module):\n def __init__(self):\n super(Net, self).__init__()\n self.conv1 = nn.Conv2d(3, 6, 5)\n self.pool = nn.MaxPool2d(2, 2)\n self.conv2 = nn.Conv2d(6, 16, 5)\n self.fc1 = nn.Linear(16 * 5 * 5, 120)\n self.fc2 = nn.Linear(120, 84)\n self.fc3 = nn.Linear(84, 10)\n\n def forward(self, x):\n x = self.pool(F.relu(self.conv1(x)))\n x = self.pool(F.relu(self.conv2(x)))\n x = x.view(-1, 16 * 5 * 5)\n x = F.relu(self.fc1(x))\n x = F.relu(self.fc2(x))\n x = self.fc3(x)\n return x\n\nnet = Net()\nprint(net)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "3. Initialize the optimizer\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nWe will use SGD with momentum.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "4. Save the general checkpoint\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nCollect all relevant information and build your dictionary.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "# Additional information\nEPOCH = 5\nPATH = \"model.pt\"\nLOSS = 0.4\n\ntorch.save({\n 'epoch': EPOCH,\n 'model_state_dict': net.state_dict(),\n 'optimizer_state_dict': optimizer.state_dict(),\n 'loss': LOSS,\n }, PATH)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "5. Load the general checkpoint\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nRemember to first initialize the model and optimizer, then load the\ndictionary locally.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "model = Net()\noptimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)\n\ncheckpoint = torch.load(PATH)\nmodel.load_state_dict(checkpoint['model_state_dict'])\noptimizer.load_state_dict(checkpoint['optimizer_state_dict'])\nepoch = checkpoint['epoch']\nloss = checkpoint['loss']\n\nmodel.eval()\n# - or -\nmodel.train()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You must call ``model.eval()`` to set dropout and batch normalization\nlayers to evaluation mode before running inference. Failing to do this\nwill yield inconsistent inference results.\n\nIf you wish to resuming training, call ``model.train()`` to ensure these\nlayers are in training mode.\n\nCongratulations! You have successfully saved and loaded a general\ncheckpoint for inference and/or resuming training in PyTorch.\n\nLearn More\n----------\n\nTake a look at these other recipes to continue your learning:\n\n- TBD\n- TBD\n\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.4" - } - }, - "nbformat": 4, - "nbformat_minor": 0 -} \ No newline at end of file diff --git a/recipes/recipes/saving_and_loading_a_general_checkpoint.py b/recipes/recipes/saving_and_loading_a_general_checkpoint.py deleted file mode 100644 index 6e0c490ec2a..00000000000 --- a/recipes/recipes/saving_and_loading_a_general_checkpoint.py +++ /dev/null @@ -1,162 +0,0 @@ -""" -Saving and loading a general checkpoint in PyTorch -================================================== -Saving and loading a general checkpoint model for inference or -resuming training can be helpful for picking up where you last left off. -When saving a general checkpoint, you must save more than just the -model’s state_dict. It is important to also save the optimizer’s -state_dict, as this contains buffers and parameters that are updated as -the model trains. Other items that you may want to save are the epoch -you left off on, the latest recorded training loss, external -``torch.nn.Embedding`` layers, and more, based on your own algorithm. - -Introduction ------------- -To save multiple checkpoints, you must organize them in a dictionary and -use ``torch.save()`` to serialize the dictionary. A common PyTorch -convention is to save these checkpoints using the ``.tar`` file -extension. To load the items, first initialize the model and optimizer, -then load the dictionary locally using torch.load(). From here, you can -easily access the saved items by simply querying the dictionary as you -would expect. - -In this recipe, we will explore how to save and load multiple -checkpoints. - -Setup ------ -Before we begin, we need to install ``torch`` if it isn’t already -available. - -:: - - pip install torch - - -""" - - - -###################################################################### -# Steps -# ----- -# -# 1. Import all necessary libraries for loading our data -# 2. Define and intialize the neural network -# 3. Initialize the optimizer -# 4. Save the general checkpoint -# 5. Load the general checkpoint -# -# 1. Import necessary libraries for loading our data -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# For this recipe, we will use ``torch`` and its subsidiaries ``torch.nn`` -# and ``torch.optim``. -# - -import torch -import torch.nn as nn -import torch.optim as optim - - -###################################################################### -# 2. Define and intialize the neural network -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# For sake of example, we will create a neural network for training -# images. To learn more see the Defining a Neural Network recipe. -# - -class Net(nn.Module): - def __init__(self): - super(Net, self).__init__() - self.conv1 = nn.Conv2d(3, 6, 5) - self.pool = nn.MaxPool2d(2, 2) - self.conv2 = nn.Conv2d(6, 16, 5) - self.fc1 = nn.Linear(16 * 5 * 5, 120) - self.fc2 = nn.Linear(120, 84) - self.fc3 = nn.Linear(84, 10) - - def forward(self, x): - x = self.pool(F.relu(self.conv1(x))) - x = self.pool(F.relu(self.conv2(x))) - x = x.view(-1, 16 * 5 * 5) - x = F.relu(self.fc1(x)) - x = F.relu(self.fc2(x)) - x = self.fc3(x) - return x - -net = Net() -print(net) - - -###################################################################### -# 3. Initialize the optimizer -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# We will use SGD with momentum. -# - -optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9) - - -###################################################################### -# 4. Save the general checkpoint -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# Collect all relevant information and build your dictionary. -# - -# Additional information -EPOCH = 5 -PATH = "model.pt" -LOSS = 0.4 - -torch.save({ - 'epoch': EPOCH, - 'model_state_dict': net.state_dict(), - 'optimizer_state_dict': optimizer.state_dict(), - 'loss': LOSS, - }, PATH) - - -###################################################################### -# 5. Load the general checkpoint -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# Remember to first initialize the model and optimizer, then load the -# dictionary locally. -# - -model = Net() -optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9) - -checkpoint = torch.load(PATH) -model.load_state_dict(checkpoint['model_state_dict']) -optimizer.load_state_dict(checkpoint['optimizer_state_dict']) -epoch = checkpoint['epoch'] -loss = checkpoint['loss'] - -model.eval() -# - or - -model.train() - - -###################################################################### -# You must call ``model.eval()`` to set dropout and batch normalization -# layers to evaluation mode before running inference. Failing to do this -# will yield inconsistent inference results. -# -# If you wish to resuming training, call ``model.train()`` to ensure these -# layers are in training mode. -# -# Congratulations! You have successfully saved and loaded a general -# checkpoint for inference and/or resuming training in PyTorch. -# -# Learn More -# ---------- -# -# Take a look at these other recipes to continue your learning: -# -# - TBD -# - TBD diff --git a/recipes/recipes/saving_and_loading_a_general_checkpoint.rst b/recipes/recipes/saving_and_loading_a_general_checkpoint.rst deleted file mode 100644 index 2669f620781..00000000000 --- a/recipes/recipes/saving_and_loading_a_general_checkpoint.rst +++ /dev/null @@ -1,216 +0,0 @@ -.. note:: - :class: sphx-glr-download-link-note - - Click :ref:`here ` to download the full example code -.. rst-class:: sphx-glr-example-title - -.. _sphx_glr_recipes_recipes_saving_and_loading_a_general_checkpoint.py: - - -Saving and loading a general checkpoint in PyTorch -================================================== -Saving and loading a general checkpoint model for inference or -resuming training can be helpful for picking up where you last left off. -When saving a general checkpoint, you must save more than just the -model’s state_dict. It is important to also save the optimizer’s -state_dict, as this contains buffers and parameters that are updated as -the model trains. Other items that you may want to save are the epoch -you left off on, the latest recorded training loss, external -``torch.nn.Embedding`` layers, and more, based on your own algorithm. - -Introduction ------------- -To save multiple checkpoints, you must organize them in a dictionary and -use ``torch.save()`` to serialize the dictionary. A common PyTorch -convention is to save these checkpoints using the ``.tar`` file -extension. To load the items, first initialize the model and optimizer, -then load the dictionary locally using torch.load(). From here, you can -easily access the saved items by simply querying the dictionary as you -would expect. - -In this recipe, we will explore how to save and load multiple -checkpoints. - -Setup ------ -Before we begin, we need to install ``torch`` if it isn’t already -available. - -:: - - pip install torch -Steps ------ - -1. Import all necessary libraries for loading our data -2. Define and intialize the neural network -3. Initialize the optimizer -4. Save the general checkpoint -5. Load the general checkpoint - -1. Import necessary libraries for loading our data -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -For this recipe, we will use ``torch`` and its subsidiaries ``torch.nn`` -and ``torch.optim``. - - - -.. code-block:: default - - - import torch - import torch.nn as nn - import torch.optim as optim - - - -2. Define and intialize the neural network -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -For sake of example, we will create a neural network for training -images. To learn more see the Defining a Neural Network recipe. - - - -.. code-block:: default - - - class Net(nn.Module): - def __init__(self): - super(Net, self).__init__() - self.conv1 = nn.Conv2d(3, 6, 5) - self.pool = nn.MaxPool2d(2, 2) - self.conv2 = nn.Conv2d(6, 16, 5) - self.fc1 = nn.Linear(16 * 5 * 5, 120) - self.fc2 = nn.Linear(120, 84) - self.fc3 = nn.Linear(84, 10) - - def forward(self, x): - x = self.pool(F.relu(self.conv1(x))) - x = self.pool(F.relu(self.conv2(x))) - x = x.view(-1, 16 * 5 * 5) - x = F.relu(self.fc1(x)) - x = F.relu(self.fc2(x)) - x = self.fc3(x) - return x - - net = Net() - print(net) - - - -3. Initialize the optimizer -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -We will use SGD with momentum. - - - -.. code-block:: default - - - optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9) - - - -4. Save the general checkpoint -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Collect all relevant information and build your dictionary. - - - -.. code-block:: default - - - # Additional information - EPOCH = 5 - PATH = "model.pt" - LOSS = 0.4 - - torch.save({ - 'epoch': EPOCH, - 'model_state_dict': net.state_dict(), - 'optimizer_state_dict': optimizer.state_dict(), - 'loss': LOSS, - }, PATH) - - - -5. Load the general checkpoint -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Remember to first initialize the model and optimizer, then load the -dictionary locally. - - - -.. code-block:: default - - - model = Net() - optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9) - - checkpoint = torch.load(PATH) - model.load_state_dict(checkpoint['model_state_dict']) - optimizer.load_state_dict(checkpoint['optimizer_state_dict']) - epoch = checkpoint['epoch'] - loss = checkpoint['loss'] - - model.eval() - # - or - - model.train() - - - -You must call ``model.eval()`` to set dropout and batch normalization -layers to evaluation mode before running inference. Failing to do this -will yield inconsistent inference results. - -If you wish to resuming training, call ``model.train()`` to ensure these -layers are in training mode. - -Congratulations! You have successfully saved and loaded a general -checkpoint for inference and/or resuming training in PyTorch. - -Learn More ----------- - -Take a look at these other recipes to continue your learning: - -- TBD -- TBD - - -.. rst-class:: sphx-glr-timing - - **Total running time of the script:** ( 0 minutes 0.000 seconds) - - -.. _sphx_glr_download_recipes_recipes_saving_and_loading_a_general_checkpoint.py: - - -.. only :: html - - .. container:: sphx-glr-footer - :class: sphx-glr-footer-example - - - - .. container:: sphx-glr-download - - :download:`Download Python source code: saving_and_loading_a_general_checkpoint.py ` - - - - .. container:: sphx-glr-download - - :download:`Download Jupyter notebook: saving_and_loading_a_general_checkpoint.ipynb ` - - -.. only:: html - - .. rst-class:: sphx-glr-signature - - `Gallery generated by Sphinx-Gallery `_ diff --git a/recipes/recipes/saving_and_loading_models_for_inference.ipynb b/recipes/recipes/saving_and_loading_models_for_inference.ipynb deleted file mode 100644 index 7fc3fa2e3c3..00000000000 --- a/recipes/recipes/saving_and_loading_models_for_inference.ipynb +++ /dev/null @@ -1,140 +0,0 @@ -{ - "cells": [ - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "%matplotlib inline" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\nSaving and loading models for inference in PyTorch\n==================================================\nThere are two approaches for saving and loading models for inference in\nPyTorch. The first is saving and loading the ``state_dict``, and the\nsecond is saving and loading the entire model.\n\nIntroduction\n------------\nSaving the model\u2019s ``state_dict`` with the ``torch.save()`` function\nwill give you the most flexibility for restoring the model later. This\nis the recommended method for saving models, because it is only really\nnecessary to save the trained model\u2019s learned parameters.\nWhen saving and loading an entire model, you save the entire module\nusing Python\u2019s\n`pickle `__ module. Using\nthis approach yields the most intuitive syntax and involves the least\namount of code. The disadvantage of this approach is that the serialized\ndata is bound to the specific classes and the exact directory structure\nused when the model is saved. The reason for this is because pickle does\nnot save the model class itself. Rather, it saves a path to the file\ncontaining the class, which is used during load time. Because of this,\nyour code can break in various ways when used in other projects or after\nrefactors.\nIn this recipe, we will explore both ways on how to save and load models\nfor inference.\n\nSetup\n-----\nBefore we begin, we need to install ``torch`` if it isn\u2019t already\navailable.\n\n\n::\n\n pip install torch\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Steps\n-----\n\n1. Import all necessary libraries for loading our data\n2. Define and intialize the neural network\n3. Initialize the optimizer\n4. Save and load the model via ``state_dict``\n5. Save and load the entire model\n\n1. Import necessary libraries for loading our data\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nFor this recipe, we will use ``torch`` and its subsidiaries ``torch.nn``\nand ``torch.optim``.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "import torch\nimport torch.nn as nn\nimport torch.optim as optim" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "2. Define and intialize the neural network\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nFor sake of example, we will create a neural network for training\nimages. To learn more see the Defining a Neural Network recipe.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "class Net(nn.Module):\n def __init__(self):\n super(Net, self).__init__()\n self.conv1 = nn.Conv2d(3, 6, 5)\n self.pool = nn.MaxPool2d(2, 2)\n self.conv2 = nn.Conv2d(6, 16, 5)\n self.fc1 = nn.Linear(16 * 5 * 5, 120)\n self.fc2 = nn.Linear(120, 84)\n self.fc3 = nn.Linear(84, 10)\n\n def forward(self, x):\n x = self.pool(F.relu(self.conv1(x)))\n x = self.pool(F.relu(self.conv2(x)))\n x = x.view(-1, 16 * 5 * 5)\n x = F.relu(self.fc1(x))\n x = F.relu(self.fc2(x))\n x = self.fc3(x)\n return x\n\nnet = Net()\nprint(net)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "3. Initialize the optimizer\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nWe will use SGD with momentum.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "4. Save and load the model via ``state_dict``\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nLet\u2019s save and load our model using just ``state_dict``.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "# Specify a path\nPATH = \"state_dict_model.pt\"\n\n# Save\ntorch.save(net.state_dict(), PATH)\n\n# Load\nmodel = Net()\nmodel.load_state_dict(torch.load(PATH))\nmodel.eval()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "A common PyTorch convention is to save models using either a ``.pt`` or\n``.pth`` file extension.\n\nNotice that the ``load_state_dict()`` function takes a dictionary\nobject, NOT a path to a saved object. This means that you must\ndeserialize the saved state_dict before you pass it to the\n``load_state_dict()`` function. For example, you CANNOT load using\n``model.load_state_dict(PATH)``.\n\nRemember too, that you must call ``model.eval()`` to set dropout and\nbatch normalization layers to evaluation mode before running inference.\nFailing to do this will yield inconsistent inference results.\n\n5. Save and load entire model\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nNow let\u2019s try the same thing with the entire model.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "# Specify a path\nPATH = \"entire_model.pt\"\n\n# Save\ntorch.save(net, PATH)\n\n# Load\nmodel = torch.load(PATH)\nmodel.eval()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Again here, remember that you must call model.eval() to set dropout and\nbatch normalization layers to evaluation mode before running inference.\n\nCongratulations! You have successfully saved and load models for\ninference in PyTorch.\n\nLearn More\n----------\n\nTake a look at these other recipes to continue your learning:\n\n- `Saving and loading a general checkpoint in PyTorch `__\n- `Saving and loading multiple models in one file using PyTorch `__\n\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.4" - } - }, - "nbformat": 4, - "nbformat_minor": 0 -} \ No newline at end of file diff --git a/recipes/recipes/saving_and_loading_models_for_inference.py b/recipes/recipes/saving_and_loading_models_for_inference.py deleted file mode 100644 index 22ae01aec76..00000000000 --- a/recipes/recipes/saving_and_loading_models_for_inference.py +++ /dev/null @@ -1,168 +0,0 @@ -""" -Saving and loading models for inference in PyTorch -================================================== -There are two approaches for saving and loading models for inference in -PyTorch. The first is saving and loading the ``state_dict``, and the -second is saving and loading the entire model. - -Introduction ------------- -Saving the model’s ``state_dict`` with the ``torch.save()`` function -will give you the most flexibility for restoring the model later. This -is the recommended method for saving models, because it is only really -necessary to save the trained model’s learned parameters. -When saving and loading an entire model, you save the entire module -using Python’s -`pickle `__ module. Using -this approach yields the most intuitive syntax and involves the least -amount of code. The disadvantage of this approach is that the serialized -data is bound to the specific classes and the exact directory structure -used when the model is saved. The reason for this is because pickle does -not save the model class itself. Rather, it saves a path to the file -containing the class, which is used during load time. Because of this, -your code can break in various ways when used in other projects or after -refactors. -In this recipe, we will explore both ways on how to save and load models -for inference. - -Setup ------ -Before we begin, we need to install ``torch`` if it isn’t already -available. - - -:: - - pip install torch - - -""" - - -###################################################################### -# Steps -# ----- -# -# 1. Import all necessary libraries for loading our data -# 2. Define and intialize the neural network -# 3. Initialize the optimizer -# 4. Save and load the model via ``state_dict`` -# 5. Save and load the entire model -# -# 1. Import necessary libraries for loading our data -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# For this recipe, we will use ``torch`` and its subsidiaries ``torch.nn`` -# and ``torch.optim``. -# - -import torch -import torch.nn as nn -import torch.optim as optim - - -###################################################################### -# 2. Define and intialize the neural network -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# For sake of example, we will create a neural network for training -# images. To learn more see the Defining a Neural Network recipe. -# - -class Net(nn.Module): - def __init__(self): - super(Net, self).__init__() - self.conv1 = nn.Conv2d(3, 6, 5) - self.pool = nn.MaxPool2d(2, 2) - self.conv2 = nn.Conv2d(6, 16, 5) - self.fc1 = nn.Linear(16 * 5 * 5, 120) - self.fc2 = nn.Linear(120, 84) - self.fc3 = nn.Linear(84, 10) - - def forward(self, x): - x = self.pool(F.relu(self.conv1(x))) - x = self.pool(F.relu(self.conv2(x))) - x = x.view(-1, 16 * 5 * 5) - x = F.relu(self.fc1(x)) - x = F.relu(self.fc2(x)) - x = self.fc3(x) - return x - -net = Net() -print(net) - - -###################################################################### -# 3. Initialize the optimizer -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# We will use SGD with momentum. -# - -optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9) - - -###################################################################### -# 4. Save and load the model via ``state_dict`` -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# Let’s save and load our model using just ``state_dict``. -# - -# Specify a path -PATH = "state_dict_model.pt" - -# Save -torch.save(net.state_dict(), PATH) - -# Load -model = Net() -model.load_state_dict(torch.load(PATH)) -model.eval() - - -###################################################################### -# A common PyTorch convention is to save models using either a ``.pt`` or -# ``.pth`` file extension. -# -# Notice that the ``load_state_dict()`` function takes a dictionary -# object, NOT a path to a saved object. This means that you must -# deserialize the saved state_dict before you pass it to the -# ``load_state_dict()`` function. For example, you CANNOT load using -# ``model.load_state_dict(PATH)``. -# -# Remember too, that you must call ``model.eval()`` to set dropout and -# batch normalization layers to evaluation mode before running inference. -# Failing to do this will yield inconsistent inference results. -# -# 5. Save and load entire model -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# Now let’s try the same thing with the entire model. -# - -# Specify a path -PATH = "entire_model.pt" - -# Save -torch.save(net, PATH) - -# Load -model = torch.load(PATH) -model.eval() - - -###################################################################### -# Again here, remember that you must call model.eval() to set dropout and -# batch normalization layers to evaluation mode before running inference. -# -# Congratulations! You have successfully saved and load models for -# inference in PyTorch. -# -# Learn More -# ---------- -# -# Take a look at these other recipes to continue your learning: -# -# - `Saving and loading a general checkpoint in PyTorch `__ -# - `Saving and loading multiple models in one file using PyTorch `__ diff --git a/recipes/recipes/saving_and_loading_models_for_inference.rst b/recipes/recipes/saving_and_loading_models_for_inference.rst deleted file mode 100644 index 8766c3edda9..00000000000 --- a/recipes/recipes/saving_and_loading_models_for_inference.rst +++ /dev/null @@ -1,223 +0,0 @@ -.. note:: - :class: sphx-glr-download-link-note - - Click :ref:`here ` to download the full example code -.. rst-class:: sphx-glr-example-title - -.. _sphx_glr_recipes_recipes_saving_and_loading_models_for_inference.py: - - -Saving and loading models for inference in PyTorch -================================================== -There are two approaches for saving and loading models for inference in -PyTorch. The first is saving and loading the ``state_dict``, and the -second is saving and loading the entire model. - -Introduction ------------- -Saving the model’s ``state_dict`` with the ``torch.save()`` function -will give you the most flexibility for restoring the model later. This -is the recommended method for saving models, because it is only really -necessary to save the trained model’s learned parameters. -When saving and loading an entire model, you save the entire module -using Python’s -`pickle `__ module. Using -this approach yields the most intuitive syntax and involves the least -amount of code. The disadvantage of this approach is that the serialized -data is bound to the specific classes and the exact directory structure -used when the model is saved. The reason for this is because pickle does -not save the model class itself. Rather, it saves a path to the file -containing the class, which is used during load time. Because of this, -your code can break in various ways when used in other projects or after -refactors. -In this recipe, we will explore both ways on how to save and load models -for inference. - -Setup ------ -Before we begin, we need to install ``torch`` if it isn’t already -available. - - -:: - - pip install torch -Steps ------ - -1. Import all necessary libraries for loading our data -2. Define and intialize the neural network -3. Initialize the optimizer -4. Save and load the model via ``state_dict`` -5. Save and load the entire model - -1. Import necessary libraries for loading our data -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -For this recipe, we will use ``torch`` and its subsidiaries ``torch.nn`` -and ``torch.optim``. - - - -.. code-block:: default - - - import torch - import torch.nn as nn - import torch.optim as optim - - - -2. Define and intialize the neural network -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -For sake of example, we will create a neural network for training -images. To learn more see the Defining a Neural Network recipe. - - - -.. code-block:: default - - - class Net(nn.Module): - def __init__(self): - super(Net, self).__init__() - self.conv1 = nn.Conv2d(3, 6, 5) - self.pool = nn.MaxPool2d(2, 2) - self.conv2 = nn.Conv2d(6, 16, 5) - self.fc1 = nn.Linear(16 * 5 * 5, 120) - self.fc2 = nn.Linear(120, 84) - self.fc3 = nn.Linear(84, 10) - - def forward(self, x): - x = self.pool(F.relu(self.conv1(x))) - x = self.pool(F.relu(self.conv2(x))) - x = x.view(-1, 16 * 5 * 5) - x = F.relu(self.fc1(x)) - x = F.relu(self.fc2(x)) - x = self.fc3(x) - return x - - net = Net() - print(net) - - - -3. Initialize the optimizer -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -We will use SGD with momentum. - - - -.. code-block:: default - - - optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9) - - - -4. Save and load the model via ``state_dict`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Let’s save and load our model using just ``state_dict``. - - - -.. code-block:: default - - - # Specify a path - PATH = "state_dict_model.pt" - - # Save - torch.save(net.state_dict(), PATH) - - # Load - model = Net() - model.load_state_dict(torch.load(PATH)) - model.eval() - - - -A common PyTorch convention is to save models using either a ``.pt`` or -``.pth`` file extension. - -Notice that the ``load_state_dict()`` function takes a dictionary -object, NOT a path to a saved object. This means that you must -deserialize the saved state_dict before you pass it to the -``load_state_dict()`` function. For example, you CANNOT load using -``model.load_state_dict(PATH)``. - -Remember too, that you must call ``model.eval()`` to set dropout and -batch normalization layers to evaluation mode before running inference. -Failing to do this will yield inconsistent inference results. - -5. Save and load entire model -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Now let’s try the same thing with the entire model. - - - -.. code-block:: default - - - # Specify a path - PATH = "entire_model.pt" - - # Save - torch.save(net, PATH) - - # Load - model = torch.load(PATH) - model.eval() - - - -Again here, remember that you must call model.eval() to set dropout and -batch normalization layers to evaluation mode before running inference. - -Congratulations! You have successfully saved and load models for -inference in PyTorch. - -Learn More ----------- - -Take a look at these other recipes to continue your learning: - -- `Saving and loading a general checkpoint in PyTorch `__ -- `Saving and loading multiple models in one file using PyTorch `__ - - -.. rst-class:: sphx-glr-timing - - **Total running time of the script:** ( 0 minutes 0.000 seconds) - - -.. _sphx_glr_download_recipes_recipes_saving_and_loading_models_for_inference.py: - - -.. only :: html - - .. container:: sphx-glr-footer - :class: sphx-glr-footer-example - - - - .. container:: sphx-glr-download - - :download:`Download Python source code: saving_and_loading_models_for_inference.py ` - - - - .. container:: sphx-glr-download - - :download:`Download Jupyter notebook: saving_and_loading_models_for_inference.ipynb ` - - -.. only:: html - - .. rst-class:: sphx-glr-signature - - `Gallery generated by Sphinx-Gallery `_ diff --git a/recipes/recipes/saving_multiple_models_in_one_file.ipynb b/recipes/recipes/saving_multiple_models_in_one_file.ipynb deleted file mode 100644 index c2aaaaaa148..00000000000 --- a/recipes/recipes/saving_multiple_models_in_one_file.ipynb +++ /dev/null @@ -1,140 +0,0 @@ -{ - "cells": [ - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "%matplotlib inline" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\nSaving and loading multiple models in one file using PyTorch\n============================================================\nSaving and loading multiple models can be helpful for reusing models\nthat you have previously trained.\n\nIntroduction\n------------\nWhen saving a model comprised of multiple ``torch.nn.Modules``, such as\na GAN, a sequence-to-sequence model, or an ensemble of models, you must\nsave a dictionary of each model\u2019s state_dict and corresponding\noptimizer. You can also save any other items that may aid you in\nresuming training by simply appending them to the dictionary.\nTo load the models, first initialize the models and optimizers, then\nload the dictionary locally using ``torch.load()``. From here, you can\neasily access the saved items by simply querying the dictionary as you\nwould expect.\nIn this recipe, we will demonstrate how to save multiple models to one\nfile using PyTorch.\n\nSetup\n-----\nBefore we begin, we need to install ``torch`` if it isn\u2019t already\navailable.\n\n::\n\n pip install torch\n \n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Steps\n-----\n\n1. Import all necessary libraries for loading our data\n2. Define and intialize the neural network\n3. Initialize the optimizer\n4. Save multiple models\n5. Load multiple models\n\n1. Import necessary libraries for loading our data\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nFor this recipe, we will use ``torch`` and its subsidiaries ``torch.nn``\nand ``torch.optim``.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "import torch\nimport torch.nn as nn\nimport torch.optim as optim" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "2. Define and intialize the neural network\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nFor sake of example, we will create a neural network for training\nimages. To learn more see the Defining a Neural Network recipe. Build\ntwo variables for the models to eventually save.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "class Net(nn.Module):\n def __init__(self):\n super(Net, self).__init__()\n self.conv1 = nn.Conv2d(3, 6, 5)\n self.pool = nn.MaxPool2d(2, 2)\n self.conv2 = nn.Conv2d(6, 16, 5)\n self.fc1 = nn.Linear(16 * 5 * 5, 120)\n self.fc2 = nn.Linear(120, 84)\n self.fc3 = nn.Linear(84, 10)\n\n def forward(self, x):\n x = self.pool(F.relu(self.conv1(x)))\n x = self.pool(F.relu(self.conv2(x)))\n x = x.view(-1, 16 * 5 * 5)\n x = F.relu(self.fc1(x))\n x = F.relu(self.fc2(x))\n x = self.fc3(x)\n return x\n\nnetA = Net()\nnetB = Net()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "3. Initialize the optimizer\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nWe will use SGD with momentum to build an optimizer for each model we\ncreated.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "optimizerA = optim.SGD(netA.parameters(), lr=0.001, momentum=0.9)\noptimizerB = optim.SGD(netB.parameters(), lr=0.001, momentum=0.9)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "4. Save multiple models\n~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nCollect all relevant information and build your dictionary.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "# Specify a path to save to\nPATH = \"model.pt\"\n\ntorch.save({\n 'modelA_state_dict': netA.state_dict(),\n 'modelB_state_dict': netB.state_dict(),\n 'optimizerA_state_dict': optimizerA.state_dict(),\n 'optimizerB_state_dict': optimizerB.state_dict(),\n }, PATH)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "4. Load multiple models\n~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nRemember to first initialize the models and optimizers, then load the\ndictionary locally.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "modelA = Net()\nmodelB = Net()\noptimModelA = optim.SGD(modelA.parameters(), lr=0.001, momentum=0.9)\noptimModelB = optim.SGD(modelB.parameters(), lr=0.001, momentum=0.9)\n\ncheckpoint = torch.load(PATH)\nmodelA.load_state_dict(checkpoint['modelA_state_dict'])\nmodelB.load_state_dict(checkpoint['modelB_state_dict'])\noptimizerA.load_state_dict(checkpoint['optimizerA_state_dict'])\noptimizerB.load_state_dict(checkpoint['optimizerB_state_dict'])\n\nmodelA.eval()\nmodelB.eval()\n# - or -\nmodelA.train()\nmodelB.train()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You must call ``model.eval()`` to set dropout and batch normalization\nlayers to evaluation mode before running inference. Failing to do this\nwill yield inconsistent inference results.\n\nIf you wish to resuming training, call ``model.train()`` to ensure these\nlayers are in training mode.\n\nCongratulations! You have successfully saved and loaded multiple models\nin PyTorch.\n\nLearn More\n----------\n\nTake a look at these other recipes to continue your learning:\n\n- TBD\n- TBD\n\n\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.4" - } - }, - "nbformat": 4, - "nbformat_minor": 0 -} \ No newline at end of file diff --git a/recipes/recipes/saving_multiple_models_in_one_file.py b/recipes/recipes/saving_multiple_models_in_one_file.py deleted file mode 100644 index b2f38247b4f..00000000000 --- a/recipes/recipes/saving_multiple_models_in_one_file.py +++ /dev/null @@ -1,162 +0,0 @@ -""" -Saving and loading multiple models in one file using PyTorch -============================================================ -Saving and loading multiple models can be helpful for reusing models -that you have previously trained. - -Introduction ------------- -When saving a model comprised of multiple ``torch.nn.Modules``, such as -a GAN, a sequence-to-sequence model, or an ensemble of models, you must -save a dictionary of each model’s state_dict and corresponding -optimizer. You can also save any other items that may aid you in -resuming training by simply appending them to the dictionary. -To load the models, first initialize the models and optimizers, then -load the dictionary locally using ``torch.load()``. From here, you can -easily access the saved items by simply querying the dictionary as you -would expect. -In this recipe, we will demonstrate how to save multiple models to one -file using PyTorch. - -Setup ------ -Before we begin, we need to install ``torch`` if it isn’t already -available. - -:: - - pip install torch - -""" - - - -###################################################################### -# Steps -# ----- -# -# 1. Import all necessary libraries for loading our data -# 2. Define and intialize the neural network -# 3. Initialize the optimizer -# 4. Save multiple models -# 5. Load multiple models -# -# 1. Import necessary libraries for loading our data -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# For this recipe, we will use ``torch`` and its subsidiaries ``torch.nn`` -# and ``torch.optim``. -# - -import torch -import torch.nn as nn -import torch.optim as optim - - -###################################################################### -# 2. Define and intialize the neural network -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# For sake of example, we will create a neural network for training -# images. To learn more see the Defining a Neural Network recipe. Build -# two variables for the models to eventually save. -# - -class Net(nn.Module): - def __init__(self): - super(Net, self).__init__() - self.conv1 = nn.Conv2d(3, 6, 5) - self.pool = nn.MaxPool2d(2, 2) - self.conv2 = nn.Conv2d(6, 16, 5) - self.fc1 = nn.Linear(16 * 5 * 5, 120) - self.fc2 = nn.Linear(120, 84) - self.fc3 = nn.Linear(84, 10) - - def forward(self, x): - x = self.pool(F.relu(self.conv1(x))) - x = self.pool(F.relu(self.conv2(x))) - x = x.view(-1, 16 * 5 * 5) - x = F.relu(self.fc1(x)) - x = F.relu(self.fc2(x)) - x = self.fc3(x) - return x - -netA = Net() -netB = Net() - - -###################################################################### -# 3. Initialize the optimizer -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# We will use SGD with momentum to build an optimizer for each model we -# created. -# - -optimizerA = optim.SGD(netA.parameters(), lr=0.001, momentum=0.9) -optimizerB = optim.SGD(netB.parameters(), lr=0.001, momentum=0.9) - - -###################################################################### -# 4. Save multiple models -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# Collect all relevant information and build your dictionary. -# - -# Specify a path to save to -PATH = "model.pt" - -torch.save({ - 'modelA_state_dict': netA.state_dict(), - 'modelB_state_dict': netB.state_dict(), - 'optimizerA_state_dict': optimizerA.state_dict(), - 'optimizerB_state_dict': optimizerB.state_dict(), - }, PATH) - - -###################################################################### -# 4. Load multiple models -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# Remember to first initialize the models and optimizers, then load the -# dictionary locally. -# - -modelA = Net() -modelB = Net() -optimModelA = optim.SGD(modelA.parameters(), lr=0.001, momentum=0.9) -optimModelB = optim.SGD(modelB.parameters(), lr=0.001, momentum=0.9) - -checkpoint = torch.load(PATH) -modelA.load_state_dict(checkpoint['modelA_state_dict']) -modelB.load_state_dict(checkpoint['modelB_state_dict']) -optimizerA.load_state_dict(checkpoint['optimizerA_state_dict']) -optimizerB.load_state_dict(checkpoint['optimizerB_state_dict']) - -modelA.eval() -modelB.eval() -# - or - -modelA.train() -modelB.train() - - -###################################################################### -# You must call ``model.eval()`` to set dropout and batch normalization -# layers to evaluation mode before running inference. Failing to do this -# will yield inconsistent inference results. -# -# If you wish to resuming training, call ``model.train()`` to ensure these -# layers are in training mode. -# -# Congratulations! You have successfully saved and loaded multiple models -# in PyTorch. -# -# Learn More -# ---------- -# -# Take a look at these other recipes to continue your learning: -# -# - TBD -# - TBD -# diff --git a/recipes/recipes/saving_multiple_models_in_one_file.rst b/recipes/recipes/saving_multiple_models_in_one_file.rst deleted file mode 100644 index 1e73a7c8783..00000000000 --- a/recipes/recipes/saving_multiple_models_in_one_file.rst +++ /dev/null @@ -1,218 +0,0 @@ -.. note:: - :class: sphx-glr-download-link-note - - Click :ref:`here ` to download the full example code -.. rst-class:: sphx-glr-example-title - -.. _sphx_glr_recipes_recipes_saving_multiple_models_in_one_file.py: - - -Saving and loading multiple models in one file using PyTorch -============================================================ -Saving and loading multiple models can be helpful for reusing models -that you have previously trained. - -Introduction ------------- -When saving a model comprised of multiple ``torch.nn.Modules``, such as -a GAN, a sequence-to-sequence model, or an ensemble of models, you must -save a dictionary of each model’s state_dict and corresponding -optimizer. You can also save any other items that may aid you in -resuming training by simply appending them to the dictionary. -To load the models, first initialize the models and optimizers, then -load the dictionary locally using ``torch.load()``. From here, you can -easily access the saved items by simply querying the dictionary as you -would expect. -In this recipe, we will demonstrate how to save multiple models to one -file using PyTorch. - -Setup ------ -Before we begin, we need to install ``torch`` if it isn’t already -available. - -:: - - pip install torch - -Steps ------ - -1. Import all necessary libraries for loading our data -2. Define and intialize the neural network -3. Initialize the optimizer -4. Save multiple models -5. Load multiple models - -1. Import necessary libraries for loading our data -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -For this recipe, we will use ``torch`` and its subsidiaries ``torch.nn`` -and ``torch.optim``. - - - -.. code-block:: default - - - import torch - import torch.nn as nn - import torch.optim as optim - - - -2. Define and intialize the neural network -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -For sake of example, we will create a neural network for training -images. To learn more see the Defining a Neural Network recipe. Build -two variables for the models to eventually save. - - - -.. code-block:: default - - - class Net(nn.Module): - def __init__(self): - super(Net, self).__init__() - self.conv1 = nn.Conv2d(3, 6, 5) - self.pool = nn.MaxPool2d(2, 2) - self.conv2 = nn.Conv2d(6, 16, 5) - self.fc1 = nn.Linear(16 * 5 * 5, 120) - self.fc2 = nn.Linear(120, 84) - self.fc3 = nn.Linear(84, 10) - - def forward(self, x): - x = self.pool(F.relu(self.conv1(x))) - x = self.pool(F.relu(self.conv2(x))) - x = x.view(-1, 16 * 5 * 5) - x = F.relu(self.fc1(x)) - x = F.relu(self.fc2(x)) - x = self.fc3(x) - return x - - netA = Net() - netB = Net() - - - -3. Initialize the optimizer -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -We will use SGD with momentum to build an optimizer for each model we -created. - - - -.. code-block:: default - - - optimizerA = optim.SGD(netA.parameters(), lr=0.001, momentum=0.9) - optimizerB = optim.SGD(netB.parameters(), lr=0.001, momentum=0.9) - - - -4. Save multiple models -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Collect all relevant information and build your dictionary. - - - -.. code-block:: default - - - # Specify a path to save to - PATH = "model.pt" - - torch.save({ - 'modelA_state_dict': netA.state_dict(), - 'modelB_state_dict': netB.state_dict(), - 'optimizerA_state_dict': optimizerA.state_dict(), - 'optimizerB_state_dict': optimizerB.state_dict(), - }, PATH) - - - -4. Load multiple models -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Remember to first initialize the models and optimizers, then load the -dictionary locally. - - - -.. code-block:: default - - - modelA = Net() - modelB = Net() - optimModelA = optim.SGD(modelA.parameters(), lr=0.001, momentum=0.9) - optimModelB = optim.SGD(modelB.parameters(), lr=0.001, momentum=0.9) - - checkpoint = torch.load(PATH) - modelA.load_state_dict(checkpoint['modelA_state_dict']) - modelB.load_state_dict(checkpoint['modelB_state_dict']) - optimizerA.load_state_dict(checkpoint['optimizerA_state_dict']) - optimizerB.load_state_dict(checkpoint['optimizerB_state_dict']) - - modelA.eval() - modelB.eval() - # - or - - modelA.train() - modelB.train() - - - -You must call ``model.eval()`` to set dropout and batch normalization -layers to evaluation mode before running inference. Failing to do this -will yield inconsistent inference results. - -If you wish to resuming training, call ``model.train()`` to ensure these -layers are in training mode. - -Congratulations! You have successfully saved and loaded multiple models -in PyTorch. - -Learn More ----------- - -Take a look at these other recipes to continue your learning: - -- TBD -- TBD - - - -.. rst-class:: sphx-glr-timing - - **Total running time of the script:** ( 0 minutes 0.000 seconds) - - -.. _sphx_glr_download_recipes_recipes_saving_multiple_models_in_one_file.py: - - -.. only :: html - - .. container:: sphx-glr-footer - :class: sphx-glr-footer-example - - - - .. container:: sphx-glr-download - - :download:`Download Python source code: saving_multiple_models_in_one_file.py ` - - - - .. container:: sphx-glr-download - - :download:`Download Jupyter notebook: saving_multiple_models_in_one_file.ipynb ` - - -.. only:: html - - .. rst-class:: sphx-glr-signature - - `Gallery generated by Sphinx-Gallery `_ diff --git a/recipes/recipes/tensorboard_with_pytorch.ipynb b/recipes/recipes/tensorboard_with_pytorch.ipynb deleted file mode 100644 index 0aca76ce383..00000000000 --- a/recipes/recipes/tensorboard_with_pytorch.ipynb +++ /dev/null @@ -1,125 +0,0 @@ -{ - "cells": [ - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "%matplotlib inline" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\nHow to use TensorBoard with PyTorch\n===================================\nTensorBoard is a visualization toolkit for machine learning experimentation. \nTensorBoard allows tracking and visualizing metrics such as loss and accuracy, \nvisualizing the model graph, viewing histograms, displaying images and much more. \nIn this tutorial we are going to cover TensorBoard installation, \nbasic usage with PyTorch, and how to visualize data you logged in TensorBoard UI.\n\nInstallation\n----------------------\nPyTorch should be installed to log models and metrics into TensorBoard log \ndirectory. The following command will install PyTorch 1.4+ via \nAnaconda (recommended):\n\n::\n\n $ conda install pytorch torchvision -c pytorch \n \n\nor pip\n\n::\n\n $ pip install torch torchvision\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Using TensorBoard in PyTorch\n-----\n\nLet\u2019s now try using TensorBoard with PyTorch! Before logging anything, \nwe need to create a ``SummaryWriter`` instance.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "import torch\nfrom torch.utils.tensorboard import SummaryWriter\nwriter = SummaryWriter()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Writer will output to ``./runs/`` directory by default.\n\n\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Log scalars\n-----\n\nIn machine learning, it\u2019s important to understand key metrics such as \nloss and how they change during training. Scalar helps to save \nthe loss value of each training step, or the accuracy after each epoch. \n\nTo log a scalar value, use \n``add_scalar(tag, scalar_value, global_step=None, walltime=None)``. \nFor example, lets create a simple linear regression training, and \nlog loss value using ``add_scalar``\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "x = torch.arange(-5, 5, 0.1).view(-1, 1)\ny = -5 * x + 0.1 * torch.randn(x.size())\n\nmodel = torch.nn.Linear(1, 1)\ncriterion = torch.nn.MSELoss()\noptimizer = torch.optim.SGD(model.parameters(), lr = 0.1)\n\ndef train_model(iter):\n for epoch in range(iter):\n y1 = model(x)\n loss = criterion(y1, y)\n writer.add_scalar(\"Loss/train\", loss, epoch)\n optimizer.zero_grad()\n loss.backward()\n optimizer.step()\n \ntrain_model(10)\nwriter.flush()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Call ``flush()`` method to make sure that all pending events \nhave been written to disk.\n\nSee `torch.utils.tensorboard tutorials `_ \nto find more TensorBoard visualization types you can log.\n\nIf you do not need the summary writer anymore, call ``close()`` method.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "writer.close()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Run TensorBoard\n-----\n\nInstall TensorBoard through the command line to visualize data you logged\n\n::\n\n $ pip install tensorboard\n\n\nNow, start TensorBoard, specifying the root log directory you used above. \nArgument ``logdir`` points to directory where TensorBoard will look to find \nevent files that it can display. TensorBoard will recursively walk \nthe directory structure rooted at logdir, looking for .*tfevents.* files.\n\n::\n\n $ tensorboard --logdir=runs\n\nGo to the URL it provides OR to `http://localhost:6006/ `_\n\n![](../../_static/img/thumbnails/tensorboard_scalars.png)\n\n :scale: 40 %\n\nThis dashboard shows how the loss and accuracy change with every epoch. \nYou can use it to also track training speed, learning rate, and other \nscalar values. It\u2019s helpful to compare these metrics across different \ntraining runs to improve your model.\n\n\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Share TensorBoard dashboards\n-----\n\n`TensorBoard.dev `_ lets you upload and share \nyour ML experiment results with anyone. Use TensorBoard.dev to host, \ntrack, and share your TensorBoard dashboards.\n\nInstall the latest version of TensorBoard to use the uploader.\n\n:: \n\n $ pip install tensorboard --upgrade\n\nUse a simple command to upload and share your TensorBoard.\n\n:: \n\n $ tensorboard dev upload --logdir runs \\\n --name \"My latest experiment\" \\ # optional\n --description \"Simple comparison of several hyperparameters\" # optional\n\nFor help, run ``$ tensorboard dev --help``.\n\n**Note:** Uploaded TensorBoards are public and visible to everyone. \nDo not upload sensitive data.\n\nView your TensorBoard live at URL provided in your terminal. \nE.g. `https://tensorboard.dev/experiment/AdYd1TgeTlaLWXx6I8JUbA `_\n\n\n![](../../_static/img/thumbnails/tensorboard_dev.png)\n\n :scale: 40 %\n\n\n

Note

TensorBoard.dev currently supports only scalars dashboard.

\n\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Learn More\n----------------------------\n\n- `torch.utils.tensorboard `_ docs\n- `Visualizing models, data, and training with TensorBoard `_ tutorial\n\n\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.4" - } - }, - "nbformat": 4, - "nbformat_minor": 0 -} \ No newline at end of file diff --git a/recipes/recipes/tensorboard_with_pytorch.py b/recipes/recipes/tensorboard_with_pytorch.py deleted file mode 100644 index c51a24728ad..00000000000 --- a/recipes/recipes/tensorboard_with_pytorch.py +++ /dev/null @@ -1,168 +0,0 @@ -""" -How to use TensorBoard with PyTorch -=================================== -TensorBoard is a visualization toolkit for machine learning experimentation. -TensorBoard allows tracking and visualizing metrics such as loss and accuracy, -visualizing the model graph, viewing histograms, displaying images and much more. -In this tutorial we are going to cover TensorBoard installation, -basic usage with PyTorch, and how to visualize data you logged in TensorBoard UI. - -Installation ----------------------- -PyTorch should be installed to log models and metrics into TensorBoard log -directory. The following command will install PyTorch 1.4+ via -Anaconda (recommended): - -:: - - $ conda install pytorch torchvision -c pytorch - - -or pip - -:: - - $ pip install torch torchvision - -""" - -###################################################################### -# Using TensorBoard in PyTorch -# ----- -# -# Let’s now try using TensorBoard with PyTorch! Before logging anything, -# we need to create a ``SummaryWriter`` instance. -# - -import torch -from torch.utils.tensorboard import SummaryWriter -writer = SummaryWriter() - -###################################################################### -# Writer will output to ``./runs/`` directory by default. -# - - -###################################################################### -# Log scalars -# ----- -# -# In machine learning, it’s important to understand key metrics such as -# loss and how they change during training. Scalar helps to save -# the loss value of each training step, or the accuracy after each epoch. -# -# To log a scalar value, use -# ``add_scalar(tag, scalar_value, global_step=None, walltime=None)``. -# For example, lets create a simple linear regression training, and -# log loss value using ``add_scalar`` -# - -x = torch.arange(-5, 5, 0.1).view(-1, 1) -y = -5 * x + 0.1 * torch.randn(x.size()) - -model = torch.nn.Linear(1, 1) -criterion = torch.nn.MSELoss() -optimizer = torch.optim.SGD(model.parameters(), lr = 0.1) - -def train_model(iter): - for epoch in range(iter): - y1 = model(x) - loss = criterion(y1, y) - writer.add_scalar("Loss/train", loss, epoch) - optimizer.zero_grad() - loss.backward() - optimizer.step() - -train_model(10) -writer.flush() - - -###################################################################### -# Call ``flush()`` method to make sure that all pending events -# have been written to disk. -# -# See `torch.utils.tensorboard tutorials `_ -# to find more TensorBoard visualization types you can log. -# -# If you do not need the summary writer anymore, call ``close()`` method. -# - -writer.close() - -###################################################################### -# Run TensorBoard -# ----- -# -# Install TensorBoard through the command line to visualize data you logged -# -# :: -# -# $ pip install tensorboard -# -# -# Now, start TensorBoard, specifying the root log directory you used above. -# Argument ``logdir`` points to directory where TensorBoard will look to find -# event files that it can display. TensorBoard will recursively walk -# the directory structure rooted at logdir, looking for .*tfevents.* files. -# -# :: -# -# $ tensorboard --logdir=runs -# -# Go to the URL it provides OR to `http://localhost:6006/ `_ -# -# .. image:: ../../_static/img/thumbnails/tensorboard_scalars.png -# :scale: 40 % -# -# This dashboard shows how the loss and accuracy change with every epoch. -# You can use it to also track training speed, learning rate, and other -# scalar values. It’s helpful to compare these metrics across different -# training runs to improve your model. -# - - -###################################################################### -# Share TensorBoard dashboards -# ----- -# -# `TensorBoard.dev `_ lets you upload and share -# your ML experiment results with anyone. Use TensorBoard.dev to host, -# track, and share your TensorBoard dashboards. -# -# Install the latest version of TensorBoard to use the uploader. -# -# :: -# -# $ pip install tensorboard --upgrade -# -# Use a simple command to upload and share your TensorBoard. -# -# :: -# -# $ tensorboard dev upload --logdir runs \ -# --name "My latest experiment" \ # optional -# --description "Simple comparison of several hyperparameters" # optional -# -# For help, run ``$ tensorboard dev --help``. -# -# **Note:** Uploaded TensorBoards are public and visible to everyone. -# Do not upload sensitive data. -# -# View your TensorBoard live at URL provided in your terminal. -# E.g. `https://tensorboard.dev/experiment/AdYd1TgeTlaLWXx6I8JUbA `_ -# -# -# .. image:: ../../_static/img/thumbnails/tensorboard_dev.png -# :scale: 40 % -# -# -# .. note:: -# TensorBoard.dev currently supports only scalars dashboard. - -######################################################################## -# Learn More -# ---------------------------- -# -# - `torch.utils.tensorboard `_ docs -# - `Visualizing models, data, and training with TensorBoard `_ tutorial -# diff --git a/recipes/recipes/tensorboard_with_pytorch.rst b/recipes/recipes/tensorboard_with_pytorch.rst deleted file mode 100644 index c7e7013323f..00000000000 --- a/recipes/recipes/tensorboard_with_pytorch.rst +++ /dev/null @@ -1,212 +0,0 @@ -.. note:: - :class: sphx-glr-download-link-note - - Click :ref:`here ` to download the full example code -.. rst-class:: sphx-glr-example-title - -.. _sphx_glr_recipes_recipes_tensorboard_with_pytorch.py: - - -How to use TensorBoard with PyTorch -=================================== -TensorBoard is a visualization toolkit for machine learning experimentation. -TensorBoard allows tracking and visualizing metrics such as loss and accuracy, -visualizing the model graph, viewing histograms, displaying images and much more. -In this tutorial we are going to cover TensorBoard installation, -basic usage with PyTorch, and how to visualize data you logged in TensorBoard UI. - -Installation ----------------------- -PyTorch should be installed to log models and metrics into TensorBoard log -directory. The following command will install PyTorch 1.4+ via -Anaconda (recommended): - -:: - - $ conda install pytorch torchvision -c pytorch - - -or pip - -:: - - $ pip install torch torchvision -Using TensorBoard in PyTorch ------ - -Let’s now try using TensorBoard with PyTorch! Before logging anything, -we need to create a ``SummaryWriter`` instance. - - - -.. code-block:: default - - - import torch - from torch.utils.tensorboard import SummaryWriter - writer = SummaryWriter() - - -Writer will output to ``./runs/`` directory by default. - - -Log scalars ------ - -In machine learning, it’s important to understand key metrics such as -loss and how they change during training. Scalar helps to save -the loss value of each training step, or the accuracy after each epoch. - -To log a scalar value, use -``add_scalar(tag, scalar_value, global_step=None, walltime=None)``. -For example, lets create a simple linear regression training, and -log loss value using ``add_scalar`` - - - -.. code-block:: default - - - x = torch.arange(-5, 5, 0.1).view(-1, 1) - y = -5 * x + 0.1 * torch.randn(x.size()) - - model = torch.nn.Linear(1, 1) - criterion = torch.nn.MSELoss() - optimizer = torch.optim.SGD(model.parameters(), lr = 0.1) - - def train_model(iter): - for epoch in range(iter): - y1 = model(x) - loss = criterion(y1, y) - writer.add_scalar("Loss/train", loss, epoch) - optimizer.zero_grad() - loss.backward() - optimizer.step() - - train_model(10) - writer.flush() - - - -Call ``flush()`` method to make sure that all pending events -have been written to disk. - -See `torch.utils.tensorboard tutorials `_ -to find more TensorBoard visualization types you can log. - -If you do not need the summary writer anymore, call ``close()`` method. - - - -.. code-block:: default - - - writer.close() - - -Run TensorBoard ------ - -Install TensorBoard through the command line to visualize data you logged - -:: - - $ pip install tensorboard - - -Now, start TensorBoard, specifying the root log directory you used above. -Argument ``logdir`` points to directory where TensorBoard will look to find -event files that it can display. TensorBoard will recursively walk -the directory structure rooted at logdir, looking for .*tfevents.* files. - -:: - - $ tensorboard --logdir=runs - -Go to the URL it provides OR to `http://localhost:6006/ `_ - -.. image:: ../../_static/img/thumbnails/tensorboard_scalars.png - :scale: 40 % - -This dashboard shows how the loss and accuracy change with every epoch. -You can use it to also track training speed, learning rate, and other -scalar values. It’s helpful to compare these metrics across different -training runs to improve your model. - - -Share TensorBoard dashboards ------ - -`TensorBoard.dev `_ lets you upload and share -your ML experiment results with anyone. Use TensorBoard.dev to host, -track, and share your TensorBoard dashboards. - -Install the latest version of TensorBoard to use the uploader. - -:: - - $ pip install tensorboard --upgrade - -Use a simple command to upload and share your TensorBoard. - -:: - - $ tensorboard dev upload --logdir runs \ - --name "My latest experiment" \ # optional - --description "Simple comparison of several hyperparameters" # optional - -For help, run ``$ tensorboard dev --help``. - -**Note:** Uploaded TensorBoards are public and visible to everyone. -Do not upload sensitive data. - -View your TensorBoard live at URL provided in your terminal. -E.g. `https://tensorboard.dev/experiment/AdYd1TgeTlaLWXx6I8JUbA `_ - - -.. image:: ../../_static/img/thumbnails/tensorboard_dev.png - :scale: 40 % - - -.. note:: - TensorBoard.dev currently supports only scalars dashboard. - -Learn More ----------------------------- - -- `torch.utils.tensorboard `_ docs -- `Visualizing models, data, and training with TensorBoard `_ tutorial - - - -.. rst-class:: sphx-glr-timing - - **Total running time of the script:** ( 0 minutes 0.000 seconds) - - -.. _sphx_glr_download_recipes_recipes_tensorboard_with_pytorch.py: - - -.. only :: html - - .. container:: sphx-glr-footer - :class: sphx-glr-footer-example - - - - .. container:: sphx-glr-download - - :download:`Download Python source code: tensorboard_with_pytorch.py ` - - - - .. container:: sphx-glr-download - - :download:`Download Jupyter notebook: tensorboard_with_pytorch.ipynb ` - - -.. only:: html - - .. rst-class:: sphx-glr-signature - - `Gallery generated by Sphinx-Gallery `_ diff --git a/recipes/recipes/warmstarting_model_using_parameters_from_a_different_model.ipynb b/recipes/recipes/warmstarting_model_using_parameters_from_a_different_model.ipynb deleted file mode 100644 index a32b3603529..00000000000 --- a/recipes/recipes/warmstarting_model_using_parameters_from_a_different_model.ipynb +++ /dev/null @@ -1,122 +0,0 @@ -{ - "cells": [ - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "%matplotlib inline" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\nWarmstarting model using parameters from a different model in PyTorch\n=====================================================================\nPartially loading a model or loading a partial model are common\nscenarios when transfer learning or training a new complex model.\nLeveraging trained parameters, even if only a few are usable, will help\nto warmstart the training process and hopefully help your model converge\nmuch faster than training from scratch.\n\nIntroduction\n------------\nWhether you are loading from a partial ``state_dict``, which is missing\nsome keys, or loading a ``state_dict`` with more keys than the model\nthat you are loading into, you can set the strict argument to ``False``\nin the ``load_state_dict()`` function to ignore non-matching keys.\nIn this recipe, we will experiment with warmstarting a model using\nparameters of a different model.\n\nSetup\n-----\nBefore we begin, we need to install ``torch`` if it isn\u2019t already\navailable.\n\n::\n\n pip install torch\n \n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Steps\n-----\n\n1. Import all necessary libraries for loading our data\n2. Define and intialize the neural network A and B\n3. Save model A\n4. Load into model B\n\n1. Import necessary libraries for loading our data\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nFor this recipe, we will use ``torch`` and its subsidiaries ``torch.nn``\nand ``torch.optim``.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "import torch\nimport torch.nn as nn\nimport torch.optim as optim" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "2. Define and intialize the neural network A and B\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nFor sake of example, we will create a neural network for training\nimages. To learn more see the Defining a Neural Network recipe. We will\ncreate two neural networks for sake of loading one parameter of type A\ninto type B.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "class NetA(nn.Module):\n def __init__(self):\n super(NetA, self).__init__()\n self.conv1 = nn.Conv2d(3, 6, 5)\n self.pool = nn.MaxPool2d(2, 2)\n self.conv2 = nn.Conv2d(6, 16, 5)\n self.fc1 = nn.Linear(16 * 5 * 5, 120)\n self.fc2 = nn.Linear(120, 84)\n self.fc3 = nn.Linear(84, 10)\n\n def forward(self, x):\n x = self.pool(F.relu(self.conv1(x)))\n x = self.pool(F.relu(self.conv2(x)))\n x = x.view(-1, 16 * 5 * 5)\n x = F.relu(self.fc1(x))\n x = F.relu(self.fc2(x))\n x = self.fc3(x)\n return x\n\nnetA = NetA()\n\nclass NetB(nn.Module):\n def __init__(self):\n super(NetB, self).__init__()\n self.conv1 = nn.Conv2d(3, 6, 5)\n self.pool = nn.MaxPool2d(2, 2)\n self.conv2 = nn.Conv2d(6, 16, 5)\n self.fc1 = nn.Linear(16 * 5 * 5, 120)\n self.fc2 = nn.Linear(120, 84)\n self.fc3 = nn.Linear(84, 10)\n\n def forward(self, x):\n x = self.pool(F.relu(self.conv1(x)))\n x = self.pool(F.relu(self.conv2(x)))\n x = x.view(-1, 16 * 5 * 5)\n x = F.relu(self.fc1(x))\n x = F.relu(self.fc2(x))\n x = self.fc3(x)\n return x\n\nnetB = NetB()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "3. Save model A\n~~~~~~~~~~~~~~~~~~~\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "# Specify a path to save to\nPATH = \"model.pt\"\n\ntorch.save(netA.state_dict(), PATH)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "4. Load into model B\n~~~~~~~~~~~~~~~~~~~~~~~~\n\nIf you want to load parameters from one layer to another, but some keys\ndo not match, simply change the name of the parameter keys in the\nstate_dict that you are loading to match the keys in the model that you\nare loading into.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "netB.load_state_dict(torch.load(PATH), strict=False)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You can see that all keys matched successfully!\n\nCongratulations! You have successfully warmstarted a model using\nparameters from a different model in PyTorch.\n\nLearn More\n----------\n\nTake a look at these other recipes to continue your learning:\n\n- `Saving and loading multiple models in one file using PyTorch `__\n- `Saving and loading models across devices in PyTorch `__\n\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.4" - } - }, - "nbformat": 4, - "nbformat_minor": 0 -} \ No newline at end of file diff --git a/recipes/recipes/warmstarting_model_using_parameters_from_a_different_model.py b/recipes/recipes/warmstarting_model_using_parameters_from_a_different_model.py deleted file mode 100644 index 410bc992c6e..00000000000 --- a/recipes/recipes/warmstarting_model_using_parameters_from_a_different_model.py +++ /dev/null @@ -1,142 +0,0 @@ -""" -Warmstarting model using parameters from a different model in PyTorch -===================================================================== -Partially loading a model or loading a partial model are common -scenarios when transfer learning or training a new complex model. -Leveraging trained parameters, even if only a few are usable, will help -to warmstart the training process and hopefully help your model converge -much faster than training from scratch. - -Introduction ------------- -Whether you are loading from a partial ``state_dict``, which is missing -some keys, or loading a ``state_dict`` with more keys than the model -that you are loading into, you can set the strict argument to ``False`` -in the ``load_state_dict()`` function to ignore non-matching keys. -In this recipe, we will experiment with warmstarting a model using -parameters of a different model. - -Setup ------ -Before we begin, we need to install ``torch`` if it isn’t already -available. - -:: - - pip install torch - -""" - - - -###################################################################### -# Steps -# ----- -# -# 1. Import all necessary libraries for loading our data -# 2. Define and intialize the neural network A and B -# 3. Save model A -# 4. Load into model B -# -# 1. Import necessary libraries for loading our data -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# For this recipe, we will use ``torch`` and its subsidiaries ``torch.nn`` -# and ``torch.optim``. -# - -import torch -import torch.nn as nn -import torch.optim as optim - - -###################################################################### -# 2. Define and intialize the neural network A and B -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# For sake of example, we will create a neural network for training -# images. To learn more see the Defining a Neural Network recipe. We will -# create two neural networks for sake of loading one parameter of type A -# into type B. -# - -class NetA(nn.Module): - def __init__(self): - super(NetA, self).__init__() - self.conv1 = nn.Conv2d(3, 6, 5) - self.pool = nn.MaxPool2d(2, 2) - self.conv2 = nn.Conv2d(6, 16, 5) - self.fc1 = nn.Linear(16 * 5 * 5, 120) - self.fc2 = nn.Linear(120, 84) - self.fc3 = nn.Linear(84, 10) - - def forward(self, x): - x = self.pool(F.relu(self.conv1(x))) - x = self.pool(F.relu(self.conv2(x))) - x = x.view(-1, 16 * 5 * 5) - x = F.relu(self.fc1(x)) - x = F.relu(self.fc2(x)) - x = self.fc3(x) - return x - -netA = NetA() - -class NetB(nn.Module): - def __init__(self): - super(NetB, self).__init__() - self.conv1 = nn.Conv2d(3, 6, 5) - self.pool = nn.MaxPool2d(2, 2) - self.conv2 = nn.Conv2d(6, 16, 5) - self.fc1 = nn.Linear(16 * 5 * 5, 120) - self.fc2 = nn.Linear(120, 84) - self.fc3 = nn.Linear(84, 10) - - def forward(self, x): - x = self.pool(F.relu(self.conv1(x))) - x = self.pool(F.relu(self.conv2(x))) - x = x.view(-1, 16 * 5 * 5) - x = F.relu(self.fc1(x)) - x = F.relu(self.fc2(x)) - x = self.fc3(x) - return x - -netB = NetB() - - -###################################################################### -# 3. Save model A -# ~~~~~~~~~~~~~~~~~~~ -# - -# Specify a path to save to -PATH = "model.pt" - -torch.save(netA.state_dict(), PATH) - - -###################################################################### -# 4. Load into model B -# ~~~~~~~~~~~~~~~~~~~~~~~~ -# -# If you want to load parameters from one layer to another, but some keys -# do not match, simply change the name of the parameter keys in the -# state_dict that you are loading to match the keys in the model that you -# are loading into. -# - -netB.load_state_dict(torch.load(PATH), strict=False) - - -###################################################################### -# You can see that all keys matched successfully! -# -# Congratulations! You have successfully warmstarted a model using -# parameters from a different model in PyTorch. -# -# Learn More -# ---------- -# -# Take a look at these other recipes to continue your learning: -# -# - `Saving and loading multiple models in one file using PyTorch `__ -# - `Saving and loading models across devices in PyTorch `__ diff --git a/recipes/recipes/warmstarting_model_using_parameters_from_a_different_model.rst b/recipes/recipes/warmstarting_model_using_parameters_from_a_different_model.rst deleted file mode 100644 index a2308a4d566..00000000000 --- a/recipes/recipes/warmstarting_model_using_parameters_from_a_different_model.rst +++ /dev/null @@ -1,194 +0,0 @@ -.. note:: - :class: sphx-glr-download-link-note - - Click :ref:`here ` to download the full example code -.. rst-class:: sphx-glr-example-title - -.. _sphx_glr_recipes_recipes_warmstarting_model_using_parameters_from_a_different_model.py: - - -Warmstarting model using parameters from a different model in PyTorch -===================================================================== -Partially loading a model or loading a partial model are common -scenarios when transfer learning or training a new complex model. -Leveraging trained parameters, even if only a few are usable, will help -to warmstart the training process and hopefully help your model converge -much faster than training from scratch. - -Introduction ------------- -Whether you are loading from a partial ``state_dict``, which is missing -some keys, or loading a ``state_dict`` with more keys than the model -that you are loading into, you can set the strict argument to ``False`` -in the ``load_state_dict()`` function to ignore non-matching keys. -In this recipe, we will experiment with warmstarting a model using -parameters of a different model. - -Setup ------ -Before we begin, we need to install ``torch`` if it isn’t already -available. - -:: - - pip install torch - -Steps ------ - -1. Import all necessary libraries for loading our data -2. Define and intialize the neural network A and B -3. Save model A -4. Load into model B - -1. Import necessary libraries for loading our data -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -For this recipe, we will use ``torch`` and its subsidiaries ``torch.nn`` -and ``torch.optim``. - - - -.. code-block:: default - - - import torch - import torch.nn as nn - import torch.optim as optim - - - -2. Define and intialize the neural network A and B -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -For sake of example, we will create a neural network for training -images. To learn more see the Defining a Neural Network recipe. We will -create two neural networks for sake of loading one parameter of type A -into type B. - - - -.. code-block:: default - - - class NetA(nn.Module): - def __init__(self): - super(NetA, self).__init__() - self.conv1 = nn.Conv2d(3, 6, 5) - self.pool = nn.MaxPool2d(2, 2) - self.conv2 = nn.Conv2d(6, 16, 5) - self.fc1 = nn.Linear(16 * 5 * 5, 120) - self.fc2 = nn.Linear(120, 84) - self.fc3 = nn.Linear(84, 10) - - def forward(self, x): - x = self.pool(F.relu(self.conv1(x))) - x = self.pool(F.relu(self.conv2(x))) - x = x.view(-1, 16 * 5 * 5) - x = F.relu(self.fc1(x)) - x = F.relu(self.fc2(x)) - x = self.fc3(x) - return x - - netA = NetA() - - class NetB(nn.Module): - def __init__(self): - super(NetB, self).__init__() - self.conv1 = nn.Conv2d(3, 6, 5) - self.pool = nn.MaxPool2d(2, 2) - self.conv2 = nn.Conv2d(6, 16, 5) - self.fc1 = nn.Linear(16 * 5 * 5, 120) - self.fc2 = nn.Linear(120, 84) - self.fc3 = nn.Linear(84, 10) - - def forward(self, x): - x = self.pool(F.relu(self.conv1(x))) - x = self.pool(F.relu(self.conv2(x))) - x = x.view(-1, 16 * 5 * 5) - x = F.relu(self.fc1(x)) - x = F.relu(self.fc2(x)) - x = self.fc3(x) - return x - - netB = NetB() - - - -3. Save model A -~~~~~~~~~~~~~~~~~~~ - - - -.. code-block:: default - - - # Specify a path to save to - PATH = "model.pt" - - torch.save(netA.state_dict(), PATH) - - - -4. Load into model B -~~~~~~~~~~~~~~~~~~~~~~~~ - -If you want to load parameters from one layer to another, but some keys -do not match, simply change the name of the parameter keys in the -state_dict that you are loading to match the keys in the model that you -are loading into. - - - -.. code-block:: default - - - netB.load_state_dict(torch.load(PATH), strict=False) - - - -You can see that all keys matched successfully! - -Congratulations! You have successfully warmstarted a model using -parameters from a different model in PyTorch. - -Learn More ----------- - -Take a look at these other recipes to continue your learning: - -- `Saving and loading multiple models in one file using PyTorch `__ -- `Saving and loading models across devices in PyTorch `__ - - -.. rst-class:: sphx-glr-timing - - **Total running time of the script:** ( 0 minutes 0.000 seconds) - - -.. _sphx_glr_download_recipes_recipes_warmstarting_model_using_parameters_from_a_different_model.py: - - -.. only :: html - - .. container:: sphx-glr-footer - :class: sphx-glr-footer-example - - - - .. container:: sphx-glr-download - - :download:`Download Python source code: warmstarting_model_using_parameters_from_a_different_model.py ` - - - - .. container:: sphx-glr-download - - :download:`Download Jupyter notebook: warmstarting_model_using_parameters_from_a_different_model.ipynb ` - - -.. only:: html - - .. rst-class:: sphx-glr-signature - - `Gallery generated by Sphinx-Gallery `_ diff --git a/recipes/recipes/what_is_state_dict.ipynb b/recipes/recipes/what_is_state_dict.ipynb deleted file mode 100644 index d597dbcd0c9..00000000000 --- a/recipes/recipes/what_is_state_dict.ipynb +++ /dev/null @@ -1,122 +0,0 @@ -{ - "cells": [ - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "%matplotlib inline" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\nWhat is a state_dict in PyTorch\n===============================\nIn PyTorch, the learnable parameters (i.e. weights and biases) of a\n``torch.nn.Module`` model are contained in the model\u2019s parameters\n(accessed with ``model.parameters()``). A ``state_dict`` is simply a\nPython dictionary object that maps each layer to its parameter tensor.\n\nIntroduction\n------------\nA ``state_dict`` is an integral entity if you are interested in saving\nor loading models from PyTorch.\nBecause ``state_dict`` objects are Python dictionaries, they can be\neasily saved, updated, altered, and restored, adding a great deal of\nmodularity to PyTorch models and optimizers.\nNote that only layers with learnable parameters (convolutional layers,\nlinear layers, etc.) and registered buffers (batchnorm\u2019s running_mean)\nhave entries in the model\u2019s ``state_dict``. Optimizer objects\n(``torch.optim``) also have a ``state_dict``, which contains information\nabout the optimizer\u2019s state, as well as the hyperparameters used.\nIn this recipe, we will see how ``state_dict`` is used with a simple\nmodel.\n\nSetup\n-----\nBefore we begin, we need to install ``torch`` if it isn\u2019t already\navailable.\n\n::\n\n pip install torchaudio\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Steps\n-----\n\n1. Import all necessary libraries for loading our data\n2. Define and intialize the neural network\n3. Initialize the optimizer\n4. Access the model and optimizer ``state_dict``\n\n1. Import necessary libraries for loading our data\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nFor this recipe, we will use ``torch`` and its subsidiaries ``torch.nn``\nand ``torch.optim``.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "import torch\nimport torch.nn as nn\nimport torch.optim as optim" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "2. Define and intialize the neural network\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nFor sake of example, we will create a neural network for training\nimages. To learn more see the Defining a Neural Network recipe.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "class Net(nn.Module):\n def __init__(self):\n super(Net, self).__init__()\n self.conv1 = nn.Conv2d(3, 6, 5)\n self.pool = nn.MaxPool2d(2, 2)\n self.conv2 = nn.Conv2d(6, 16, 5)\n self.fc1 = nn.Linear(16 * 5 * 5, 120)\n self.fc2 = nn.Linear(120, 84)\n self.fc3 = nn.Linear(84, 10)\n\n def forward(self, x):\n x = self.pool(F.relu(self.conv1(x)))\n x = self.pool(F.relu(self.conv2(x)))\n x = x.view(-1, 16 * 5 * 5)\n x = F.relu(self.fc1(x))\n x = F.relu(self.fc2(x))\n x = self.fc3(x)\n return x\n\nnet = Net()\nprint(net)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "3. Initialize the optimizer\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nWe will use SGD with momentum.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "4. Access the model and optimizer ``state_dict``\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nNow that we have constructed our model and optimizer, we can understand\nwhat is preserved in their respective ``state_dict`` properties.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "# Print model's state_dict\nprint(\"Model's state_dict:\")\nfor param_tensor in net.state_dict():\n print(param_tensor, \"\\t\", net.state_dict()[param_tensor].size())\n\nprint()\n\n# Print optimizer's state_dict\nprint(\"Optimizer's state_dict:\")\nfor var_name in optimizer.state_dict():\n print(var_name, \"\\t\", optimizer.state_dict()[var_name])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This information is relevant for saving and loading the model and\noptimizers for future use.\n\nCongratulations! You have successfully used ``state_dict`` in PyTorch.\n\nLearn More\n----------\n\nTake a look at these other recipes to continue your learning:\n\n- `Saving and loading models for inference in PyTorch `__\n- `Saving and loading a general checkpoint in PyTorch `__\n\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.4" - } - }, - "nbformat": 4, - "nbformat_minor": 0 -} \ No newline at end of file diff --git a/recipes/recipes/what_is_state_dict.py b/recipes/recipes/what_is_state_dict.py deleted file mode 100644 index 8e718e9071e..00000000000 --- a/recipes/recipes/what_is_state_dict.py +++ /dev/null @@ -1,132 +0,0 @@ -""" -What is a state_dict in PyTorch -=============================== -In PyTorch, the learnable parameters (i.e. weights and biases) of a -``torch.nn.Module`` model are contained in the model’s parameters -(accessed with ``model.parameters()``). A ``state_dict`` is simply a -Python dictionary object that maps each layer to its parameter tensor. - -Introduction ------------- -A ``state_dict`` is an integral entity if you are interested in saving -or loading models from PyTorch. -Because ``state_dict`` objects are Python dictionaries, they can be -easily saved, updated, altered, and restored, adding a great deal of -modularity to PyTorch models and optimizers. -Note that only layers with learnable parameters (convolutional layers, -linear layers, etc.) and registered buffers (batchnorm’s running_mean) -have entries in the model’s ``state_dict``. Optimizer objects -(``torch.optim``) also have a ``state_dict``, which contains information -about the optimizer’s state, as well as the hyperparameters used. -In this recipe, we will see how ``state_dict`` is used with a simple -model. - -Setup ------ -Before we begin, we need to install ``torch`` if it isn’t already -available. - -:: - - pip install torchaudio - -""" - - - -###################################################################### -# Steps -# ----- -# -# 1. Import all necessary libraries for loading our data -# 2. Define and intialize the neural network -# 3. Initialize the optimizer -# 4. Access the model and optimizer ``state_dict`` -# -# 1. Import necessary libraries for loading our data -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# For this recipe, we will use ``torch`` and its subsidiaries ``torch.nn`` -# and ``torch.optim``. -# - -import torch -import torch.nn as nn -import torch.optim as optim - - -###################################################################### -# 2. Define and intialize the neural network -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# For sake of example, we will create a neural network for training -# images. To learn more see the Defining a Neural Network recipe. -# - -class Net(nn.Module): - def __init__(self): - super(Net, self).__init__() - self.conv1 = nn.Conv2d(3, 6, 5) - self.pool = nn.MaxPool2d(2, 2) - self.conv2 = nn.Conv2d(6, 16, 5) - self.fc1 = nn.Linear(16 * 5 * 5, 120) - self.fc2 = nn.Linear(120, 84) - self.fc3 = nn.Linear(84, 10) - - def forward(self, x): - x = self.pool(F.relu(self.conv1(x))) - x = self.pool(F.relu(self.conv2(x))) - x = x.view(-1, 16 * 5 * 5) - x = F.relu(self.fc1(x)) - x = F.relu(self.fc2(x)) - x = self.fc3(x) - return x - -net = Net() -print(net) - - -###################################################################### -# 3. Initialize the optimizer -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# We will use SGD with momentum. -# - -optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9) - - -###################################################################### -# 4. Access the model and optimizer ``state_dict`` -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# Now that we have constructed our model and optimizer, we can understand -# what is preserved in their respective ``state_dict`` properties. -# - -# Print model's state_dict -print("Model's state_dict:") -for param_tensor in net.state_dict(): - print(param_tensor, "\t", net.state_dict()[param_tensor].size()) - -print() - -# Print optimizer's state_dict -print("Optimizer's state_dict:") -for var_name in optimizer.state_dict(): - print(var_name, "\t", optimizer.state_dict()[var_name]) - - -###################################################################### -# This information is relevant for saving and loading the model and -# optimizers for future use. -# -# Congratulations! You have successfully used ``state_dict`` in PyTorch. -# -# Learn More -# ---------- -# -# Take a look at these other recipes to continue your learning: -# -# - `Saving and loading models for inference in PyTorch `__ -# - `Saving and loading a general checkpoint in PyTorch `__ diff --git a/recipes/recipes/what_is_state_dict.rst b/recipes/recipes/what_is_state_dict.rst deleted file mode 100644 index 223a0556cbc..00000000000 --- a/recipes/recipes/what_is_state_dict.rst +++ /dev/null @@ -1,183 +0,0 @@ -.. note:: - :class: sphx-glr-download-link-note - - Click :ref:`here ` to download the full example code -.. rst-class:: sphx-glr-example-title - -.. _sphx_glr_recipes_recipes_what_is_state_dict.py: - - -What is a state_dict in PyTorch -=============================== -In PyTorch, the learnable parameters (i.e. weights and biases) of a -``torch.nn.Module`` model are contained in the model’s parameters -(accessed with ``model.parameters()``). A ``state_dict`` is simply a -Python dictionary object that maps each layer to its parameter tensor. - -Introduction ------------- -A ``state_dict`` is an integral entity if you are interested in saving -or loading models from PyTorch. -Because ``state_dict`` objects are Python dictionaries, they can be -easily saved, updated, altered, and restored, adding a great deal of -modularity to PyTorch models and optimizers. -Note that only layers with learnable parameters (convolutional layers, -linear layers, etc.) and registered buffers (batchnorm’s running_mean) -have entries in the model’s ``state_dict``. Optimizer objects -(``torch.optim``) also have a ``state_dict``, which contains information -about the optimizer’s state, as well as the hyperparameters used. -In this recipe, we will see how ``state_dict`` is used with a simple -model. - -Setup ------ -Before we begin, we need to install ``torch`` if it isn’t already -available. - -:: - - pip install torchaudio -Steps ------ - -1. Import all necessary libraries for loading our data -2. Define and intialize the neural network -3. Initialize the optimizer -4. Access the model and optimizer ``state_dict`` - -1. Import necessary libraries for loading our data -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -For this recipe, we will use ``torch`` and its subsidiaries ``torch.nn`` -and ``torch.optim``. - - - -.. code-block:: default - - - import torch - import torch.nn as nn - import torch.optim as optim - - - -2. Define and intialize the neural network -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -For sake of example, we will create a neural network for training -images. To learn more see the Defining a Neural Network recipe. - - - -.. code-block:: default - - - class Net(nn.Module): - def __init__(self): - super(Net, self).__init__() - self.conv1 = nn.Conv2d(3, 6, 5) - self.pool = nn.MaxPool2d(2, 2) - self.conv2 = nn.Conv2d(6, 16, 5) - self.fc1 = nn.Linear(16 * 5 * 5, 120) - self.fc2 = nn.Linear(120, 84) - self.fc3 = nn.Linear(84, 10) - - def forward(self, x): - x = self.pool(F.relu(self.conv1(x))) - x = self.pool(F.relu(self.conv2(x))) - x = x.view(-1, 16 * 5 * 5) - x = F.relu(self.fc1(x)) - x = F.relu(self.fc2(x)) - x = self.fc3(x) - return x - - net = Net() - print(net) - - - -3. Initialize the optimizer -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -We will use SGD with momentum. - - - -.. code-block:: default - - - optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9) - - - -4. Access the model and optimizer ``state_dict`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Now that we have constructed our model and optimizer, we can understand -what is preserved in their respective ``state_dict`` properties. - - - -.. code-block:: default - - - # Print model's state_dict - print("Model's state_dict:") - for param_tensor in net.state_dict(): - print(param_tensor, "\t", net.state_dict()[param_tensor].size()) - - print() - - # Print optimizer's state_dict - print("Optimizer's state_dict:") - for var_name in optimizer.state_dict(): - print(var_name, "\t", optimizer.state_dict()[var_name]) - - - -This information is relevant for saving and loading the model and -optimizers for future use. - -Congratulations! You have successfully used ``state_dict`` in PyTorch. - -Learn More ----------- - -Take a look at these other recipes to continue your learning: - -- `Saving and loading models for inference in PyTorch `__ -- `Saving and loading a general checkpoint in PyTorch `__ - - -.. rst-class:: sphx-glr-timing - - **Total running time of the script:** ( 0 minutes 0.000 seconds) - - -.. _sphx_glr_download_recipes_recipes_what_is_state_dict.py: - - -.. only :: html - - .. container:: sphx-glr-footer - :class: sphx-glr-footer-example - - - - .. container:: sphx-glr-download - - :download:`Download Python source code: what_is_state_dict.py ` - - - - .. container:: sphx-glr-download - - :download:`Download Jupyter notebook: what_is_state_dict.ipynb ` - - -.. only:: html - - .. rst-class:: sphx-glr-signature - - `Gallery generated by Sphinx-Gallery `_ diff --git a/recipes/recipes/zeroing_out_gradients.ipynb b/recipes/recipes/zeroing_out_gradients.ipynb deleted file mode 100644 index 3314d6c622e..00000000000 --- a/recipes/recipes/zeroing_out_gradients.ipynb +++ /dev/null @@ -1,140 +0,0 @@ -{ - "cells": [ - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "%matplotlib inline" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\nZeroing out gradients in PyTorch\n================================\nIt is beneficial to zero out gradients when building a neural network.\nThis is because by default, gradients are accumulated in buffers (i.e,\nnot overwritten) whenever ``.backward()`` is called.\n\nIntroduction\n------------\nWhen training your neural network, models are able to increase their\naccuracy through gradient decent. In short, gradient descent is the\nprocess of minimizing our loss (or error) by tweaking the weights and\nbiases in our model.\n\n``torch.Tensor`` is the central class of PyTorch. When you create a\ntensor, if you set its attribute ``.requires_grad`` as ``True``, the\npackage tracks all operations on it. This happens on subsequent backward\npasses. The gradient for this tensor will be accumulated into ``.grad``\nattribute. The accumulation (or sum) of all the gradients is calculated\nwhen .backward() is called on the loss tensor.\n\nThere are cases where it may be necessary to zero-out the gradients of a\ntensor. For example: when you start your training loop, you should zero\nout the gradients so that you can perform this tracking correctly.\nIn this recipe, we will learn how to zero out gradients using the\nPyTorch library. We will demonstrate how to do this by training a neural\nnetwork on the ``CIFAR10`` dataset built into PyTorch.\n\nSetup\n-----\nSince we will be training data in this recipe, if you are in a runable\nnotebook, it is best to switch the runtime to GPU or TPU.\nBefore we begin, we need to install ``torch`` and ``torchvision`` if\nthey aren\u2019t already available.\n\n::\n\n pip install torchvision\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Steps\n-----\n\nSteps 1 through 4 set up our data and neural network for training. The\nprocess of zeroing out the gradients happens in step 5. If you already\nhave your data and neural network built, skip to 5.\n\n1. Import all necessary libraries for loading our data\n2. Load and normalize the dataset\n3. Build the neural network\n4. Define the loss function\n5. Zero the gradients while training the network\n\n1. Import necessary libraries for loading our data\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nFor this recipe, we will just be using ``torch`` and ``torchvision`` to\naccess the dataset.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "import torch\n\nimport torch.nn as nn\nimport torch.nn.functional as F\n\nimport torch.optim as optim\n\nimport torchvision\nimport torchvision.transforms as transforms" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "2. Load and normalize the dataset\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nPyTorch features various built-in datasets (see the Loading Data recipe\nfor more information).\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "transform = transforms.Compose(\n [transforms.ToTensor(),\n transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])\n\ntrainset = torchvision.datasets.CIFAR10(root='./data', train=True,\n download=True, transform=transform)\ntrainloader = torch.utils.data.DataLoader(trainset, batch_size=4,\n shuffle=True, num_workers=2)\n\ntestset = torchvision.datasets.CIFAR10(root='./data', train=False,\n download=True, transform=transform)\ntestloader = torch.utils.data.DataLoader(testset, batch_size=4,\n shuffle=False, num_workers=2)\n\nclasses = ('plane', 'car', 'bird', 'cat',\n 'deer', 'dog', 'frog', 'horse', 'ship', 'truck')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "3. Build the neural network\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nWe will use a convolutional neural network. To learn more see the\nDefining a Neural Network recipe.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "class Net(nn.Module):\n def __init__(self):\n super(Net, self).__init__()\n self.conv1 = nn.Conv2d(3, 6, 5)\n self.pool = nn.MaxPool2d(2, 2)\n self.conv2 = nn.Conv2d(6, 16, 5)\n self.fc1 = nn.Linear(16 * 5 * 5, 120)\n self.fc2 = nn.Linear(120, 84)\n self.fc3 = nn.Linear(84, 10)\n\n def forward(self, x):\n x = self.pool(F.relu(self.conv1(x)))\n x = self.pool(F.relu(self.conv2(x)))\n x = x.view(-1, 16 * 5 * 5)\n x = F.relu(self.fc1(x))\n x = F.relu(self.fc2(x))\n x = self.fc3(x)\n return x" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "4. Define a Loss function and optimizer\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nLet\u2019s use a Classification Cross-Entropy loss and SGD with momentum.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "net = Net()\ncriterion = nn.CrossEntropyLoss()\noptimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "5. Zero the gradients while training the network\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nThis is when things start to get interesting. We simply have to loop\nover our data iterator, and feed the inputs to the network and optimize.\n\nNotice that for each entity of data, we zero out the gradients. This is\nto ensure that we aren\u2019t tracking any unnecessary information when we\ntrain our neural network.\n\n\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "for epoch in range(2): # loop over the dataset multiple times\n\n running_loss = 0.0\n for i, data in enumerate(trainloader, 0):\n # get the inputs; data is a list of [inputs, labels]\n inputs, labels = data\n\n # zero the parameter gradients\n optimizer.zero_grad()\n\n # forward + backward + optimize\n outputs = net(inputs)\n loss = criterion(outputs, labels)\n loss.backward()\n optimizer.step()\n\n # print statistics\n running_loss += loss.item()\n if i % 2000 == 1999: # print every 2000 mini-batches\n print('[%d, %5d] loss: %.3f' %\n (epoch + 1, i + 1, running_loss / 2000))\n running_loss = 0.0\n\nprint('Finished Training')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You can also use ``model.zero_grad()``. This is the same as using\n``optimizer.zero_grad()`` as long as all your model parameters are in\nthat optimizer. Use your best judgement to decide which one to use.\n\nCongratulations! You have successfully zeroed out gradients PyTorch.\n\nLearn More\n----------\n\nTake a look at these other recipes to continue your learning:\n\n- `Loading data in PyTorch `__\n- `Saving and loading models across devices in PyTorch `__\n\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.4" - } - }, - "nbformat": 4, - "nbformat_minor": 0 -} \ No newline at end of file diff --git a/recipes/recipes/zeroing_out_gradients.py b/recipes/recipes/zeroing_out_gradients.py deleted file mode 100644 index 687120dcfcb..00000000000 --- a/recipes/recipes/zeroing_out_gradients.py +++ /dev/null @@ -1,193 +0,0 @@ -""" -Zeroing out gradients in PyTorch -================================ -It is beneficial to zero out gradients when building a neural network. -This is because by default, gradients are accumulated in buffers (i.e, -not overwritten) whenever ``.backward()`` is called. - -Introduction ------------- -When training your neural network, models are able to increase their -accuracy through gradient decent. In short, gradient descent is the -process of minimizing our loss (or error) by tweaking the weights and -biases in our model. - -``torch.Tensor`` is the central class of PyTorch. When you create a -tensor, if you set its attribute ``.requires_grad`` as ``True``, the -package tracks all operations on it. This happens on subsequent backward -passes. The gradient for this tensor will be accumulated into ``.grad`` -attribute. The accumulation (or sum) of all the gradients is calculated -when .backward() is called on the loss tensor. - -There are cases where it may be necessary to zero-out the gradients of a -tensor. For example: when you start your training loop, you should zero -out the gradients so that you can perform this tracking correctly. -In this recipe, we will learn how to zero out gradients using the -PyTorch library. We will demonstrate how to do this by training a neural -network on the ``CIFAR10`` dataset built into PyTorch. - -Setup ------ -Since we will be training data in this recipe, if you are in a runable -notebook, it is best to switch the runtime to GPU or TPU. -Before we begin, we need to install ``torch`` and ``torchvision`` if -they aren’t already available. - -:: - - pip install torchvision - - -""" - - -###################################################################### -# Steps -# ----- -# -# Steps 1 through 4 set up our data and neural network for training. The -# process of zeroing out the gradients happens in step 5. If you already -# have your data and neural network built, skip to 5. -# -# 1. Import all necessary libraries for loading our data -# 2. Load and normalize the dataset -# 3. Build the neural network -# 4. Define the loss function -# 5. Zero the gradients while training the network -# -# 1. Import necessary libraries for loading our data -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# For this recipe, we will just be using ``torch`` and ``torchvision`` to -# access the dataset. -# - -import torch - -import torch.nn as nn -import torch.nn.functional as F - -import torch.optim as optim - -import torchvision -import torchvision.transforms as transforms - - -###################################################################### -# 2. Load and normalize the dataset -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# PyTorch features various built-in datasets (see the Loading Data recipe -# for more information). -# - -transform = transforms.Compose( - [transforms.ToTensor(), - transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))]) - -trainset = torchvision.datasets.CIFAR10(root='./data', train=True, - download=True, transform=transform) -trainloader = torch.utils.data.DataLoader(trainset, batch_size=4, - shuffle=True, num_workers=2) - -testset = torchvision.datasets.CIFAR10(root='./data', train=False, - download=True, transform=transform) -testloader = torch.utils.data.DataLoader(testset, batch_size=4, - shuffle=False, num_workers=2) - -classes = ('plane', 'car', 'bird', 'cat', - 'deer', 'dog', 'frog', 'horse', 'ship', 'truck') - - -###################################################################### -# 3. Build the neural network -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# We will use a convolutional neural network. To learn more see the -# Defining a Neural Network recipe. -# - -class Net(nn.Module): - def __init__(self): - super(Net, self).__init__() - self.conv1 = nn.Conv2d(3, 6, 5) - self.pool = nn.MaxPool2d(2, 2) - self.conv2 = nn.Conv2d(6, 16, 5) - self.fc1 = nn.Linear(16 * 5 * 5, 120) - self.fc2 = nn.Linear(120, 84) - self.fc3 = nn.Linear(84, 10) - - def forward(self, x): - x = self.pool(F.relu(self.conv1(x))) - x = self.pool(F.relu(self.conv2(x))) - x = x.view(-1, 16 * 5 * 5) - x = F.relu(self.fc1(x)) - x = F.relu(self.fc2(x)) - x = self.fc3(x) - return x - - -###################################################################### -# 4. Define a Loss function and optimizer -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# Let’s use a Classification Cross-Entropy loss and SGD with momentum. -# - -net = Net() -criterion = nn.CrossEntropyLoss() -optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9) - - -###################################################################### -# 5. Zero the gradients while training the network -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# -# This is when things start to get interesting. We simply have to loop -# over our data iterator, and feed the inputs to the network and optimize. -# -# Notice that for each entity of data, we zero out the gradients. This is -# to ensure that we aren’t tracking any unnecessary information when we -# train our neural network. -# - -for epoch in range(2): # loop over the dataset multiple times - - running_loss = 0.0 - for i, data in enumerate(trainloader, 0): - # get the inputs; data is a list of [inputs, labels] - inputs, labels = data - - # zero the parameter gradients - optimizer.zero_grad() - - # forward + backward + optimize - outputs = net(inputs) - loss = criterion(outputs, labels) - loss.backward() - optimizer.step() - - # print statistics - running_loss += loss.item() - if i % 2000 == 1999: # print every 2000 mini-batches - print('[%d, %5d] loss: %.3f' % - (epoch + 1, i + 1, running_loss / 2000)) - running_loss = 0.0 - -print('Finished Training') - - -###################################################################### -# You can also use ``model.zero_grad()``. This is the same as using -# ``optimizer.zero_grad()`` as long as all your model parameters are in -# that optimizer. Use your best judgement to decide which one to use. -# -# Congratulations! You have successfully zeroed out gradients PyTorch. -# -# Learn More -# ---------- -# -# Take a look at these other recipes to continue your learning: -# -# - `Loading data in PyTorch `__ -# - `Saving and loading models across devices in PyTorch `__ diff --git a/recipes/recipes/zeroing_out_gradients.rst b/recipes/recipes/zeroing_out_gradients.rst deleted file mode 100644 index ecfcf7f4a02..00000000000 --- a/recipes/recipes/zeroing_out_gradients.rst +++ /dev/null @@ -1,248 +0,0 @@ -.. note:: - :class: sphx-glr-download-link-note - - Click :ref:`here ` to download the full example code -.. rst-class:: sphx-glr-example-title - -.. _sphx_glr_recipes_recipes_zeroing_out_gradients.py: - - -Zeroing out gradients in PyTorch -================================ -It is beneficial to zero out gradients when building a neural network. -This is because by default, gradients are accumulated in buffers (i.e, -not overwritten) whenever ``.backward()`` is called. - -Introduction ------------- -When training your neural network, models are able to increase their -accuracy through gradient decent. In short, gradient descent is the -process of minimizing our loss (or error) by tweaking the weights and -biases in our model. - -``torch.Tensor`` is the central class of PyTorch. When you create a -tensor, if you set its attribute ``.requires_grad`` as ``True``, the -package tracks all operations on it. This happens on subsequent backward -passes. The gradient for this tensor will be accumulated into ``.grad`` -attribute. The accumulation (or sum) of all the gradients is calculated -when .backward() is called on the loss tensor. - -There are cases where it may be necessary to zero-out the gradients of a -tensor. For example: when you start your training loop, you should zero -out the gradients so that you can perform this tracking correctly. -In this recipe, we will learn how to zero out gradients using the -PyTorch library. We will demonstrate how to do this by training a neural -network on the ``CIFAR10`` dataset built into PyTorch. - -Setup ------ -Since we will be training data in this recipe, if you are in a runable -notebook, it is best to switch the runtime to GPU or TPU. -Before we begin, we need to install ``torch`` and ``torchvision`` if -they aren’t already available. - -:: - - pip install torchvision -Steps ------ - -Steps 1 through 4 set up our data and neural network for training. The -process of zeroing out the gradients happens in step 5. If you already -have your data and neural network built, skip to 5. - -1. Import all necessary libraries for loading our data -2. Load and normalize the dataset -3. Build the neural network -4. Define the loss function -5. Zero the gradients while training the network - -1. Import necessary libraries for loading our data -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -For this recipe, we will just be using ``torch`` and ``torchvision`` to -access the dataset. - - - -.. code-block:: default - - - import torch - - import torch.nn as nn - import torch.nn.functional as F - - import torch.optim as optim - - import torchvision - import torchvision.transforms as transforms - - - -2. Load and normalize the dataset -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -PyTorch features various built-in datasets (see the Loading Data recipe -for more information). - - - -.. code-block:: default - - - transform = transforms.Compose( - [transforms.ToTensor(), - transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))]) - - trainset = torchvision.datasets.CIFAR10(root='./data', train=True, - download=True, transform=transform) - trainloader = torch.utils.data.DataLoader(trainset, batch_size=4, - shuffle=True, num_workers=2) - - testset = torchvision.datasets.CIFAR10(root='./data', train=False, - download=True, transform=transform) - testloader = torch.utils.data.DataLoader(testset, batch_size=4, - shuffle=False, num_workers=2) - - classes = ('plane', 'car', 'bird', 'cat', - 'deer', 'dog', 'frog', 'horse', 'ship', 'truck') - - - -3. Build the neural network -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -We will use a convolutional neural network. To learn more see the -Defining a Neural Network recipe. - - - -.. code-block:: default - - - class Net(nn.Module): - def __init__(self): - super(Net, self).__init__() - self.conv1 = nn.Conv2d(3, 6, 5) - self.pool = nn.MaxPool2d(2, 2) - self.conv2 = nn.Conv2d(6, 16, 5) - self.fc1 = nn.Linear(16 * 5 * 5, 120) - self.fc2 = nn.Linear(120, 84) - self.fc3 = nn.Linear(84, 10) - - def forward(self, x): - x = self.pool(F.relu(self.conv1(x))) - x = self.pool(F.relu(self.conv2(x))) - x = x.view(-1, 16 * 5 * 5) - x = F.relu(self.fc1(x)) - x = F.relu(self.fc2(x)) - x = self.fc3(x) - return x - - - -4. Define a Loss function and optimizer -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Let’s use a Classification Cross-Entropy loss and SGD with momentum. - - - -.. code-block:: default - - - net = Net() - criterion = nn.CrossEntropyLoss() - optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9) - - - -5. Zero the gradients while training the network -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -This is when things start to get interesting. We simply have to loop -over our data iterator, and feed the inputs to the network and optimize. - -Notice that for each entity of data, we zero out the gradients. This is -to ensure that we aren’t tracking any unnecessary information when we -train our neural network. - - - -.. code-block:: default - - - for epoch in range(2): # loop over the dataset multiple times - - running_loss = 0.0 - for i, data in enumerate(trainloader, 0): - # get the inputs; data is a list of [inputs, labels] - inputs, labels = data - - # zero the parameter gradients - optimizer.zero_grad() - - # forward + backward + optimize - outputs = net(inputs) - loss = criterion(outputs, labels) - loss.backward() - optimizer.step() - - # print statistics - running_loss += loss.item() - if i % 2000 == 1999: # print every 2000 mini-batches - print('[%d, %5d] loss: %.3f' % - (epoch + 1, i + 1, running_loss / 2000)) - running_loss = 0.0 - - print('Finished Training') - - - -You can also use ``model.zero_grad()``. This is the same as using -``optimizer.zero_grad()`` as long as all your model parameters are in -that optimizer. Use your best judgement to decide which one to use. - -Congratulations! You have successfully zeroed out gradients PyTorch. - -Learn More ----------- - -Take a look at these other recipes to continue your learning: - -- `Loading data in PyTorch `__ -- `Saving and loading models across devices in PyTorch `__ - - -.. rst-class:: sphx-glr-timing - - **Total running time of the script:** ( 0 minutes 0.000 seconds) - - -.. _sphx_glr_download_recipes_recipes_zeroing_out_gradients.py: - - -.. only :: html - - .. container:: sphx-glr-footer - :class: sphx-glr-footer-example - - - - .. container:: sphx-glr-download - - :download:`Download Python source code: zeroing_out_gradients.py ` - - - - .. container:: sphx-glr-download - - :download:`Download Jupyter notebook: zeroing_out_gradients.ipynb ` - - -.. only:: html - - .. rst-class:: sphx-glr-signature - - `Gallery generated by Sphinx-Gallery `_ diff --git a/recipes/recipes_index.rst b/recipes/recipes_index.rst deleted file mode 100644 index 192cce28d2f..00000000000 --- a/recipes/recipes_index.rst +++ /dev/null @@ -1,160 +0,0 @@ -PyTorch Recipes ---------------------------------------------- -Recipes are bite-sized bite-sized, actionable examples of how to use specific PyTorch features, different from our full-length tutorials. - -.. raw:: html - -
- - -
- - - -
- -
- -
-
- -.. Add recipe cards below this line - -.. Basics - -.. customcarditem:: - :header: Loading data in PyTorch - :card_description: Learn how to use PyTorch packages to prepare and load common datasets for your model. - :image: ../_static/img/thumbnails/cropped/loading-data.png - :link: ../recipes/recipes/loading_data_recipe.html - :tags: Basics - - -.. customcarditem:: - :header: Defining a Neural Network - :card_description: Learn how to use PyTorch's torch.nn package to create and define a neural network the MNIST dataset. - :image: ../_static/img/thumbnails/cropped/defining-a-network.png - :link: ../recipes/recipes/defining_a_neural_network.html - :tags: Basics - -.. customcarditem:: - :header: What is a state_dict in PyTorch - :card_description: Learn how state_dict objects, Python dictionaries, are used in saving or loading models from PyTorch. - :image: ../_static/img/thumbnails/cropped/what-is-a-state-dict.png - :link: ../recipes/recipes/what_is_state_dict.html - :tags: Basics - -.. customcarditem:: - :header: Saving and loading models for inference in PyTorch - :card_description: Learn about the two approaches for saving and loading models for inference in PyTorch - via the state_dict and via the entire model. - :image: ../_static/img/thumbnails/cropped/saving-and-loading-models-for-inference.png - :link: ../recipes/recipes/saving_and_loading_models_for_inference.html - :tags: Basics - - -.. customcarditem:: - :header: Saving and loading a general checkpoint in PyTorch - :card_description: Saving and loading a general checkpoint model for inference or resuming training can be helpful for picking up where you last left off. In this recipe, explore how to save and load multiple checkpoints. - :image: ../_static/img/thumbnails/cropped/saving-and-loading-general-checkpoint.png - :link: ../recipes/recipes/saving_and_loading_a_general_checkpoint.html - :tags: Basics - -.. customcarditem:: - :header: Saving and loading multiple models in one file using PyTorch - :card_description: In this recipe, learn how saving and loading multiple models can be helpful for reusing models that you have previously trained. - :image: ../_static/img/thumbnails/cropped/saving-multiple-models.png - :link: ../recipes/recipes/saving_multiple_models_in_one_file.html - :tags: Basics - -.. customcarditem:: - :header: Warmstarting model using parameters from a different model in PyTorch - :card_description: Learn how warmstarting the training process by partially loading a model or loading a partial model can help your model converge much faster than training from scratch. - :image: ../_static/img/thumbnails/cropped/warmstarting-models.png - :link: ../recipes/recipes/warmstarting_model_using_parameters_from_a_different_model.html - :tags: Basics - -.. customcarditem:: - :header: Saving and loading models across devices in PyTorch - :card_description: Learn how saving and loading models across devices (CPUs and GPUs) is relatively straightforward using PyTorch. - :image: ../_static/img/thumbnails/cropped/saving-and-loading-models-across-devices.png - :link: ../recipes/recipes/save_load_across_devices.html - :tags: Basics - -.. customcarditem:: - :header: Zeroing out gradients in PyTorch - :card_description: Learn when you should zero out graidents and how doing so can help increase the accuracy of your model. - :image: ../_static/img/thumbnails/cropped/zeroing-out-gradients.png - :link: ../recipes/recipes/zeroing_out_gradients.html - :tags: Basics - -.. Customization - -.. customcarditem:: - :header: Custom Datasets, Transforms & Dataloaders - :card_description: Learn how to leverage the PyTorch dataset API to easily create a custom dataset and custom dataloader. - :image: ../_static/img/thumbnails/cropped/custom-datasets-transforms-and-dataloaders.png - :link: ../recipes/recipes/custom_dataset_transforms_loader.html - :tags: Data-Customization - -.. Interpretability - -.. customcarditem:: - :header: Model Interpretability using Captum - :card_description: Learn how to use Captum attribute the predictions of an image classifier to their corresponding image features and visualize the attribution results. - :image: ../_static/img/thumbnails/cropped/model-interpretability-using-captum.png - :link: ../recipes/recipes/Captum_Recipe.html - :tags: Interpretability,Captum - -.. customcarditem:: - :header: How to use TensorBoard with PyTorch - :card_description: Learn basic usage of TensorBoard with PyTorch, and how to visualize data in TensorBoard UI - :image: ../_static/img/thumbnails/tensorboard_scalars.png - :link: ../recipes/recipes/tensorboard_with_pytorch.html - :tags: Visualization,TensorBoard - -.. Quantization - -.. customcarditem:: - :header: Dynamic Quantization - :card_description: Apply dynamic quantization to a simple LSTM model. - :image: ../_static/img/thumbnails/cropped/using-dynamic-post-training-quantization.png - :link: ../recipes/recipes/dynamic_quantization.html - :tags: Quantization,Text,Model-Optimization - - -.. Production Development - -.. customcarditem:: - :header: TorchScript for Deployment - :card_description: Learn how to export your trained model in TorchScript format and how to load your TorchScript model in C++ and do inference. - :image: ../_static/img/thumbnails/cropped/torchscript_overview.png - :link: ../recipes/torchscript_inference.html - :tags: TorchScript - -.. customcarditem:: - :header: Deploying with Flask - :card_description: Learn how to use Flask, a lightweight web server, to quickly setup a web API from your trained PyTorch model. - :image: ../_static/img/thumbnails/cropped/using-flask-create-restful-api.png - :link: ../recipes/deployment_with_flask.html - :tags: Production,TorchScript - - -.. End of tutorial card section - -.. raw:: html - -
- - - -
- -
diff --git a/recipes/torchscript_inference.rst b/recipes/torchscript_inference.rst deleted file mode 100644 index 54068e70723..00000000000 --- a/recipes/torchscript_inference.rst +++ /dev/null @@ -1,197 +0,0 @@ -TorchScript for Deployment -========================== - -In this recipe, you will learn: - -- What TorchScript is -- How to export your trained model in TorchScript format -- How to load your TorchScript model in C++ and do inference - -Requirements ------------- - -- PyTorch 1.5 -- TorchVision 0.6.0 -- libtorch 1.5 -- C++ compiler - -The instructions for installing the three PyTorch components are -available at `pytorch.org`_. The C++ compiler will depend on your -platform. - -What is TorchScript? --------------------- - -**TorchScript** is an intermediate representation of a PyTorch model -(subclass of ``nn.Module``) that can then be run in a high-performance -environment like C++. It’s a high-performance subset of Python that is -meant to be consumed by the **PyTorch JIT Compiler,** which performs -run-time optimization on your model’s computation. TorchScript is the -recommended model format for doing scaled inference with PyTorch models. -For more information, see the PyTorch `Introduction to TorchScript -tutorial`_, the `Loading A TorchScript Model in C++ tutorial`_, and the -`full TorchScript documentation`_, all of which are available on -`pytorch.org`_. - -How to Export Your Model ------------------------- - -As an example, let’s take a pretrained vision model. All of the -pretrained models in TorchVision are compatible with TorchScript. - -Run the following Python 3 code, either in a script or from the REPL: - -.. code:: python3 - - import torch - import torch.nn.functional as F - import torchvision.models as models - - r18 = models.resnet18(pretrained=True) # We now have an instance of the pretrained model - r18_scripted = torch.jit.script(r18) # *** This is the TorchScript export - dummy_input = torch.rand(1, 3, 224, 224) # We should run a quick test - -Let’s do a sanity check on the equivalence of the two models: - -:: - - unscripted_output = r18(dummy_input) # Get the unscripted model's prediction... - scripted_output = r18_scripted(dummy_input) # ...and do the same for the scripted version - - unscripted_top5 = F.softmax(unscripted_output, dim=1).topk(5).indices - scripted_top5 = F.softmax(scripted_output, dim=1).topk(5).indices - - print('Python model top 5 results:\n {}'.format(unscripted_top5)) - print('TorchScript model top 5 results:\n {}'.format(scripted_top5)) - -You should see that both versions of the model give the same results: - -:: - - Python model top 5 results: - tensor([[463, 600, 731, 899, 898]]) - TorchScript model top 5 results: - tensor([[463, 600, 731, 899, 898]]) - -With that check confirmed, go ahead and save the model: - -:: - - r18_scripted.save('r18_scripted.pt') - -Loading TorchScript Models in C++ ---------------------------------- - -Create the following C++ file and name it ``ts-infer.cpp``: - -.. code:: cpp - - #include - #include - - - int main(int argc, const char* argv[]) { - if (argc != 2) { - std::cerr << "usage: ts-infer \n"; - return -1; - } - - std::cout << "Loading model...\n"; - - // deserialize ScriptModule - torch::jit::script::Module module; - try { - module = torch::jit::load(argv[1]); - } catch (const c10::Error& e) { - std::cerr << "Error loading model\n"; - std::cerr << e.msg_without_backtrace(); - return -1; - } - - std::cout << "Model loaded successfully\n"; - - torch::NoGradGuard no_grad; // ensures that autograd is off - module.eval(); // turn off dropout and other training-time layers/functions - - // create an input "image" - std::vector inputs; - inputs.push_back(torch::rand({1, 3, 224, 224})); - - // execute model and package output as tensor - at::Tensor output = module.forward(inputs).toTensor(); - - namespace F = torch::nn::functional; - at::Tensor output_sm = F::softmax(output, F::SoftmaxFuncOptions(1)); - std::tuple top5_tensor = output_sm.topk(5); - at::Tensor top5 = std::get<1>(top5_tensor); - - std::cout << top5[0] << "\n"; - - std::cout << "\nDONE\n"; - return 0; - } - -This program: - -- Loads the model you specify on the command line -- Creates a dummy “image” input tensor -- Performs inference on the input - -Also, notice that there is no dependency on TorchVision in this code. -The saved version of your TorchScript model has your learning weights -*and* your computation graph - nothing else is needed. - -Building and Running Your C++ Inference Engine ----------------------------------------------- - -Create the following ``CMakeLists.txt`` file: - -:: - - cmake_minimum_required(VERSION 3.0 FATAL_ERROR) - project(custom_ops) - - find_package(Torch REQUIRED) - - add_executable(ts-infer ts-infer.cpp) - target_link_libraries(ts-infer "${TORCH_LIBRARIES}") - set_property(TARGET ts-infer PROPERTY CXX_STANDARD 11) - -Make the program: - -:: - - cmake -DCMAKE_PREFIX_PATH= - make - -Now, we can run inference in C++, and verify that we get a result: - -:: - - $ ./ts-infer r18_scripted.pt - Loading model... - Model loaded successfully - 418 - 845 - 111 - 892 - 644 - [ CPULongType{5} ] - - DONE - -Important Resources -------------------- - -- `pytorch.org`_ for installation instructions, and more documentation - and tutorials. -- `Introduction to TorchScript tutorial`_ for a deeper initial - exposition of TorchScript -- `Full TorchScript documentation`_ for complete TorchScript language - and API reference - -.. _pytorch.org: https://pytorch.org/ -.. _Introduction to TorchScript tutorial: https://pytorch.org/tutorials/beginner/Intro_to_TorchScript_tutorial.html -.. _Full TorchScript documentation: https://pytorch.org/docs/stable/jit.html -.. _Loading A TorchScript Model in C++ tutorial: https://pytorch.org/tutorials/advanced/cpp_export.html -.. _full TorchScript documentation: https://pytorch.org/docs/stable/jit.html diff --git a/recipes_source/recipes/dynamic_quantization.py b/recipes_source/recipes/dynamic_quantization.py index 73811daa94c..78dc1f5408a 100644 --- a/recipes_source/recipes/dynamic_quantization.py +++ b/recipes_source/recipes/dynamic_quantization.py @@ -2,14 +2,13 @@ Dynamic Quantization ==================== --------------- - In this recipe you will see how to take advantage of Dynamic Quantization to accelerate inference on an LSTM-style recurrent neural network. This reduces the size of the model weights and speeds up model execution. -**Introduction** +Introduction +------------- There are a number of trade-offs that can be made when designing neural networks. During model developmenet and training you can alter the @@ -24,7 +23,8 @@ your model size significantly and may get a significant latency reduction without losing a lot of accuracy. -**What is dynamic quantization?** +What is dynamic quantization? +------------- Quantizing a network means converting it to use a reduced precision integer representation for the weights and/or activations. This saves on @@ -55,15 +55,18 @@ it well suited to be added into production pipelines as a standard part of converting LSTM models to deployment. --------------- -**Note: Limitations on the approach taken here** -This recipe provides a quick introduction to the dynamic quantization -features in PyTorch and the workflow for using it. Our focus is on -explaining the specific functions used to convert the model. We will -make a number of significant simplifications in the interest of brevity -and clarity +.. note:: + Limitations on the approach taken here + + + This recipe provides a quick introduction to the dynamic quantization + features in PyTorch and the workflow for using it. Our focus is on + explaining the specific functions used to convert the model. We will + make a number of significant simplifications in the interest of brevity + and clarity + 1. You will start with a minimal LSTM network 2. You are simply going to initialize the network with a random hidden @@ -85,36 +88,26 @@ to the `advanced dynamic quantization tutorial `__. --------------- - -**Recipe Structure** +Steps +------------- This recipe has 5 steps. -1. Set up - - Here you define a very simple LSTM, import modules, and establish +1. Set Up - Here you define a very simple LSTM, import modules, and establish some random input tensors. -2. Do the quantization - - Here you instantiate a floating point model and then create quantized +2. Do the Quantization - Here you instantiate a floating point model and then create quantized version of it. -3. Look at model size - - Here you show that the model size gets smaller. +3. Look at Model Size - Here you show that the model size gets smaller. -4. Look at latency +4. Look at Latency - Here you run the two models and compare model runtime (latency). - Here you run the two models and compare model runtime (latency). +5. Look at Accuracy - Here you run the two models and compare outputs. -5. Look at accuracy - - Here you run the two models and compare outputs. - -**Step 1: set up** +1: Set Up +~~~~~~~~~~~~~~~ This is a straightfoward bit of code to set up for the rest of the recipe. @@ -165,7 +158,8 @@ def forward(self,inputs,hidden): ###################################################################### -# **Step 2: Do the quantization** +# 2: Do the Quantization +# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ # # Now we get to the fun part. First we create an instance of the model # called float\_lstm then we are going to quantize it. We're going to use @@ -202,8 +196,8 @@ def forward(self,inputs,hidden): ###################################################################### -# **Step 4. Look at the model size** -# +# 3. Look at Model Size +# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ # Ok, so we've quantized the model. What does that get us? Well the first # benefit is that we've replaced the FP32 model parameters with INT8 # values (and some recorded scale factors). This means about 75% less data @@ -231,8 +225,8 @@ def print_size_of_model(model, label=""): ###################################################################### -# **Step 4: Look at latency** -# +# 4. Look at Latency +# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ # The second benefit is that the quantized model will typically run # faster. This is due to a combinations of effects including at least: # @@ -254,8 +248,8 @@ def print_size_of_model(model, label=""): ###################################################################### -# **Step 5: Look at accuracy** -# +# 5: Look at Accuracy +# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ # We are not going to do a careful look at accuracy here because we are # working with a randomly initialized network rather than a properly # trained one. However, I think it is worth quickly showing that the @@ -282,32 +276,31 @@ def print_size_of_model(model, label=""): ###################################################################### -# **Summary** -# -# We've explained what dynamic quantization is, what beenefits it brings, +# Learn More +# ------------ +# We've explained what dynamic quantization is, what benefits it brings, # and you have used the ``torch.quantization.quantize_dynamic()`` function # to quickly quantize a simple LSTM model. # -# **To continue learning about dynamic quantization** -# # This was a fast and high level treatment of this material; for more -# detail please continue learning with: -# -# https://pytorch.org/tutorials/advanced/dynamic\_quantization\_tutorial.html +# detail please continue learning with `(experimental) Dynamic Quantization on an LSTM Word Language Model Tutorial `_. # -# **Other resources:** # -# Docs +# Additional Resources +# ========= +# Documentation +# ~~~~~~~~~~~~~~ # -# https://pytorch.org/docs/stable/quantization.html +# `Quantization API Documentaion `_ # # Tutorials +# ~~~~~~~~~~~~~~ # -# https://pytorch.org/tutorials/intermediate/dynamic\_quantization\_bert\_tutorial.html +# `(experimental) Dynamic Quantization on BERT `_ # -# https://pytorch.org/tutorials/advanced/dynamic\_quantization\_tutorial.html +# `(experimental) Dynamic Quantization on an LSTM Word Language Model `_ # # Blogs +# ~~~~~~~~~~~~~~ +# ` Introduction to Quantization on PyTorch `_ # -# https://pytorch.org/blog/introduction-to-quantization-on-pytorch/ -# \ No newline at end of file