Merge branch 'master' into patch-1

Author: xksteven

.gitignore

@@ -3,6 +3,7 @@ beginner
 intermediate
 advanced
 pytorch_basics
+recipes
 
 #data things
 _data/
@@ -31,6 +32,7 @@ __pycache__/
 *.so
 
 # Distribution / packaging
+src/
 .Python
 env/
 build/

Makefile

@@ -109,7 +109,7 @@ docs:
 
 html-noplot:
 	$(SPHINXBUILD) -D plot_gallery=0 -b html $(SPHINXOPTS) "$(SOURCEDIR)" "$(BUILDDIR)/html"
-	bash .jenkins/remove_invisible_code_block_batch.sh "$(BUILDDIR)/html"
+	# bash .jenkins/remove_invisible_code_block_batch.sh "$(BUILDDIR)/html"
 	@echo
 	@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
 

README.md

@@ -11,11 +11,11 @@ All the tutorials are now presented as sphinx style documentation at:
 
 We use sphinx-gallery's [notebook styled examples](https://sphinx-gallery.github.io/stable/tutorials/index.html) to create the tutorials. Syntax is very simple. In essence, you write a slightly well formatted python file and it shows up as documentation page.
 
-Here's how to create a new tutorial:
+Here's how to create a new tutorial or recipe:
 1. Create a notebook styled python file. If you want it executed while inserted into documentation, save the file with suffix `tutorial` so that file name is `your_tutorial.py`.
-2. Put it in one of the beginner_source, intermediate_source, advanced_source based on the level.
-2. Include it in the right TOC tree at index.rst
-3. Create a thumbnail in the index file using a command like `.. galleryitem:: beginner/your_tutorial.py`. (This is a custom directive. See `custom_directives.py` for more info.) 
+2. Put it in one of the beginner_source, intermediate_source, advanced_source based on the level. If it is a recipe, add to recipes_source.
+2. For Tutorials, include it in the TOC tree at index.rst
+3. For Tutorials, create a thumbnail in the [index.rst file](https://github.com/pytorch/tutorials/blob/master/index.rst) using a command like `.. customcarditem:: beginner/your_tutorial.html`. For Recipes, create a thumbnail in the [recipes_index.rst](https://github.com/pytorch/tutorials/blob/master/recipes_source/recipes_index.rst)
 
 In case you prefer to write your tutorial in jupyter, you can use [this script](https://gist.github.com/chsasank/7218ca16f8d022e02a9c0deb94a310fe) to convert the notebook to python file. After conversion and addition to the project, please make sure the sections headings etc are in logical order.
 
@@ -28,4 +28,4 @@ In case you prefer to write your tutorial in jupyter, you can use [this script](
 - Then you can build using `make docs`. This will download the data, execute the tutorials and build the documentation to `docs/` directory. This will take about 60-120 min for systems with GPUs. If you do not have a GPU installed on your system, then see next step.
 - You can skip the computationally intensive graph generation by running `make html-noplot` to build basic html documentation to `_build/html`. This way, you can quickly preview your tutorial.
 
-> If you get **ModuleNotFoundError: No module named 'pytorch_sphinx_theme' make: *** [html-noplot] Error 2**, from /tutorials/src/pytorch_sphinx_theme run `python setup.py install`. 
+> If you get **ModuleNotFoundError: No module named 'pytorch_sphinx_theme' make: *** [html-noplot] Error 2**, from /tutorials/src/pytorch-sphinx-theme run `python setup.py install`. 

advanced_source/dynamic_quantization_tutorial.py

@@ -1,5 +1,5 @@
 """
-(experimental) Dynamic Quantization on an LSTM Word Language Model
+(beta) Dynamic Quantization on an LSTM Word Language Model
 ==================================================================
 
 **Author**: `James Reed <https://github.com/jamesr66a>`_
@@ -13,7 +13,7 @@
 to int, which can result in smaller model size and faster inference with only a small
 hit to accuracy.
 
-In this tutorial, we'll apply the easiest form of quantization - 
+In this tutorial, we'll apply the easiest form of quantization -
 `dynamic quantization <https://pytorch.org/docs/stable/quantization.html#torch.quantization.quantize_dynamic>`_ -
 to an LSTM-based next word-prediction model, closely following the
 `word language model <https://github.com/pytorch/examples/tree/master/word_language_model>`_

advanced_source/static_quantization_tutorial.py

@@ -1,5 +1,5 @@
 """
-(experimental) Static Quantization with Eager Mode in PyTorch
+(beta) Static Quantization with Eager Mode in PyTorch
 =========================================================
 
 **Author**: `Raghuraman Krishnamoorthi <https://github.com/raghuramank100>`_

conf.py

@@ -35,6 +35,8 @@
 import glob
 import shutil
 from custom_directives import IncludeDirective, GalleryItemDirective, CustomGalleryItemDirective, CustomCalloutItemDirective, CustomCardItemDirective
+import distutils.file_util
+import re
 
 
 try:
@@ -63,10 +65,20 @@
     'examples_dirs': ['beginner_source', 'intermediate_source',
                       'advanced_source', 'recipes_source'],
     'gallery_dirs': ['beginner', 'intermediate', 'advanced', 'recipes'],
-    'filename_pattern': os.environ.get('GALLERY_PATTERN', r'tutorial.py'),
+    'filename_pattern': 'tutorial.py',
     'backreferences_dir': False
 }
 
+if os.getenv('GALLERY_PATTERN'):
+    # GALLERY_PATTERN is to be used when you want to work on a single
+    # tutorial.  Previously this was fed into filename_pattern, but
+    # if you do that, you still end up parsing all of the other Python
+    # files which takes a few seconds.  This strategy is better, as
+    # ignore_pattern also skips parsing.
+    # See https://github.com/sphinx-gallery/sphinx-gallery/issues/721
+    # for a more detailed description of the issue.
+    sphinx_gallery_conf['ignore_pattern'] = r'/(?!' + re.escape(os.getenv('GALLERY_PATTERN')) + r')[^/]+$'
+
 for i in range(len(sphinx_gallery_conf['examples_dirs'])):
     gallery_dir = sphinx_gallery_conf['gallery_dirs'][i]
     source_dir = sphinx_gallery_conf['examples_dirs'][i]
@@ -78,7 +90,7 @@
 
     # Copy rst files from source dir to gallery dir
     for f in glob.glob(os.path.join(source_dir, '*.rst')):
-        shutil.copy(f, gallery_dir)
+        distutils.file_util.copy_file(f, gallery_dir, update=True)
 
 
 # Add any paths that contain templates here, relative to this directory.

index.rst

@@ -9,7 +9,7 @@ Welcome to PyTorch Tutorials
 .. Add callout items below this line
 
 .. customcalloutitem::
-   :description: The 60 min blitz is the most common starting point and provides a broad view on how to use PyTorch. It covers the basics all to the way constructing deep neural networks.
+   :description: The 60 min blitz is the most common starting point and provides a broad view on how to use PyTorch. It covers the basics all the way to constructing deep neural networks.
    :header: New to PyTorch?
    :button_link: beginner/deep_learning_60min_blitz.html
    :button_text: Start 60-min blitz
@@ -203,14 +203,14 @@ Welcome to PyTorch Tutorials
 .. Frontend APIs
 
 .. customcarditem::
-   :header: (experimental) Introduction to Named Tensors in PyTorch
+   :header: (prototype) Introduction to Named Tensors in PyTorch
    :card_description: Learn how to use PyTorch to train a Deep Q Learning (DQN) agent on the CartPole-v0 task from the OpenAI Gym.
    :image: _static/img/thumbnails/cropped/experimental-Introduction-to-Named-Tensors-in-PyTorch.png
    :link: intermediate/memory_format_tutorial.html
    :tags: Frontend-APIs,Named-Tensor,Best-Practice
 
 .. customcarditem::
-   :header: (experimental) Channels Last Memory Format in PyTorch
+   :header: (beta) Channels Last Memory Format in PyTorch
    :card_description: Get an overview of Channels Last memory format and understand how it is used to order NCHW tensors in memory preserving dimensions.
    :image: _static/img/thumbnails/cropped/experimental-Channels-Last-Memory-Format-in-PyTorch.png
    :link: intermediate/memory_format_tutorial.html
@@ -261,28 +261,28 @@ Welcome to PyTorch Tutorials
    :tags: Model-Optimization,Best-Practice
 
 .. customcarditem::
-   :header: (experimental) Dynamic Quantization on an LSTM Word Language Model
+   :header: (beta) Dynamic Quantization on an LSTM Word Language Model
    :card_description: Apply dynamic quantization, the easiest form of quantization, to a LSTM-based next word prediction model.
    :image: _static/img/thumbnails/cropped/experimental-Dynamic-Quantization-on-an-LSTM-Word-Language-Model.png
    :link: advanced/dynamic_quantization_tutorial.html
    :tags: Text,Quantization,Model-Optimization
 
 .. customcarditem::
-   :header: (experimental) Dynamic Quantization on BERT
+   :header: (beta) Dynamic Quantization on BERT
    :card_description: Apply the dynamic quantization on a BERT (Bidirectional Embedding Representations from Transformers) model.
    :image: _static/img/thumbnails/cropped/experimental-Dynamic-Quantization-on-BERT.png
    :link: intermediate/dynamic_quantization_bert_tutorial.html
    :tags: Text,Quantization,Model-Optimization
 
 .. customcarditem::
-   :header: (experimental) Static Quantization with Eager Mode in PyTorch
+   :header: (beta) Static Quantization with Eager Mode in PyTorch
    :card_description: Learn techniques to impove a model's accuracy =  post-training static quantization, per-channel quantization, and quantization-aware training.
    :image: _static/img/thumbnails/cropped/experimental-Static-Quantization-with-Eager-Mode-in-PyTorch.png
    :link: advanced/static_quantization_tutorial.html
    :tags: Image/Video,Quantization,Model-Optimization
 
 .. customcarditem::
-   :header: (experimental) Quantized Transfer Learning for Computer Vision Tutorial
+   :header: (beta) Quantized Transfer Learning for Computer Vision Tutorial
    :card_description: Learn techniques to impove a model's accuracy -  post-training static quantization, per-channel quantization, and quantization-aware training.
    :image: _static/img/thumbnails/cropped/experimental-Quantized-Transfer-Learning-for-Computer-Vision-Tutorial.png
    :link: advanced/static_quantization_tutorial.html

intermediate_source/dist_tuto.rst

@@ -394,29 +394,28 @@ using point-to-point collectives.
 
     """ Implementation of a ring-reduce with addition. """
     def allreduce(send, recv):
-        rank = dist.get_rank()
-        size = dist.get_world_size()
-        send_buff = th.zeros(send.size())
-        recv_buff = th.zeros(send.size())
-        accum = th.zeros(send.size())
-        accum[:] = send[:]
-
-        left = ((rank - 1) + size) % size
-        right = (rank + 1) % size
-
-        for i in range(size - 1):
-            if i % 2 == 0:
-                # Send send_buff
-                send_req = dist.isend(send_buff, right)
-                dist.recv(recv_buff, left)
-                accum[:] += recv[:]
-            else:
-                # Send recv_buff
-                send_req = dist.isend(recv_buff, right)
-                dist.recv(send_buff, left)
-                accum[:] += send[:]
-            send_req.wait()
-        recv[:] = accum[:]
+       rank = dist.get_rank()
+       size = dist.get_world_size()
+       send_buff = send.clone()
+       recv_buff = send.clone()
+       accum = send.clone()
+
+       left = ((rank - 1) + size) % size
+       right = (rank + 1) % size
+
+       for i in range(size - 1):
+           if i % 2 == 0:
+               # Send send_buff
+               send_req = dist.isend(send_buff, right)
+               dist.recv(recv_buff, left)
+               accum[:] += recv_buff[:]
+           else:
+               # Send recv_buff
+               send_req = dist.isend(recv_buff, right)
+               dist.recv(send_buff, left)
+               accum[:] += send_buff[:]
+           send_req.wait()
+       recv[:] = accum[:]
 
 In the above script, the ``allreduce(send, recv)`` function has a
 slightly different signature than the ones in PyTorch. It takes a

intermediate_source/dynamic_quantization_bert_tutorial.rst

@@ -1,10 +1,10 @@
-(experimental) Dynamic Quantization on BERT
+(beta) Dynamic Quantization on BERT
 ===========================================
 
 .. tip::
-   To get the most of this tutorial, we suggest using this 
+   To get the most of this tutorial, we suggest using this
    `Colab Version <https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/dynamic_quantization_bert_tutorial.ipynb>`_. This will allow you to experiment with the information presented below.
- 
+
 **Author**: `Jianyu Huang <https://github.com/jianyuh>`_
 
 **Reviewed by**: `Raghuraman Krishnamoorthi <https://github.com/raghuramank100>`_
@@ -71,7 +71,7 @@ built-in F1 score calculation helper function.
    pip install transformers
 
 
-Because we will be using the experimental parts of the PyTorch, it is
+Because we will be using the beta parts of the PyTorch, it is
 recommended to install the latest version of torch and torchvision. You
 can find the most recent instructions on local installation `here
 <https://pytorch.org/get-started/locally/>`_. For example, to install on

intermediate_source/memory_format_tutorial.py

@@ -1,6 +1,6 @@
 # -*- coding: utf-8 -*-
 """
-(experimental) Channels Last Memory Format in PyTorch
+(beta) Channels Last Memory Format in PyTorch
 *******************************************************
 **Author**: `Vitaly Fedyunin <https://github.com/VitalyFedyunin>`_
 
@@ -11,12 +11,12 @@
 
 For example, classic (contiguous) storage of NCHW tensor (in our case it is two 2x2 images with 3 color channels) look like this:
 
-.. figure:: /_static/img/classic_memory_format.png   
+.. figure:: /_static/img/classic_memory_format.png
    :alt: classic_memory_format
 
 Channels Last memory format orders data differently:
 
-.. figure:: /_static/img/channels_last_memory_format.png   
+.. figure:: /_static/img/channels_last_memory_format.png
    :alt: channels_last_memory_format
 
 Pytorch supports memory formats (and provides back compatibility with existing models including eager, JIT, and TorchScript) by utilizing  existing strides structure.
@@ -34,7 +34,7 @@
 # Memory Format API
 # -----------------------
 #
-# Here is how to convert tensors between contiguous and channels 
+# Here is how to convert tensors between contiguous and channels
 # last memory formats.
 
 ######################################################################
@@ -104,9 +104,9 @@
 ######################################################################
 # Performance Gains
 # -------------------------------------------------------------------------------------------
-# The most significant performance gains are observed on NVidia's hardware with 
+# The most significant performance gains are observed on Nvidia's hardware with
 # Tensor Cores support. We were able to archive over 22%  perf gains while running '
-# AMP (Automated Mixed Precision) training scripts supplied by NVidia https://github.com/NVIDIA/apex.
+# AMP (Automated Mixed Precision) training scripts supplied by Nvidia https://github.com/NVIDIA/apex.
 #
 # ``python main_amp.py -a resnet50 --b 200 --workers 16 --opt-level O2  ./data``
 
@@ -144,7 +144,7 @@
 
 ######################################################################
 # Passing ``--channels-last true`` allows running a model in Channels Last format with observed 22% perf gain.
-# 
+#
 # ``python main_amp.py -a resnet50 --b 200 --workers 16 --opt-level O2 --channels-last true ./data``
 
 # opt_level = O2
@@ -192,7 +192,7 @@
 # Converting existing models
 # --------------------------
 #
-# Channels Last support not limited by existing models, as any model can be converted to Channels Last and propagate format through the graph as soon as input formatted correctly. 
+# Channels Last support not limited by existing models, as any model can be converted to Channels Last and propagate format through the graph as soon as input formatted correctly.
 #
 
 # Need to be done once, after model initialization (or load)
@@ -203,12 +203,12 @@
 output = model(input)
 
 #######################################################################
-# However, not all operators fully converted to support Channels Last (usually returning 
-# contiguous output instead). That means you need to verify the list of used operators 
-# against supported operators list https://github.com/pytorch/pytorch/wiki/Operators-with-Channels-Last-support, 
+# However, not all operators fully converted to support Channels Last (usually returning
+# contiguous output instead). That means you need to verify the list of used operators
+# against supported operators list https://github.com/pytorch/pytorch/wiki/Operators-with-Channels-Last-support,
 # or introduce memory format checks into eager execution mode and run your model.
-# 
-# After running the code below, operators will raise an exception if the output of the 
+#
+# After running the code below, operators will raise an exception if the output of the
 # operator doesn't match the memory format of the input.
 #
 #
@@ -282,7 +282,7 @@ def attribute(m):
 
 ######################################################################
 # If you found an operator that doesn't support Channels Last tensors
-# and you want to contribute, feel free to use following developers 
+# and you want to contribute, feel free to use following developers
 # guide https://github.com/pytorch/pytorch/wiki/Writing-memory-format-aware-operators.
 #
 

intermediate_source/named_tensor_tutorial.py

@@ -1,6 +1,6 @@
 # -*- coding: utf-8 -*-
 """
-(experimental) Introduction to Named Tensors in PyTorch
+(prototype) Introduction to Named Tensors in PyTorch
 *******************************************************
 **Author**: `Richard Zou <https://github.com/zou3519>`_
 

intermediate_source/quantized_transfer_learning_tutorial.rst

@@ -1,10 +1,10 @@
-(experimental) Quantized Transfer Learning for Computer Vision Tutorial
+(beta) Quantized Transfer Learning for Computer Vision Tutorial
 ========================================================================
 
 .. tip::
-   To get the most of this tutorial, we suggest using this 
-   `Colab Version <https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/quantized_transfer_learning_tutorial.ipynb>`_. 
-   This will allow you to experiment with the information presented below. 
+   To get the most of this tutorial, we suggest using this
+   `Colab Version <https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/quantized_transfer_learning_tutorial.ipynb>`_.
+   This will allow you to experiment with the information presented below.
 
 **Author**: `Zafar Takhirov <https://github.com/z-a-f>`_
 
@@ -62,7 +62,7 @@ such as installations and data loading/visualizations.
 Installing the Nightly Build
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Because you will be using the experimental parts of the PyTorch, it is
+Because you will be using the beta parts of the PyTorch, it is
 recommended to install the latest version of ``torch`` and
 ``torchvision``. You can find the most recent instructions on local
 installation `here <https://pytorch.org/get-started/locally/>`_.

intermediate_source/rpc_tutorial.rst

@@ -5,7 +5,7 @@ Getting Started with Distributed RPC Framework
 
 This tutorial uses two simple examples to demonstrate how to build distributed
 training with the `torch.distributed.rpc <https://pytorch.org/docs/master/rpc.html>`__
-package which is first introduced as an experimental feature in PyTorch v1.4.
+package which is first introduced as a prototype feature in PyTorch v1.4.
 Source code of the two examples can be found in
 `PyTorch examples <https://github.com/pytorch/examples>`__.