Merge branch 'master' into master

Author: brianjo

advanced_source/cpp_export.rst

@@ -1,8 +1,6 @@
 Loading a TorchScript Model in C++
 =====================================
 
-**This tutorial was updated to work with PyTorch 1.2**
-
 As its name suggests, the primary interface to PyTorch is the Python
 programming language. While Python is a suitable and preferred language for
 many scenarios requiring dynamism and ease of iteration, there are equally many

advanced_source/cpp_frontend.rst

@@ -698,110 +698,40 @@ The Generator Module
 ********************
 
 We begin by defining the generator module, which consists of a series of
-transposed 2D convolutions, batch normalizations and ReLU activation units. Like
-in Python, PyTorch here provides two APIs for model definition: a functional one
-where inputs are passed through successive functions, and a more object-oriented
-one where we build a ``Sequential`` module containing the entire model as
-submodules. Let's see how our generator looks with either API, and you can
-decide for yourself which one you prefer. First, using ``Sequential``:
+transposed 2D convolutions, batch normalizations and ReLU activation units.
+We explicitly pass inputs (in a functional way) between modules in the
+``forward()`` method of a module we define ourselves:
 
 .. code-block:: cpp
 
-  using namespace torch;
-
-  nn::Sequential generator(
-      // Layer 1
-      nn::Conv2d(nn::Conv2dOptions(kNoiseSize, 256, 4)
-                     .with_bias(false)
-                     .transposed(true)),
-      nn::BatchNorm(256),
-      nn::Functional(torch::relu),
-      // Layer 2
-      nn::Conv2d(nn::Conv2dOptions(256, 128, 3)
-                     .stride(2)
-                     .padding(1)
-                     .with_bias(false)
-                     .transposed(true)),
-      nn::BatchNorm(128),
-      nn::Functional(torch::relu),
-      // Layer 3
-      nn::Conv2d(nn::Conv2dOptions(128, 64, 4)
-                     .stride(2)
-                     .padding(1)
-                     .with_bias(false)
-                     .transposed(true)),
-      nn::BatchNorm(64),
-      nn::Functional(torch::relu),
-      // Layer 4
-      nn::Conv2d(nn::Conv2dOptions(64, 1, 4)
-                     .stride(2)
-                     .padding(1)
-                     .with_bias(false)
-                     .transposed(true)),
-      nn::Functional(torch::tanh));
-
-.. tip::
-
-	A ``Sequential`` module simply performs function composition. The output of
-	the first submodule becomes the input of the second, the output of the third
-	becomes the input of the fourth and so on.
-
-The particular modules chosen, like ``nn::Conv2d`` and ``nn::BatchNorm``,
-follows the structure outlined earlier. The ``kNoiseSize`` constant determines
-the size of the input noise vector and is set to ``100``. Notice also that we
-use the ``torch::nn::Functional`` module for our activation functions, passing
-it ``torch::relu`` for inner layers and ``torch::tanh`` as the final activation.
-Hyperparameters were, of course, found via grad student descent.
-
-.. note::
-
-	The Python frontend has one module for each activation function, like
-	``torch.nn.ReLU`` or ``torch.nn.Tanh``. In C++, we instead only provide the
-	``Functional`` module, to which you can pass any C++ function that will be
-	called inside the ``Functional``'s ``forward()`` method.
-
-.. attention::
-
-	No grad students were harmed in the discovery of hyperparameters. They were
-	fed Soylent regularly.
-
-For the second approach, we explicitly pass inputs (in a functional way) between
-modules in the ``forward()`` method of a module we define ourselves:
-
-.. code-block:: cpp
-
-  struct GeneratorImpl : nn::Module {
-    GeneratorImpl(int kNoiseSize)
-        : conv1(nn::Conv2dOptions(kNoiseSize, 256, 4)
-                    .with_bias(false)
-                    .transposed(true)),
+  struct DCGANGeneratorImpl : nn::Module {
+    DCGANGeneratorImpl(int kNoiseSize)
+        : conv1(nn::ConvTranspose2dOptions(kNoiseSize, 256, 4)
+                    .bias(false)),
           batch_norm1(256),
-          conv2(nn::Conv2dOptions(256, 128, 3)
+          conv2(nn::ConvTranspose2dOptions(256, 128, 3)
                     .stride(2)
                     .padding(1)
-                    .with_bias(false)
-                    .transposed(true)),
+                    .bias(false)),
           batch_norm2(128),
-          conv3(nn::Conv2dOptions(128, 64, 4)
+          conv3(nn::ConvTranspose2dOptions(128, 64, 4)
                     .stride(2)
                     .padding(1)
-                    .with_bias(false)
-                    .transposed(true)),
+                    .bias(false)),
           batch_norm3(64),
-          conv4(nn::Conv2dOptions(64, 1, 4)
+          conv4(nn::ConvTranspose2dOptions(64, 1, 4)
                     .stride(2)
                     .padding(1)
-                    .with_bias(false)
-                    .transposed(true))
+                    .bias(false))
    {
      // register_module() is needed if we want to use the parameters() method later on
      register_module("conv1", conv1);
      register_module("conv2", conv2);
      register_module("conv3", conv3);
      register_module("conv4", conv4);
      register_module("batch_norm1", batch_norm1);
-     register_module("batch_norm2", batch_norm1);
-     register_module("batch_norm3", batch_norm1);
+     register_module("batch_norm2", batch_norm2);
+     register_module("batch_norm3", batch_norm3);
    }
 
    torch::Tensor forward(torch::Tensor x) {
@@ -812,25 +742,34 @@ modules in the ``forward()`` method of a module we define ourselves:
      return x;
    }
 
-   nn::Conv2d conv1, conv2, conv3, conv4;
-   nn::BatchNorm batch_norm1, batch_norm2, batch_norm3;
+   nn::ConvTranspose2d conv1, conv2, conv3, conv4;
+   nn::BatchNorm2d batch_norm1, batch_norm2, batch_norm3;
   };
-  TORCH_MODULE(Generator);
+  TORCH_MODULE(DCGANGenerator);
+
+  DCGANGenerator generator(kNoiseSize);
+
+We can now invoke ``forward()`` on the ``DCGANGenerator`` to map a noise sample to an image.
+
+The particular modules chosen, like ``nn::ConvTranspose2d`` and ``nn::BatchNorm2d``,
+follows the structure outlined earlier. The ``kNoiseSize`` constant determines
+the size of the input noise vector and is set to ``100``. Hyperparameters were,
+of course, found via grad student descent.
 
-  Generator generator;
+.. attention::
 
-Whichever approach we use, we can now invoke ``forward()`` on the ``Generator`` to
-map a noise sample to an image.
+	No grad students were harmed in the discovery of hyperparameters. They were
+	fed Soylent regularly.
 
 .. note::
 
 	A brief word on the way options are passed to built-in modules like ``Conv2d``
 	in the C++ frontend: Every module has some required options, like the number
-	of features for ``BatchNorm``. If you only need to configure the required
+	of features for ``BatchNorm2d``. If you only need to configure the required
 	options, you can pass them directly to the module's constructor, like
-	``BatchNorm(128)`` or ``Dropout(0.5)`` or ``Conv2d(8, 4, 2)`` (for input
+	``BatchNorm2d(128)`` or ``Dropout(0.5)`` or ``Conv2d(8, 4, 2)`` (for input
 	channel count, output channel count, and kernel size). If, however, you need
-	to modify other options, which are normally defaulted, such as ``with_bias``
+	to modify other options, which are normally defaulted, such as ``bias``
 	for ``Conv2d``, you need to construct and pass an *options* object. Every
 	module in the C++ frontend has an associated options struct, called
 	``ModuleOptions`` where ``Module`` is the name of the module, like
@@ -845,36 +784,42 @@ and activations. However, the convolutions are now regular ones instead of
 transposed, and we use a leaky ReLU with an alpha value of 0.2 instead of a
 vanilla ReLU. Also, the final activation becomes a Sigmoid, which squashes
 values into a range between 0 and 1. We can then interpret these squashed values
-as the probabilities the discriminator assigns to images being real:
+as the probabilities the discriminator assigns to images being real.
+
+To build the discriminator, we will try something different: a `Sequential` module.
+Like in Python, PyTorch here provides two APIs for model definition: a functional one
+where inputs are passed through successive functions (e.g. the generator module example),
+and a more object-oriented one where we build a `Sequential` module containing the
+entire model as submodules. Using `Sequential`, the discriminator would look like:
 
 .. code-block:: cpp
 
   nn::Sequential discriminator(
     // Layer 1
     nn::Conv2d(
-        nn::Conv2dOptions(1, 64, 4).stride(2).padding(1).with_bias(false)),
-    nn::Functional(torch::leaky_relu, 0.2),
+        nn::Conv2dOptions(1, 64, 4).stride(2).padding(1).bias(false)),
+    nn::LeakyReLU(nn::LeakyReLUOptions().negative_slope(0.2)),
     // Layer 2
     nn::Conv2d(
-        nn::Conv2dOptions(64, 128, 4).stride(2).padding(1).with_bias(false)),
-    nn::BatchNorm(128),
-    nn::Functional(torch::leaky_relu, 0.2),
+        nn::Conv2dOptions(64, 128, 4).stride(2).padding(1).bias(false)),
+    nn::BatchNorm2d(128),
+    nn::LeakyReLU(nn::LeakyReLUOptions().negative_slope(0.2)),
     // Layer 3
     nn::Conv2d(
-        nn::Conv2dOptions(128, 256, 4).stride(2).padding(1).with_bias(false)),
-    nn::BatchNorm(256),
-    nn::Functional(torch::leaky_relu, 0.2),
+        nn::Conv2dOptions(128, 256, 4).stride(2).padding(1).bias(false)),
+    nn::BatchNorm2d(256),
+    nn::LeakyReLU(nn::LeakyReLUOptions().negative_slope(0.2)),
     // Layer 4
     nn::Conv2d(
-        nn::Conv2dOptions(256, 1, 3).stride(1).padding(0).with_bias(false)),
-    nn::Functional(torch::sigmoid));
+        nn::Conv2dOptions(256, 1, 3).stride(1).padding(0).bias(false)),
+    nn::Sigmoid());
 
-.. note::
+.. tip::
+
+  A ``Sequential`` module simply performs function composition. The output of
+  the first submodule becomes the input of the second, the output of the third
+  becomes the input of the fourth and so on.
 
-	When the function we pass to ``Functional`` takes more arguments than a single
-	tensor, we can pass them to the ``Functional`` constructor, which will forward
-	them to each function call. For the leaky ReLU above, this means
-	``torch::leaky_relu(previous_output_tensor, 0.2)`` is called.
 
 Loading Data
 ------------