fine tuned intel_extension_for_pytorch. 1. moved intel_extension_for_pytorch.py to intel_extension_for_pytorch.rst to make it compatible to cpp source codes. 2. fixed bug of Float32 level missing in intel_extension_for_pytorch (#1704)

Author: jingxu10

recipes_source/intel_extension_for_pytorch.rst

@@ -0,0 +1,339 @@
+Intel® Extension for PyTorch*
+=============================
+
+Intel Extension for PyTorch* extends PyTorch with optimizations for extra
+performance boost on Intel hardware. Most of the optimizations will be
+included in stock PyTorch releases eventually, and the intention of the
+extension is to deliver up to date features and optimizations for PyTorch
+on Intel hardware, examples include AVX-512 Vector Neural Network
+Instructions (AVX512 VNNI) and Intel® Advanced Matrix Extensions (Intel® AMX).
+
+Intel® Extension for PyTorch* has been released as an open–source project
+at `Github <https://github.com/intel/intel-extension-for-pytorch>`_.
+
+Features
+--------
+
+- **Ease-of-use Python API:** Intel® Extension for PyTorch* provides simple
+  frontend Python APIs and utilities for users to get performance optimizations
+  such as graph optimization and operator optimization with minor code changes.
+  Typically, only 2 to 3 clauses are required to be added to the original code.
+- **Channels Last:** Comparing to the default NCHW memory format, channels_last
+  (NHWC) memory format could further accelerate convolutional neural networks.
+  In Intel® Extension for PyTorch*, NHWC memory format has been enabled for
+  most key CPU operators, though not all of them have been merged to PyTorch
+  master branch yet. They are expected to be fully landed in PyTorch upstream
+  soon.
+- **Auto Mixed Precision (AMP):** Low precision data type BFloat16 has been
+  natively supported on the 3rd Generation Xeon scalable Servers (aka Cooper
+  Lake) with AVX512 instruction set and will be supported on the next
+  generation of Intel® Xeon® Scalable Processors with Intel® Advanced Matrix
+  Extensions (Intel® AMX) instruction set with further boosted performance. The
+  support of Auto Mixed Precision (AMP) with BFloat16 for CPU and BFloat16
+  optimization of operators have been massively enabled in Intel® Extension
+  for PyTorch*, and partially upstreamed to PyTorch master branch. Most of
+  these optimizations will be landed in PyTorch master through PRs that are
+  being submitted and reviewed.
+- **Graph Optimization:** To optimize performance further with torchscript,
+  Intel® Extension for PyTorch* supports fusion of frequently used operator
+  patterns, like Conv2D+ReLU, Linear+ReLU, etc. The benefit of the fusions are
+  delivered to users in a transparent fashion. Detailed fusion patterns
+  supported can be found `here <https://github.com/intel/intel-extension-for-pytorch>`_.
+  The graph optimization will be up-streamed to PyTorch with the introduction
+  of oneDNN Graph API.
+- **Operator Optimization:** Intel® Extension for PyTorch* also optimizes
+  operators and implements several customized operators for performance. A few
+  ATen operators are replaced by their optimized counterparts in Intel®
+  Extension for PyTorch* via ATen registration mechanism. Moreover, some
+  customized operators are implemented for several popular topologies. For
+  instance, ROIAlign and NMS are defined in Mask R-CNN. To improve performance
+  of these topologies, Intel® Extension for PyTorch* also optimized these
+  customized operators.
+
+Getting Started
+---------------
+
+Minor code changes are required for users to get start with Intel® Extension
+for PyTorch*. Both PyTorch imperative mode and TorchScript mode are
+supported. This section introduces usage of Intel® Extension for PyTorch* API
+functions for both imperative mode and TorchScript mode, covering data type
+Float32 and BFloat16. C++ usage will also be introduced at the end.
+
+You just need to import Intel® Extension for PyTorch* package and apply its
+optimize function against the model object. If it is a training workload, the
+optimize function also needs to be applied against the optimizer object.
+
+For training and inference with BFloat16 data type, torch.cpu.amp has been
+enabled in PyTorch upstream to support mixed precision with convenience, and
+BFloat16 datatype has been enabled excessively for CPU operators in PyTorch
+upstream and Intel® Extension for PyTorch*. Running torch.cpu.amp will match
+each operator to its appropriate datatype and returns the best possible
+performance.
+
+The code changes that are required for Intel® Extension for PyTorch* are
+highlighted with comments in a line above.
+
+Training
+~~~~~~~~
+
+Float32
+^^^^^^^
+
+.. code:: python3
+
+   import torch
+   import torch.nn as nn
+   # Import intel_extension_for_pytorch
+   import intel_extension_for_pytorch as ipex
+   
+   class Model(nn.Module):
+       def __init__(self):
+           super(Model, self).__init__()
+           self.linear = nn.Linear(4, 5)
+   
+       def forward(self, input):
+           return self.linear(input)
+   
+   model = Model()
+   model.set_state_dict(torch.load(PATH))
+   optimizer.set_state_dict(torch.load(PATH))
+   # Invoke optimize function against the model object and optimizer object
+   model, optimizer = ipex.optimize(model, optimizer, dtype=torch.float32)
+   
+   for images, label in train_loader():
+       # Setting memory_format to torch.channels_last could improve performance with 4D input data. This is optional.
+       images = images.to(memory_format=torch.channels_last)
+       loss = criterion(model(images), label)
+       loss.backward()
+       optimizer.step()
+   torch.save(model.state_dict(), PATH)
+   torch.save(optimizer.state_dict(), PATH)
+
+BFloat16
+^^^^^^^^
+
+.. code:: python3
+
+   import torch
+   import torch.nn as nn
+   # Import intel_extension_for_pytorch
+   import intel_extension_for_pytorch as ipex
+   
+   class Model(nn.Module):
+       def __init__(self):
+           super(Model, self).__init__()
+           self.linear = nn.Linear(4, 5)
+   
+       def forward(self, input):
+           return self.linear(input)
+   
+   model = Model()
+   model.set_state_dict(torch.load(PATH))
+   optimizer.set_state_dict(torch.load(PATH))
+   # Invoke optimize function against the model object and optimizer object with data type set to torch.bfloat16
+   model, optimizer = ipex.optimize(model, optimizer, dtype=torch.bfloat16)
+   
+   for images, label in train_loader():
+       with torch.cpu.amp.autocast():
+           # Setting memory_format to torch.channels_last could improve performance with 4D input data. This is optional.
+           images = images.to(memory_format=torch.channels_last)
+           loss = criterion(model(images), label)
+       loss.backward()
+       optimizer.step()
+   torch.save(model.state_dict(), PATH)
+   torch.save(optimizer.state_dict(), PATH)
+
+Inference - Imperative Mode
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Float32
+^^^^^^^
+
+.. code:: python3
+
+   import torch
+   import torch.nn as nn
+   # Import intel_extension_for_pytorch
+   import intel_extension_for_pytorch as ipex
+   
+   class Model(nn.Module):
+       def __init__(self):
+           super(Model, self).__init__()
+           self.linear = nn.Linear(4, 5)
+   
+       def forward(self, input):
+           return self.linear(input)
+   
+   input = torch.randn(2, 4)
+   model = Model()
+   model.eval()
+   # Invoke optimize function against the model object
+   model = ipex.optimize(model, dtype=torch.float32)
+   res = model(input)
+
+BFloat16
+^^^^^^^^
+
+.. code:: python3
+
+   import torch
+   import torch.nn as nn
+   # Import intel_extension_for_pytorch
+   import intel_extension_for_pytorch as ipex
+   
+   class Model(nn.Module):
+       def __init__(self):
+           super(Model, self).__init__()
+           self.linear = nn.Linear(4, 5)
+   
+       def forward(self, input):
+           return self.linear(input)
+   
+   input = torch.randn(2, 4)
+   model = Model()
+   model.eval()
+   # Invoke optimize function against the model object with data type set to torch.bfloat16
+   model = ipex.optimize(model, dtype=torch.bfloat16)
+   with torch.cpu.amp.autocast():
+       res = model(input)
+ 
+Inference - TorchScript Mode
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+TorchScript mode makes graph optimization possible, hence improves
+performance for some topologies. Intel® Extension for PyTorch* enables most
+commonly used operator pattern fusion, and users can get the performance
+benefit without additional code changes.
+
+Float32
+^^^^^^^
+
+.. code:: python3
+
+   import torch
+   import torch.nn as nn
+   # Import intel_extension_for_pytorch
+   import intel_extension_for_pytorch as ipex
+   
+   # oneDNN graph fusion is enabled by default, uncomment the line below to disable it explicitly
+   # ipex.enable_onednn_fusion(False)
+   
+   class Model(nn.Module):
+       def __init__(self):
+           super(Model, self).__init__()
+           self.linear = nn.Linear(4, 5)
+   
+       def forward(self, input):
+           return self.linear(input)
+   
+   input = torch.randn(2, 4)
+   model = Model()
+   model.eval()
+   # Invoke optimize function against the model object
+   model = ipex.optimize(model, dtype=torch.float32)
+   model = torch.jit.trace(model, torch.randn(2, 4))
+   model = torch.jit.freeze(model)
+   res = model(input)
+
+BFloat16
+^^^^^^^^
+
+.. code:: python3
+
+   import torch
+   import torch.nn as nn
+   # Import intel_extension_for_pytorch
+   import intel_extension_for_pytorch as ipex
+   
+   # oneDNN graph fusion is enabled by default, uncomment the line below to disable it explicitly
+   # ipex.enable_onednn_fusion(False)
+   
+   class Model(nn.Module):
+       def __init__(self):
+           super(Model, self).__init__()
+           self.linear = nn.Linear(4, 5)
+   
+       def forward(self, input):
+           return self.linear(input)
+   
+   input = torch.randn(2, 4)
+   model = Model()
+   model.eval()
+   # Invoke optimize function against the model with data type set to torch.bfloat16
+   model = ipex.optimize(model, dtype=torch.bfloat16)
+   with torch.cpu.amp.autocast():
+       model = torch.jit.trace(model, torch.randn(2, 4))
+       model = torch.jit.freeze(model)
+       res = model(input)
+
+C++
+~~~
+
+To work with libtorch, C++ library of PyTorch, Intel® Extension for PyTorch*
+provides its C++ dynamic library as well. The C++ library is supposed to handle
+inference workload only, such as service deployment. For regular development,
+please use Python interface. Comparing to usage of libtorch, no specific code
+changes are required, except for converting input data into channels last data
+format. Compilation follows the recommended methodology with CMake. Detailed
+instructions can be found in `PyTorch tutorial <https://pytorch.org/tutorials/advanced/cpp_export.html#depending-on-libtorch-and-building-the-application>`_.
+During compilation, Intel optimizations will be activated automatically
+once C++ dynamic library of Intel® Extension for PyTorch* is linked.
+
+**example-app.cpp**
+
+.. code:: cpp
+
+   #include <torch/script.h>
+   #include <iostream>
+   #include <memory>
+   
+   int main(int argc, const char* argv[]) {
+       torch::jit::script::Module module;
+       try {
+           module = torch::jit::load(argv[1]);
+       }
+       catch (const c10::Error& e) {
+           std::cerr << "error loading the model\n";
+           return -1;
+       }
+       std::vector<torch::jit::IValue> inputs;
+       // make sure input data are converted to channels last format
+       inputs.push_back(torch::ones({1, 3, 224, 224}).to(c10::MemoryFormat::ChannelsLast));
+   
+       at::Tensor output = module.forward(inputs).toTensor();
+   
+       return 0;
+   }
+
+**CMakeList.txt**
+
+::
+
+   cmake_minimum_required(VERSION 3.0 FATAL_ERROR)
+   project(example-app)
+   
+   find_package(Torch REQUIRED)
+   set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${TORCH_CXX_FLAGS} -Wl,--no-as-needed")
+   
+   add_executable(example-app example-app.cpp)
+   # Link the binary against the C++ dynamic library file of Intel® Extension for PyTorch*
+   target_link_libraries(example-app "${TORCH_LIBRARIES}" "${INTEL_EXTENSION_FOR_PYTORCH_PATH}/lib/libintel-ext-pt-cpu.so")
+
+   set_property(TARGET example-app PROPERTY CXX_STANDARD 14)
+
+**Note:** Since Intel® Extension for PyTorch* is still under development, name of
+the c++ dynamic library in the master branch may defer to
+*libintel-ext-pt-cpu.so* shown above. Please check the name out in the
+installation folder. The so file name starts with *libintel-*.
+
+**Command for compilation**
+
+::
+
+   $ cmake -DCMAKE_PREFIX_PATH=<LIBPYTORCH_PATH> -DINTEL_EXTENSION_FOR_PYTORCH_PATH=<INTEL_EXTENSION_FOR_PYTORCH_INSTALLATION_PATH> ..
+   $ make
+
+Tutorials
+---------
+
+Please visit `Intel® Extension for PyTorch* Github repo <https://github.com/intel/intel-extension-for-pytorch>`_ for more tutorials.