{{ post.date | date: '%B %d, %Y' }}
-- {{ post.title }} -
-{{ post.excerpt | remove: '
' | remove: '
' | truncate: 500}} -
@@ -128,8 +133,8 @@ torchvision (0.10.0a0+300a8a4)
Although Jetson Inference includes models already converted to the TensorRT engine file format, you can fine-tune the models by following the steps in Transfer Learning with PyTorch (for Jetson Inference) [here](https://github.com/dusty-nv/jetson-inference/blob/master/docs/pytorch-transfer-learning.md).
### Using TensorRT
-[TensorRT](https://docs.nvidia.com/deeplearning/tensorrt/) is an SDK for high-performance inference from NVIDIA. Jetson Nano supports TensorRT via the Jetpack SDK, included in the SD Card image used to set up Jetson Nano. To confirm that TensorRT is already installed in Nano, `run dpkg -l|grep -i tensorrt`:
+[TensorRT](https://docs.nvidia.com/deeplearning/tensorrt/) is an SDK for high-performance inference from NVIDIA. Jetson Nano supports TensorRT via the Jetpack SDK, included in the SD Card image used to set up Jetson Nano. To confirm that TensorRT is already installed in Nano, `run dpkg -l|grep -i tensorrt`:
@@ -139,7 +144,7 @@ Theoretically, TensorRT can be used to “take a trained PyTorch model and optim
1. How to convert the model from PyTorch to ONNX;
-2. How to convert the ONNX model to a TensorRT engine file;
+2. How to convert the ONNX model to a TensorRT engine file;
3. How to run the engine file with the TensorRT runtime for performance improvement: inference time improved from the original 31.5ms/19.4ms (FP32/FP16 precision) to 6.28ms (TensorRT).
@@ -147,7 +152,7 @@ You can replace the Resnet50 model in the notebook code with another PyTorch mod
`Error Code 1: Cuda Runtime (all CUDA-capable devices are busy or unavailable)`
-You may also see an error when converting a PyTorch model to ONNX model, which may be fixed by replacing:
+You may also see an error when converting a PyTorch model to ONNX model, which may be fixed by replacing:
`torch.onnx.export(resnet50, dummy_input, "resnet50_pytorch.onnx", verbose=False)`
@@ -155,12 +160,13 @@ with:
`torch.onnx.export(model, dummy_input, "deeplabv3_pytorch.onnx", opset_version=11, verbose=False)`
-### Using PyTorch
+### Using PyTorch
+
First, to download and install PyTorch 1.9 on Nano, run the following commands (see [here](https://forums.developer.nvidia.com/t/pytorch-for-jetson-version-1-10-now-available/72048) for more information):
```
wget https://nvidia.box.com/shared/static/p57jwntv436lfrd78inwl7iml6p13fzh.whl -O torch-1.8.0-cp36-cp36m-linux_aarch64.whl -O torch-1.9.0-cp36-cp36m-linux_aarch64.whl
-sudo apt-get install python3-pip libopenblas-base libopenmpi-dev
+sudo apt-get install python3-pip libopenblas-base libopenmpi-dev
pip3 install Cython
pip3 install numpy torch-1.9.0-cp36-cp36m-linux_aarch64.whl
```
@@ -173,17 +179,18 @@ pip3 install torchvision-0.10.0a0+300a8a4-cp36-cp36m-linux_aarch64.whl
```
After the steps above, run this to confirm:
+
```
$ pip3 list|grep torch
torch (1.9.0)
torchvision (0.10.0)
```
-You can also use the docker image described in the section *Using Jetson Inference* (which also has PyTorch and torchvision installed), to skip the manual steps above.
+You can also use the docker image described in the section _Using Jetson Inference_ (which also has PyTorch and torchvision installed), to skip the manual steps above.
The official [YOLOv5](https://github.com/ultralytics/yolov5) repo is used to run the PyTorch YOLOv5 model on Jetson Nano. After logging in to Jetson Nano, follow the steps below:
-* Get the repo and install what’s required:
+- Get the repo and install what’s required:
```
git clone https://github.com/ultralytics/yolov5
@@ -191,13 +198,13 @@ cd yolov5
pip install -r requirements.txt
```
-* Run `python3 detect.py`, which by default uses the PyTorch yolov5s.pt model. You should see something like:
+- Run `python3 detect.py`, which by default uses the PyTorch yolov5s.pt model. You should see something like:
```
detect: weights=yolov5s.pt, source=data/images, imgsz=[640, 640], conf_thres=0.25, iou_thres=0.45, max_det=1000, device=, view_img=False, save_txt=False, save_conf=False, save_crop=False, nosave=False, classes=None, agnostic_nms=False, augment=False, visualize=False, update=False, project=runs/detect, name=exp, exist_ok=False, line_thickness=3, hide_labels=False, hide_conf=False, half=False
YOLOv5 🚀 v5.0-499-g48b00db torch 1.9.0 CUDA:0 (NVIDIA Tegra X1, 3956.1015625MB)
-Fusing layers...
+Fusing layers...
Model Summary: 224 layers, 7266973 parameters, 0 gradients
image 1/5 /home/jeff/repos/yolov5-new/yolov5/data/images/bus.jpg: 640x480 4 persons, 1 bus, 1 fire hydrant, Done. (0.142s)
...
@@ -206,6 +213,7 @@ image 1/5 /home/jeff/repos/yolov5-new/yolov5/data/images/bus.jpg: 640x480 4 pers
**The inference time on Jetson Nano GPU is about 140ms, more than twice as fast as the inference time on iOS or Android (about 330ms).**
If you get an error `“ImportError: The _imagingft C module is not installed.”` then you need to reinstall pillow:
+
```
sudo apt-get install libpng-dev
sudo apt-get install libfreetype6-dev
@@ -231,29 +239,30 @@ Using the same test files used in the PyTorch iOS YOLOv5 demo app or Android YOL
@@ -68,18 +72,18 @@ Figure 3: Excerpt of an AI-generated medical report. HPI stands for History of p
[PyTorch](https://pytorch.org/) is an open-source machine learning framework developed by Facebook that helps researchers prototype Deep Learning models. The [Fairseq](https://github.com/pytorch/fairseq) toolkit is built on top of PyTorch and focuses on sequence generation tasks, such as Neural Machine Translation (NMT) or Text Summarization. Fairseq features an active community that is continuously providing reference implementations of state-of-the-art models. It contains many built-in components (model architectures, modules, loss functions, and optimizers) and is easily extendable with plugins.
-Text summarization constitutes a significant challenge in NLP. We need models capable of generating a short version of a document while retaining the key points and avoiding uninformative content. These challenges can be addressed with different approaches. 1). Abstractive text summarization aimed at training models that can generate a summary in narrative form. 2). Extractive methods where the models are trained to select the most important parts from the input text. 3). A combination of the two, where the essential parts from the input are selected and then summarized in an abstractive fashion. Hence, summarization can be accomplished via a single end-to-end network or as a pipeline of extractive and abstractive components. To that end, Fairseq provides all the necessary tools to be successful in our endeavor. It features either end-to-end models such as the classical Transformer, different types of Language Models and pre-trained versions that enable researchers to focus on what matters most—to build state-of-the-art models that generate valuable reports.
+Text summarization constitutes a significant challenge in NLP. We need models capable of generating a short version of a document while retaining the key points and avoiding uninformative content. These challenges can be addressed with different approaches. 1). Abstractive text summarization aimed at training models that can generate a summary in narrative form. 2). Extractive methods where the models are trained to select the most important parts from the input text. 3). A combination of the two, where the essential parts from the input are selected and then summarized in an abstractive fashion. Hence, summarization can be accomplished via a single end-to-end network or as a pipeline of extractive and abstractive components. To that end, Fairseq provides all the necessary tools to be successful in our endeavor. It features either end-to-end models such as the classical Transformer, different types of Language Models and pre-trained versions that enable researchers to focus on what matters most—to build state-of-the-art models that generate valuable reports.
However, we are not just summarizing the transcribed conversation; we generate high-quality medical reports, which have many considerations.
-* Every section of a medical report is different in terms of content, structure, fluency, etc.
-* All medical facts mentioned in the conversation should be present in the report, for example, a particular treatment or dosage.
-* In the healthcare domain, the vocabulary is extensive, and models need to deal with medical terminology.
-* Patient-doctor conversations are usually much longer than the final report.
+- Every section of a medical report is different in terms of content, structure, fluency, etc.
+- All medical facts mentioned in the conversation should be present in the report, for example, a particular treatment or dosage.
+- In the healthcare domain, the vocabulary is extensive, and models need to deal with medical terminology.
+- Patient-doctor conversations are usually much longer than the final report.
All these challenges require our researchers to run a battery of extensive experiments. Thanks to the flexibility of PyTorch and Fairseq, their productivity has greatly increased. Further, the ecosystem offers an easy path from ideation, implementation, experimentation, and final roll-out to production. Using multiple GPUs or CPUs is as simple as providing an additional argument to the tools, and because of the tight Python integration, PyTorch code can be easily debugged.
-In our continuous effort to contribute to the open-source community, features have been developed at Nuance and pushed to the Fairseq GitHub repository. These try to overcome some of the challenges mentioned such as, facilitating copying of, especially rare or unseen, words from the input to summary, training speedups by improving Tensor Core utilization, and ensuring TorchScript compatibility of different Transformer configurations. Following, we will show an example of how to train a Transformer model with a Pointer Generator mechanism (Transformer-PG), which can copy words from the input.
+In our continuous effort to contribute to the open-source community, features have been developed at Nuance and pushed to the Fairseq GitHub repository. These try to overcome some of the challenges mentioned such as, facilitating copying of, especially rare or unseen, words from the input to summary, training speedups by improving Tensor Core utilization, and ensuring TorchScript compatibility of different Transformer configurations. Following, we will show an example of how to train a Transformer model with a Pointer Generator mechanism (Transformer-PG), which can copy words from the input.
## How to build a Transformer model with a Pointer Generator mechanism
@@ -158,8 +162,8 @@ fairseq-preprocess --task "translation" \
--cpu \
--joined-dictionary \
--destdir 
@@ -13,11 +17,11 @@ In collaboration with the Metal engineering team at Apple, we are excited to ann
## Metal Acceleration
-Accelerated GPU training is enabled using Apple’s Metal Performance Shaders (MPS) as a backend for PyTorch. The MPS backend extends the PyTorch framework, providing scripts and capabilities to set up and run operations on Mac. MPS optimizes compute performance with kernels that are fine-tuned for the unique characteristics of each Metal GPU family. The new device maps machine learning computational graphs and primitives on the MPS Graph framework and tuned kernels provided by MPS.
+Accelerated GPU training is enabled using Apple’s Metal Performance Shaders (MPS) as a backend for PyTorch. The MPS backend extends the PyTorch framework, providing scripts and capabilities to set up and run operations on Mac. MPS optimizes compute performance with kernels that are fine-tuned for the unique characteristics of each Metal GPU family. The new device maps machine learning computational graphs and primitives on the MPS Graph framework and tuned kernels provided by MPS.
## Training Benefits on Apple Silicon
-Every Apple silicon Mac has a unified memory architecture, providing the GPU with direct access to the full memory store. This makes Mac a great platform for machine learning, enabling users to train larger networks or batch sizes locally. This reduces costs associated with cloud-based development or the need for additional local GPUs. The Unified Memory architecture also reduces data retrieval latency, improving end-to-end performance.
+Every Apple silicon Mac has a unified memory architecture, providing the GPU with direct access to the full memory store. This makes Mac a great platform for machine learning, enabling users to train larger networks or batch sizes locally. This reduces costs associated with cloud-based development or the need for additional local GPUs. The Unified Memory architecture also reduces data retrieval latency, improving end-to-end performance.
In the graphs below, you can see the performance speedup from accelerated GPU training and evaluation compared to the CPU baseline:
@@ -29,11 +33,10 @@ In the graphs below, you can see the performance speedup from accelerated GPU tr
Accelerated GPU training and evaluation speedups over CPU-only (times faster)